Feature #384

Add support for interfacing with display via Android

Added by Hammel about 3 years ago. Updated over 2 years ago.

Status:In ProgressStart date:09 Sep 2014
Priority:HighDue date:
Assignee:Hammel% Done:

50%

Category:Design
Target version:PiBox - 0.12.0
Severity:01 - Critical

Description

This essentially allows sending appmgr the command to run. Appmgr would need to reply that the app is started so the Adroid app can switch to a management tool specific to the app now running on PiBox.

piboid-main.png - Main keypad (143 KB) Hammel, 09 Jan 2015 09:51

piboid-numkeys.png - Numeric/special keys keyboard (144 KB) Hammel, 09 Jan 2015 09:51

piboid-alphakeys.png - Alphabetic keyboard. (141 KB) Hammel, 09 Jan 2015 09:51

68
69
70

Associated revisions

Revision bb1a6276
Added by Hammel almost 3 years ago

RM #384: Force Android SDK level 11 or later.

Revision d4ab366d
Added by Hammel almost 3 years ago

RM #384: Initial UI layouts and flowcharting for PiBoid and its comm with PiBox Media systems.

Revision 26fb09ac
Added by Hammel almost 3 years ago

RM #384: Add Action Bar support (but it doesn't do anything yet).

Revision e9f50038
Added by Hammel almost 3 years ago

RM #384: Removed build artifacts that shouldn't be under source control.

Revision e8119ecd
Added by Hammel almost 3 years ago

RM #384: Initial integration of settings activity.

Revision f00610e3
Added by Hammel almost 3 years ago

RM #384: Integrated settings page activity stub.

Revision 53af6263
Added by Hammel almost 3 years ago

RM #384: Added prerequisite list for Android development on Fedora.

Revision 20b8b212
Added by Hammel almost 3 years ago

RM #384: Added input field and button for adding a device via IP address. Changed the Settings menu option to Select a Device.

Revision 1fa43c77
Added by Hammel almost 3 years ago

RM #384: Renamed "Settings" to "SelectDevice" to avoid confusion.

Revision eebcd148
Added by Hammel almost 3 years ago

RM #384: Added support for saving device ip and symbollic name to a file.

Revision 037bf50d
Added by Hammel almost 3 years ago

RM #384: Added loading of devices files.

Revision fe079fb4
Added by Hammel almost 3 years ago

RM #384: Notify user if there are no configured devices yet. Also, code cleanup - tabs to spaces.

Revision f4e77da7
Added by Hammel almost 3 years ago

RM #384: Add dialog display to select state (connect, delete, cancel) for selected device.

Revision fd4f9478
Added by Hammel almost 3 years ago

RM #384: Add support for deleting a configured device.

Revision 4ab52c79
Added by Hammel almost 3 years ago

RM #384: Added arrow key sample images and sized them larger.

Revision dc92577e
Added by Hammel almost 3 years ago

RM #384: Remade arrow keys so their lighting is consistent. Added Home, Enter, Tab, Letters and Number keys.

Revision c2884639
Added by Hammel almost 3 years ago

RM #384: Updated to reflect current implementation and added Connect box between Select Device and Credentials to reflect expected implementation. Removed deprecated sample images that had been used as placeholders for the keypad.

Revision 1b82422c
Added by Hammel almost 3 years ago

RM #384: Removed ellipses from ABC and 123 buttons, added special characters to 123 button since it will support both types.

Revision 0ac386b6
Added by Hammel almost 3 years ago

RM #384: Implemented basic login dialog with saved credentials, sans actual login to remote device.

Revision aab2e75e
Added by Hammel almost 3 years ago

RM #384: uncomment finish() call for SelectDevice activity when no devices are configured.

Revision 93f741d1
Added by Hammel almost 3 years ago

RM #384: Added a text view to display current connection, but it's kinda klunky looking at the moment.

Revision e1e3f907
Added by Hammel almost 3 years ago

RM #384: Add connected device IP to ActionBar.

Revision 7eb22035
Added by Hammel almost 3 years ago

RM #384: Updated layout for custom keyboards.

Revision 44e10afe
Added by Hammel almost 3 years ago

RM #384: Fix keypad to fit on any display. Added sample special/numeric keyboard, but sans PiBoid customizations.
Aligned keyboard to bottom of any display on popup and support closing of keyboard.

Revision a611e078
Added by Hammel almost 3 years ago

RM #384: Converted example keyboard into numbers+plus special characters keyboard and updated design doc to reflect layout changes.

Revision d8d00330
Added by Hammel almost 3 years ago

RM #384: Added alphabetic keyboard with a custom keyboard view that can change the coloring of the CAPS key depending on its state.

Revision 19ee5780
Added by Hammel almost 3 years ago

RM #384: Updatet the stock launcher icon to a modified MIOT icon.

Revision 4fa690cf
Added by Hammel almost 3 years ago

RM #384: Added a Comm class to handle submitting keycodes to PiBox via the www interface.

Revision 19b86424
Added by Hammel almost 3 years ago

RM #384: Fixed crash when Selecting Device but no devices exist by passing back an Intent with RESULT_CANCELLED before calling finish().

Revision 7437791c
Added by Hammel almost 3 years ago

RM #384: Added AsyncTask to handle http requests in Comm.

Revision c9ffac68
Added by Hammel almost 3 years ago

RM #384: Added space key to numberic soft keyboard. Completed key handline for keypad and both soft keyboards. Add class to convert keypad/keyboard keycodes to GDK keysyms before submitting to remote PiBox device.

Revision be4ce2dc
Added by Hammel almost 3 years ago

RM #384: Add support for PiBoid passthrough to AppMgr.

Revision 40867313
Added by Hammel almost 3 years ago

RM #384: Initial import of PiBox app library

Revision eb29cd35
Added by Hammel almost 3 years ago

RM #384: Updated cdtools script to change from "piboxlib" to "libpibox" for the project name.

Revision ea1cb74b
Added by Hammel almost 3 years ago

RM #384: Added libpibox to metabuild.

Revision f9448891
Added by Hammel almost 3 years ago

RM #384: Removed unused tag references.

Revision 9b7e3ab8
Added by Hammel almost 3 years ago

RM #384: Remove unused tag references.

Revision df350203
Added by Hammel almost 3 years ago

RM #384: Remove app-level functions (copied from appmgr originally) and integrate use of registration packet and its associated handler callback.

Revision 6c4150cf
Added by Hammel almost 3 years ago

RM #384: Added registration structure type definition.

Revision 3533a2b5
Added by Hammel almost 3 years ago

RM #384: Added library initialization routine utilizing new registration structure.

Revision 66b1f1e4
Added by Hammel almost 3 years ago

RM #384: Added disabled stubs for later use with PiBoid.

Revision 90c0609a
Added by Hammel almost 3 years ago

RM #384: Update message protocol for MT_KEY.

Revision 45bfbeac
Added by Hammel almost 3 years ago

RM #384: Switch to default msgProcessor port number.

Revision 47bfd040
Added by Hammel over 2 years ago

RM #384: Change RESTful interface to use version/service.

History

#1 Updated by Hammel almost 3 years ago

One way to handle this:
  1. Android app connects to PiBox web server over https (see monkey's secure ssl)
  2. A restful interface provides command input from the app
  3. The restful interface passes the commands to appmgr
  4. appmgr forwards the commands to the current app
  5. The app can use the command or ignore it

This should be implemented via a library of some kind so any app can use the same input mechanism with callbacks.

See handling http authentication on stackoverflow.

#2 Updated by Hammel almost 3 years ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 10

I now have adkt function (in androidtest) for building a test application. It contains information on how to work with Android from the command line.

Once I've completed the tutorial I'll need to create a Dia flowchart showing how the application will operate.

#3 Updated by Hammel almost 3 years ago

  • % Done changed from 10 to 20

Successfully built the GridView tutorial. This will become the baseline for the initial android app. Initially I called this PiDroid but a google search shows there are other apps/projects named PiDroid. So the official name will be PiBoid, which is PiBox for Android.

I'm going to clean up the code to make sure the app and package names are correct, then import to gitorious.

After that I'll create the Dia diagram for how this app should operate and interact with PiBox.

Note: when building the GridView tutorial I discovered that the sample images provided, which were in JPEG format, didn't work. I had to convert them to PNG. Just a personal note that the PiBox Android app should use PNGs and not JPEGs.

#4 Updated by Hammel almost 3 years ago

  • Project changed from PiBox to PiBoid
  • Category deleted (04 - Applications)

#5 Updated by Hammel almost 3 years ago

  • Category set to Design

#6 Updated by Hammel almost 3 years ago

Some important links for this project.
  1. Android Training is where to learn about android development.
  2. Android Reference has the API javadoc.
  3. Android Downloads like the Action Bar icon pack.
    1. Android Iconography
  4. Custom keyboard discussion on stackoveflow talks about the pros/cons of using a custom input method vs the soft keyboard.
  5. Soft Keyboard Examples
    1. Keyboard example shows how to use Android's Keyboard and KeyboardView to create custom keyboards.
    2. Detailed example with a clear tutorial including source code.
    3. Another example on github, but the tutorial is in italian. However, the source is very short and easy to follow.
    4. shortyz has an example keyboard layout.
    5. alanszlosek example on github (with screenshot)
    6. dhaval0122 example on github (no description or screenshot but extremely simple)
  6. Broadcast/Multicast reception
    1. WifiManager.MulticastLock is used to allow reception of multicast packets.
    2. Android Multicast Receiver example
    3. UDP Multicast on Android discusses sending as well as receiving.
    4. Network Discovery using UDP Broadcast is for Java but is probably similar in Android.
  7. Android Layouts
  8. Scrolling ListView
    1. ListView API

It appears that using Multicast over wifi may not be as easy as using over wired networks. The initial implementation will therefore fallback to using remote hosts configured by the user.

#7 Updated by Hammel almost 3 years ago

  • % Done changed from 20 to 30

I now have a 3x3 grid I can use for the main navigation commands as well as menu options for selecting and adding remote devices to control.

Next up:
  1. Save devices to a file
  2. Show the devices in the file
  3. Allow deleting devices where we select them
  4. Create icons for the 3x3 grid

#8 Updated by Hammel almost 3 years ago

Implemented:
  1. Save devices to a file
  2. Show the devices in the file

Delete option is available but callback is not implemented yet.

After that, need icons from main window.

#9 Updated by Hammel almost 3 years ago

All keypad icons are integrated into the app. I should probably test this on my phone to see if they fit.

Next up
  1. Prompt for credentials on connect request.
    1. Fields: userid, password (hidden)
    2. Buttons: login, cancel, show password
      1. Login: submits request and verifies
        1. Success Saves credentials and returns to home page
        2. Failure: stays on credentials
      2. Cancel: returns to home page.
      3. Show password: toggles password visibility
    3. Uses saved credentials but allows editing
  2. Front page should show which device it is connected to (text field above keypad)
  3. Create custom soft keyboards, attach to keypad buttons and create keyhandler stubs
    1. ABC
    2. 123
    3. Special characters

#10 Updated by Hammel almost 3 years ago

  • % Done changed from 30 to 40

Userid/password prompt for login to remote host has been added and credentials are saved between uses.
Added the connected device to the ActionBar, when connected.
Created two custom keyboards: alphabetic and numeric/special keys. These have support for capturing their keycodes, which match ASCII values for the printed characters.

Next up:
  1. Perform the actual login to the remote device's web service.
  2. Create a service that can send through the connection the keycodes as they are pressed.

Note that the main keypad should map to the keycodes used by GTK+ for the arrow, Enter, Tab and ESC keys.

#11 Updated by Hammel almost 3 years ago

I've completed the core/stubs of the Android app through to the point of having it send keycodes from the keypad to the PiBox server. I can see the request arrive at the Monkey web server but don't have any code on the web server to handle it yet. That comes next as I develop the PiBox side of this project.

What's missing on the Android side is to handle all keystrokes from the keypad and the two soft keyboards. I've currently implemented some keypad keys as GDK keysyms, such as KP_Left or Tab (see /usr/include/gtk-2.0/gdk/gdkkeysyms.h). To continue this I'll need to create a lookup table for each keystroke on the keypad or soft keyboards. There are only about 50 or 60 keys total so this is just a bit of busy work.

Using the keysyms should make it easier to work with a library on the PiBox side to translate into keystrokes under GTK+/GDK.

I've attached a few screenshots showing the main keypad and the custom soft keyboards. They don't show the device configuration and login but those are simple text fields and buttons, nothing special there. Note that it's possible to switch software keyboards using the keypad if your device doesn't cover the keypad by the keyboards. Alternatively you can close the keyboards using the keyboard button at the bottom/center of each keyboard.

Main keypad
Numeric/special keys keyboard
Alphabetic keyboard.

#12 Updated by Hammel almost 3 years ago

All keycodes are now sent properly using GDK keysyms. I added a KeyCodes class to do the conversion from ASCII codes to the keysyms.

Now on to the PiBox side support.

#13 Updated by Hammel almost 3 years ago

Implemented passthrough from piboxwww to AppMgr. This was pretty simple. Network throughput is pretty slow at my house but the router config here requires sending two floors down and then two floors back up. I suspect it will be a bit faster in production in a travel trailer.

#14 Updated by Hammel almost 3 years ago

  • Target version changed from 0.10.0 to 0.12.0

The PiBox side of this task involves creating a new app library that can handle inbound messages automatically and pass them to a registered app-specific handler. This turns out to be a big deal and can require lots of changes to all the apps and appmgr. It's a very good idea because it allows message passing at a low level using a very unrestrictive messaging protocol and can (or so I expect) integrate with GTK+ to make it easier for the Android app to apply keystrokes to the currently running app. Even nicer: if the app doesn't support the library then the remote simply has no effect. It doesn't break anything.

However, it's also apparent that so many changes potentially break many things and, at the moment, the current codebase is very stable. So I've decided to make the 0.10.0 release with the current stable codebase but without the Android app, add a new release just for that (and the new app library) and push out the other two releases (hardware focus and TV/music apps) currently scheduled. That way I'll have a nice stable release I can use to do demos or use to introduce people to the project.

Pushing this task to 0.11.0.

#15 Updated by Hammel almost 3 years ago

  • Priority changed from High to Immediate

#16 Updated by Hammel over 2 years ago

There might be a way to simplify the PiBox app side of this: xdotool. This is a user space tool for faking keyboard input. There is a library associated with this that can be linked from C programs.

I think the trick here will be that the primary navigation will be a different action than the specialized keyboards. The navigation will cause xdo_keysequence() to be called while the keyboards need to cause xdo_type() to be called.

If this works then I can wrap this library in the pibox app library and have it receive remote commands and issue these xdo calls from the appmgr without having to modify the apps themselves. A very handy solution, if it works.

A simpler solution that doesn't require an external library may be possible as well.

#17 Updated by Hammel over 2 years ago

  • Priority changed from Immediate to Urgent

#18 Updated by Hammel over 2 years ago

  • Priority changed from Urgent to High
  • Severity changed from 03 - Medium to 01 - Critical

Also available in: Atom PDF