Mobile Phone Writing in Air: PhonePoint Pen Study

1. Using Mobile Phones To Write In Air Chris Coykendall – ODU CS495

2. Introduction • Researchers from Duke University and University of Illinois collaborated to develop a prototype system called PhonePoint Pen on the Nokia N95 platform • The PhonePoint Pen system aims to allow humans to write messages or diagrams in the air by using a phone as a pen.

3. The Nokia N95 device Source: Engadget

4. Some Motivations • Make “post-it” notes on the fly • Assistive communications for disabled persons and/or the elderly • Sketching complex ideas, equations and non-standard information • Emergency operations and first responders

5. So Why A New System? • User experience on existing mobile technologies leaves much to be desired • Mobile keyboards difficult to use for motor impaired individuals • Voice recording technologies are cumbersome to search through (and speech-to-text software can be hit or miss)

6. Primary Goals • To explore the viability of using mobile phone accelerometers to write in the air. • To understand and overcome limitations with character recognition. • To develop a prototype on the Nokia N95 platform and perform a test study.

7. Challenges • Lack of a gyroscope • Many systems such as the Wii, Kinect and others are more resourceful in hardware and have a gyroscope for filtering rotation. (This study was done shortly before many phones with gyroscopes, such as the iPhone 4, were released.) • In this study, users were constrained to using the phone by holding it in a steady, non-rotating grip like a pen. They were also asked to write in slow, deliberate motions and make each character around 12 inches square.

8. Challenges • Suppressing background vibration “noise” • Jitter is caused by natural hand vibrations and the accelerometer error itself. • The researchers approached this issue by implementing a moving ∆velocity over last 7 readings. Any acceleration samples < .05m/s2 were ignored.

9. Challenges • Computing displacement of device • The physical position in the air is necessary to know the size of the characters and their relative positions. • This was overcome by looking at the velocities measured suppressing the background vibrations. If n readings were measured as all background noise, it is likely a pause and the velocity is reset to zero.

10. (From source article)

11. Challenges • Multi-stroke characters and transitions • Letters such as the letter ‘A’ require two distinct strokes to write. The letter ‘B’ looks remarkably similar to writing ‘13’. With no frame of reference for position, this is difficult to accommodate. • The study exploited user natural motion of “picking” up the pen on the Z-axis. • They also relied on a combination of heuristics such as delimiters and grammar algorithms they developed to predict what the next stroke was intended to be.

12. (From source article)

13. Implementation • PrototypedonNokia N95 phone platform. • A server-side implementation was developed in MATLAB, which they were able to use basic libraries for the signal processing. • The on-phone processing version was implemented in Python, but stripped-down to only one character at a time and simpler signal processing methods.

14. User Evaluation • Tests were conducted with 10 average users to write the English alphabet: mainly CS and Engineering students. • 4 subjects were trained on the system, while the other 6 “novice” subjects had writing less than ten characters. • Another 5 individuals took part in 8-character tests at Duke University Hospital under supervision. These patients either suffered from cognitive disorders or motion impairments.

15. Result Highlights • Human readability accuracy (HRA) of the characters was around 83-85% for the student test subjects. Character recognition accuracy (CRA) was 91.9% for trained users and 78.2% for novices • The hospital patient 8-character HRA test results were: User 1 – 1/8User 2 – 1/8 User 3 – 1/8 User 4 – 5/8 User 5 – 0/8 (could not operate button)

16. (From source article)

17. Patient Barriers • The PhonePoint Pen required a button to be pressed to begin and end the writing. • Shoulder, elbow, and wrist coordination for large 12in characters can be difficult. • Familiarity with mobile devices. • IRB restrictions make it difficult to exhaustively test patient use.

18. Improvements To Be Made • Faster writing (only 3.02s per letter on average) • Writing longer words/drawings • Cursive handwriting • Writing while moving • More diverse test subjects (CS/Enginering majors likely more technologically-inclined.) • More advanced algorithms (Bayesian Networks, Hdden Markov Models, etc.)

19. Conclusions and Related Research • PhonePoint Pen compares well to other research in this area considering its limited processing power and sensor hardware (just the accelerometer) • Air-gestures with 3D accelerometers (uWave) • Vision-based gestures (Microsoft write-in-the-air) • Stylus-based sketch recognition(SketchREAD) • Wiimote, Logitech Air-Mouse, Nokia NiiMe • Smart Pen and SmartQuill

20. Source • S. Agarwal et. al., "Using Mobile Phones to Write in Air", MobiSys'11