deWristified : Handwriting Inference Using Wrist-Based Motion Sensors Revisited

1. deWristified: Handwriting Inference Using Wrist-Based Motion Sensors Revisited Raveen Wijewickrama raveen.wijewickrama@utsa.edu Anindya Maiti a.maiti@ieee.org Murtuza Jadliwala murtuza.jadliwala@utsa.edu University of Texas at San Antonio

2. Wrist Wearables • Extends the functionality of traditional wristwatches beyond timekeeping. • Captures rich contextual information about the wearer. • Enables several novel context-based applications. SPriTELab @ UTSA

3. Motion Sensors • Two main types of motion or inertial sensors: • Accelerometer: records device acceleration. • Gyroscope: records device angular rotation. • Accessing motion sensors on wearable devices: • All applications have access to motion sensors by default (also referred to as zero-permissionsensors) on most wearable OSs. • Applications’ access to motion sensors cannot be regulated on most wearable OSs – we can’t turn them off! • Can an adversary take advantage of motion sensor data from a wrist-wearable device to infer private information inputted by the user’s device-wearing hand? SPriTELab @ UTSA

4. Inferring Private User Inputs (Using Wrist Wearables) SPriTELab @ UTSA

5. State-of-the-Art in Handwriting Recognition (Using Wrist Wearables) Airwriting (Amma et al.) Whiteboard writing (Arduser et al.) Finger writing (Xu et al.) Pen(cil) writing (Xia et al.) SPriTELab @ UTSA

6. Adversary Model • Adversary has knowledge of the type of handwriting. • Adversary is able to record data from the target smartwatch’s accelerometer and gyroscope sensors. • Could employ a Trojan app for this! • Adversary’s Goal: To infer handwritten information using target user’s smartwatch sensors. SPriTELab @ UTSA

7. Limitations of Earlier Handwriting Recognition Studies (Using Wrist Wearables) • Airwriting (Amma et al.) • Custom-designed hand glove with very high precision sensors. • Our adversary relies on target user’s smartwatch or fitness band. • Only uppercase words. • Whiteboard writing (Arduser et al.) • Not generalized (training and testing data not from different participants). • Only uppercase alphabets. • No handwriting activity detection. • Finger writing (Xu et al.) • Use of Shimmer, a specialized sensing device intended for lab studies. • Not generalized (training and testing data not from different participants). • Pen(cil) writing (Xia et al.) • Only lowercase alphabets. • Controlled data collection. • No handwriting activity detection. SPriTELab @ UTSA

8. Our Research • How practical is handwriting inference when • Using consumer-grade wrist wearables, • Using generalized training and testing, • Writing in a uncontrolled and unconstrained manner, and • Both upper and lowercase alphabets are modeled ? New Uncontrolled and Unconstrained Writing Data Existing Models SPriTELab @ UTSA

9. Handwriting Inference Framework SPriTELab @ UTSA

10. Experimental Setup • 28 participants for the four writing scenarios. • 18 to 30 years of age • 13 male, 15 female • Two different wrist-wearables. • Sony Smartwatch 3, LG Watch Urbane • Accelerometer and gyroscope recorded at 200Hz. • Participants provided with appropriate writing apparatus. SPriTELab @ UTSA

11. Writing Tasks (In-Lab) • Alphabets. • Individual alphabets one at a time. • Covered all 26 English alphabets in random order. • Each alphabet was written 10 times. • Both upper and lower cases. • Words. • 4-8 alphabet words, from a vocabulary (Goldhahn et al. 2012). • Each participant wrote 20 words, in both upper and lower cases. • Sentence. • "the five boxing wizards jump quickly" in both upper and lower cases. SPriTELab @ UTSA

12. Writing Activity Recognition (Out of Lab) • 2 participants. • Wore a smartwatch for an entire day. • Performed the four writing scenarios at random times. • Adversary’s Goal: To infer handwriting activity first, and then classify the handwritten text. SPriTELab @ UTSA

13. Replicated Inference Frameworks • Airwriting • Hidden Markov Model (HMM) • Whiteboard writing • Dynamic Time Warping (DTW) • Finger writing • Naive Bayes, Logistic Regression and Decision Tree classifiers • Pen(cil) writing • Random Forest classifier SPriTELab @ UTSA

14. Personalized Inference Accuracy Writing Activity Detection: 56% recall and 57% precision for air and finger writing 39% recall and 47% precision for pencil writing 23% recall and 34% precision for whiteboard writing SPriTELab @ UTSA

15. Personalized Inference Accuracy(Whiteboard Writing) Lowercase Uppercase SPriTELab @ UTSA

16. Generalized Inference Accuracy Writing Activity Detection: 35-40% recall for airwriting, whiteboard writing and pencil writing Only 8% recall for finger writing SPriTELab @ UTSA

17. Factors Affecting Inference Accuracy • Number of Strokes. SPriTELab @ UTSA

18. Factors Affecting Inference Accuracy Number of strokes for the same letter for different participants (lowercase). Number of strokes for the same letter for different participants (uppercase). SPriTELab @ UTSA

19. Factors Affecting Inference Accuracy Lowercase Uppercase Variance in number of strokes per alphabet per participant, averaged for all participants SPriTELab @ UTSA

20. Factors Affecting Inference Accuracy • Number of Strokes. • Order of Strokes. • Direction of Strokes. SPriTELab @ UTSA

21. Factors Affecting Inference Accuracy SPriTELab @ UTSA

22. Factors Affecting Inference Accuracy • Number of Strokes. • Order of Strokes. • Direction of Strokes. • Uppercase vs Lowercase. • Specialized Devices. Airwriting (Amma et al.) SPriTELab @ UTSA

23. Conclusion • We investigated how wrist-wearable based handwriting inference attacks perform in realistic day-to-day writing situations. • Such inference attacks are unlikely to pose a substantial threat to users of current consume-grade smartwatches and fitness bands. • Primarily due to highly varying nature of handwriting. • Replicable artifacts: https://sprite.utsa.edu/art/dewristified SPriTELab @ UTSA