Lecture #1 0 Multimodal Interfaces and Accessibility

1. Lecture #10Multimodal Interfaces and Accessibility *39TUR

2. Multimodal User Interfaces • Attempts to use human communication skills • Provide user with multiple modalities • May be simultaneous or not • Multiple styles of interaction This presentation contains materials byGeorge G. Robertson / Microsoft Research and Adam Sporka / University of Trento TUR 2012

3. Why Multimodal User Interfaces • Use more of user’s senses • Not just vision • Sound • Tactile feedback • Users perceive multiple things at once • e.g., vision and sound TUR 2012

4. Modalities • Visual • Primary in the UI • Acoustic • Often used • Tactile • Complementary, but still important • Used by the vision-impaired people (Braille lines) TUR 2012

5. Modalities • Olfactory (smell), gustatory (taste) • Experiments • “When you smell burning plastic, you know the machine has failed.” • Food simulator • http://www.siggraph.org/s2003/conference/etech/food.html TUR 2012

6. Modality • Many styles exist • Command interface • Natural Language • Direct manipulation (WIMP and non-WIMP) • Conversational (with an interface agent) • Collaborative • Styles vs. I/O devices • Mixed styles produce multimodal UI • Direct manipulation and conversational agent TUR 2012

7. Examples • Put That There (1970s, early 1980s) • Speech and gestures used simultaneously • http://www.youtube.com/watch?v=sC5Zg0fU2e8 • Typical GPS devices in cars • Visual modality (screen) • Acoustic modality (speech output) • Computer games TUR 2012

8. Multimodal versus Multimedia • Multimedia is about media channels • Text, graphics, animation, video: all visual media • Multimodal is about sensory modalities • Visual, auditory, tactile, … TUR 2012

9. Modalities PERFORMANCE ARTICULATION  TYPING CONTROLLER POINTING  CONTROLLER SPEECH RECOGNITION  H C  HEARING TTS  VISION GUI  VISION OBSERVATION PRINT PRESENTATION TUR 2012

10. People with Disabilities • Politically correct terminology evolves over time • Handicapped • The blind people • Vision-impaired, motor-impaired • People with Special Needs • Differently able • Mark Haddon (2003)The Curious Incident of the Dog in the Night-time • Can be permanent, can be temporary TUR 2012

11. Accessibility • Usability • “Users know how to use the tool” • Accessibility • “Users possess senses and dexterity to operate the tool” • Being able to be accessed by as many people as possible TUR 2012

12. Vision-Impaired People  TYPING CONTROLLER POINTING CONTROLLER SPEECH RECOGNITION  H C  HEARING TTS VISION GUI  BRAILLE TOUCH TUR 2012

13. Motor-Impaired People TYPING CONTROLLER POINTING CONTROLLER SPEECH RECOGNITION  H C  HEARING TTS  VISION GUI  VISION PRINT TUR 2012

14. Assistive Technology • An umbrella term for various techniques and devices that compensate the lack of dexterity and/or senses. • Various approaches • Screen readers • Speech recognition • Eye trackers • Head tracers TUR 2012

15. Accessible technology Accessible technology is technology that is flexible and adjustable to an individual’s visual, mobility, hearing, language, and learning needs It enables computer access for people with impairments – regardless of their functional abilities Accessible technology solutions enable people of all abilities to realize their potential TUR 2012

16. Modality Substitution  SPEECH HEARING H H  HEARING SPEECH Environment / Condition (Danninger et al., 2007) TUR 2012

17. Modality Substitution (Danninger et al., 2007) TUR 2012

18. Jan Volný (2011)

19. Modality Conversion HEARING H  C  GUI VISION (Kildal et al., 2007) TUR 2012

20. User Interfaces for the Elderly

21. Condition • Reduced • Motor skills • Perception • Vision, hearing, touch, mobility • Speed • Memory • Other needs • Technology experience is varied (How many grandmothers use email? mothers?) • Uninformed on how technology could help them • Practice skills (hand-eye, problem solving, etc.) • Touch screens, larger fonts, louder sounds TUR 2012

22. Usability for Senior Citizens • Provide large targets • Reduce the number of clicks • Do not require double-clicks • Do not use pull-down menus • Do not have a deep page hierarchy • Concentrate important information at page top • Avoid the need to scroll • Place most links in bulleted, not tightly clustered list • Clearly differentiated visited/non-visited links TUR 2012

23. Usability for Senior Citizens • Use few colors, avoiding blue and green tones • For text to be read or scanned, use: • 12- to 14-point sans serif font (Helvetica, Arial)? • Black text on white background • Left alignment • Increased spacing (leading) between lines • Sentence style mixed-case letters • Appropriate large headings in a 14- to 16-point sans serif font TUR 2012

24. Vision Impairments

25. Colorblindness • Normal vision • Perception of three wave-lengths, corresponding to red, green, and blue • Perception of a color  “RGB ratio” • Trichromacy Stockman, MacLeod & Johnson (1993) TUR 2012

26. Flatla & Gutwin (2012) http://80.dx.doi.org.dialog.cvut.cz/10.1145/2384916.2384946

27. Colorblindness • Colorblindness • Inability to perceive all three “channels” • Males: 8 %, Females: <1 % • Types • Monochromacy • “Grayscale vision” • Very rare • Dichromacy • One type of cones is absent • Protanopia (R), Deteranopia (G), Tritanopia (B) • Anomalous trichromacy • One type of cones is anomalous TUR 2012

28. Colorblindness • User Interface • Visited / unvisited links • Alert messages do not stand out • Everyday life problems • “Is the fruit ripe?” • “Is the meat rare or well-done?” TUR 2012

29. Simulation Images made using http://www.vischeck.com/ TUR 2012

30. Simulation TUR 2012

31. Simulation TUR 2012

32. Personalized Simulation Original Personalized Generic Flatla & Gutwin (2012) http://80.dx.doi.org.dialog.cvut.cz/10.1145/2384916.2384946 TUR 2012

33. Personalized Simulation Flatla & Gutwin (2012) http://80.dx.doi.org.dialog.cvut.cz/10.1145/2384916.2384946 TUR 2012

34. Personalized Simulation Flatla & Gutwin (2012) http://80.dx.doi.org.dialog.cvut.cz/10.1145/2384916.2384946 TUR 2012

35. Compensation • Traffic lights • Workaround: Determining the R/G light by the position • Replacement of green with blue (Japan) • Shape coding (Canada) • Visualization • The color should not be the singlecarrier of information • Maximize the contrast • Use some texture, use captions • Allow the users set up the color scheme • Daltonization http://raynefeather.deviantart.com/art/Rainbow-Dash-quot-Grrr-quot-Face-272751637 TUR 2012

36. Daltonization • Maximize the differences between colors TUR 2012

37. Blindness • Severity • Normal vision • Vision corrected to normal • Loss of acuity • Counting Fingers, Hand Motion, Light Perception • No Light Perception TUR 2012

38. Various Anomalies • Tunnel vision • Loss of peripheral vision • Can count coins, can’t cross the street • Macular degeneration • Loss of vision in the center of the visual field • Akinetopia • Inability to perceive moving objects • Hemispatial neglect • Inability to perceive objects on one side TUR 2012

39. Impacts on the UI • Reduced ability to perceive feedback from the machine • Reduced ability to perceive the physical device • Layout of the buttons • Compensation • Modality substitution • Simplification of the machine TUR 2012

40. Motor-impaired Users

41. Causes • Congenital disorders • Injury • Spinal cord • Loss of limbs • Other medical conditions • Cerebral palsy • Arthritis TUR 2012

42. Consequences • Poor muscle control • Difficulty sensing, grasping, reaching • Difficulty making fine motor movements with fingers • Difficulty doing compound manipulations • Inability to exert much force on controls TUR 2012

43. Consequences • Limited movement of arms • No function of arms • Limited or no articulation in fingers • Standard devices present a challenge • Keyboard, Mouse, Joystick • No or very slow use TUR 2012

44. Effects on the UI • Some people don’t type fast • E.g. Friedreich’s Ataxia patients – 15 to 30 characters per minute • This affects any text input, including chat and search • Some people don’t use mouse • Limited use of keyboard or use of a dedicated hardware instead. • This affects the navigation in the software • Some people can’t use touch screens TUR 2012

45. Cognitive Disabilities

46. Cognitive disabilities TUR 2012 Memory Problem-solving Attention Reading, linguistic, and verbal comprehension Math comprehension Visual comprehension (46)

47. Cognitive disabilities http://webaim.org/articles/cognitive/ What is being said in this phrase? Tob eornot obe TUR 2012 (47)

48. Accessibility Standards and Legislation

49. Motivation • Everybody is allowed to receive information from public institutions • Everybody is required at some point to communicated with public institutions • Bureaus, offices, ministries, etc. • People with disabilities need other people to access these services • Unless, the services are accessible • Accessibility is mandatory for public services in many countries worldwide TUR 2012

50. Motivation • Accessibility • Transportation • Buildings • Software • Websites TUR 2012