Constructing Images Eyes-free: A Grid-based Dynamic Drawing Tool for the Blind

1. Constructing Images Eyes-free: A Grid-based Dynamic DrawingTool for the Blind Hesham M. Kamel James A. Landay Group for User Interface Research EECS Department University of California, Berkeley

2. Why do blind people need to draw? Drawing by a totally blind user of the Integrated Communication 2 Draw (IC2D) tool

3. Presentation Overview • Motivation • Other Graphical Systems for the Blind • The Integrated Communication 2 Draw (IC2D) • IC2D Usability Study • Results and Discussion • Live Demo • Conclusions

4. Motivations Behind Creating a Drawing Tool for the Blind • Misconception: you can’t carry out visual tasks in a non-visual environment • Truth: there can be a bridge between mediums • Making graphical information accessible for the blind is a challenge, not a problem • the challenge is to match the physical environment with a proper user interface • Our goal: allow the manipulation of on-screen objects in a non-visual format

5. Related Work: Creation & Access • Creating graphics • TDraw (Kurze ‘96) • Communicating graphics • Mercator (Mynatt ‘95)

6. Related Work:Image Sonification Systems • Auditory shape perception (Hollander ‘94) • use of a virtual speaker array to represent shapes • Scene sonification (Meijer ‘92) • pixel-based image-to-sound conversion

7. Major Problems with Drawing UIs • Graphical user interfaces • total reliance on visual feedback • imagine drawing with the monitor off • where is the cursor? • what’s on the screen? • how do I get back to where I was? • let’s do a little experiment… • Haptic user interfaces • hard to carry • expensive

8. 1 2 3 4 5 6 7 8 9 Grid-based Drawing Model • Based on telephone keypad • known by most blind individuals • nine fixed cells navigated by keyboard • directional keys or numbers 1-9 • each cell is a unique point of reference & can be selected • equivalent to point & click • The grid supports • feedback • finding relative & absolute locations • measuring distances • determining common angles (Kamel & Landay 2000)

9. Grid Recursion • Allows more precise point selections • 3 levels of recursion • resolution of 27 x 27 cells • permits drawing objects at different scales • Objects drawn at full screen resolution

10. IC2D Provides a Medium for Accommodation & Collaboration • Blind accustomed to screen readers • IC2D outputs voice & non-speech sounds • Users can create & share drawings reflecting self-expression • Facilitates sighted users communicating graphically w/ blind users & vice versa (Kamel & Landay 2001) • Allows users to enhance semanticsof drawings

11. Labeling in IC2D • Allows objects & groups of objects to be annotated • Uses a hierarchical labeling structure This car, created with help of a sighted user, self-describes its “parts” (Kamel & Landay 2001)

12. Circle Line Line Example of Labeling (Part 1) • Draw a wheel • consists of a circle & two lines

13. Example of Labeling (Part 2) • Label tire & spokes Front Tire Spoke1 Spoke2 Circle Circle Line Line Line Line

14. Front Spokes Group Example of Labeling (Part 3) • Group individual spokes together • Label that group Front Tire Spoke1 Spoke2 Circle Circle Line Line Line Line

15. Front Spokes Front Wheel Group Group Example of Labeling (Part 4) • Group front tire & front spokes together • Label that group Front Tire Spoke1 Spoke2 Circle Circle Line Line Line Line

16. Evaluation Goals • Test if users could carry out instructed drawing with precision • Test users’ recognition of interrelational representations of objects • recognize objects in relation to grid positions & to each other • Test labeling as method for understanding drawing via semantic enhancement • Test if users could draw representations of real-world objects

17. Methodology • Used keyboard input & voice synthesizer output • 16 participants: • 8 sighted • 8 visually impaired (5 partially sighted, 3 congenitally blind) • 19-49 years old • sighted & partially sighted participants wore blindfolds • Equal amount of familiarization time • Used a verbal protocol & recorded along with output • Results evaluated on • task completion time • participants’ self-assessed confidence level • quality of the drawing, as rated by sighted third-party judge

18. Three circles drawn & described by this participant Task 1 - Part A • Draw 3 circles, describe spatial arrangement • Tests interralational drawing • 38-year-old congenitally blind participant: “I can understand the grid concept abstractly, but not in a visual sense, because I have never been able to see.” • 21-year-old sighted participant: “I can almost visualize everything on the screen without having to look at it.”

19. Task 1 - Part B • Draw triangle at smallest grid level in bottom right, draw rectangle in top left, then connect with line • Tests instructed drawing & interralational reps

20. Task 2 • Explore & describe 3 labeled drawings • 3 different arrangements of a man, tree, and house • one visually impaired participant said: “I liked when it told you what the picture was, and where it was. It was very easy to picture that in my mind.” • Tests whether labels match mental model of the image As the participants navigated the cursor around this figure, they heard “top of tree”, “bottom of tree”, “my man”, etc.

21. Task 3 • Draw a representation of a physical cube • Tests if users can draw real world objects • blindfolded partially sighted participant successfully represented a physical cube by counting grid positions (t=10 minutes) • blindfolded sighted participant drew this representation of the cube by visualizing it (t=9 minutes) a) b)

22. Time for Task Completion (in minutes) • Tasks 1a, 1b, & 3: no significant differences in time, user confidence, or the judge’s rating • Task 2: visually impaired group significantly faster, more confident, & rated higher

23. Self-Assessed Confidence Rating Blind participants more acclimated to voice output Sighted participants found the experience “intense” One blind participant: “…the way this grid is laid out is not complicated, you will be able to know exactly where you are all the time.”

24. Performance Rating (by judge) Visually impaired participants on average performed at least as well as sighted participants

25. Drawing by partially sighted participant using IC2D Discussion • Grid-based model can be comprehended by sighted & blind participants • Input/output • keyboard effective navigational device for interacting w/ graphics • voice synthesizer doesn’t hinder communicating graphical info • Labeling technique effective method for blind users to build mental models of images

26. Live Demo

27. Conclusions • Designed IC2D interface to match users’ abilities • Grid-based model is a useful schema for creating graphics in a non-visual environment • Labeling technique effective for users building mental models of visual information • IC2D can be used to increase communication between sighted & blind • Creating & accessing graphics in a non-visual environment is not a barrier, it is a challenge

28. For More Information http://guir.berkeley.edu/ic2d

29. Summary of Statistics