1 / 35

See.ColOr: Experiments in Enhancing Perception for the Blind

This research explores the use of the See.ColOr interface, which converts colors and depth into sound, to enhance perception and mobility for blind individuals. The experiments demonstrate the effectiveness of the interface in tasks such as recognizing objects, following paths, and grasping objects.

gracecastro
Download Presentation

See.ColOr: Experiments in Enhancing Perception for the Blind

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. “It's always incumbent on a researcher to make the effort to reach out to the community to engage with the population they're serving.”

  2. Experiments with the See ColOr Interface Guido Bologna CVML, University of Geneva, Switzerland

  3. Outline • Introduction • Local Perception Module • Experiments in the past • New experiments • Conclusion

  4. Introduction • Blind people 40 millions in 2020 (90% after the age of 1 year) 4 millions in OCDE countries • Long term goal : sensing the environment with sounds Action is key to perception • A number of pixels are directional sound sources

  5. Local Perception Module

  6. See ColOr Seeing Colours with an Orchestra head-centric colour encoding 2D sound spatialization distance stereovision video piano saxophone 25 points headphones

  7. Colour encoding • Colour system: Hue-Saturation-Luminance • Hue: musical instrument • Saturation: 4 instrument notes • Luminance: 2 instruments with different notes

  8. Oboe Viola Saxophone Violin pizzicato Piano Flute Trumpet Hue

  9. Saturation and luminance • 4 levels of saturation • 8 levels of luminance Low High Double bass Human voice

  10. Prototypes • Stereoscopic or Kinect camera • Sonification of a row containing 25 points • CIPIC Kemar HRTFs (2D spatialization) • HSL colour coding • Two depth coding schemes • Lower peripheral resolution [15 12 9 7 5 3 3 2 2 1 1 1 1 1 1 1 2 2 3 3 5 7 9 12 15]

  11. Depth coding (first mode) For all sonified pixels (25) : • 90 ms for undetermined depth (D); • 160 ms for 0 <D<1 • 207 ms for 1<D<2 • 254 ms for 2<D<3 • 300 ms for D > 3

  12. Depth coding (second mode) • The strategy here is to sonify the dominant colour of the row with only 1 sound source (providing average depth) • The centroid of the first red area is laterally spatialized [R R R R R B B J B B B R R G G GJ J B B B G G G G] [R R R R R ] • The volume V after 3 meters starts to decrease: F(V) = V * exp(-k*D)

  13. Experiments in the Past

  14. Left picture : the 6 experiment participants interpreted the major colours as the sky the sea and the sun; clouds were more difficult to infer (2 subjects); instead of ducks, all the subjects found an island (with yellow sand) or a ship. Right picture : 4 participants correctly recognised the tree with the sky and the grass; one subject qualified the tree as a strange dark object and finally the last individual imagined a nuclear explosion !

  15. All individuals found one of the red doors in a time range between 4 and 9 minutes.

  16. Pairing colored socks

  17. Participant Time (mn) Success rate (pairs) P1 12 5 P2 24 5 P3 75 P4 6 3 P5 16 5 Average13.0 (7.3) 4.6 (0.9) Participant Time (mn) Success rate (pairs) P6 5 5 P7 10 5 P8 45 P9 4 5 P10 18 5 Average8.2 (6.0) 5.0 (0.0)

  18. Following a red path • In this experiment the goal was to follow a coloured path in an outdoor environment.

  19. Participant Path Length (m) Speed Average (m/h) P1 M+C = 88 723 P2 M = 84 710 P3 M+B = 110 485 P4 M+A = 93 656 P5 M = 84 484 P6 M+A+C = 97 600 P7 M+A+C = 97 451 P8 M = 84 869 P9 M = 84 1326 P10 M = 84 1096 Average 90.5(8.7) 740.0(284.6)

  20. Find a red tee shirt

  21. New Experiments

  22. Alerting System Object Recognition Local Perception Text reading Current Prototype with Kinect Global Perception

  23. See ColOr – CVML (2013)

  24. EXPERIMENT ONE: The Spinning Chair

  25. EXPERIMENT THREE: Finding Juan

  26. EXPERIMENT TWO: Walking Towards a Wall

  27. EXPERIMENT FOUR: Grasping Objects

  28. Conclusions

  29. Sonification of two very important vision features (colour and depth) made it possible to achieve a number of tasks related to mobility and recognition. • We will pursue to define new challenging experiments requiring the activation of several See ColOr modules. • We hope that new small 3D RGB cameras will be in the market soon (like the new Kinect Capri) !

More Related