1 / 47

Can you see me now? Disability assistance in mobile devices

Can you see me now? Disability assistance in mobile devices. Mark Stump Graduate Student Department of Computer Engineering Rochester Institute of Technology. Agenda. Introduction Computer Vision Android Android/CV Fusion and Challenges Problem Preliminary Implementation and Results

lisle
Download Presentation

Can you see me now? Disability assistance in mobile devices

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Can you see me now? Disability assistance in mobile devices Mark Stump Graduate Student Department of Computer Engineering Rochester Institute of Technology

  2. Agenda • Introduction • Computer Vision • Android • Android/CV Fusion and Challenges • Problem • Preliminary Implementation and Results • Questions/Answers

  3. Computer Vision • How computers see and perceive the world • Very diverse field with much up and coming development Image Source: http://startupblog.files.wordpress.com/2009/10/webcam.jpg Image Source: http://www.elexp.com/test/ping.jpg

  4. Computer Vision • Subject Areas • Feature Detection • Edge Detection • Blob Detection • Corner Detection • Recognition • Optical Character Recognition (OCR) • Pose Estimation • Gesture Recognition • And many, many more Image Source: http://stevehanov.ca/blog/index.php?id=62

  5. Face Detection • Recognizes a face in an image or video feed • Has many applications • Digital Cameras • Autofocus • Security • Face as a security Code • Advertising • Energy Saving • Televisions detecting when someone is not looking at the screen and decreasing the brightness Image Source: http://blog.cordiner.net/wp-content/uploads/2010/02/lenna_output.jpg

  6. Face Detection Image Source: http://cdn.thefrisky.com/images/uploads/domestic_violence_ad_m.jpg

  7. Face Detection • Various algorithms developed • Find faces through various methods • Comparing against a large training set • Looking for components of a face in a scene (eyes, mouth, etc) • With further development, can extract other information • Gender • Mood Image Source: http://www.dcviews.com/press/images/Face-detection.jpg

  8. Face Tracking • After a face is detected • Follow it as it moves through a scene • Completed many ways • Fast face detection • Color Analysis • Body analysis • Template Matching Image Source: http://www.globalspec.com/NpaPics/38/121978_020520089775_ExhibitPic.JPG

  9. Image Source: http://scherle.com/wp-content/uploads/2009/03/500px-android-logosvg.png

  10. Android Introduction • Created by Google • “Software stack for mobile devices that includes an operating system, middleware and key applications” • Generally used in Cell Phones • HTC Hero • HTC Magic • Motorola Droid • Also beginning to branch other devices • Dell Mini 5 Tablet PC • Acer Aspire One D250 Netbook • Barnes and Noble Nook eReader Image Source: http://en.wikipedia.org/wiki/File:Android_and_cupcake.jpg

  11. Image Source: http://developer.android.com/images/system-architecture.jpg

  12. Android SDK • Freely available • Currently in release 2.2 (FroYo) • Provides for interfacing with various portions of an Android device • Screen • Wireless (Bluetooth, Wifi) • Motion Sensing • GPS • Camera • Telephony Image Source: http://developer.android.com/assets/images/home/froyo-android.png

  13. Platform • Applications primarily written in Java • Plugin available for Eclipse IDE • Runs of native “Dalvik Virtual Machine” • Register-based Architecture • Used for platforms with limited resources (like a phone) • Applications can also be written in C/C++ • Android Native Development Kit (NDK)

  14. Device Development (HTC Magic) • Android Phone, marketed by T-Mobile as ‘myTouch 3G’ • Specifications • Operating System: Android 1.6 • Processor: Qualcomm® MSM7200A™, 528 MHz • Memory • 512 MB ROM • 288 MB RAM • Display: 3.2-inch TFT-LCD flat touch-sensitive screen with 320x480 HVGA resolution • Connectivity • Bluetooth 2.0 • Wi-Fi 802.11 b/g • Proprietary USB Interface (Mini-USB 2.0/Audio Jack in one) • 3.2 Megapixel Camera with auto focus

  15. Image Source: http://www.thomy.com.br/blog/uploads/htc-magic-white.jpg

  16. Device Development (Motorola Droid) • Android Phone, marketed by Motorola on Verizon Network • Specifications • Operating System: Android 2.1 • Processor: Arm® Cortex™, 550 MHz • Memory • 512 MB ROM • 256 MB RAM • Display: 3.7-inch 16:9 Widescreen touch-sensitive screen with 480x854 WVGA resolution • Connectivity • Bluetooth 2.1 + EDR • Wi-Fi 802.11 b/g • Micro USB Interface • 5 Megapixel Camera with auto focus and dual-LED Flash

  17. Image Source: http://www.softsailor.com/wp-content/uploads/2009/10/Motorola-Droid-Multimedia-Station.jpg Image Source: http://smartphonesfans.com/wp-content/uploads/2010/03/motorola-droid.jpg

  18. Device Development (HTC Evo) • Android Phone, marketed by HTC on Sprint Network • Specifications • Operating System: Android 2.1 • Processor: Qualcomm® Snapdragon™ 8650, 1 GHz • Memory • 1 GB ROM • 512 MB RAM • Display: 4.3-inch touch-sensitive screen with 800x480 WVGA resolution • Connectivity • Bluetooth 2.1 + EDR • Wi-Fi 802.11 b/g • Micro USB Interface • HDMI Output • 8 Megapixel Camera with auto focus and dual-LED Flash • With 1.3 MP front-facing camera

  19. Image Source: http://nexus404.com/Blog/wp-content/uploads2/2010/03/Sprint-HTC-EVO-4G-1.jpg Image Source: http://www.hypebeast.com/image/2010/03/htc-evo-phone-4.jpg

  20. Android/Computer Vision • Potential for almost any application to be developed • Numerous Augmented Reality applications already available for platform Image Source: http://www.uwplatt.edu/web/presentations/PennState/ar/pix/augmented-reality-hud.jpg

  21. Google Goggles • Application created by Google Labs • “Visual Search Application” • Can take pictures of various objects and complete searches based on them • Landmarks • Books • Artwork • Places • Wine • Logos • Business Cards • Can pull phone numbers from a card to import into phone to call, store, etc.

  22. Google Goggles - Landmark Image Source: http://www.google.com/mobile/images/labs/goggles/goggles_landmark.jpg

  23. Google Goggles – Business Card Image Source: http://www.google.com/mobile/images/labs/goggles/goggles_contact.png

  24. Google Goggles – Wine Image Source: http://www.google.com/mobile/images/labs/goggles/goggles_wine.jpg

  25. Scotoma (Blind Spots) • “A spot in the visual field in which vision is absent or deficient” • Causes • Macular Degeneration • Degeneration of the eye • Optic neuritis • Inflammation, Degeneration of the Optic Nerve • And others.. Image Source: http://en.wikipedia.org/wiki/File:CentralScotoma.jpg

  26. Scotoma Treatments • Treatments center around stopping the spot from growing larger • For most cases, no cure • Effects (Very Situational) • Impairs functional acuity • Can affect color perception Image Source: http://www.incm.cnrs-mrs.fr/pperso/portraits/scotoma.png

  27. Scotoma Effects (Continued) • ‘Eccentric Viewing’ • Also called Preferred Retinal Loci (PRL) • “This is similar to looking slightly away from an object at night in order to see it better” • Dan Roberts, via Macular Generation Support, www.mdsupport.org Image Source: http://www.acbvi.org/albums/Vision/images/112p_donut.jpg

  28. My Goal! • To develop a system that will aid a person in compensating with various types of scotoma • Through the use of mobile devices

  29. Finally, my development • Program to help those with various scotoma interact • Limit (and possibly eliminate) ‘eccentric viewing’ • Goals • Run efficiently and quickly • Provide easy-to-use and configurable interface • Be implementable on a mobile device • Implement cutting-edge face detection methods developed in-house • ‘Random Projection’-Based Methods

  30. Current Status • Prototype Implemented on a lab machine • Relatively Inefficient Implementation • Relies heavily on OpenCV graphical libraries for interface • OpenCV is a freely available computer vision library that provides many standard algorithms and aids for implementation • Originally launched by Intel in 1999 • Uses ‘off-the-shelf’ algorithms for face detection • Viola-Jones Detection

  31. Prototype Images

  32. Prototype Images

  33. Prototype Images

  34. Prototype Images

  35. Prototype Images

  36. Prototype Images

  37. Prototype Images

  38. Initial Android Development • Program written to perform edge detection on various images • Uses standard edge detection method • Sobel Kernel • Further extended to implement the on-board camera on the phone

  39. Initial Android Development

  40. Initial Android Development Android Emulator HTC Magic Motorola Droid

  41. Initial Android Development Android Emulator HTC Magic Motorola Droid

  42. Initial Android Development Android Emulator HTC Magic Motorola Droid

  43. Initial Android Development HTC Magic (Java-Based Implementation) HTC Magic (OpenCV-Based Implementation)

  44. Initial Android Development Motorola Droid (Java-Based Implementation) Motorola Droid (OpenCV-Based Implementation)

  45. Future • Research into optimizing hand-offs between native libraries and interface • Optimize/Port previous lab-machine code to Android • Experiment with different interface types for configuration and use

  46. Questions/Answers

  47. References/Further Information • Merriam-Webster’s Online Dictionary • http://www.merriam-webster.com/dictionary • Causes of Scotoma • http://www.wrongdiagnosis.com/s/scotoma/causes.htm • Scotoma – Texas School for the Blind and Visually Impaired • http://www.tsbvi.edu/Education/anomalies/scotoma.htm • Eccentric Viewing • http://www.mdsupport.org/library/eccentric.html • Energy saving of TV by face detection • http://portal.acm.org.offcampus.lib.washington.edu/citation.cfm?doid=1389586.1389693 • Google Goggles Information • http://www.google.com/mobile/goggles/

More Related