1 / 16

Sign Language Recognition using Webcams

Sign Language Recognition using Webcams. Overview. Average person’s typing speed Composing: ~19 words per minute Transcribing: ~33 words per minute Sign speaker Full sign language: ~200 words per minute Spelling out: estimate: 50 words per minute Up to 3x faster. Purpose and Scope.

alvaro
Download Presentation

Sign Language Recognition using Webcams

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 Sign Language Recognition using Webcams

  2. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 Overview • Average person’s typing speed • Composing: ~19 words per minute • Transcribing: ~33 words per minute • Sign speaker • Full sign language: ~200 words per minute • Spelling out: estimate: 50 words per minute • Up to 3x faster

  3. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 Purpose and Scope • Native signers can input faster • Benefits: • Hearing & speaking disabled • Sign interpreters • Just letters & numbers for now • Additional complexity too much to handle • Would require smaller distinctions

  4. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 Research • Related projects • Using mechanical gloves, colored gloves • Tracking body parts • Neural network-based application • Still images: 92% accuracy • Motion: less than 50% accuracy • Feature vector-based application • Also about 90% accuracy on stills • No motion tests

  5. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 More Research • Image techniques: • Edge detection (Robert’s Cross)‏ • Line detection (Hough transform)‏ • Line interpretation methods • Chaining groups of lines • Macro-scale templates • Residual math • Memory management

  6. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 Testing Model • Human interaction necessary • General testing model: bhood@testing:~/syslab-tech$ \ > ./main images/hand.png [DEBUG] Edge detect time: 29 ms Errors:0Warnings:0

  7. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 Program Architecture SERVER PROCESS Webcam capture Edge detection Line detection Interpretation Attribute matching IMAGE FEATURE OUTLINE LINE LIST FINGER POSITONS

  8. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 Edge Detection Results • Results: • Outlines the important edges and not much besides • Robert’s Cross balances detection of major and minor lines Original image (800 x 703)‏ Final image (800 x 703)‏

  9. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 Cropping Results • Remove useless rows & columns with no features • Better contrast • Very large optimization • Memory savings • Area difference means order n2 Original (800 x 703)‏ Result (633 x 645)‏

  10. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 Line detection • Finished! • Recently finished tweaking sensitivities • Still a few potential memory issues

  11. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 Line Grouping • Part of line detection • Large optimization • Iterate over an order of magnitude fewer items • Easier to handle, more pronounced trends Examples of line groups, called “chains”

  12. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 Line Interpretation • Chaining groups of lines • Templates • Generation • Template-based comparison • Line residuals • Use point coordinate averages • Calculate average offset from average • Easy to find height of finger

  13. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 Sample Output • After a typical run: 10days,6:12:19until graduation!! bhood@testing~/syslab-tech/src$ ./main hand.png Edge detection took 0.04 sec Image cropping took 0.00 sec Line detection took 0.17 sec (detected 1424 lines)‏ Line chaining took 0.25 sec (detected 130 chains)‏ Getting orientation took -0.08 src (1 => ORIENTATION_FORWARD)‏ Getting pinky pos. took 0.00 sec (2 => FINGER_BENT)‏ Getting ring pos. took 0.00 sec (2 => FINGER_BENT)‏ Getting middle pos. took 0.01 sec (2 => FINGER_BENT)‏ Getting index pos. took -0.00 sec (4 => FINGER_TUCKED)‏ Overall process took 0.47 sec [TOTALCOUNT] allocated: 10718901, freed: 10364880; leaked: 354021.

  14. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 Timing • Timing data from runs: • To nearest hundredth of a second • Edge detection: 0.04 sec • Image cropping: 0.00 sec • Line detection: 0.17 sec • Line chaining: 0.25 sec • orientation: 0.08 sec • Pinky finger: 0.00 sec • Ring finger: 0.00 sec • Middle finger: 0.01 sec • Index finger: 0.00 sec • Overall process: 0.47 sec • A little slow considering goal of real-time

  15. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 The Mysterious Future • Perfect line interpretation • Work on memory management • Am leaking large quantities (~50K) of memory • Aggressive profiling needed • Finish camera-computer interaction • Device control must be precise, picky

  16. Byron Hood | version 0.4 Computer Systems Lab Project2007-2008 The End! • Code will be available to future years • Contact me for a copy: byron@phareware.com

More Related