Sign Classification

1. University of Southern California Sign Classification using Boosted Cascade of Classifiers Computer Vision group: Thang Dinh thang.dinh@usc.edu Eunyoung Kim eunyoung.kim@usc.edu Li Zhang li.zhang@usc.edu Yuping Lin yuping.lin@usc.edu

2. Content 1. Sign classification: Applications and Challenges 2. Problem statement 3. Why Boosted Cascade of Classifiers ? 4. Sign detection and classification with Cascade 5. Experimental Results Jan 2007

3. Sign Classification Robot Control Virtual Reality Sign language Sign classification: Applications & Challenges Applications …….. Jan 2007

4. Sign classification: Applications & Challenges Challenges 1. Detection Rate An interpreting system that interprets wrongly, a robot that always misunderstands commands… can not be employed. 2. Computation Time Robots that need 30s to understand a command telling them to do something immediately, Systems that need 30s to interpret each word of speakers… will not be employed either. need a Fast and Robust system Jan 2007

5. Content 1. Sign classification: Applications and Challenges 2. Problem statement 3. Why Boosted Cascade of Classifiers ? 4. Sign detection and classification with Cascade 5. Experimental Results Jan 2007

6. Problem statement Human Gesture: • Dynamic Gesture: requires motion of hand, body… • Static Gesture: static pose of hand, body… • Finger Spelling (ASL System) Jan 2007

7. Problem statement Implemented system: Challenges: • Detection rate, Computation time • Many similar signs …. Jan 2007

8. Content 1. Sign classification: Applications and Challenges 2. Problem statement 3. Why Boosted Cascade of Classifiers ? 4. Sign detection and classification with Cascade 5. Experimental Results Jan 2007

9. Why Boosted Cascade of Classifiers • Cascade of boosted classifiers with Haar wavelet features (Viola and Jones) is currently ‘The state of the art’ in face detection. • Cascade was brought to field of hand detection by Eng-Jon with impressive results. Jan 2007

10. Why Boosted Cascade of Classifiers Face detection system of Viola and Jones is: • 15 times faster than Rowley’s (double layer Neural Network) • 600 times faster than Schneiderman-Kanade’s (Statistics) Cascade of boosted classifiers seems to be a good approach for the Sign classification problem. Jan 2007

11. Content 1. Sign classification: Applications and Challenges 2. Problem statement 3. Why Boosted Cascade of Classifiers ? 4. Sign detection and classification with Cascade 5. Experimental Results Jan 2007

12. Haar wavelet features • Haar wavelet features (Papageorgiou ) • Feature set was enlarged by Lien Hart to 45º rotated. • We proposed Double-L for Sign detection Jan 2007

13. Integral Image – Fast calculation of Haar features • There are millions of features that need processing and each feature itself needs time to be calculated makes the training process time-consuming • Viola and Jones proposed Integral Image in order to reduce the feature calculation complexity, which results in less computation time Definition of intergral image at point (x, y) Then: Sum(D) = 1 + 4 – (2 + 3) Jan 2007

14. Adaboost • Proposed by Freund and Schapire • Combine many ‘weak’ hypothesis to form a ‘strong’ one • ‘weak’ classifier ht only need to be better than chance Jan 2007

15. Adaboost with Haar wavelet features Advantages: • Adaboost adjusts adaptively the errors of the weak hypotheses • Weak Hypothesis learnt from this algorithm can run very fast because of simple calculation of Haar feature, which speeds up the whole system Disadvantages: • Require large amount of training samples • The training process is rather time-consuming because the algorithm needs to check through millions of features extracted from thousands of samples. Jan 2007

16. Cascade of classifiers • Cascade of boosted classifiers is a tree of classifiers where classifier lying at each stage is better than the last • Only those input patterns having passed through all the layers are considered objects Simple backgrounds can be easily rejected by one-feature classifier Jan 2007

17. Sign Classifiers • Each Sign Detectors Di is a cascade of boosted classifier evaluating an input image to give out value vi which is then compared to the threshold of it • 24 Sign Detectors are combined to form a Sign Classifier C = signi | Di = signi and g(Di)=max{g(Dj)|j=1, 2,… 24} Where g(Di) = |vi – i| Jan 2007

18. Sign Classifiers Jan 2007

19. Content 1. Sign classification: Applications and Challenges 2. Problem statement 3. Why Boosted Cascade of Classifiers ? 4. Sign detection and classification with Cascade 5. Experimental Results Jan 2007

20. Experimental Results Samples Collection • Collect raw image • Find the ‘key’ point (which make it different from others) • Cut and Align the image (base on ‘key’ point) • Aligned images are finally made greyscale Image B - Raw Image B - Focus Image B - Aligned Jan 2007

21. Experimental Results Training Process • First stage: • Positive: Thousands of aligned sign images • Negative: Background only – Buildings, paintings, trees, other parts of human body… (also non-sign) Fast eliminate simple background • Second stage: • Positive: also thousands of aligned sign images • Negative: aligned images of other signs Distinguish each sign from others Jan 2007

22. Experimental Results Results 1. Detectors • We have trained 24 detectors for 24 signs • Average number of stages: 14 • Total features: approx 123 • Detection rate: 90% - 100% • Test Images were divided into 2 groups • Simple background • Complex background DR = 100% FA = 0% DR = … FA = … Jan 2007

23. Experimental Results Results 1. Detectors (cont) Hit rate diagram Jan 2007

24. Experimental Results Results 1. Detectors (cont) False alarm High FA rate due to similarity between signs Jan 2007

25. Experimental Results Results 2. Classifiers Test Images: 600 Correct rate: 83% Jan 2007

26. Thank you Thank you Jan 2007