Recognition of Traffic Lights in Live Video Streams on Mobile Devices

1. Recognition of Traffic Lights in Live Video Streams on Mobile Devices Jan Roters Xiaoyi Jiang Kai Rothaus 2011 IEEE Transactions on CSVT

2. Outline • Introduction • Problems • System Architecture • Identification • Classification • Video Analysis • Time-Based Verification • Experiment Results • Evaluations • Conclusion

3. Introduction • People with visual disabilities are limited in mobility. • Orientate pedestrians with zebra crossings at intersections • Portable PC with a digital camera and a pair of auricular stereo • Present a system for mobile devices to help sightless people cross roads.

4. Problems • Program usage • Real world conditions • Camera resolution • Different appearances

5. Problems • The scale of traffic lights • Many traffic lights • Occluded • Illumination • Rotation

6. Pedestrian Lights in Germany • Installation • Shape • Color arrangement • Circuitry • Background

7. Mobile Device & Databases • Nokia N95 • 330MHZ ARM processor • 18Mb RAM • 320240 • 2 publicly available database • Ground truth segmentation was made manually

8. System Architecture 1. 2. 4. 3.

9. 1. Localization

10. Red and Green Color Filter(1/3) • Analyze the data

11. Red and Green Color Filter(2/3) • Design the filter rules (ex : red traffic light) • The Gaussian distribution of the red cluster is defined by its mean color = (0.48,0.06,0.07) and has three eigenvectors • A color c = (r, g, b) is a red traffic light color when

12. Red and Green Color Filter(3/3) • Optimize parameters • different parameter settings for each color • Use 300 images to train • Measure the quality of each setting by TP, FP, FN Recall = , Precision =

13. Size/Circuitry Filter • Assume the traffic light is 4 to 24 meters away • Fixed camera focal length and possible aspect ratios • Filter out regions that are too small or too large • Vertical neighbor should not have different color

14. Background Color Filter • Inspect the region under a red light candidate or above a green light candidate • If there are no dark pixels within search region, refuse this candidate Search region Search region

15. Validation of Localization • Validate the localization results with 201 images Error = 33.7%

16. 2. Classification • TLC is the broadest • TLC has the smallest distance to the top of image • No other traffic light has similar height with TLC

17. Performance of Classification

19. 3. Video Analysis(1/2) • Temporary Occlusion • Falsified Colors • Contradictory Scene • Repeating Results

20. 3. Video Analysis(2/2) • Find the motion vector between two frames • Use KLT tracker to track feature points • Only search in a small area around crucial traffic light candidate (30 pixels in each direction) • Correlate the features by using SAD Crucial traffic light Candidate region Feature point Search region

21. 4. Time-Based Verification • Reduce the false positive detections by comparing 2 kinds of results • Use state queue with 4 scenarios • Identification and video analysis are both successful and the locations match with each other. • Identification and video analysis are successful but the locations are different. • Video analysis succeeds but identification fails. • Video analysis fails but identification succeeds.

22. Experiment Results • and • Compute at least 5 frames per second • At least consecutive correct detection with the same color

23. Experiment Results

24. Evaluations • Reliability • Prevent false positive green light detection

25. Evaluations • Interactivity • Temporal analysis reduce the interactivity • The feedback is normally given within 2 seconds

26. Conclusion • The system can be helpful on driver assistance systems • Limited computational power on mobile devices • The verification ideas can be improved