Background Subtraction for Urban Traffic Monitoring using Webcams

1. Background Subtraction for Urban Traffic Monitoring using Webcams Master Graduation Project Final Presentation Supervisor: Rein van den Boomgaard Mark Smids December 12th 2006

2. Overview • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demo’s • Background Subtraction in action • Shadow Detector in action • Smart Surveillance using Video Summarization • Evaluation • Conclusions

3. Introduction • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • Traditional ways of traffic monitoring • using magnetic loops

4. Introduction • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • Traditional ways of traffic monitoring • using magnetic loops • Limitations: • These systems only count, very costly

5. Introduction • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • Traditional ways of traffic monitoring • using magnetic loops • Limitations: • These systems only count, very costly • For extended traffic monitoring we want to measure: • road density, queue detection, vehicle speed, exact location of vehicles

6. Introduction • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • Traditional ways of traffic monitoring • using magnetic loops • Limitations: • These systems only count, very costly • For extended traffic monitoring we want to measure: • road density, queue detection, vehicle speed, exact location of vehicles • Solution: use cameras to monitor traffic automatically

7. Introduction • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • Why focus on an urban setting? • Most research focused on a highway setting • More challenging tasks

8. Introduction • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • Why focus on an urban setting? • Most research focused on a highway setting • More challenging tasks • Components of a vision based traffic monitoring system: • cameras, calibration, background subtraction, tracking, shadow detection, parameter extraction, video summarization, …

9. Introduction • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • Why focus on an urban setting? • Most research focused on a highway setting • More challenging tasks • Components of a vision based traffic monitoring system: • cameras, calibration, background subtraction, tracking, shadow detection, parameter extraction, video summarization, …

10. Background Subtraction • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • Deterministic approach • Create an initial background model from the first N frames • For each new frame, subtract it from the background model to obtain a binary mask • for all x,y: if I(x,y) – B(x,y) > T then M(x,y) = 1 • else M(x,y) = 0 • Update the background model: for all x,y: if M(x,y) = 0 then B(x,y) = I(x,y)

11. Background Subtraction • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • Statistical approach • Model each pixel in the background model by a mixture of Gaussians

12. Background Subtraction • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • Statistical approach • Model each pixel in the background model by a mixture of Gaussians • How to determine those components that model the background? • Observation: these Gaussians have the most supporting evidence and lowest variances • Order the K distributions in the mixture by the value of • The first B distributions are chosen as the background model, where:

13. Shadow Detection • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • Shadows: cast and self shadows

14. Shadow Detection • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • Shadows: cast and self shadows • Elimination of cast shadows can improve background subtraction results very much…

15. Shadow Detection • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • Shadows: cast and self shadows • Elimination of cast shadows can improve background subtraction results very much…

16. Shadow Detection • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • Consider the set of pixels classified as foreground pixels

17. Shadow Detection • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • Consider the set of pixels classified as foreground pixels • A pixel is a candidate shadow pixel when the pixel value has a significant lower value than it’s corresponding background value

18. Shadow Detection • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • Consider the set of pixels classified as foreground pixels • A pixel is a candidate shadow pixel when the pixel value has a significant lower value than it’s corresponding background value • Extend this idea: let c = (R,G,B) and • Rate of similarity:

19. Shadow Detection • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • Consider the set of pixels classified as foreground pixels • A pixel is a candidate shadow pixel when the pixel value has a significant lower value than it’s corresponding background value • Extend this idea: let c = (R,G,B) and • Rate of similarity: • If tau < < 1 then pixel is a shadow pixel

20. Video Summarization • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • Application: smart vision based surveillance system

21. Video Summarization • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • Application: smart vision based surveillance system • Record only frames which includes relevant foreground objects

22. Video Summarization • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • Application: smart vision based surveillance system • Record only frames which includes relevant foreground objects • How to guarantee that a full trajectory of a vehicle is recorded?

23. Demos • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • Shadow Detector in action – 1 | 2 • Background Subtraction in action • det 1 | stat 1 - det 2 | stat 2 • Smart Surveillance using Video Sum. - 1

24. Evaluation • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • Test videos: three different weather conditions (5 minutes each) • Goal: test both background subtraction algorithms on these videos • Limitation: no ground truth available!

25. Evaluation • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • Evaluation on another level: using the video summarization component. • A frame level ground truth is created • For each algorithm a score can be computed

26. Evaluation • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions

27. Evaluation • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions

28. Conclusions • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • For all weather conditions: the statistical approach outperforms the deterministic approach (5-10%)

29. Conclusions • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • For all weather conditions: the statistical approach outperforms the deterministic approach (5-10%) • Wind is the hardest problem from both algorithms

30. Conclusions • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • For all weather conditions: the statistical approach outperforms the deterministic approach (5-10%) • Wind is the hardest problem from both algorithms • Statistical approach performs much better in the sunny settings

31. Conclusions • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • For all weather conditions: the statistical approach outperforms the deterministic approach (5-10%) • Wind is the hardest problem from both algorithms • Statistical approach performs much better in the sunny settings • Future work: create a pixel-level ground truth and evaluate both algorithms

32. Questions? • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • Questions? http://www.science.uva.nl/~msmids/afstuderen/master

33. MoG details • Introduction • Background Subtraction • Shadow Detection • Video Summarization • Demos • Evaluation • Conclusions • Update Equations: • MoG: Z. Zivkovic, “Improved Adaptive Gaussian Mixture Model for Background Subtraction”