Vision-Based Detection, Tracking and Classification of Vehicles using Features and Patterns with Automatic Camera Calibr

1. Vision-Based Detection, Tracking and Classification of Vehicles using Features and Patterns with Automatic Camera Calibration Neeraj K. Kanhere Committee members Dr. Stanley Birchfield (Advisor) Dr. John Gowdy Dr. Robert Schalkoff Dr. Wayne Sarasua Clemson University July 10th 2008

3. Inductive loop detectors • Piezoelectric and Fiber Optic sensors • The Infra-Red Traffic Logger (TIRTL) • Radar • Laser • No traffic disruption for installation and maintenance • Wide area detection with a single sensor • Rich in information for manual inspection Vehicle tracking Why detect and track vehicles ? • Intelligent Transportation Systems (ITS) • Data collection for transportation engineering applications • Incident detection and emergency response Non-vision sensors Vision-based sensors

4. Autoscope (Econolite) Citilog Vantage (Iteris) Traficon Available video commercial systems

5. Problems with commercial systems Video

6. Related research Region/contour (Magee 04, Gupte et al. 02) • Computationally efficient • Good results when vehicles are well separated 3D model (Ferryman et al. 98) • Large number of models needed for different vehicle types • Limited experimental results Markov random field (Kamijo et al. 01) • Good results on low angle sequences • Accuracy drops by 50% when sequence is processed • in true order Feature tracking (Kim 08, Beymer et al. 97) • Handles partial occlusions • Good accuracy for free flowing as well as congested traffic conditions

7. Overview of the research Scope of this research includes three problems Vehicle detection and tracking Camera calibration Features Patterns Vehicle classification and traffic parameter extraction

8. Overview of the research Scope of this research includes three problems Vehicle detection and tracking Camera calibration Features Patterns Vehicle classification and traffic parameter extraction

9. Problem of depth ambiguity Image plane Focal point Road • Pinhole camera model • All points along the ray map to the same image location

10. Problem of depth ambiguity 1 2 3 4 Top view Perspective view An image point on the roof of the trailer is in the second lane

11. Problem of depth ambiguity 1 2 3 4 Top view Perspective view The same image point is now in the last lane

12. Problem of depth ambiguity 1 2 3 4 1 2 3 4

13. Problem of scale change Grouping based on pixel distances fails when there is a large scale change in the scene.

14. Feature segmentation using 3D coordinates Background model Calibration 1 Background subtraction 5 Correspondence 4 Normalized cuts on affinity matrix 2 Single frame estimation 3 Rigid motion constraint Neeraj Kanhere, Stanley Birchfield and Shrinivas Pundlik (CVPR 2005) Neeraj Kanhere, Stanley Birchfield and Wayne Sarasua (TRR 2006)

15. Vehicle trajectories and data Improved real-time implementation Image frame Featuretracking Backgroundsubtraction Filtering Group stable features PLP estimation Correspondence, Validation and Classification Group unstable features Calibration Neeraj Kanhere and Stanley Birchfield (IEEE Transactions on Intelligent Transportation Systems, 2008)

16. Offline camera calibration 1) User draws two lines (red) corresponding to the edges of the road 2) User draws a line (green) corresponding to a known length along the road 3) Using either road width or camera height, a calibrated detection zone is computed

17. Background subtraction and filtering Background features Vehicle features Shadow features Only vehicles features are considered in further processing, reducing distraction from shadows

18. Plumb line projection (PLP) • PLP is the projection of a feature on the road in the foreground image. • With this projection, an estimate of 3D location of the feature is obtained.

19. Error in 3D estimation with PLP

20. Selecting stable features Feature is stable if & • Features are stable if • close to the ground, and • slope is small at plumb line projection

21. Grouping of stable features Within each lane: Seed growing is used to group features with similar Y coordinate Across lanes: Groups with similar Y coordinate are merged if their combined width is acceptable

22. Grouping unstable features Location of an unstable feature is estimated with respect to each stable group using rigid motion constraint. Centroid of a stable feature group Unstable feature

23. Grouping unstable features Likelihood of the unstable feature is computed based on the estimated 3D location. score for group i validity of location bias terms for large vehicles Unlikely to belong to any other group Unstable feature is assigned to the group if it is likely to belong to that group & a is best matching stable group. bis second best matching stable group.

24. Overview of the research Scope of this research includes three problems Vehicle detection and tracking Camera calibration Features Patterns Vehicle classification and traffic parameter extraction

25. Combining pattern recognition Feature grouping Pattern recognition • Needs a trained detector for • significantly different viewpoints • Works under varying camera • placement • Eliminates false counts due to shadows but headlight reflections are still a problem • Does not get distracted by headlight reflections • Does not need calibration • Needs calibration • Handles back-to-back occlusions but difficult to handle lateral occlusions • Handles lateral occlusions but fails in case of back-to-back occlusions

26. B B A A Combining pattern recognition Lateral occlusion Back-to-back occlusion • Handles lateral occlusions but fails in case of back-to-back occlusions • Handles back-to-back occlusions but difficult to handle lateral occlusions

27. Boosted Cascade Vehicle Detector (BCVD) Vehicles detected in new images Offline supervised training of the detector using training images Run-time Training Positive training samples BCVD Negative training samples Cascadearchitecture Detection Stage n Stage 1 Stage 2 …. Rejected sub-windows

28. sum(A) = val(1) sum(A+B) = val(2) sum(A+C) = val(3) sum(A+B+C+D) = val(4) sum(D) = val(4) – val(3) – val(2) + val(1) A B 1 2 C D 3 4 Rectangular features with Integral images Haar-like rectangular features Fast computation and fast scaling Viola and Jones, CVPR 2001

29. Sample results for static vehicle detection

30. Overview of the research Scope of this research includes three problems Vehicle detection and tracking Camera calibration Vehicle classification and traffic parameter extraction

31. Two calibration approaches Image-world correspondences f, h, Φ, θ … M[3x4] M[3x4] Direct estimation of projective transform Estimation of parameters for the assumed camera model • Goal is to estimate 11 elements of a matrix which transforms points in 3D to a 2D plane • Harder to incorporate scene-specific knowledge • Goal is to estimate camera parameters such as focal length and pose • Easier to incorporate known quantities and constraints

32. Direct estimation of projective matrix Atleast six points are required to estimate the 11 unknown parametes of the projective matrix

33. Camera calibration modes Assumptions: Flat road surface, zero skew, square pixels, and principal point at image center Known quantities: Width (W) or, Length (L), or Camera height (H)

34. Camera calibration modes Assumptions: Flat road surface, zero skew, square pixels, principal point at image center, and zero roll angle Known quantities: W or L or H

35. Camera calibration modes Assumptions: Flat road surface, zero skew, square pixels, principal point at image center, and zero roll angle Known quantities: {W, L} or {W, H} or {L, H}

36. Previous approaches to automatic calibration Schoepflin and Dailey (2003) Dailey et al. (2000) Song et al. (2006) Zhang et al. (2008) • Previous approaches: • Need background image • Sensitive to image processing parameters • Affected by spillover • Do not work at night

37. Our approach to automatic calibration • Does not depend on road markings • Does not require scene specific parameters such as lane dimensions • Works in presence of significant spill-over (low height) • Works under night-time condition (no ambient light) Neeraj Kanhere, Stanley Birchfield and Wayne Sarasua (TRR 2008)

38. Estimating vanishing points Vanishing point in the direction of travel is estimated using vehicle tracks Orthogonal vanishing point is estimated using strong gradients or headlights

39. Automatic calibration algorithm Focal length (pixels) Pan angle Tilt angle Camera height

40. Overview of the research Vehicle detection and tracking Camera calibration Vehicle classification and traffic parameter extraction

41. Motorcycles • Passenger cars • Other two-axle, four-tire single unit vehicles • Buses • Two-axle, six-tire, single-unit trucks • Three-axle single-unit trucks • Four or more axle single-unit trucks • Four or fewer axle single-trailer trucks • Five-axle single-trailer trucks • Six or more axle single-trailer trucks • Five or fewer axle multi-trailer trucks • Six axle multi-trailer trucks • Seven or more axle multi-trailer trucks Vehicle classification based on axle counts FHWA highway manual lists 13 vehicle classes based on axle counts:

42. Vehicle classification based on length Thanks to Steven Jessberger (FHWA)

43. Vehicle classification based on length Four classes for length-based classification • Motorcycles • Passenger cars • Other two-axle, four-tire single unit vehicles • Buses • Two-axle, six-tire, single-unit trucks • Three-axle single-unit trucks • Four or more axle single-unit trucks • Four or fewer axle single-trailer trucks • Five-axle single-trailer trucks • Six or more axle single-trailer trucks • Five or fewer axle multi-trailer trucks • Six axle multi-trailer trucks • Seven or more axle multi-trailer trucks

44. Traffic parameters • Volumes • Lane counts • Speeds • Classification (three classes)

45. Results

47. Quantitative results

48. Results for automatic camera calibration

49. Demo

50. Conclusion • Research contributions: • A system for detection, tracking and classification of vehicles • Combination of feature tracking and background subtraction to group features in 3D • Pattern recognition-based approach to detection and tracking of vehicles • Automatic camera calibration technique which doesn’t need pavement markings and works even in absence of ambient light • Future work should be aimed at: • Extending automatic calibration to handle non-zero roll • Improving and extending vehicle classification • Long term testing of the system in day and night conditions • A framework for combining pattern recognition with features