Generic, Deformable Models for 3-D Vehicle Surveillance

1. Generic, Deformable Models for 3-D Vehicle Surveillance • Matthew Leotta • Ph.D Thesis Defense • September 2, 2009

2. Motivating Example 1:3-D Traffic Monitoring • 3-d road models are used in traffic reports • Raw traffic video shows what is really happening • Can the actual traffic images be mapped into a 3-d traffic model? [Triangle Software]

3. Motivating Example 2:3-D Facility Monitoring • Monitor vehicle traffic for security on a campus or base • Want to recognize vehicles and track them from camera to camera • Can this be automated and integrated into a single world map?

4. 3-D Surveillance Objectives • Primary thesis objective: • Dense 3-d shape recovery of vehicles from 2-d images • Secondary thesis objectives: • Vehicle tracking in 3-d (jointly with shape estimation) • Vehicle class recognition (from recovered shape) four-door sedan

5. Overview • The approach: A 3-d deformable model • Learning the shape deformations • Fitting the model to images (shape and pose) • Tracking the model in video • Vehicle class recognition in shape space

6. Overview • The approach: A 3-d deformable model • Learning the shape deformations • Fitting the model to images (shape and pose) • Tracking the model in video • Vehicle class recognition in shape space

7. Previous Work:Deformable 3-D Vehicles • Deformable 3-d model fits many vehicle classes • Model edges are matched to lines, which are fit from detected edges Ferryman et al. BVMC 1995 Koller et al. IJCV 1993 [Kolleret al. IJCV 1993, Ferryman et al. BMVC 1995]

8. Previous Work:Complex 3-D Vehicle Meshes • Use highly complex vehicle meshes (~100K faces) from CAD • Predict vehicle high resolution appearance over viewpoint change Guo et al. CVPR 2008 [Guoet. al. CVPR 2008]

9. Which CAD model do I use?

10. E3D CAD Models • 79 Passenger Vehicles from DARPA’s Exploitation of 3-D Data (E3D) project • Six vehicle classes • SUV • Minivan • Pickup truck • Two-door sedan • Four-door sedan • Station wagon

11. Detailed, Deformable Vehicle • The proposed model • Requirements • Adaptive mesh complexity • Deformable 3-d mesh • Deformable 2-d vehicle parts • Small number of shape parameters

12. Subdivision Surface Vehicle generic, but too simple too specific, too complex • Subdivision adapts the complexity of the mesh as needed generic with adaptable complexity

13. Texture Mapped Parts • Texture space is a mapping of the mesh vertices to the unit square • Parts are polygons in 2-d texture space • Part boundaries are not restricted to mesh edges Texture Space

14. Overview • The approach: A 3-d deformable model • Learning the shape deformations • Fitting the model to images (shape and pose) • Tracking the model in video • Vehicle class recognition in shape space

15. Mesh Model Learning • Shrink-wrap mesh to implicit surface of polygon soup[Shenet al. SIGGRAPH 2004] • Recursively fit the subdivided mesh body

16. Mesh Model Learning • Shrink-wrap mesh to implicit surface of polygon soup[Shenet al. SIGGRAPH 2004] • Recursively fit the subdivided mesh body

17. Mesh Model Learning • Shrink-wrap mesh to implicit surface of polygon soup[Shenet al. SIGGRAPH 2004] • Recursively fit the subdivided mesh body

18. Mesh Model Learning • Shrink-wrap mesh to implicit surface of polygon soup[Shenet al. SIGGRAPH 2004] • Recursively fit the subdivided mesh body

19. Training Part Extraction • Map CAD model parts to the closest point on the deformable mesh • Map boundaries into texture space

20. Part Model Learning • Apply the analogous 2-d shrink wrap to the parts • Implicit surface is not needed • Convex hull used for initial fit

21. Part Model Learning • Apply the analogous 2-d shrink wrap to the parts • Implicit surface is not needed • Convex hull used for initial fit

22. Fit Training Vehicles

23. Principal Components • PCA reduces the space of shape parameters • The top 5 principal component of the vehicle model deviation 1 2 3 4 5

24. Overview • The approach: A 3-d deformable model • Learning the shape deformations • Fitting the model to images (shape and pose) • Tracking the model in video • Vehicle class recognition in shape space

25. Edge Prediction • Predict the most salient intensity edges with • Occluding contours • Part boundaries

26. Jacobian Error Vector Parameter Update Optimizing Parameters • Shape and pose • Gauss-Newton minimization • Iterative least-squares to minimize edge distances

27. Identity Matrix Negative of Current Parameters Regularization • Use Tikhonov regularization • A Gaussian prior in Bayesian terms

28. Scaled Inverse Standard Deviations Regularization (Normalized) • controls the amount of regularization • normalizes the shape parameters • zeroed for pose parameters

29. Diagonal Weight Matrix Robust Estimation • Iteratively reweighted least-squares • Beaton-Tukey M-estimator • Other heuristic weights

30. Multi-Scale Optimization Scale = 8 Scale = 4 • Plot show magnitude of the residual: • Start with coarse scale and progress to fine scale. Scale = 2 Scale = 1

31. Fitting to Multiple Images

32. Poor Initialization High Initial Uncertainty Good Initialization Low Initial Uncertainty Ferryman Dodecahedral Detailed 2 Detailed 1 Detailed 3 Number of PCA Parameters • 5 shape parameters balance expressiveness with over fitting

33. Chevrolet S10 Blazer Dodge Caravan Dodge Stratus Nissan Maxima Toyota 4Runner Toyota Tundra VW Beetle (black) VW Beetle (yellow) Volvo V70XC Multi-view Fitting

34. Chevrolet S10 Blazer Dodge Caravan Dodge Stratus Nissan Maxima Toyota 4Runner Toyota Tundra VW Beetle (black) VW Beetle (yellow) Volvo V70XC Multi-view Fitting

35. - RMS Error (meters) - RMS Error (meters) Z Angle (degrees) Z Angle (degrees) X Distance (meters) X Distance (meters) Pose Convergence Regions Dodge Stratus • Initial pose sampled on a regular grid in a 2-d subspace • X distance • Z angle • A plateau indicates a region of stable convergence • Toyota 4Runner has a dual plateau Z X Toyota 4Runner

36. Convergence Results • Mean Initial Shape • Random Initial Pose • Uniform in angle mag. • Uniform in distance • Uniform directions on the sphere • 10 samples per bin Number Converged Distance (meters) Angle (degrees)

37. Overview • The approach: A 3-d deformable model • Learning the shape deformations • Fitting the model to images (shape and pose) • Tracking the model in video • Vehicle class recognition in shape space

38. Tracking Stages • Initialize • Predict • Correct • Repeat 2 & 3 predict correct correct predict predict initialize correct

39. Track Initialization • Kanade-Lucas-Tomasi (KLT) sparse optical flow helps to initially orient the vehicle model[Shi and Tomasi, CVPR 1994] • Mixture of Gaussians background modeling helps to detect and position the vehicle model[Stauffer and Grimson, CVPR 1999]

40. Shadow Prediction • Project the vehicle into the ground plane • Trace the silhouette of the vehicle and shadow together Direct illumination Overcast illumination

41. Track Initialization 2. BG boundary, centroid, flow vectors 1. Initial frame 3. Back-projected initialization 4. Centroid alignment 5. Boundary alignment 6. Complete initialization

42. State Prediction • A ground plane constraint reduces pose parameters from 6 to 3 • Velocities are added to the state vector • Predict motion with a constant velocity circular arc • Predict constant shape ground position Z angle forward velocity angular velocity shape Prediction function Jacobian of State covariance estimate Predicted estimate Corrected estimate

43. State Correction • Extended Iterated Kalman Filter [Kolleret al. IJCV 1993] • Correct the prediction by fitting to the current frame with Gauss-Newton minimization • Use the prediction as a prior model

44. State Correction • Regularized Extended Iterated Kalman Filter • Use both the predicted state prior and the time independent PCA prior • Reduces drift toward invalid vehicle shapes

45. Tracking with Fixed Shape • White PT Cruiser • Proposed vehicle model (3rd resolution) • Shape learned from CAD model

46. Tracking with Shape Estimation • White PT Cruiser • Proposed vehicle model (3rd resolution) • Unknown shape a priori

47. Tracking with Shape Estimation • White PT Cruiser • Ferryman’s model • Unknown shape a priori

48. Tracking with Shape Estimation • Dark red PT Cruiser • Proposed vehicle model (3rd resolution) • Unknown shape a priori

49. Tracking with Shape Estimation • Grey crossover SUV (Toyota RAV4) • Proposed vehicle model (3rd resolution) • Unknown shape a priori

50. Tracking with Shape Estimation • White pickup truck • Proposed vehicle model (3rd resolution) • Unknown shape a priori