Latent SVMs for Human Detection with a Locally Affine Deformation Field

1. Latent SVMs for Human Detection with a Locally Affine Deformation Field Ľubor Ladický1 Phil Torr2 Andrew Zisserman1 1 University of Oxford 2 Oxford Brookes University

2. Human Detection • Find all objects of interest • Enclose them tightly in a bounding box

3. Human Detection • Find all objects of interest • Enclose them tightly in a bounding box

4. HOG Detector • Sliding window using learnt HOG template • Post-processing using non-maxima suppression Dalal & Triggs CVPR05

5. HOG Detector • Sliding window using learnt HOG template • Post-processing using non-maxima suppression Dalal & Triggs CVPR05

6. HOG Detector • Sliding window using learnt HOG template • Post-processing using non-maxima suppression Dalal & Triggs CVPR05

7. HOG Detector • Sliding window using learnt HOG template • Post-processing using non-maxima suppression Dalal & Triggs CVPR05

8. HOG Detector • Sliding window using learnt HOG template • Post-processing using non-maxima suppression Dalal & Triggs CVPR05

9. HOG Detector • Sliding window using learnt HOG template • Post-processing using non-maxima suppression Dalal & Triggs CVPR05

10. HOG Detector • Sliding window using learnt HOG template • Post-processing using non-maxima suppression Dalal & Triggs CVPR05

11. HOG Detector Does not fit well ! • Sliding window using learnt HOG template • Post-processing using non-maxima suppression Dalal & Triggs CVPR05

12. Deformable Part-based Model • Allows parts to move relative to the centre • Effectively allows the template to deform • Multiple models based on an aspect ratio Felzenszwalb et al. CVPR08

13. Deformable Part-based Model • Allows parts to move relative to the centre • Effectively allows the template to deform • Multiple models based on an aspect ratio Felzenszwalb et al. CVPR08

14. Comparison with other approaches HOG template (no deformation) Part-based model (rigid movable parts) Our model (deformation field)

15. HOG Detector Classifier response : weights bias HOG feature over the set of cells c Dalal & Triggs CVPR05

16. HOG Detector Classifier response : The weights w* and the bias b* learnt using the Linear SVM as: regularization number of training samples hinge loss ground truth labels Dalal & Triggs CVPR05

17. Detector with Deformation Field Cells c displaced by the deformation field d:

18. Detector with Deformation Field Cells c displaced by the deformation field d: Classifier response : HOG feature over the deformed template Regularization of the deformation field

19. Detector with Deformation Field Cells c displaced by the deformation field d: Classifier response : Regularisation takes the form of a smoothness MRF prior: Pairwise weight Pairwise cost

20. Detector with Deformation Field Cells c displaced by the deformation field d: Classifier response : The weights w* and the bias b* learnt using the Latent Linear SVM as:

21. Detector with Deformation Field Why hasn’t anyone tried it before?

22. Detector with Deformation Field Why hasn’t anyone tried it before? • Latent models with many latent variables tend to over-fit • Inference not feasible for a sliding window

23. Detector with Deformation Field • To resolve these problems we propose: • Flexible constraints on the deformation field which avoid over-fitting • Feasible inference method under these constraints • Clustering of the training data into multiple models

24. Locally Affine Deformation Field We restrict the deformation field to be locally affine ( ):

25. Locally Affine Deformation Field We restrict the deformation field to be locally affine ( ):

26. Locally Affine Deformation Field We restrict the deformation field to be locally affine ( ):

27. Locally Affine Deformation Field We restrict the deformation field to be locally affine ( ):

28. Locally Affine Deformation Field We restrict the deformation field to be locally affine ( ):

29. Locally Affine Deformation Field We restrict the deformation field to be locally affine ( ):

30. Locally Affine Deformation Field We restrict the deformation field to be locally affine ( ):

31. Locally Affine Deformation Field We restrict the deformation field to be locally affine ( ):

32. Locally Affine Deformation Field We restrict the deformation field to be locally affine ( ):

33. Locally Affine Deformation Field We restrict the deformation field to be locally affine ( ):

34. Optimisation Weights / bias (w*, b*) and the deformation fields dk estimated iteratively

35. Optimisation Weights / bias (w*, b*) and the deformation fields dk estimated iteratively Given the deformation fields the problem is a standard linear SVM:

36. Optimisation Given (w*, b*) the problem is a constrained MRF optimisation:

37. Optimisation Given (w*, b*) the problem is a constrained MRF optimisation: The last can be decomposed as :

38. Optimisation Given (w*, b*) the problem is a constrained MRF optimisation: The last can be decomposed as : By defining the optimisation becomes:

39. Optimisation Given (w*, b*) the problem is a constrained MRF optimisation: The last can be decomposed as : By defining the optimisation becomes: Didn’t we make the problem harder ?

40. Optimisation • The location of the cells in the first row and in the first column fully determine the location of each cell

41. Optimisation • The location of the cells in the first row and in the first column fully determine the location of each cell • Any locally affine deformation field can be reached by two moves : • move all columns where each column i can move by (Δcdix ,Δcdiy) • move all rows where each row j can move by (Δrdjx ,Δrdjy)

42. Optimisation Columns move

43. Optimisation Rows move

44. Optimisation • The location of the cells in the first row and in the first column fully determine the location of each cell • Any locally affine deformation field can be reached by two moves : • move all columns where each column i can move by (Δcdix ,Δcdiy) • move all rows where each row j can move by (Δrdjx ,Δrdjy) • These moves do not alter the local affinity Columns move Rows move

45. Optimisation • The location of the cells in the first row and in the first column fully determine the location of each cell • Any locally affine deformation field can be reached by two moves : • move all columns where each column i can move by (Δcdix ,Δcdiy) • move all rows where each row j can move by (Δrdjx ,Δrdjy) • These moves do not alter the local affinity • Both moves can be solved quickly using dynamic programming

46. Optimisation • The location of the cells in the first row and in the first column fully determine the location of each cell • Any locally affine deformation field can be reached by two moves : • move all columns where each column i can move by (Δcdix ,Δcdiy) • move all rows where each row j can move by (Δrdjx ,Δrdjy) • These moves do not alter the local affinity • Both moves can be solved quickly using dynamic programming • Works for any form of pairwise potentials

47. Learning multiple poses / viewpoints • We define a similarity measure between two training samples as : where

48. Learning multiple poses / viewpoints • We define a similarity measure between two training samples as : where • K-medoid clustering of S matrix clusters the data into multi models

49. Experiments • Buffy dataset (typically used for pose estimation) • Contains large variety of poses, viewpoints and aspect ratios • Consists of 748 images • Episode s5e3 used for training • Episode s5e4 used for validation • Episodes s5e2, s5e5 and s5e6 used for testing Ferrari et al. CVPR08

50. Clustering of training samples Each row corresponds to one model (out of 10 models)