Max-Margin Latent Variable Models

1. Max-Margin Latent Variable Models M. Pawan Kumar

2. Max-Margin Latent Variable Models M. Pawan Kumar Kevin Miller, Rafi Witten, Tim Tang, Danny Goodman, HaithemTurki, Dan Preston, Dan Selsam, Andrej Karpathy Ben Packer Daphne Koller

3. Computer Vision Data Log (Size) ~ 2000 Segmentation Information

4. Computer Vision Data Log (Size) ~ 12000 ~ 2000 Bounding Box Segmentation Information

5. Computer Vision Data > 14 M Log (Size) Image-Level ~ 12000 ~ 2000 Bounding Box Segmentation Information “Chair” “Car”

6. Computer Vision Data > 6 B Noisy Label > 14 M Log (Size) Image-Level ~ 12000 ~ 2000 Bounding Box Segmentation Information Learn with missing information (latent variables)

7. Outline • Two Types of Problems • Latent SVM (Background) • Self-Paced Learning • Max-Margin Min-Entropy Models • Discussion

8. Annotation Mismatch Learn to classify an image Image x h Annotation a = “Deer” Mismatch between desired and available annotations Exact value of latent variable is not “important”

9. Annotation Mismatch Learn to classify a DNA sequence Sequence x Latent Variables h Annotation a {+1, -1} Mismatch between desired and possible annotations Exact value of latent variable is not “important”

10. Output Mismatch Learn to segment an image Image x Output y

11. Output Mismatch Learn to segment an image (x, a) (a, h) Bird

12. Output Mismatch Learn to segment an image (x, a) (a, h) Cow Mismatch between desired output and available annotations Exact value of latent variable is important

13. Output Mismatch Learn to classify actions (x, y)

14. Output Mismatch Learn to classify actions x ha = +1 hb + “jumping”

15. Output Mismatch Learn to classify actions hb x ha = -1 + “jumping” Mismatch between desired output and available annotations Exact value of latent variable is important

16. Outline • Two Types of Problems • Latent SVM (Background) • Self-Paced Learning • Max-Margin Min-Entropy Models • Discussion

17. Latent SVM Andrews et al, 2001; Smola et al, 2005; Felzenszwalb et al, 2008; Yu and Joachims, 2009 Image x Features (x,a,h) h Parameters w Annotation a = “Deer” (a(w),h(w)) = maxa,h wT(x,a,h)

18. Parameter Learning Score of Ground-Truth Best Completion of > Score of All Other Outputs

19. Parameter Learning maxhwT(xi,ai,h) > wT(x,a,h)

20. Parameter Learning min ||w||2 + CΣiξi maxhwT(xi,ai,h) ≥ wT(x,a,h) - ξi + Δ(ai,a) Annotation Mismatch

21. Optimization hi* = argmaxhwT(xi,ai,h) Update Update wby solving a convex problem min ||w||2 + C∑i i wT(xi,ai,hi*) - wT(xi,a,h) ≥ (ai, a) - i Repeat until convergence

22. Outline • Two Types of Problems • Latent SVM (Background) • Self-Paced Learning • Max-Margin Min-Entropy Models • Discussion

23. Self-Paced Learning Kumar, Packer and Koller, NIPS 2010 Math is for losers !! 1 + 1 = 2 1/3 + 1/6 = 1/2 eiπ+1 = 0 FAILURE … BAD LOCAL MINIMUM

24. Self-Paced Learning Kumar, Packer and Koller, NIPS 2010 Euler was a Genius!! 1 + 1 = 2 1/3 + 1/6 = 1/2 eiπ+1 = 0 SUCCESS … GOOD LOCAL MINIMUM

25. Optimization hi* = argmaxhwT(xi,ai,h) Update Update wby solving a convex problem vi  {0,1} min ||w||2 + C∑ii vi • - λ∑ivi wT(xi,ai,hi*) - wT(xi,a,h) ≥ (ai, a) - i λλμ Repeat until convergence

26. Image Classification Mammals Dataset 271 images, 6 classes 90/10 train/test split 5 folds

27. Image Classification Kumar, Packer and Koller, NIPS 2010 CCCP CCCP SPL SPL HOG-Based Model. Dalal and Triggs, 2005

28. Image Classification PASCAL VOC 2007 Dataset ~ 5000 images Car vs. Not-Car 50/50 train/test split 5 folds

29. Image Classification Witten, Miller, Kumar, Packer and Koller, In Preparation Objective HOG + Dense SIFT + Dense Color SIFT SPL+ – Different features choose different “easy” samples

30. Image Classification Witten, Miller, Kumar, Packer and Koller, In Preparation Mean Average Precision HOG + Dense SIFT + Dense Color SIFT SPL+ – Different features choose different “easy” samples

31. Motif Finding UniProbe Dataset ~ 40,000 sequences Binding vs. Not-Binding 50/50 train/test split 5 folds

32. Motif Finding Kumar, Packer and Koller, NIPS 2010 CCCP CCCP SPL SPL Motif + Markov Background Model. Yu and Joachims, 2009

33. Semantic Segmentation VOC Segmentation 2009 Stanford Background + Train - 1274 images Validation - 225 images Test - 750 images Train - 572 images Validation - 53 images Test - 90 images

34. Semantic Segmentation VOC Detection 2009 ImageNet + Bounding Box Data Image-Level Data Train - 1564 images Train - 1000 images

35. Semantic Segmentation Kumar, Turki, Preston and Koller, ICCV 2011 SPL SPL CCCP CCCP SUP SUP SUP – Supervised Learning (Segmentation Data Only) Region-based Model. Gould, Fulton and Koller, 2009

36. Action Classification PASCAL VOC 2011 + Bounding Box Data Noisy Data Train – 3000 instances Train - 10000 images Test – 3000 instances

37. Action Classification Packer, Kumar, Tang and Koller, In Preparation SPL CCCP SUP Poselet-based Model. Maji, Bourdev and Malik, 2011

38. Self-Paced Multiple Kernel Learning Kumar, Packer and Koller, In Preparation 1 + 1 = 2 Integers Rational Numbers 1/3 + 1/6 = 1/2 Imaginary Numbers eiπ+1 = 0 USE A FIXED MODEL

39. Self-Paced Multiple Kernel Learning Kumar, Packer and Koller, In Preparation 1 + 1 = 2 Integers Rational Numbers 1/3 + 1/6 = 1/2 Imaginary Numbers eiπ+1 = 0 ADAPT THE MODEL COMPLEXITY

40. Optimization hi* = argmaxhwT(xi,ai,h) Update and c Update wby solving a convex problem vi  {0,1} ^ min ||w||2 + C∑ii vi • - λ∑ivi wT(xi,ai,hi*) - wT(xi,a,h) ≥ (ai, a) - i Kij= (xi,ai,hi)T(xj,aj,hj) K= ΣkckKk λλμ Repeat until convergence

41. Image Classification Mammals Dataset 271 images, 6 classes 90/10 train/test split 5 folds

42. Image Classification Kumar, Packer and Koller, In Preparation FIXED FIXED SPMKL SPMKL HOG-Based Model. Dalal and Triggs, 2005

43. Motif Finding UniProbe Dataset ~ 40,000 sequences Binding vs. Not-Binding 50/50 train/test split 5 folds

44. Motif Finding Kumar, Packer and Koller, NIPS 2010 FIXED FIXED SPMKL SPMKL Motif + Markov Background Model. Yu and Joachims, 2009

45. Outline • Two Types of Problems • Latent SVM (Background) • Self-Paced Learning • Max-Margin Min-Entropy Models • Discussion

46. MAP Inference Pr(a,h|x) = exp(wT(x,a,h)) Z(x) Pr(a1,h|x)

47. MAP Inference Pr(a,h|x) = exp(wT(x,a,h)) Z(x) mina,h – log (Pr(a,h|x)) Value of latent variable? Pr(a1,h|x) Pr(a2,h|x)

48. Min-Entropy Inference mina – log (Pr(a|x)) + Hα (Pr(h|a,x)) Renyi entropy of generalized distribution Q(a; x, w) = Set of all {Pr(a,h|x)} mina Hα(Q(a; x, w))

49. Max-Margin Min-Entropy Models Miller, Kumar, Packer, Goodman and Koller, AISTATS 2012 min ||w||2 + C∑ii Hα(Q(a; x, w))- Hα(Q(ai; x, w)) ≥ (ai, a) - i i≥ 0 Like latent SVM, minimizes (ai, ai(w)) In fact, when α = ∞...

50. Max-Margin Min-Entropy Models Miller, Kumar, Packer, Goodman and Koller, AISTATS 2012 min ||w||2 + C∑ii maxhwT(x,ai,h)-maxhwT(x,a,h) ≥ (ai, a) - i i≥ 0 Like latent SVM, minimizes (ai, ai(w)) Latent SVM In fact, when α = ∞...