1 / 62

Describing People: A Poselet-Based Approach to Attribute Classification

Describing People: A Poselet-Based Approach to Attribute Classification. Lubomir Bourdev 1,2 Subhransu Maji 1 Jitendra Malik 1 1 EECS U.C. Berkeley 2 Adobe Systems Inc. Goal: Extract attributes from images of people. Who has long hair?. Who has short pants?.

erik
Download Presentation

Describing People: A Poselet-Based Approach to Attribute Classification

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Describing People: A Poselet-Based Approach to Attribute Classification Lubomir Bourdev1,2 Subhransu Maji1 Jitendra Malik1 • 1EECS U.C. Berkeley 2Adobe Systems Inc.

  2. Goal: Extract attributes from images of people

  3. Who has long hair?

  4. Who has short pants?

  5. Male or female?

  6. Prior work on poselets and on attributes

  7. Prior work on Poselets • Introduced by [Bourdev and Malik, ICCV09] • Detection with poselets [Bourdev et al, ECCV10] • Applications • Segmentation [Brox et al, ECCV10] [Maire et al, ICCV 11] • Actions [Yang et al, CVPR10] [Maji et al, CVPR11] [Yao et al, ICCV11] • Human parsing [Wang et al, CVPR11] • Semantic contours [Hariharan et al, ICCV11] • Subordinate level categorization [Farrell et al, ICCV11]

  8. Prior work on Poselets • Introduced by [Bourdev and Malik, ICCV09] • Detection with poselets [Bourdev et al, ECCV10] • Applications • Segmentation [Brox et al, ECCV10] [Maire et al, ICCV 11] • Actions [Yang et al, CVPR10] [Maji et al, CVPR11] [Yao et al, ICCV11] • Human parsing [Wang et al, CVPR11] • Semantic contours [Hariharan et al, ICCV11] • Subordinate level categorization [Farrell et al, ICCV11]

  9. Prior work on Poselets • Introduced by [Bourdev and Malik, ICCV09] • Detection with poselets [Bourdev et al, ECCV10] • Applications • Segmentation [Brox et al, ECCV10] [Maire et al, ICCV 11] • Actions [Yang et al, CVPR10] [Maji et al, CVPR11] [Yao et al, ICCV11] • Human parsing [Wang et al, CVPR11] • Semantic contours [Hariharan et al, ICCV11] • Subordinate level categorization [Farrell et al, ICCV11]

  10. Prior work on Poselets • Introduced by [Bourdev and Malik, ICCV09] • Detection with poselets [Bourdev et al, ECCV10] • Applications • Segmentation [Brox et al, ECCV10] [Maire et al, ICCV 11] • Actions [Yang et al, CVPR10] [Maji et al, CVPR11] [Yao et al, ICCV11] • Human parsing [Wang et al, CVPR11] • Semantic contours [Hariharan et al, ICCV11] • Subordinate level categorization [Farrell et al, ICCV11]

  11. Prior work on Poselets • Introduced by [Bourdev and Malik, ICCV09] • Detection with poselets [Bourdev et al, ECCV10] • Applications • Segmentation [Brox et al, ECCV10] [Maire et al, ICCV 11] • Actions [Yang et al, CVPR10] [Maji et al, CVPR11] [Yao et al, ICCV11] • Human parsing [Wang et al, CVPR11] • Semantic contours [Hariharan et al, ICCV11] • Subordinate level categorization [Farrell et al, ICCV11]

  12. Prior work on Attributes Attributes as intermediate parts Discovering attributes from text Discovering attributes from images Attributes from motion capture Joint learning of classes & attributes Image retrieval with attributes Attributes and actions Active learning with attributes Attributes of people Gender attribute [Cottrell and Medcalfe, NIPS90] [Golomb et al, NIPS90] [Moghaddam & Yang, PAMI02] [Ferrari & Zisserman, NIPS07] [Kumar et al, ECCV08] [Gallagher and Chen, CVPR08] [Cao et al, ACM08] [Lampert et al, CVPR09] [Farhadi et al, CVPR 09] [Wang et al, BMVC09] [Wang and Forsyth, ICCV09] [Kumar et al, ICCV09] [Farhadi et al, CVPR10] [Berg et al, ECCV10] [Wang and Mori, ECCV10] [Sigal et al, ECCV10] [Branson el al, ECCV10] [Hwang et al, CVPR11] [Parikh and Grauman, CVPR11] [Douze et al, CVPR11] [Kovashka et al, ICCV11] [Liu et al, CVPR11] [Qiu et al, ICCV11] [Yao et al, ICCV11] [Dhar et al, CVPR11] [Parikh and Grauman, ICCV11] [Siddiquie et al, CVPR11]

  13. Prior work on Attributes Attributes as intermediate parts Discovering attributes from text Discovering attributes from images Attributes from motion capture Joint learning of classes & attributes Image retrieval with attributes Attributes and actions Active learning with attributes Attributes of people Gender attribute [Cottrell and Medcalfe, NIPS90] [Golomb et al, NIPS90] [Moghaddam & Yang, PAMI02] [Ferrari & Zisserman, NIPS07] [Kumar et al, ECCV08] [Gallagher and Chen, CVPR08] [Cao et al, ACM08] [Lampert et al, CVPR09] [Farhadi et al, CVPR 09] [Wang et al, BMVC09] [Wang and Forsyth, ICCV09] [Kumar et al, ICCV09] [Farhadi et al, CVPR10] [Berg et al, ECCV10] [Wang and Mori, ECCV10] [Sigal et al, ECCV10] [Branson el al, ECCV10] [Hwang et al, CVPR11] [Parikh and Grauman, CVPR11] [Douze et al, CVPR11] [Kovashka et al, ICCV11] [Liu et al, CVPR11] [Qiu et al, ICCV11][Yao et al, ICCV11] [Dhar et al, CVPR11] [Parikh and Grauman, ICCV11] [Siddiquie et al, CVPR11]

  14. Prior work on Attributes Attributes as intermediate parts Discovering attributes from text Discovering attributes from images Attributes from motion capture Joint learning of classes & attributes Image retrieval with attributes Attributes and actions Active learning with attributes Attributes of people Gender attribute [Cottrell and Medcalfe, NIPS90] [Golomb et al, NIPS90] [Moghaddam & Yang, PAMI02] [Ferrari & Zisserman, NIPS07] [Kumar et al, ECCV08] [Gallagher and Chen, CVPR08] [Cao et al, ACM08] [Lampert et al, CVPR09] [Farhadi et al, CVPR 09] [Wang et al, BMVC09] [Wang and Forsyth, ICCV09] [Kumar et al, ICCV09] [Farhadi et al, CVPR10] [Berg et al, ECCV10] [Wang and Mori, ECCV10] [Sigal et al, ECCV10] [Branson el al, ECCV10] [Hwang et al, CVPR11] [Parikh and Grauman, CVPR11] [Douze et al, CVPR11] [Kovashka et al, ICCV11] [Liu et al, CVPR11] [Qiu et al, ICCV11] [Yao et al, ICCV11] [Dhar et al, CVPR11] [Parikh and Grauman, ICCV11] [Siddiquie et al, CVPR11]

  15. Prior work on Attributes Attributes as intermediate parts Discovering attributes from text Discovering attributes from images Attributes from motion capture Joint learning of classes & attributes Image retrieval with attributes Attributes and actions Active learning with attributes Attributes of people Gender attribute [Cottrell and Medcalfe, NIPS90] [Golomb et al, NIPS90] [Moghaddam & Yang, PAMI02] [Ferrari & Zisserman, NIPS07] [Kumar et al, ECCV08] [Gallagher and Chen, CVPR08] [Cao et al, ACM08] [Lampert et al, CVPR09] [Farhadi et al, CVPR 09] [Wang et al, BMVC09] [Wang and Forsyth, ICCV09] [Kumar et al, ICCV09] [Farhadi et al, CVPR10] [Berg et al, ECCV10] [Wang and Mori, ECCV10] [Sigal et al, ECCV10] [Branson el al, ECCV10] [Hwang et al, CVPR11] [Parikh and Grauman, CVPR11] [Douze et al, CVPR11] [Kovashka et al, ICCV11] [Liu et al, CVPR11] [Qiu et al, ICCV11] [Yao et al, ICCV11] [Dhar et al, CVPR11] [Parikh and Grauman, ICCV11] [Siddiquie et al, CVPR11]

  16. Prior work on Attributes Attributes as intermediate parts Discovering attributes from text Discovering attributes from images Attributes from motion capture Joint learning of classes & attributes Image retrieval with attributes Attributes and actions Active learning with attributes Attributes of people Gender attribute [Cottrell and Medcalfe, NIPS90] [Golomb et al, NIPS90] [Moghaddam & Yang, PAMI02] [Ferrari & Zisserman, NIPS07] [Kumar et al, ECCV08] [Gallagher and Chen, CVPR08] [Cao et al, ACM08] [Lampert et al, CVPR09] [Farhadi et al, CVPR 09] [Wang et al, BMVC09] [Wang and Forsyth, ICCV09] [Kumar et al, ICCV09] [Farhadi et al, CVPR10] [Berg et al, ECCV10] [Wang and Mori, ECCV10] [Sigal et al, ECCV10] [Branson el al, ECCV10] [Hwang et al, CVPR11] [Parikh and Grauman, CVPR11] [Douze et al, CVPR11] [Kovashka et al, ICCV11] [Liu et al, CVPR11] [Qiu et al, ICCV11] [Yao et al, ICCV11] [Dhar et al, CVPR11] [Parikh and Grauman, ICCV11] [Siddiquie et al, CVPR11]

  17. Prior work on Attributes Attributes as intermediate parts Discovering attributes from text Discovering attributes from images Attributes from motion capture Joint learning of classes & attributes Image retrieval with attributes Attributes and actions Active learning with attributes Attributes of people Gender attribute [Cottrell and Medcalfe, NIPS90] [Golomb et al, NIPS90] [Moghaddam & Yang, PAMI02] [Ferrari & Zisserman, NIPS07] [Kumar et al, ECCV08] [Gallagher and Chen, CVPR08] [Cao et al, ACM08] [Lampert et al, CVPR09] [Farhadi et al, CVPR 09] [Wang et al, BMVC09] [Wang and Forsyth, ICCV09] [Kumar et al, ICCV09] [Farhadi et al, CVPR10] [Berg et al, ECCV10] [Wang and Mori, ECCV10] [Sigal et al, ECCV10] [Branson el al, ECCV10] [Hwang et al, CVPR11] [Parikh and Grauman, CVPR11] [Douze et al, CVPR11] [Kovashka et al, ICCV11] [Liu et al, CVPR11] [Qiu et al, ICCV11] [Yao et al, ICCV11] [Dhar et al, CVPR11] [Parikh and Grauman, ICCV11] [Siddiquie et al, CVPR11]

  18. Prior work on Attributes Attributes as intermediate parts Discovering attributes from text Discovering attributes from images Attributes from motion capture Joint learning of classes & attributes Image retrieval with attributes Attributes and actions Active learning with attributes Attributes of people Gender attribute [Cottrell and Medcalfe, NIPS90] [Golomb et al, NIPS90] [Moghaddam & Yang, PAMI02] [Ferrari & Zisserman, NIPS07] [Kumar et al, ECCV08] [Gallagher and Chen, CVPR08] [Cao et al, ACM08] [Lampert et al, CVPR09] [Farhadi et al, CVPR 09] [Wang et al, BMVC09] [Wang and Forsyth, ICCV09] [Kumar et al, ICCV09] [Farhadi et al, CVPR10][Berg et al, ECCV10] [Wang and Mori, ECCV10] [Sigal et al, ECCV10] [Branson el al, ECCV10] [Hwang et al, CVPR11] [Parikh and Grauman, CVPR11] [Douze et al, CVPR11] [Kovashka et al, ICCV11] [Liu et al, CVPR11] [Qiu et al, ICCV11] [Yao et al, ICCV11] [Dhar et al, CVPR11] [Parikh and Grauman, ICCV11] [Siddiquie et al, CVPR11]

  19. Prior work on Attributes Attributes as intermediate parts Discovering attributes from text Discovering attributes from images Attributes from motion capture Joint learning of classes & attributes Image retrieval with attributes Attributes and actions Active learning with attributes Attributes of people Gender attribute [Cottrell and Medcalfe, NIPS90] [Golomb et al, NIPS90] [Moghaddam & Yang, PAMI02] [Ferrari & Zisserman, NIPS07] [Kumar et al, ECCV08] [Gallagher and Chen, CVPR08] [Cao et al, ACM08] [Lampert et al, CVPR09] [Farhadi et al, CVPR 09] [Wang et al, BMVC09] [Wang and Forsyth, ICCV09] [Kumar et al, ICCV09] [Farhadi et al, CVPR10] [Berg et al, ECCV10] [Wang and Mori, ECCV10] [Sigal et al, ECCV10] [Branson el al, ECCV10] [Hwang et al, CVPR11] [Parikh and Grauman, CVPR11] [Douze et al, CVPR11] [Kovashka et al, ICCV11] [Liu et al, CVPR11] [Qiu et al, ICCV11] [Yao et al, ICCV11] [Dhar et al, CVPR11] [Parikh and Grauman, ICCV11] [Siddiquie et al, CVPR11]

  20. Prior work on Attributes Attributes as intermediate parts Discovering attributes from text Discovering attributes from images Attributes from motion capture Joint learning of classes & attributes Image retrieval with attributes Attributes and actions Active learning with attributes Attributes of people Gender attribute [Cottrell and Medcalfe, NIPS90] [Golomb et al, NIPS90] [Moghaddam & Yang, PAMI02] [Ferrari & Zisserman, NIPS07] [Kumar et al, ECCV08] [Gallagher and Chen, CVPR08] [Cao et al, ACM08] [Lampert et al, CVPR09] [Farhadi et al, CVPR 09] [Wang et al, BMVC09] [Wang and Forsyth, ICCV09] [Kumar et al, ICCV09] [Farhadi et al, CVPR10] [Berg et al, ECCV10] [Wang and Mori, ECCV10] [Sigal et al, ECCV10] [Branson el al, ECCV10] [Hwang et al, CVPR11] [Parikh and Grauman, CVPR11] [Douze et al, CVPR11] [Kovashka et al, ICCV11] [Liu et al, CVPR11] [Qiu et al, ICCV11] [Yao et al, ICCV11] [Dhar et al, CVPR11] [Parikh and Grauman, ICCV11] [Siddiquie et al, CVPR11]

  21. Prior work on Attributes Attributes as intermediate parts Discovering attributes from text Discovering attributes from images Attributes from motion capture Joint learning of classes & attributes Image retrieval with attributes Attributes and actions Active learning with attributes Attributes of people Gender attribute [Cottrell and Medcalfe, NIPS90] [Golomb et al, NIPS90] [Moghaddam & Yang, PAMI02] [Ferrari & Zisserman, NIPS07] [Kumar et al, ECCV08] [Gallagher and Chen, CVPR08] [Cao et al, ACM08] [Lampert et al, CVPR09] [Farhadi et al, CVPR 09] [Wang et al, BMVC09] [Wang and Forsyth, ICCV09] [Kumar et al, ICCV09] [Farhadi et al, CVPR10] [Berg et al, ECCV10] [Wang and Mori, ECCV10] [Sigal et al, ECCV10] [Branson el al, ECCV10] [Hwang et al, CVPR11] [Parikh and Grauman, CVPR11] [Douze et al, CVPR11] [Kovashka et al, ICCV11] [Liu et al, CVPR11] [Qiu et al, ICCV11] [Yao et al, ICCV11] [Dhar et al, CVPR11] [Parikh and Grauman, ICCV11] [Siddiquie et al, CVPR11]

  22. Prior work on Attributes Attributes as intermediate parts Discovering attributes from text Discovering attributes from images Attributes from motion capture Joint learning of classes & attributes Image retrieval with attributes Attributes and actions Active learning with attributes Attributes of people Gender attribute [Cottrell and Medcalfe, NIPS90] [Golomb et al, NIPS90] [Moghaddam & Yang, PAMI02] [Ferrari & Zisserman, NIPS07] [Kumar et al, ECCV08] [Gallagher and Chen, CVPR08] [Cao et al, ACM08] [Lampert et al, CVPR09] [Farhadi et al, CVPR 09] [Wang et al, BMVC09] [Wang and Forsyth, ICCV09] [Kumar et al, ICCV09] [Farhadi et al, CVPR10] [Berg et al, ECCV10] [Wang and Mori, ECCV10] [Sigal et al, ECCV10] [Branson el al, ECCV10] [Hwang et al, CVPR11] [Parikh and Grauman, CVPR11] [Douze et al, CVPR11] [Kovashka et al, ICCV11] [Liu et al, CVPR11] [Qiu et al, ICCV11] [Yao et al, ICCV11] [Dhar et al, CVPR11] [Parikh and Grauman, ICCV11] [Siddiquie et al, CVPR11]

  23. Poselets for Attribute Classification

  24. Male or female?

  25. Gender recognition is easier if we factor out the pose

  26. Poselets [Bourdev & Malik ICCV09]

  27. Poselets Examples may differ visually but have common semantics

  28. How do we train a poselet?

  29. Finding correspondences at training time Given part of a human pose How do we find a similar pose configuration in the training set?

  30. Finding correspondences at training time Left Shoulder Left Hip We use keypoints to annotate the joints, eyes, nose, etc. of people

  31. Finding correspondences at training time Residual Error

  32. Training poselet classifiers Residual Error: Given a seed patch Find the closest patch for every other person Sort them by residual error Threshold them 0.15 0.20 0.10 0.35 0.15 0.85

  33. Training poselet classifiers Given a seed patch Find the closest patch for every other person Sort them by residual error Threshold them Use them as positive training examples to train a linear SVM with HOG features

  34. Attribute Classification Algorithm at Test Time

  35. Goal: Extract attributes of this person

  36. Goal: Extract attributes of this person • Target person bounds • Bounds of other nearby people Input:

  37. Step 1: Detect poselet activations • [Bourdev et al, ECCV10]

  38. Step 2: Cluster the activations • [Bourdev et al, ECCV10]

  39. Step 3: Predict person bounds • [Bourdev et al, ECCV10]

  40. Step 4: Identify the correct cluster • Max-flow in bipartite graph

  41. Start with its poselet activations Poselet Activations

  42. Features • Pyramid HOG • LAB histogram • Skin features • Hands-skin • Legs-skin Poselet patch Skin mask Arms mask B .* C Features Features Poselet Activations

  43. Attribute Classification Overview Poselet-level Attribute Classifiers Features Poselet Activations

  44. Attribute Classification Overview Person-level Attribute Classifiers Poselet-level Attribute Classifiers Features Poselet Activations

  45. Attribute Classification Overview Context-level Attribute Classifiers Person-level Attribute Classifiers Poselet-level Attribute Classifiers Features Poselet Activations

  46. Results

  47. Our dataset • Source: VOC 2010 trainval for Person + H3D • ~8000 annotations (4000 train + 4000 test) • 9 binary attributes specified by 5 independent annotators via AMT • Ground truth label: If 4 of the 5 agree • Dataset will be made publicly available

  48. Visual search on our test set “Wears hat” “Female”

  49. “Has long hair” “Wears glasses”

  50. “Wears shorts” “Has long sleeves”

More Related