160 likes | 187 Views
Recovering Human Body Configurations using Pairwise Constraints between Parts. Xiaofeng Ren, Alex Berg, Jitendra Malik. Finding People. Challenges: Pose, Clothing, Lighting, Clutter, …. Previous Work. Related Domains Tracking People Detecting Pedestrians ... … Localizing Human Figures
E N D
Recovering Human Body Configurations using Pairwise Constraints between Parts Xiaofeng Ren, Alex Berg, Jitendra Malik 1
Finding People Challenges: Pose, Clothing, Lighting, Clutter, … 2
Previous Work • Related Domains • Tracking People • Detecting Pedestrians • ... … • Localizing Human Figures • Exemplar-based: [Toyama & Blake 01], [Mori & Malik 02], [Sullivan & Carlsson 02], [Shakhnarovich, Viola & Darrell 03], … • Part-based: [Felzenswalb & Huttenlocher 00], [Ioffe & Forsyth 01], [Song, Goncalves & Perona 03], [Mori, Ren, Efros & Malik 04], … • … … 3
Beyond “Trees” ? • A hard problem! More information is needed. • Important cues that are NOT in the tree model: • Symmetry of clothing/color • “V-shape” formed by the upper legs • Distance/smooth connection between arms and legs • …… 4
Our Approach • Preprocessing with Constrained Delaunay Triangulation • Detecting Candidate Parts from Bottom-up • Learning Pairwise Constraints between Parts • Assembling Parts by Integer Quadratic Programming (IQP) 5
Constrained Delaunay Triangulation • Detect edges with Pb (Probability of Boundary) • Trace contours with Canny’s hysteresis • Recursively split contours into piecewise straight lines • Complete the partial graph with Constrained Delaunay Triangulation 6
Detecting Parts using Parallelism N C1 (L1,1) C2 (L2,2) T • Candidate parts as parallel line segments (Ebenbreite) • (Scale-invariant) Features for parallelism: |Pb1+Pb2|/2, |1-2|, |L1-L2|/|L1+L2|, |(C1-C2)T|/|L1+L2|, |(C1-C2)N|/|L1+L2| • Logistic Classifier 7
C1 C2 Pairwise Constraints between Parts • Scale (width) consistency • Use anthropometric data as groundtruth • Symmetry of appearance (color) • Orientation consistency • Connectivity • Short distance between adjacent parts • “Smooth” connection between non-adjacent parts • short “gaps” on shortest path (on CDT graph) • small maximum angle on the shortest path • few T-junctions/turns on the shortest path 8
Learning Pairwise Constraints 15 hand-labeled images from a skating sequence Empirical distributions of some pairwise features For simplicity, assume all features are Gaussian (future work here as they are clearly non-Gaussian) 9
Assembling Parts as Assignment (Lj1,Ci1=(Lj1)) (Lj2,Ci2=(Lj2)) Candidates {Ci} Parts {Lj} assignment Cost for a partial assignment {(Lj1,Ci1), (Lj2,Ci2)}: 10
Assignment by IQP Q(x)=xTHx • Suppose there are m parts and n candidates, the optimal assignment minimizes a quadratic function where x is a mn1 indicator vector and H is of size mnmn. • This is a well-formulated Integer Quadratic Programming (IQP) problem and has efficient approximate solutions. • We choose an approximation scheme which solves mn linear programs followed by gradient descent. • The approximate scheme produces a ranked list of torso candidates. We consider the top 5 torso candidates and solve the corresponding 5 IQP problems. • We have m=9 and n~150; the total time is less than a minute. 11
Conclusion • To find people under general conditions, we need to go beyond the traditional tree-based model; • Most important constraints for the human body are between pairs of body parts; • Pairwise constraints may be learned from a small set of training examples; • Integer Quadratic Programming (IQP) efficiently finds optimal configurations under pairwise constraints. 15