Classifying Visual Objects Regardless of Depictive Style

1. Classifying Visual Objects Regardless of Depictive Style Qi Wu, Peter Hall Department of Computer Science University of Bath

2. Summary • Conventional Comp.Vis. classifiers do not generalise well across depictive styles. • We propose new visual class model,one invariant to depictive style. • Experiments validate our model.

3. People can see objects in a wide variety of depictive styles. Photos Artwork

4. Literature Gap: BoW does not generalise across depictive styles 47% (Dense SIFT) Photos Artwork 51% (Dense SIFT) Photos Artwork

5. Our solution: A new Visual Class Model that does generalise across styles. 47% (Dense SIFT) Photos Artwork 64% 51% (Dense SIFT) Photos Artwork 67%

6. A New Visual Class Model • We assume an object class is characterised by: • the qualitative shape of object parts, • the structural arrangement of those parts. • A hierarchical graph model per image: • coarse-to-fine representation (layered), • nodes labelled by primitive shapes, • abstracting region shape brings greater robustness. • arcs labelled with displacement vectors • Median graph models: • aggregates models from several instances, • single class model.

7. Making a VCM (a): An input collection. (b): Probability maps for each input image, and graph models for each map. (c): The median graph model for the whole class. (d): The refined median graph as the final class model

8. A schematic VCM • A hierarchical description • Berkeley segmentation • Filtering process using cLge • A graph • Arcs at same level denote touching neighbours. • Arcs between layers link parent – children. • Nodes label • A 6-elements probability vector. • The probability that a region belongs to a given prime shape class.

9. Prime Shapes,BMVC 2012

10. Prime Shapes • Does a set of elementary planar shapes appear commonly in the world ? • Art provides strong anecdotal evidence “yes” • 20th century Western Art --- Cubism

11. Determine Prime Shapes • A fully unsupervised framework

12. Determine Prime Shapes

13. Back to our model…

14. Build graphs, one for each image Left: graph model. Right: Object broken in primitive shapes

15. Compute an initial Visual Class Model • Median Graph • First compute the graph distance between each pair. • Using the distance matrix to embed graph into a vector space • Compute the Euclidean Median of all the data points. • Transfer the median vector back to graph using a state-of-art method proposed in [Ferrer and Valveny, 2008]

16. Refine the Visual Class Model The initial model contains nodes and arcs that derive from visual clutter in back ground of images in the training set • Refine the model • Match the median back into each training image. • Count the number of times a given node in the model appears in the training data. • Delete all nodes below a frequency threshold., which is computed via maximising matching score.

17. Some Examples

18. Experiments • Compare with other two methods • PHOW features (Dense SIFT) [Bosch and Zisserman, ICCV 2007] • Local PAS features [Ferrari and Jurie, IJCV 2010] • Structure Only [Bai and Song, CVIU 2011]

19. Results

20. Conclusions • It’s possible to learn models of objects classes that generalise across depictive styles. • Many applications are promised. • Just a first step • Simplify the model, still too much nodes and arcs. • Time consuming. • Additional labelling • Move to object localisation.

21. Questions?

22. One application of Prime shapes