Sketch Tokens: A Learned Mid-level Representation for Contour and Object Detection

1. Sketch Tokens: A Learned Mid-level Representation for Contour and Object Detection Joseph J. Lim (MIT), C. Lawrence Zitnick (MSR), Piotr Dollár (MSR) Contour Detection (BSDS 500) Object Detection on PASCAL2007 Overview Method Goal: learn and detect local contour-based representation for mid-level features Defining Sketch Tokens Detecting Sketch Tokens Given a set of sketch token classes, our goal is to detect them in color images. Each color patch’s ground truth class is assigned to one of Sketch Token or background class. We used random forest classifier with various features (e.g. CIE-LUV intensity, orientation, and self-similarity). We are given a set of images, I, and its corresponding set of binary contour images, S. • Sketch Tokens: • Local edge structures (e.g. straight lines, t-junctions, y-junctions) • Discovered from human-generated image sketches t1 Sketch Tokens are clusters of extracted patches from the binary contour images S. t2 t3 t5 t7 t4 t6 • We demonstrate our approach on both top-down and bottom-up tasks. • State-of-the-art result on contour detection, while 200x faster • Large improvements on object and pedestrian detection. • Each patch has a fixed size of 35x35, and its center pixel must be on a labeled contour • 150 clusters are extracted using K-means on Daisy descriptors computed on binary patches. t8 t9 t14 t15 Sketch Tokens INRIA Pedestrian Detection In addition to standard features used in Dollár et. al.’s implementation, we added Sketch Token responses. We used Sketch Token responses (150 st + 1 bg dimension) on images as additional features to the deformable parts model detector. On average, we improved 3.8 AP. Conclusion 200x faster! MATLAB code is available on the website