INTRODUCTION

1. Tensor-based High-order Semantic Relation Transfer for Semantic Scene Segmentation Heesoo Myeong and Kyoung Mu LeeDepartment of ECE, ASRI, Seoul National University, Seoul, Korea sky sky http://cv.snu.ac.kr tree tree building building building building car car car car road road road sidewalk Objective Function PROPOSED METHOD EXPERIMENTS CONCLUSION INTRODUCTION Results on Jain et al. Dataset sky tree tree tree mountain sky • We separately deal with the semantic relations transfer problem with respect to • The quadratic objective function with respect to as where is the triplet-wise similarity between two region triplets and building sky building tree Overview sky sky Goal building bison tree person bison • Semantic scene segmentation: identifying and segmenting all objects in a scene road road grass building building road grass person car road car sky High-order relations in the training dataset road tree car tree car sky sky sky sky sky sky sky tree building building building building tree tree car Query Ground truth Proposed Query Ground truth Proposed Inference person Results on LMO Dataset car road car building mountain mountain car car road Test image Semantic scene segmentation … • Use fully connected third-order Markov random field (MRF) model: Key idea road road sidewalk sidewalk Query image Integrated high-order relation Semantic segmentation • Exploiting high-order(mostly third-order) semantic relation sky sky sky sky building building tree building where represents data term, represents pairwise term, and is confidence score by the semantic relation transfer algorithm building window door car building 1. For a test image, retrieve similar training images using global features 2. Apply semantic relation transfer algorithm to transfer third-order semantic relation from each training image to the test image 3. Integrate the high-order score into MRF optimization framework and obtain semantic scene segmentation building road car sidewalk window door Groundtruth window Retrieved images Annotations of retrieved images Predicted top scored high-order relation Quantitative Results on Standard Datasets Previous works & Limitations Query Ground truth Proposed Query Ground truth Proposed Semantic Relation Transfer Algorithm Results on Polo Dataset • Jain et al. dataset (Jain et al., ECCV10): • 250 training images, 100 test images, 19 labels • SIFT Flow dataset (Liu et al., CVPR09): • 2,488 training images, 200 test images, 33 labels • Polo dataset (Zhang et al., CVPR11): • 80 training images, 237 test images, 6 labels • Table 1: Per-pixel classification rates and (average per-class rates) person • Conventional context models mainly focus on learning pairwise relationships between objects • Pairwise relations are not enough to represent high-level contextual knowledge within images person horse horse horse horse Problem Statement grass grass grass grass • Among superpixels(is the number of total superpixels, is the number of object classes), third-order semantic relation is defined as a number of semantic tensors • The variable indicates confidence score of how likely the region triplet would be labeled as , respectively • To describe observed third-order semantic relation within the retrieved image, we define another number of semantic tensors • where denotes the ground truth class of region • Now, the semantic relation transfer problem is reformulated as the problem of estimating the magnitude of confidence scores for all superpixel triplets and for all object class triplets based on person person person person horse horse horse horse grass grass grass grass Query Ground truth Proposed Query Ground truth Proposed Pairwise semantic relation Our Contributions • We have presented a novel approach to learn high-order semantic relations of regions in a nonparametric manner • We develop a novel semantic tensor representation of the high-order semantic relations • We cast the high-order semantic relation transfer problem as a quadratic objective function of semantic tensors and propose an efficient approximate algorithm • The use of high-order semantic relations for semantic segmentation • A novel tensor-based representation of high-order semantic relations • A quadratic objective function for learning the semantic tensor and an efficient approximate algorithm