210 likes | 381 Views
Street-to-Shop: Cross-Scenario Clothing Retrieval via Parts Alignment and Auxiliary Set. Si Liu+, Zheng Song, Guangcan Liu , Changsheng Xu +, Hanqing Lu+, Shuicheng Yan* * ECE Department, National University of Singapore + NLPR, Institute of Automation, Chinese Academy of Science.
E N D
Street-to-Shop: Cross-Scenario Clothing Retrievalvia Parts Alignment and Auxiliary Set Si Liu+, ZhengSong, Guangcan Liu, ChangshengXu+, Hanqing Lu+, ShuichengYan* * ECE Department, National University of Singapore + NLPR, Institute of Automation, Chinese Academy of Science
Outline Introduction Related Work Dataset Construction Framework Towards Cross-Scenario by Human Parts Alignment Towards Cross-Scenario by Auxiliary Set Experiments Conclusions and Future Work
Introduction Given a human photo captured on street, finding similar clothing from online shops Collect a large online shopping dataset and daily photo dataset
Introduction • Two-step calculation to handle the cross-scenario discrepancies • human parts alignment • auxiliary set
Related Work Clothing Study Parts Appearance based Human Attribute Analysis Unsupervised Transfer Learning
Dataset Construction • Clothing Image Collection • Online Shopping (OS) dataset • Daily Photo (DP) dataset • Clothing Attribute Labeling • global • upper-body • lower-body
Towards Cross-Scenario by Human PartsAlignment • Train one human upper body and one human lower body detector • Extract 5 kinds of features from the upper-body parts and lower-body parts • HOG, LBP, Color moment, Color histogram and skin descriptor
Towards Cross-Scenario by Auxiliary Set Aim is to find a new representation of daily photo, so that it can be directly compared with clothing in the online shopping dataset.
Towards Cross-Scenario by Auxiliary Set as image features from the OS dataset as image features from the auxiliary DP dataset : sparse matrix : reconstruction error convert into the following equivalent problem then minimize the following augmented Lagrange function
Experiments • Experimental Setting • Evaluation Criterion q :query image Rel(i) : the groundtruth relevance between q and the ith ranked image k : top k retrieved datum N : normalization constant
Experiments • Within-Scenario Vs. Cross-Scenario • using the OS query set and the DP query set to retrieve images in the OS training set
Experiments Performances of Different Features and Parts
Experiments • Parts Alignment for Cross-Scenario Retrieval • To validate the effectiveness, we compare our method with a baseline using global features • Auxiliary Set for Cross-Scenario Retrieval • implement the Algorithm 1 with λ1= 0.1 and λ2 =0.01 to learn the transfer matrix
Experiments Exemplar Retrieval Results
Experiments Extension: Interactive Clothing Retrieval
Conclusions and Future Work • We propose a solution including two key components • human/clothing parts alignment to handle human pose variation • an auxiliary daily photo dataset to bridge cross-scenario discrepancies • In the future, our system can be extended as an online learning system