1 / 22

Context-based Visual Concept Detection Using Domain Adaptive Semantic Diffusion

Context-based Visual Concept Detection Using Domain Adaptive Semantic Diffusion. Yu-Gang Jiang ‡ , Jun Wang ‡ , Shih-Fu Chang ‡ , Chong-Wah Ngo † † VIREO Research Group (VIREO), City University of Hong Kong ‡ Digital Video and Multimedia Lab (DVMM), Columbia University.

annis
Download Presentation

Context-based Visual Concept Detection Using Domain Adaptive Semantic Diffusion

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Context-based Visual Concept Detection Using Domain Adaptive Semantic Diffusion Yu-Gang Jiang‡, Jun Wang‡, Shih-Fu Chang‡, Chong-Wah Ngo† †VIREO Research Group (VIREO), City University of Hong Kong ‡ Digital Video and Multimedia Lab (DVMM), Columbia University NIST TRECVID Workshop, Nov. 2009

  2. Overview: framework Local Feature Global Feature SVM Classifiers 6 5 VIREO-374: 374 LSCOM concept detectors Domain Adaptive Semantic Diffusion 1-4

  3. Overview: performance DASD Local + global features Local feature alone • Local feature is still the most powerful component (MAP=0.150) • Global features help a little bit (MAP=0.156) • DASD further contributes incrementally to the final detection

  4. Overview: framework Local Feature Global Feature SVM Classifiers 6 5 VIREO-374: 374 LSCOM concept detectors Domain Adaptive Semantic Diffusion 1-4

  5. Local feature representation Chang et al TRECVID 2008; Jiang, Yang, Ngo & Hauptmann, IEEE TMM, to appear

  6. Context-based concept detection Local Feature Global Feature SVM Classifiers 6 5 VIREO-374: 374 LSCOM concept detectors DASD: Domain Adaptive Semantic Diffusion 1-4

  7. DASD - motivation • Most existing methods aim at the assignment of concept labels individually • but concepts do not occur in isolation! military personnel smoke building explosion_fire vehicle road outdoor

  8. DASD - motivation • Most existing methods aim at the assignment of concept labels individually • but concepts do not occur in isolation! • Domain change between training and testing data was not considered Broadcast News Videos Documentary Videos

  9. DASD - overview road vehicle sky water 0.01 0.36 0.53 0.17 0.23 0.01 0.12 0.91 0.18 0.05 0.11 0.58 0.10 0.13 0.02 0.05 0.19 0.80 0.46 0.13 Jiang, Wang, Chang & Ngo, ICCV 2009

  10. DASD - overview • Domain adaptive semantic diffusion (DASD) • Semantic graph • Nodes are concepts • Edges represent concept correlation • Graph diffusion • Smooth concept detection scores w.r.t the concept correlation road 0.1 0.6 0.1 0.1 0.0 … 0.8 0.0 0.4 0.5 0.2 0.8 … 0.7 0.0 0.1 0.9 0.2 0.1 … 0.3 0.1 0.2 0.8 0.5 0.1 … 0.4 vehicle Water sky

  11. DASD - formulation • Energy function Detection score of concept cion test samples Concept affinity

  12. DASD - formulation (cont.) • Gradually smooth the function makes the detection scores in accordance with the concept relationships Detection score smoothing process

  13. DASD - formulation (cont.) • Graph adaptation Graph adaptation process

  14. Graph adaptation - example Iteration: 8 Iteration: 12 Iteration: 0 Iteration: 4 Iteration: 16 Iteration: 20 Broadcast news video domain Documentary video domain

  15. Experiments on TV ’05-’07 • Baseline detectors • VIREO-374 • Graph construction: • Ground-truth labels on TRECVID 2005 TRECVID 05/06 (Broadcast News Videos) TRECVID 07 (Documentary Videos) WALKING MAP SPORTS WEATHER SPORTS WEATHER OFFICE BUS CLASSROOM PEOPLE-MARCHING CORP. LEADER DESERT MOUNTAIN DESERT WATER NIGHT TIME TELEPHONE MOUNTAIN EXPLOSION- FIRE TRUCK OFFICE BUILDING ANIMAL TWO PEOPLE STREET MILITARY POLICE

  16. Results on TV ’05-’07 • Performance gain on TRECVID 05-07 Datasets • SD: semantic diffusion (without graph adaptation) • Consistent improvement over all 3 data sets • DASD: domain adaptive semantic diffusion • Graph adaptation further improves the performance

  17. Results on TV ’05-’07 (cont.) TRECVID 2006 Test Data Comparison with the state-of-the-arts

  18. Results on TRECVID ’09 10% 5% 30%

  19. Results on TRECVID ’09 (cont.) • Quality of contextual detectors (VIREO-374) 5% DASD performance gain TV09 detectors 16% 18% TV07 detectors TV06 detectors Context VIREO-374

  20. DASD - computational time • Complexity is O(mn) • m: # concepts; n: # video shots • Only 2 milliseconds per shot/keyframe!

  21. Summary • A well-designed approach using local features achieves good results for concept detection. • Context information is helpful ! • Domain adaptive semantic diffusion • effective for enhancing concept detection accuracy • can alleviate the effect of data domain changes • highly efficient ! • Future directions include: • detector reliability: diffusion over directed graph • web data annotation: utilize contextual information to improve the quality of tags • Source code available for download from DVMM lab research page

  22. Thank you!

More Related