1 / 52

Hierarchical Novelty Detection for Visual Object Recognition

Hierarchical Novelty Detection for Visual Object Recognition. Kibok Lee * , Kimin Lee † , Kyle Min * , Yuting Zhang * , Jinwoo Shin † , Honglak Lee *‡ University of Michigan * , KAIST † , Google Brain ‡. Conventional novelty detection framework does not provide more

ronaldmoore
Download Presentation

Hierarchical Novelty Detection for Visual Object Recognition

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Hierarchical NoveltyDetection for Visual ObjectRecognition Kibok Lee*, Kimin Lee†, KyleMin*, Yuting Zhang*, Jinwoo Shin†, HonglakLee*‡ University of Michigan*, KAIST†, GoogleBrain‡

  2. Conventionalnoveltydetectionframeworkdoesnotprovidemore • informationthan“novelty”ofanobject. • Supposewehavetrainingdatalike… Why Hierarchical NoveltyDetection? Persiancat Siamesecat Pomeranian Welshcorgi

  3. Then,supposewehavetestimageslike… • Testimage: Why Hierarchical NoveltyDetection? Persiancat Siamesecat Pomeranian Welshcorgi

  4. Testimage: Why Hierarchical NoveltyDetection? animal cat Persiancat Siamesecat dog Pomeranian Welshcorgi

  5. Ourhierarchicalnoveltydetectionframeworkaimstofindthemost specific class label of any data on the hierarchical taxonomy built with knownlabels. Why Hierarchical NoveltyDetection? Testimage:

  6. Thisframeworkcanbepotentiallyusefulforautomaticallyor interactively organizing a customizedtaxonomy • Company’s productcatalog • Wildlifemonitoring • Personal photolibrary • bysuggestingclosestcategoriesforanimagefromnovel categories. • new consumerproducts • unregistered animalspecies • untagged scenes orplaces Why Hierarchical NoveltyDetection?

  7. Ourtaxonomyhasthreetypesofclasses. • Known leaf classes are seen duringtraining • Superclassesareancestorsofleafclasses,alsoknown • Novel classes are unseen duringtraining • Theirexpectedpredictionistheclosestsuperclassinthetaxonomyinourtask HierarchicalTaxonomy

  8. HierarchicalTaxonomy

  9. HierarchicalTaxonomy

  10. HierarchicalTaxonomy

  11. HierarchicalTaxonomy

  12. HierarchicalTaxonomy

  13. HierarchicalTaxonomy

  14. Multi-stageclassification • Untilarrivingataknownleafclass Approach - Top-down (TD)Method .2 .8 .7 .1 .2

  15. Multi-stageclassification • Untilarrivingataknownleafclass • Or classification is unconfident(novel) Approach - Top-down (TD)Method .5 .5

  16. Classificationrule: • Classification is confidentif Approach - Top-down (TD)Method

  17. Trainingobjective: Approach - Top-down (TD)Method

  18. Representallprobabilitiesofknownleafandnovelclassesina singlevector • Add virtual novelclasses Approach - FlattenMethod

  19. Representallprobabilitiesofknownleafandnovelclassesina singlevector • Add virtual novelclasses • Andthenflattenthestructure Approach - FlattenMethod

  20. Representallprobabilitiesofknownleafandnovelclassesina singlevector • Add virtual novelclasses • Andthenflattenthestructure Approach - FlattenMethod

  21. Classificationrule: • We proposetwostrategiestotrainthismodel. Approach - FlattenMethod

  22. Datarelabeling • Fill novel classes by hierarchicalrelabeling Approach - FlattenMethod animal cat dog Persiancat Siamesecat Pomeranian Welshcorgi

  23. Datarelabeling • Fill novel classes by hierarchicalrelabeling Approach - FlattenMethod animal cat dog Persiancat Siamesecat Pomeranian Welshcorgi

  24. Datarelabeling • Fill novel classes by hierarchicalrelabeling Approach - FlattenMethod animal cat dog Persiancat Siamesecat Pomeranian Welshcorgi

  25. Datarelabeling • Fill novel classes by hierarchicalrelabeling • Relabeling ratecan be chosen by validation Approach - FlattenMethod animal cat dog Persiancat Siamesecat Pomeranian Welshcorgi

  26. Datarelabeling • Trainingobjective: Approach - FlattenMethod animal cat dog Persiancat Siamesecat Pomeranian Welshcorgi

  27. Leave-one-out (LOO)strategy • Generate deficienttaxonomies • Andthentrainthemodelwiththem Approach - FlattenMethod animal cat dog Persiancat Siamesecat Pomeranian Welshcorgi

  28. Leave-one-out (LOO)strategy • e.g.,when Approach - FlattenMethod animal cat dog Persiancat Siamesecat Pomeranian Welshcorgi

  29. Leave-one-out (LOO)strategy • e.g.,when Approach - FlattenMethod animal novelanimal dog Pomeranian Welshcorgi

  30. Leave-one-out (LOO)strategy • Trainingobjective: Approach - FlattenMethod animal novelanimal dog Pomeranian Welshcorgi

  31. ComputeandenumeratetheoutputofTD • AndthenfeedittoLOO • Combinedmethodutilizestheircomplementarybenefits. • Top-down method leverages hierarchical structureinformation. • Butitsuffersfromerroraggregationoverhierarchy. • Flattenmethodavoidserroraggregationoverhierarchy. • Butitdoesnotleveragehierarchicalstructureinformation. Approach - Combined Method(TD+LOO)

  32. Quantitativeresults Experiments - Hierarchical NoveltyDetection

  33. Comparedalgorithms • Baseline: DARTS (Deng et al.,2012) • Ours: Relabel, LOO,TD+LOO Experiments - Hierarchical NoveltyDetection J.Deng,J.Krause,A.C.Berg,andL.Fei-Fei.“Hedgingyourbets:Optimizingaccuracy-specificitytrade-offsinlargescale visualrecognition.”InCVPR,2012

  34. Datasets • ImageNet: 1k known, 16k novelclasses • AwA2: 40 known, 10 novelclasses • CUB: 150 known, 50 novelclasses Experiments - Hierarchical NoveltyDetection

  35. Metrics • Novel class accuracy @ knownclass accuracy = 50% • Byaddinganappropriatescorebiastoallnovelclasses • Area under known-novel class accuracycurve • By varying the novel class scorebias Experiments - Hierarchical NoveltyDetection

  36. Metrics • Novel class accuracy @ knownclass accuracy = 50% • Byaddinganappropriatescorebiastoallnovelclasses • Area under known-novel class accuracycurve • By varying the novel class scorebias • (a)ImageNet (b)AwA2 Experiments - Hierarchical NoveltyDetection (c)CUB

  37. Metrics • Novel class accuracy @ knownclass accuracy = 50% • Byaddinganappropriatescorebiastoallnovelclasses • Area under known-novel class accuracycurve • By varying the novel class scorebias • (a)ImageNet (b)AwA2 Experiments - Hierarchical NoveltyDetection (c)CUB

  38. Metrics • Novel class accuracy @ knownclass accuracy = 50% • Byaddinganappropriatescorebiastoallnovelclasses • Area under known-novel class accuracycurve • By varying the novel class scorebias • (a)ImageNet (b)AwA2 Experiments - Hierarchical NoveltyDetection (c)CUB

  39. Qualitativeresults Experiments - Hierarchical NoveltyDetection Novel class: Americanfoxhound

  40. Qualitativeresults Experiments - Hierarchical NoveltyDetection

  41. Qualitativeresults Experiments - Hierarchical NoveltyDetection

  42. Qualitativeresults Experiments - Hierarchical NoveltyDetection

  43. Qualitativeresults Experiments - Hierarchical NoveltyDetection

  44. Quantitativeresults Experiments - Generalized Zero-ShotLearning

  45. Semanticembeddings • Attributes (numeric attributevalues) • Wordvector(similarityamongwordsinrealcoordinatespace) • Hierarchicalembedding Experiments - Generalized Zero-ShotLearning

  46. Compared hierarchicalembeddings • Baseline: Path (Akata etal., 2015) • Distance between classes onhierarchy • Ours: Top-down(TD) • Expectedoutputofourtop-downmodel Experiments - Generalized Zero-ShotLearning Z.Akata,S.Reed,D.Walter, H.Lee,andB.Schiele.“Evaluationofoutputembeddingsforfine-grainedimageclassification.”InCVPR,2015.

  47. Datasets • AwA1,2: 40 known, 10 novelclasses • CUB: 150 known, 50 novelclasses Experiments - Generalized Zero-ShotLearning

  48. Metrics • Unseen class accuracy(ZSL) • Area under seen-unseen curve(GZSL) • By varying the unseen class scorebias Experiments - Generalized Zero-ShotLearning

  49. Metrics • Unseen class accuracy(ZSL) • Area under seen-unseen curve(GZSL) • By varying the unseen class scorebias • AwA1 (b)AwA2 Experiments - Generalized Zero-ShotLearning (c)CUB

  50. Metrics • Unseen class accuracy(ZSL) • Area under seen-unseen curve(GZSL) • By varying the unseen class scorebias • AwA1 (b)AwA2 Experiments - Generalized Zero-ShotLearning (c)CUB

More Related