1 / 50

Cl a ss 5: A t t ri b utes and S e mant i c F e a tu r es R o g e rio F e ris, F e b 21, 2013

Cl a ss 5: A t t ri b utes and S e mant i c F e a tu r es R o g e rio F e ris, F e b 21, 2013 E E CS 6890 – T o p ics in I n f orm a tion P r o ce ss i n g Spr i ng 2013, Co l um b i a Un i v e r sity h t tp:// r o g er i o f er i s . c om/Visual R e c og n i t i onA n dS e a r c h.

yale
Download Presentation

Cl a ss 5: A t t ri b utes and S e mant i c F e a tu r es R o g e rio F e ris, F e b 21, 2013

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Class5:AttributesandSemanticFeatures Rogerio Feris, Feb21, 2013 EECS 6890–Topicsin InformationProcessing Spring2013, ColumbiaUniversity http://rogerioferis.com/VisualRecognitionAndSearch

  2. ProjectReport  Thanksforsendingtheprojectproposals!  Projectupdatepresentations(10min pergroup)   March14 April11  Details will beprovidedin the coursewebsite VisualRecognition And Search ColumbiaUniversity, Spring 2013

  3. Plan for Today     IntroductiontoSemanticFeatures Attribute-basedClassificationand Search Attributesfor Fine-Grained Classification RelativeAttributes  ProjectProposal Presentations VisualRecognition And Search ColumbiaUniversity, Spring 2013

  4. Semantic Features Usethescores of semanticclassifiersas high-levelfeatures InputImage Off-the-shelf Classifiers … SandClassifier WaterClassifier SkyClassifier Score Score Score Compact/ powerful descriptor withsemantic meaning(allows “explaining”thedecision) SemanticFeatures BeachClassifier VisualRecognition And Search ColumbiaUniversity, Spring 2013

  5. Semantic Features(Frame-Level) IllustrationofEarly IBMwork(multimediacommunity)describing this concept [JohnSmith etal,MultimediaSemanticIndexing UsingModelVectors, ICME 2003] Concatenation/DimensionalityReduction VisualRecognition And Search ColumbiaUniversity, Spring 2013

  6. Semantic Features(Frame-level) Systemevolvedtothe IBMMultimediaAnalysisand Retrieval System(IMARS) [RongYanet al,Model-SharedSubspace BoostingforMulti-labelClassification,KDD2007] Discriminativesemanticbasis Ensemble Learning Rapid eventmodeling, e.g.,“accidentwith high- speedskidding” VisualRecognition And Search ColumbiaUniversity, Spring 2013

  7. Classemes (Frame-level) Descriptor isformedbyconcatenating theoutputsofweakly trained classifierscalledclassemes(trained withnoisy labels) [L.Torresanietal,EfficientObjectCategoryRecognitionUsingClassemes,ECCV2010] Imagesusedtotrain the“table”classeme(from Googleimage search) Noisy Labels VisualRecognition And Search ColumbiaUniversity, Spring 2013

  8. Classemes (Frame-level) CompactandEfficientDescriptor,usefulfor large-scale classification Featuresare notreallysemantic! VisualRecognition And Search ColumbiaUniversity, Spring 2013

  9. Semantic Features(Object Level) Object Bank [Li-JiaLietal,ObjectBank:AHigh-LevelImageRepresentation forSceneClassificationandSemanticFeatureSparsification] http://vision.stanford.edu/projects/objectbank/ State-of-the-artscene classificationresults (~7secondsperimage) VisualRecognition And Search ColumbiaUniversity, Spring 2013

  10. Semantic Attributes Describing Naming ? Bald Beard RedShirt  Modifiersrather than (or inaddition to)nouns  Semanticpropertiesthat areshared amongobjects  Attributesarecategory independentand transferrable VisualRecognition And Search ColumbiaUniversity, Spring 2013

  11. Attribute-Based Search VisualRecognition And Search ColumbiaUniversity, Spring 2013

  12. PeopleSearch in SurveillanceVideos TraditionalApproaches:FaceRecognition(“Naming”)  Facerecognitionisverychallengingunderlightingchanges,posevariation,andlow- resolutionimagery(typicalconditionsinsurveillancescenarios) Attribute-basedPeople Search(“Describing”) [Vaqueroetal,Attribute-basedPeopleSearchinSurveillanceEnvironments,WACV2009]  Ratherthanrelyingonfacerecognitiononly,acomplementarypeoplesearch frameworkbasedonsemanticattributesisprovided QueryExample: “Showmeallbaldpeopleatthe42ndstreetstationlastmonthwithdarkskin,wearing sunglasses,wearingaredjacket” VisualRecognition And Search ColumbiaUniversity, Spring 2013

  13. PeopleSearch in SurveillanceVideos VisualRecognition And Search ColumbiaUniversity, Spring 2013

  14. PeopleSearch in SurveillanceVideos VisualRecognition And Search ColumbiaUniversity, Spring 2013

  15. PeopleSearch in SurveillanceVideos PeopleSearchbased on textual descriptions-It does notrequire trainingimagesforthe targetsuspect. Robustness: attributedetectorsaretrainedusinglots oftraining imagescoveringdifferentlightingconditions,pose variation,etc. Workswellinlow-resolutionimagery(typicalinvideosurveillance scenarios) VisualRecognition And Search ColumbiaUniversity, Spring 2013

  16. PeopleSearch in SurveillanceVideos Modelingattributecorrelations [Siddiquie,FerisandDavis,“ImageRankingandRetrievalBasedon Multi-AttributeQueries”,CVPR2011] VisualRecognition And Search ColumbiaUniversity, Spring 2013

  17. Attribute-Based Classification VisualRecognition And Search ColumbiaUniversity, Spring 2013

  18. Attribute-basedClassification Recognitionof Unseen Classes(Zero-ShotLearning) [Lampertet al,LearningTo DetectUnseenObject ClassesbyBetween-ClassAttribute Transfer, CVPR2009] 1) Train semanticattributeclassifiers 2) Obtainaclassifier for anunseen object(notrainingsamples) byjust specifyingwhichattributesit has VisualRecognition And Search ColumbiaUniversity, Spring 2013

  19. Attribute-basedClassification Unseencategories Flat multi-class classification Unseencategories SemanticAttribute Classifiers Attribute-based classification VisualRecognition And Search ColumbiaUniversity, Spring 2013

  20. Attribute-basedClassification Actionrecognition[Liual, CVPR2011] Faceverification[Kumar etal,ICCV2009] BirdCategorization[Farrell etal,ICCV 2011] Animal Recognition [Lampertetal, CVPR2009] PersonRe-identification [Layneetal,BMVC 2012] Many more!Significant growthin thepast few years VisualRecognition And Search ColumbiaUniversity, Spring 2013

  21. Attribute-basedClassification Note:Severalrecent methodsuse the term“attributes”to refer to non-semantic modeloutputs In this caseattributesare justmid-level features,likePCA, hidden layersin neuralnets, …(non-interpretablesplits) VisualRecognition And Search ColumbiaUniversity, Spring 2013

  22. Attribute-basedClassification http://rogerioferis.com/VisualRecognitionAndSearch/Resources.html VisualRecognition And Search ColumbiaUniversity, Spring 2013

  23. Attributesfor Fine-Grained Categorization VisualRecognition And Search ColumbiaUniversity, Spring 2013

  24. Fine-Grained Categorization VisualRecognition And Search ColumbiaUniversity, Spring 2013

  25. Fine-Grained Categorization VisualRecognition And Search ColumbiaUniversity, Spring 2013

  26. Fine-Grained Categorization VisualRecognition And Search ColumbiaUniversity, Spring 2013

  27. Fine-Grained Categorization Visipedia(http://http://visipedia.org/) Machinescollaboratingwithhumans to organizevisualknowledge,connectingtext to images, imagesto text, andimagesto images Easyannotationinterfaceforexperts(poweredbycomputervision) VisualQuery:Fine-grainedBirdCategorization Picturecredit:Serge Belongie VisualRecognition And Search ColumbiaUniversity, Spring 2013

  28. Fine-Grained Categorization African Isit anAfricanor IndianElephant? Indian Example-basedFine-GrainedCategorizationis Hard!! SlideCredit:ChristophLampert VisualRecognition And Search ColumbiaUniversity, Spring 2013

  29. Fine-Grained Categorization African Isit anAfricanor IndianElephant? Indian SmallerEars LargerEars Visualdistinctionofsubordinatecategoriesmaybequitesubtle,usually basedonParts and Attributes VisualRecognition And Search ColumbiaUniversity, Spring 2013

  30. Fine-Grained Categorization Standard classificationmethods may notbesuitablebecausethe variationbetween classesissmall… [B.Yao,CVPR2012] Codebook …and intra-classvariationisstillhigh. VisualRecognition And Search ColumbiaUniversity, Spring 2013

  31. Fine-Grained Categorization Humans rely onfield guides! Field guides usually refer to partsand attributesofthe object SlideCredit:PietroPerona VisualRecognition And Search ColumbiaUniversity, Spring 2013

  32. Fine-Grained Categorization [Branson etal, VisualRecognitionwithHumansinthe Loop,ECCV2010] VisualRecognition And Search ColumbiaUniversity, Spring 2013

  33. Fine-Grained Categorization [Branson etal, VisualRecognitionwithHumansinthe Loop,ECCV2010] Computervision reduces the amountofhuman-interaction (minimizesthe numberofquestions) VisualRecognition And Search ColumbiaUniversity, Spring 2013

  34. Fine-Grained Categorization [Wahetal, MulticlassRecognitionand PartLocalizationwithHumansin theLoop, ICCV2011] Localizedpartand attributedetectors. Questionsincludeaskingtheusertolocalizeparts. VisualRecognition And Search ColumbiaUniversity, Spring 2013

  35. Fine-Grained Categorization http://www.vision.caltech.edu/visipedia/CUB-200-2011.html VisualRecognition And Search ColumbiaUniversity, Spring 2013

  36. Fine-Grained Categorization VideoDemo: http://www.youtube.com/watch?v=_ReKVqnDXzA VisualRecognition And Search ColumbiaUniversity, Spring 2013

  37. Likea normal fieldguide… that and you can searchandsort   with visualrecognition See N.Kumar et al, "Leafsnap:AComputer VisionSystemfor Automatic PlantSpecies Identification,ECCV 2012

  38. Nearly1 million downloads  40knewuserspermonth 100kactive users million imagestaken 100knewimages/month 100kuserswith>5images   1.7    Usersfromallovertheworld Botanists,educators,kids,hobbyists, photographers,…   Slide Credit:NeerajKumar

  39. Fine-Grained Categorization Checkthefine-grainedvisual categorizationworkshop: http://www.fgvc.org/ VisualRecognition And Search ColumbiaUniversity, Spring 2013

  40. RelativeAttributes VisualRecognition And Search ColumbiaUniversity, Spring 2013

  41. Relative Attributes [Parikh &Grauman,RelativeAttributes,ICCV 2011] Smiling ??? Not smiling ??? Natural Notnatural Slidecredit:Parikh&Grauman VisualRecognition And Search ColumbiaUniversity, Spring 2013

  42. LearningRelative Attributes Foreach attribute e.g., “openness” Supervision consists of: Orderedpairs Similarpairs Slidecredit:Parikh&Grauman VisualRecognition And Search ColumbiaUniversity, Spring 2013

  43. LearningRelative Attributes Learna rankingfunction Imagefeatures Learnedparameters that best satisfies the constraints: Slidecredit:Parikh&Grauman VisualRecognition And Search ColumbiaUniversity, Spring 2013

  44. LearningRelative Attributes Max-margin learning to rank formulation 2 1 4 3 6 5 Based on[Joachims2002] RankMargin Image RelativeAttributeScore Slidecredit:Parikh&Grauman VisualRecognition And Search ColumbiaUniversity, Spring 2013

  45. Relative Zero-ShotLearning  Each image is convertedinto a vector ofrelative attributescores indicating thestrength of eachattribute  A Gaussiandistributionforeach categoryis builtintherelative attribute space.Thedistributionof unseen categoriesis estimatedbased onthe specifiedconstraints andthe distributionsof seen categories  Max-likelihood is thenusedfor classification Blue:Seen class Green:Unseen class VisualRecognition And Search ColumbiaUniversity, Spring 2013

  46. Relative Image Description Slidecredit:Parikh&Grauman VisualRecognition And Search ColumbiaUniversity, Spring 2013

  47. WhittleSearch Slidecredit:KristenGrauman VisualRecognition And Search ColumbiaUniversity, Spring 2013

  48. http://rogerioferis.com/PartsAndAttributes/ http://pub.ist.ac.at/~chl/PnA2012/ VisualRecognition And Search ColumbiaUniversity, Spring 2013

  49. Summary Semanticattributeclassifierscan beusefulfor: Describingimagesofunknownobjects [Farhadiet al,CVPR2009] Recognizingunseenclasses[Lampertet al,CVPR2009] Reducingdataset bias (trainedacross classes) Effective objectsearch insurveillancevideos[Vaqueroet al,WACV2009] Compactdescriptors/Efficientimage retrieval[Douzeetal, CVPR2011] Fine-grainedobjectcategorization [Wahetal, ICCV 2011] Face verification[Kumar etal, 2009],Actionrecognition[Liuetal, CVPR 2011], Personre-identification[Layneetal, BMVC 2012] andother classificationtasks. Otherapplications,suchassentencegeneration fromimages [Kulkarniet al, CVPR2011], imageaesthetics prediction[Dharet alCVPR2011], … VisualRecognition And Search ColumbiaUniversity, Spring 2013

  50. Summary Extensive annotationmay berequiredfor attributeclassifiers Class-attribute relations may beautomatically extractedfrom textual sources [Rohrbachet al,WhatHelpsWhere–AndWhy?SemanticRelatednessfor KnowledgeTransfer",CVPR2010];[Bergetal, AutomaticAttribute Discovery andCharacterizationfrom Noisy WebData,ECCV2008]. SemanticAttributesmay notbediscriminative Variousmethodscombinesemanticattributeswith“discriminative attributes” (non-semantic)for classification(e.g.,[Farhadietal, CVPR2009]).Construction of nameable+ discriminativeattributeshasalsobeenproposed by[Parikh & Grauman,Interactively Buildinga DiscriminativeVocabularyofNameable Attributes,CVPR2011] VisualRecognition And Search ColumbiaUniversity, Spring 2013

More Related