210 likes | 217 Views
Evaluate multi-object tracking using AMI Tracking Evaluator. User manual, data formats, implementation details for accurate assessment.
E N D
AMI Tracking Evaluation Vítězslav Beran SERVITE, 13.1.2006 Department of Computer Graphic and Multimedia Faculty of Information Technology Brno University of Technology
Content • MULTI-OBJECT TRACKING EVALUATION • Coverage test • Configuration errors • Identification errors • AMI TRACKING EVALUATOR • User manual • Data formats • Implementation • Console Evaluator Vítězslav Beran, Ami Tracking Evaluator
Multi-object Tracking Evaluation • Introduced in AMI WP4 Tracking: Evaluation scheme 1.0, 2006 • Two data sets • Ground Truth (GT) • Estimates (E) Vítězslav Beran, Ami Tracking Evaluator
Coverage test • Two measurements • Recall • Precision • F-measure Vítězslav Beran, Ami Tracking Evaluator
Configuration errors • Frame errors • False positive (FP) • False negative (FN) • Multiple trackers (MT) • Multiple objects (MO) • Configuration distance (CD) Vítězslav Beran, Ami Tracking Evaluator
GTj 1 2 3 Ei 1 0,89 0,00 0,00 1 2 0,00 0,00 0,00 0 3 0,00 0,51 0,00 1 4 0,00 0,47 0,47 2 1 2 2 Configuration errors (cont.) • tI is coverage threshold Vítězslav Beran, Ami Tracking Evaluator
Configuration errors (cont.) • Configuration Map • Configuration Errors Vítězslav Beran, Ami Tracking Evaluator
Configuration distance • Relevant only together with other errors. Vítězslav Beran, Ami Tracking Evaluator
Occlusion • t0 is occlusion threshold • when occlusion is set no error is computed Vítězslav Beran, Ami Tracking Evaluator
Normalization • for easy comparison of tracking algorithm errors Vítězslav Beran, Ami Tracking Evaluator
Identification errors • Sequence errors • Falsely identified tracker (FIT) • Falsely identified object (FIO) Vítězslav Beran, Ami Tracking Evaluator
Identification errors (cont.) • Object purity (OP) • Track purity (TP) • Not interesting for our purpose. Vítězslav Beran, Ami Tracking Evaluator
Example sequence Vítězslav Beran, Ami Tracking Evaluator
Example sequence results Vítězslav Beran, Ami Tracking Evaluator
AMI Tracking Evaluator • Windows platform application • tracking video dataevaluation • load / save data sets from file source • GT and E objects visualization • errors visualization • console version Vítězslav Beran, Ami Tracking Evaluator
User interface Vítězslav Beran, Ami Tracking Evaluator
Data formats - XML <!ELEMENT AVEvents (EventGroups?, EventTypes?, File?, Editor?)> ... <!ELEMENT File (Source*, TimeFormat?, Event*, Title*)> <!ELEMENT Source (#PCDATA)> <!ELEMENT TimeFormat (#PCDATA)> <!ELEMENT Event (ID, Time, Text?, Parameters?)> <!ELEMENT ID (#PCDATA)> <!ELEMENT Time (#PCDATA)> <!ELEMENT Text (#PCDATA)> <!ELEMENT Parameters EMPTY> <!ELEMENT Title (Time, Text)> ... <!ELEMENT Editor ANY> Vítězslav Beran, Ami Tracking Evaluator
Data formats - TXT • Three format supported: • format 1 frame frameID object objectID BoxCenterX BoxCenterY Width/2 Height/2 object ... • format 2 image[frameID].* objectID minXPos minYPos maxXPos maxYPos • format 3 frameID objectID visibility minXPos minYPos maxXPos maxYPos Vítězslav Beran, Ami Tracking Evaluator
Conclusion • tool for multi-object tracking evaluation • same tool by IDIAP is much more strict to data source formats • presentation, documentation and downloads available onhttp://www.fit.vutbr.cz/research/grants/ami/evaluator.html Vítězslav Beran, Ami Tracking Evaluator
Future work • evaluate our evaluation tool Vítězslav Beran, Ami Tracking Evaluator
Thank you for your attention. Any questions, please?