1 / 74

ETISEO

ETISEO. François BREMOND ORION Team, INRIA Sophia Antipolis, France. Fair Evaluation. Unbiased and transparent evaluation protocol Large participation Meaningful evaluation. Tasks evaluated. GT & Metrics are designed to evaluate tasks all along the video processing chain:

williegraham
Download Presentation

ETISEO

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ETISEO François BREMOND ORION Team, INRIA Sophia Antipolis, France

  2. Fair Evaluation • Unbiased and transparent evaluation protocol • Large participation • Meaningful evaluation

  3. Tasks evaluated • GT & Metrics are designed to evaluate tasks all along the video processing chain: • Task 1: Detection of physical objects, • Task 2: Localisationof physical objects, • Task 3: Classification of physical objects, • Task 4: Tracking of physical objects, • Task 5: Event recognition.

  4. Matching Computation To evaluate the matching between a candidate result and a reference data, we may use following distances: • D1-The Dice coefficient: Twice the shared, divided by the sum of both intervals: 2*card(RDC) / (card(RD) + card(C)). • D2-The overlapping: card(RDC) / card(RD). • D3-Bertozzi and al. metric: (card(RDC))^2 / (card(RD) * card(C)). • D4-The maximum deviation of the candidate object or target according to the shared frame span: Max { card(C\RD) / card(C), card(RD\C) / card(RD) }. RD C

  5. metrics (1) T1- DETECTION OF PHYSICAL OBJECTS OF INTEREST C1.1 Number of physical objects C1.2 Number of physical objects using their bounding box T2- LOCALISATION OF PHYSICAL OBJECTS OF INTEREST • C2.1 Physical objects area (pixel comparison based on BB) • C2.2 Physical object area fragmentation (splitting) • C2.3 Physical object area integration (merge) • C2.4 Physical objects localisation • 2D and 3D • Centroïd or bottom centre point of BB

  6. metrics (2) T3- TRACKING OF PHYSICAL OBJECTS OF INTEREST • C3.1 Frame-To-Frame Tracking: Link between two frames • C3.2 Number of object being tracked during time • C3.3 Detection time evaluation • C3.4 Physical object ID fragmentation • C3.5 Physical object ID confusion criterion • C3.6 Physical object 2D trajectory • C3.7 Physical object 3D trajectory T4- CLASSIFICATION OF PHYSICAL OBJECTS OF INTEREST C4.1 Object Type over the sequence C4.2 Object classification per type C4.3 Time Percentage Good Classification card{ RDC, Type(C) = Type(RD) } / card(RDC) T5- EVENT RECOGNITION C5.1 Number of Events recognized over the sequence C5.2 Scenario parameters

  7. Metric Evaluation • Distance for matching groundtruth and algorithms results • Similar measures: D1, D2, D3, D4. • Few main metrics measure general trends • Discriminant and meaningful • Detection M1.2.1: CNumberObjectsBoundingBox • Localization M2.4.3: CCentroid2DLocalisationPix. • Tracking M3.3.1: CtrackingTime • Object Classification M4.1.3: CobjectTypeOverSequenceBBoxID • Event Recognition M5.1.2: CNumberNamedEvents

  8. Metric Evaluation (cont’d) • Secondary metrics: • Complementary information • Pixel-based (M2.1.1) versus object-based (M1.2.1) metrics • Potential algorithm errors. • Example: M3.3.1 complemented (eg., about stability) by M3.2.1, M3.4.1 and M3.5.1. • Non-informative Metrics: • Add noise to the evaluation or non-discriminative • Example: M1.1.1 CNumberObjects gives the object number per frame without position information. • The same for M4.1.1 and M5.1.1.

  9. Evaluation on ETI-VS2-BE-19-C1

  10. Global Results: Video • Remarks: • For similar scenes, very dissimilar results! • For different scenes, results can spread over a large range or concentrate in a narrow range.

  11. Detection of Physical Objects (ETI-VS2-BE-19-C1.xml) M1.1.1: NumberObjects M1.2.1: NumberObjectsBoundingBoxD1

  12. Detection of Physical Objects (ETI-VS2-BE-19-C1.xml) M1.2.1:NumberObjectsBoundingBoxD1 M2.1.1: ObjectsArea

  13. Detection of Physical Objects (ETI-VS2-BE-19-C1.xml) M2.2.1: SplittingD5 M2.3.1: MergingD2

  14. Summary on Detection of Physical Objects • Main metric measures: • Detection M1.2.1: CNumberObjectsBoundingBox • Problems: static objects, contextual objects, background, masks… • Advantages: objects vs pixels, large objects and bounding boxes • Secondary metrics: • M2.1.1 (area): indication on the precision and handling shadows • Split/Merge measures (M2.2.1, M2.3.1): • Advantage: indicate potential merge • Inconvenients: threshold-dependent, non-detected objects not taken into account

  15. Localisation of Physical Objects

  16. Localisation of Physical Objects(ETI-VS2-BE-19-C1.xml) M2.3.1: MergingD2 M2.4.3: Centroid2DLocalisationPixD1

  17. Localisation of Physical Objects(ETI-VS2-BE-19-C1.xml) M2.4.1: Centroid2DLocalisationD1 M2.4.3: Centroid2DLocalisationPixD1

  18. Localisation of Physical Objects(ETI-VS2-BE-19-C1.xml) M2.4.2.: Centroid3DLocalisationD1

  19. Summary on Localisation of Physical Objects • M2.4.1, M2.4.2, M2.4.3, main metrics: • Problems: low utilisation of 3D info and calibration • Good performance: good precision on reliable TP (handling shadow and merge) • Advantages: complementary to the Detection; normalised, pixel or meter metrics

  20. Tracking of Physical Objects

  21. Tracking of Physical Objects(ETI-VS2-BE-19-C1.xml) M3.2.1: NumberObjectTrackedD1 M3.3.1: TrackingTime

  22. Tracking of Physical Objects(ETI-VS2-BE-19-C1.xml) M3.4.1: PhysicalObjectIdFragmentation M3.5.1: PhysicalObjectIdConfusion

  23. Tracking of Physical Objects(ETI-VS2-BE-19-C1.xml) M3.6.1: PhysicalObject2DTrajectories M2.4.1: Centroid2DLocalisationD1

  24. Summary on Tracking of Physical Objects • M3.3.1, main metric: • Problems: propagation of detection errors • Advantages: good global overview • M3.2.1, secondary metric: • Good performance: consistent TP over time for few TPs • Problems: not taking into account of complete FN • Fragmentation/confusion (M3.4.1, M3.5.1): • Advantage: indicate potential ID switching • Inconvenients: not discriminative; favoring under-detection (few IDs); over-detection (multiple IDs)

  25. Object Classification

  26. Object Classification(ETI-VS2-BE-19-C1.xml) M4.1.1: ObjectTypeOverSequence M4.1.1b: ObjectTypeOverSequenceBoundingBoxD1 Subtype

  27. Object Classification(ETI-VS2-BE-19-C1.xml) M4.1.3: ObjectTypeOverSequenceBoundingBoxIdD1 M4.1.2: ObjectTypeOverSequenceBoundingBoxD1

  28. Object Classification

  29. Object Classification

  30. Summary on Object Classification • M4.1.2, M4.1.3, same main metrics: • Problems: low classification of subtypes (doors, bikes, bags), favoring a few good quality TPs. • Advantage: reliable. • M4.1.1 (without BBox): • Inconvenients: wrong evaluation result in case of double errors (classified noise and FN) • Advantage: indicate potential double errors.

  31. Event Recognition

  32. Event Recognition(ETI-VS2-BE-19-C1.xml)

  33. Event Recognition(ETI-VS2-BE-19-C1.xml)

  34. Event Recognition(ETI-VS2-BE-19-C1.xml) M5.1.1: NumberEvents M5.1.2: NumberNamedEventsD1

  35. Summary on Event Recognition • M5.1.2 (with time), main metrics: • Problems: lack of understanding of ground truth definition • Advantages: good global overview per scenario type. • M5.1.1, secondary metric: • Problems: not taking into account of occurrence time

  36. Evaluation Results

  37. Evaluation on ETI-VS2-BE-19-C4

  38. Detection of Physical Objects (ETI-VS2-BE-19-C4.xml) M1.2.1: NumberObjectsBoundingBoxD1 M2.1.1: ObjectsArea

  39. Tracking of Physical Objects (ETI-VS2-BE-19-C4.xml) M3.3.1.D1: TrackingTime M3.2.1: NumberObjectTrackedD1

  40. Event Recognition (ETI-VS2-BE-19-C4.xml) M5.1.2: NumberNamedEventsD1 M5.1.1: NumberEvents

  41. Evaluation Results

  42. Evaluation on ETI-VS2-MO-1-C1

  43. Detection of Physical Objects (ETI-VS2-MO-1-C1.xml) M1.2.1: NumberObjectsBoundingBoxD1 M2.1.1: ObjectsArea

  44. Tracking of Physical Objects (ETI-VS2-MO-1-C1.xml) M3.3.1.D1: TrackingTime M3.2.1: NumberObjectTrackedD1

  45. Event Recognition (ETI-VS2-MO-1-C1.xml) M5.1.1: NumberEvents M5.1.2: NumberNamedEventsD1

  46. Evaluation Results

  47. Evaluation on ETI-VS2-RD-6-C7

  48. Detection of Physical Objects (ETI-VS2-RD-6-C7.xml) M1.2.1: NumberObjectsBoundingBoxD1 M2.1.1: ObjectsArea

  49. Detection of Physical Objects: Reference Data Filtering (ETI-VS2-RD-6-C7.xml: M1.2.1 - NumberObjectsBoundingBoxD1) No filtering With filtering

  50. Detection of Physical Objects: Reference Data Filtering (ETI-VS2-RD-6-C7.xml: M2.1.1 - ObjectsArea) No filtering With filtering

More Related