1 / 45

Parametric measures to estimate and predict performance of identification techniques

Amos Y. Johnson & Aaron Bobick. STATISTICAL METHODS FOR COMPUTATIONAL EXPERIMENTS IN VISUAL PROCESSING & COMPUTER VISION NIPS 2002. Parametric measures to estimate and predict performance of identification techniques. Setup – for example. Given a particular human identification technique.

huyen
Download Presentation

Parametric measures to estimate and predict performance of identification techniques

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Amos Y. Johnson & Aaron Bobick STATISTICAL METHODS FOR COMPUTATIONAL EXPERIMENTS IN VISUAL PROCESSING & COMPUTER VISION NIPS 2002 Parametric measures to estimate and predict performance of identification techniques

  2. Setup – for example • Given a particular human identification technique

  3. x Setup – for example • Given a particular human identification technique • This technique measures 1 feature (q) from n individuals - 1D Feature Space -

  4. x Setup – for example • Given a particular human identification technique • This technique measures 1 feature (q) from n individuals • Measure the feature again - 1D Feature Space -

  5. x Setup – for example • Given a particular human identification technique • This technique measures 1 feature (q) from n individuals • Measure the feature again Probe Gallery - 1D Feature Space -

  6. x Setup – for example • Given a particular human identification technique • This technique measures 1 feature (q) from n individuals • Measure the feature again Target Probe Gallery For template - 1D Feature Space -

  7. x Setup – for example • Given a particular human identification technique • This technique measures 1 feature (q) from n individuals • Measure the feature again Target Imposters Probe Gallery For template - 1D Feature Space -

  8. Target Imposters Probe x Gallery For template - 1D Feature Space - Question • For a given human identification technique, how should identification performance be evaluated?

  9. Target Imposters Probe x Gallery For template - 1D Feature Space - Possible ways to evaluate performance • For a given classification threshold, compute • False accept rate (FAR) of impostors • Correct accept rate (HIT) of genuine targets

  10. Possible ways to evaluate performance • For various classification thresholds, plot • MultipleFAR and HIT rates (ROC curve)

  11. Possible ways to evaluate performance • For various classification thresholds, plot • MultipleFAR and HIT rates (ROC curve) • Compute area under a ROC curve (AUROC) Probability of correct classification

  12. Probability of incorrect classification Possible ways to evaluate performance • For various classification thresholds, plot • MultipleFAR and HIT rates (ROC curve) • Compute 1 - area under a ROC curve (1 -AUROC)

  13. Problem • Database size • If the database is not of sufficient size, then results may not estimate or predict performance on a larger population of people. 1 - AUROC

  14. Our Goal • To estimate and predict identification performance with a small number subjects 1 - AUROC

  15. Our Solution • Derive two parametric measures • Expected Confusion (EC) • Transformed Expected-Confusion (EC*)

  16. Our Solution • Derive two parametric measures • Expected Confusion (EC) • Transformed Expected-Confusion (EC*) Probability that an imposter’s feature vector is within the measurement variation of a target’s template

  17. Our Solution • Derive two parametric measures • Expected Confusion (EC) • Transformed Expected-Confusion (EC*) Probability that an imposter’s feature vector is closer to a target’s template, than the target’s feature vector

  18. Our Solution • Derive two parametric measures • Expected Confusion (EC) • Transformed Expected-Confusion (EC*) EC* = 1 - AUROC

  19. Target Imposters Probe x Gallery For template - 1D Feature Space - Expected Confusion • Probability that an imposter’s feature vector is within the measurement variation of a target’s template

  20. Expected Confusion - Uniform • The templates of the n individuals, are from an uniform density • Pp(x) = 1/n P(x) Pp(x) 1/n x - 1D Feature Space -

  21. Expected Confusion - Uniform • The measurement variation of a template is also uniform • Pi(x) = 1/m P(x) Pi(x) 1/m Pp(x) 1/n x - 1D Feature Space -

  22. Expected Confusion - Uniform • The probability that an imposter’s feature vector is within the measurement variation of template q3 is the area of overlap • True if m << n P(x) Pi(x) 1/m Pp(x) 1/n x - 1D Feature Space -

  23. Expected Confusion - Uniform • The probability that an imposter’s feature vector is within the measurement variation of any template q • True if m << n P(x) Pi(x) 1/m Pp(x) 1/n x

  24. : Population density : Measurement variation Expected Confusion - Gaussian • Following the same analysis, for the multidimensional Gaussian case

  25. Expected Confusion - Gaussian • Following the same analysis, for the multidimensional Gaussian case • True if the measurement variation is significantly less then the population variation Probability that an imposter’s feature vector is within the measurement variation of a target’s template

  26. Expected Confusion - Gaussian • Relationship to other metrics • Mutual Information • The negative natural log of the EC is the mutual information of two Gaussian densities

  27. Target Imposters Probe x Gallery For template - 1D Feature Space - Transformed Expected-Confusion • Probability that an imposter’s feature vector is closer to a target’s template, than the target’s feature vector

  28. Target Imposters x k For template Transformed Expected-Confusion • First: We find the probability that a target’s feature vector is some distance k away from its template

  29. Target Imposters x k For template Transformed Expected-Confusion • Second: We find the probability that an imposter’s feature vector is less than or equal to that distance k

  30. x Transformed Expected-Confusion • Therefore: The probability that an imposter’s feature is closer to the target’s template, than the target’s feature (for a distance k) is Target Imposters k

  31. x Transformed Expected-Confusion • Therefore: The probability that an imposter’s feature is closer to the target’s template, than the target’s feature (for any distance k) is Target Imposters k

  32. Transformed Expected-Confusion • Therefore: The expected value of this probability over all target’s templates is x

  33. Transformed Expected-Confusion • Next: Replace the density of the distance between a target’s feature-vectors and its template q

  34. Transformed Expected-Confusion • Answer: Probability that an imposter’s feature vector is closer to a target’s template, than the target’s feature vector

  35. Transformed Expected-Confusion • This probability can be shown to be one minus the area under a ROC curve • Following the analysis of Green and Swets (1966)

  36. Transformed Expected-Confusion • Integrate: With these assumptions

  37. Transformed Expected-Confusion • Integrate: With these assumptions

  38. Transformed Expected-Confusion • Integrate: With these assumptions

  39. Transformed Expected-Confusion • Integrate: With these assumptions

  40. Transformed Expected-Confusion • Integrate: Probability that an imposter’s feature vector is closer to a target’s template, than the target’s feature vector

  41. Transformed Expected-Confusion • Compare: EC* with 1 - AUROC EC* = 1 - AUROC

  42. Conclusion • Derive two parametric measures • Expected Confusion (EC) • Transformed Expected-Confusion (EC*) Probability that an imposter’s feature vector is closer to a target’s template, than the target’s feature vector

  43. Conclusion • Derive two parametric measures • Expected Confusion (EC) • Transformed Expected-Confusion (EC*) Probability that an imposter’s feature vector is within the measurement variation of a target’s template Probability that an imposter’s feature vector is closer to a target’s template, than the target’s feature vector

  44. Conclusion • Derive two parametric measures • Expected Confusion (EC) • Transformed Expected-Confusion (EC*) Probability that an imposter’s feature vector is within the measurement variation of a target’s template Probability that an imposter’s feature vector is closer to a target’s template, than the target’s feature vector

  45. Future Work • Developing a mathematical model of the cumulative match characteristic (CMC) curve • Benefit: To predict how the CMC curve changes as more subjects are added

More Related