Professor Qiang Ji Spring, 2011

1. ECSE 6610 Pattern Recognition Professor QiangJiSpring, 2011

2. Pattern Recognition Overview Training Classification/ Regression Feature extraction Output Values Features Features Training Training Output Values Unknown Classifier/ Regressor Testing Raw Data Training Raw Data Testing Feature extraction Learned Classifier/ Regressor Feature extraction: extract the most discriminative features to concisely represent the original data, typically involving dimensionality reduction Training/Learning: learn a mapping function that maps input to output Classification/regression: map the input to a discrete output value for classification and to continuous output value for regression.

3. Pattern Recognition Overview (cont’d) Supervised learning Both input (feature) and output (class labels) are provided Unsupervised learning-only input is given Clustering Dimensionality reduction Density estimation Semi-supervised learning-some input has output labels and others do not have

4. Examples of Pattern Recognition Applications • Computer/Machine Vision object recognition, activity recognition, image segmentation, inspection • Medical Imaging Cell classification • Optical Character Recognition Machine or hand written character/digit recognition • Brain Computer Interface Classify human brain states from EEG signals • Speech Recognition Speaker recognition, speech understanding, language translation • Robotics Obstacle detection, scene understanding, navigation

5. Computer Vision Example: Facial Expression Recognition

6. Machine Vision Example

7. Example: Handwritten Digit Recognition

8. Probability Calculus U is the sample space X is a subset of the outcome or an event P(X ˅ Y)=P(X)+P(Y) - P(X ˄Y) ,i.e, X and Y are mutually exclusive

9. Probability Calculus (cont’d) • Conditional independence • The Chain Rule Given three events A, B, C

10. The Rules of Probability • Sum Rule • Product Rule

11. Bayes’ Theorem posterior  likelihood × prior

12. Bayes Rule A2 A3 A4 A1 p(A, B) p(B | A)p(A) = = p(A | B) p(B) p(B) • Based on definition of conditional probability • p(Ai|E) is posterior probability given evidence E • p(Ai) is the prior probability • P(E|Ai) is the likelihood of the evidence given Ai • p(E) is the probability of the evidence E A6 A5 p(E | A )p(A ) p(E | A )p(A ) = = i i i i p(A | E) å i p(E) p(E | A )p(A ) i i i

13. Bayesian Rule (cont’d) Assume E1 and E2 are independent given H, the above equation may be written as where is the prior and is the likelihood of H given E2

14. A Simple Example Consider two related variables: 1. Drug (D) with values y or n 2. Test (T) with values +ve or –ve And suppose we have the following probabilities: P(D = y) = 0.001 P(T = +ve | D = y) = 0.8 P(T = +ve | D = n) = 0.01 These probabilities are sufficient to define a joint probability distribution. Suppose an athlete tests positive. What is the probability that he has taken the drug?

15. Expectation (or Mean) • For discrete RV X • For continuous RV X • Conditional Expectation

16. Expectations Conditional Expectation (discrete) Approximate Expectation (discrete and continuous)

17. Variance • The variance of a RV X • Standard deviation • Covariance of RVs X and Y, • Chebyshev inequality

18. Independence • If X and Y are independent, then

19. Probability Densities p(x) is the density function, while P(x) is the cumulative distribution. P(x) is a non-decreasing function.

20. Transformed Densities

21. The Gaussian Distribution

22. Gaussian Mean and Variance

23. The Multivariate Gaussian m=mean vector S=covariance matrix

24. Minimum Misclassification Rate Two types of mistakes: False positive (type 1) False negative (type 2) The above is called Bayes error. Minimum Bayes error is achieved at x0

25. Generative vs Discriminative Generative approach: Model Use Bayes’ theorem Discriminative approach: Model directly