Linear Discriminant Analysis and Its Variations

1. Linear Discriminant Analysis and Its Variations Abu Minhajuddin CSE 8331 Department of Statistical Science Southern Methodist University April 27, 2002

2. Plan… • The Problem • Linear Discriminant Analysis • Quadratic Discriminant Analysis • Other Extensions • Evaluation of the Method • An Example • Summary

3. The Problem… Training Situation: Data on p predictors, Membership of one of g groups Classification Problem: Data on p predictors Unknown group membership

4. The Problem… Fisher’s Iris Data: Identify the three species?

5. Linear Discriminant Analysis Classify the item xat hand to one of J groups based on measurements on p predictors. Rule: Assign xto group j that has the closest mean j = 1, 2, …, J Distance Measure: Mahalanobis Distance…. Takes the spread of the data into Consideration

6. Linear Discriminant Analysis Distance Measure: For j = 1, 2, …, J, compute Assign x to the group for which dj is minimum is the pooled estimate of the covariance matrix

7. Linear Discriminant Analysis Linear Discriminant Analysis …or equivalently, assign x to the group for which is a maximum. (Notice the linear form of the equation!)

8. Linear Discriminant Analysis • …optimal if…. • Multivariate normal distribution for the observation in each of the groups • Equal covariance matrix for all groups • Equal prior probability for each group • Equal costs for misclassification

9. Linear Discriminant Analysis Relaxing the assumption of equal prior probabilities… being the prior probability for the jth group.

10. Linear Discriminant Analysis Relaxing the assumption of equal covariance matrices… result?…Quadratic Discriminant Analysis

11. Quadratic Discriminant Analysis Rule: assign to group j if is the largest. Optimal if the J groups of measurements are multivariate normal

12. Other Extensions & Related Methods Relaxing the assumption of normality… Kernel density based LDA and QDA Other extensions….. Regularized discriminant analysis Penalized discriminant analysis Flexible discriminant analysis

13. Other Extensions & Related Methods Related Methods: Logistic regression for binary classification Multinomial logistic regression These methods models the probability of being in a class as a linear function of the predictor.

14. Actual group Number of observations Predicted group A B A B nA nB n11 n21 n12 n22 Evaluations of the Methods • Classification Table (confusion matrix)

15. Evaluations of the Methods Apparent Error Rate (APER): APER = # misclassified/Total # of cases ….underestimates the actual error rate. Improved estimate of APER: Holdout Method or cross validation

16. Actual Group Number of Observations Predicted Group Setosa Versicolor Virginica Setosa Versicolor Virginica 50 50 50 50 0 0 0 48 1 0 2 49 An Example: Fisher’s Iris Data Table 1: Linear Discriminant Analysis (APER = 0.0200)

17. Actual Group Number of Observations Predicted Group Setosa Versicolor Virginica Setosa Versicolor Virginica 50 50 50 50 0 0 0 47 1 0 3 49 An Example: Fisher’s Iris Data An Example: Fisher’s Iris Data Table 1: Quadratic Discriminant Analysis (APER = 0.0267)

18. An Example: Fisher’s Iris Data

19. An Example: Fisher’s Iris Data

20. Summary • LDA is a powerful tool available for classification. • Widely implemented through various software • Theoretical properties well researched • SAS implementation available for large data sets.