Ch19. Evaluation Criteria for BCI Research

1. Ch19.Evaluation Criteria for BCI Research

2. Contents • Introduction • Criteria for Evaluating Trial-based BCI Data • Criteria for Evaluating Self-Paced BCI Data • Other criteria

3. Introduction • The factors affect BCI performance • Trial-based (system-paced) <-> Asynchronous mode (self-paced) • Type and number of EEG feature • Spectral parameter • Slow cortical potentials • Spatiotemporal parameters • Nonlinear features • Type of classifier • Linear and quadratic discriminate analysis • Support vector machines • Neural networks • Simple threshold detection • Target application

4. Introduction • The necessity of consistent evaluation criteria • For compare different BCI systems and approaches • Consideration for evaluation criteria • What is being evaluated • Feedback loop • The most frequently used evaluation criterion • Error rate or accuracy • Response speed of BCIs • Evaluation of asynchronous BCI data

5. Criteria for Evaluating Trial-based BCI Data • The Confusion Matrix • Classification Accuracy and Error Rate • Cohen’s Kappa Coefficient • Mutual Information of a Discrete Output

6. The Confusion Matrix Correctly classified Incorrectly classified TP: True PositiveTN: True Negative FP: False PositiveFN: False Negative FA: False Activation CR: Correct Rejection

7. Classification Accuracy and Error Rate • The most widely used evaluation criteria in BCI research • Denoted as ACC for classification ACCuracy, ERR(=1-ACC) for error rate • Can be very easily calculated and interpreted • The accuracy is 100%/M (M: number of class) • The maximum accuracy can never exceed 100% • Some limitation • The off-diagonal values of the confusion matrix are not considered • Classification accuracy of less frequent classes have small weight Correctly classified sample Total number of sample

8. Cohen’s Kappa Coefficient • Addresses several of the critiques on the accuracy measure • Use the overall agreementpo, and the chance agreement pe Sum of i-th column Sum of i-th row Predicted classes show no correlation with actual classes Perfect classification Different assignment between output and the true classes Standard error of kappa coefficient

9. Cohen’s Kappa Coefficient • Address several of the criticisms of the accuracy measure • It considers the distribution of the wrong classifications=> i.e., off-diagonal elements of the confusion matrix) • Frequency of occurrence is normalized for each class => class with less samples get the same weight as class with many samples • The standard error of the kappa coefficient easily can be used for comparing whether the results of distinct classification systems have statistically significant differences

10. Mutual Information of a Discrete Output • Assume following things • BCI system can be modeled as communication channel • Communication theory of Shannon and Weaver (1949) can be applied directly to quantify the information transfer • Farwell and Donchin (1988) • Information transfer for M classes can be calculated as • Problms • The information rate assume an error-free system • This suggestion is not useful for comparing different BCI systems

11. Mutual Information of a Discrete Output • Based on Pierce (1980), Wolpaw et al. (2000a) • Information transfer rate for M classes an ACC = pois • The formula has following limitations • M selections (classes) are possible • Each class has the same probability • The specific accuracy is same for each class • Each undesired selection must have the same probability of selection • Often these assumption are not fulfilled It does not satisfy 3rd and 4th condition

12. Mutual Information of a Discrete Output • Random variable X models the user intension • Random variable Y models the classifier output • The entropy H(X) of a discrete random variable is defined as • Nykopp(2001) derived the information transfer for a general confusion matrix Probability to classify xi as yj a priori probability for class xi Mutual information

13. Criteria for Evaluating Self-paced BCI Data Asynchronous mode BCI • The BCI system is specially designed to produce outputs in response to intentional control • HF-Difference • Hit False difference • H : hit rate, F: false detection rate

14. Other Criteria • Receiver-Operator Characteristics (ROC) • Correlation Coefficient • Evaluation of Continuous-Input and Continuous-Output Systems • Response time • High accuracy is important • But the response time is also important • Maximum Steepness of the Mutual Information is used in BCI Competition III t0 : time for the cue onset I(t) : continuous mutual information