High Transfer Rate, Real-time Brain-Computer Interface

1. High Transfer Rate, Real-time Brain-Computer Interface Machine-based learning techniques towards a practical spelling device for the completely paralyzed

2. Agenda • Brain Computer Interfaces – brief intro. • Our system • Overview, technical details • Machine learning – Support Vector Machines • Additional Bandwidth – Word Prediction • Results • Future Improvements, Q&A • Demonstration at Psychology Lab ThinQ Innovation

3. BCIs – the Need • ‘Locked-in’ patients Example: J.D. Bauby, “The Diving Bell and the Butterfly” • Persistence of life –“butterfly” • Extreme physical disability – “diving bell” ThinQ Innovation

4. BCIs – the Need • Amyotrophic Lateral Sclerosis (ALS), aka Lou Gherig’s • Degeneration of motor neurons, paralysis of voluntary muscles • 120,000 diagnosed each year worldwide • 2000 Canadians live with ALS right now • Can leave patients ‘locked-in’ • Cognitive and sensory functions remain intact ThinQ Innovation

5. BCI(1): Slow Cortical Potentials (SCPs) • Extensive training ~ 3 months using biofeedback mechanism • Tested on ALS patients, learned to control SCPs Ref: N. Birbaumer et al., “The thought translation device (TTD) for completely paralyzed patients,” IEEE Trans. Rehab. Eng., Vol. 8, pp. 190-193,June 2000. ThinQ Innovation

6. BCI(1): SCPs cont. • Most successful subject – artificially fed and respirated for 4 years • After 3 months of training, wrote letter below • Took 16 hours to write ~ 2 letters/minute • Expresses thanks, wants to have a party ThinQ Innovation

7. BCI(2): Implants - Cyberkinetics Inc. • BrainGate Neural Interface System: Mkt. cap ~$45mil. • Control of cursor on PC using implant in motor cortex • Undergoing limited clinical trials • Limb movement possibilities ThinQ Innovation

8. 8-40 uV avg. deflection 300ms P300 Spelling Device – the P300 Event Related Potential • Known as ‘oddball’ or ‘surprise’ paradigm • Inherent ThinQ Innovation

9. P300 Spelling Device – the System • Non-invasive • Inherent Response ThinQ Innovation

10. P300 Speller Terminology • Epoch = One flash of any row or column • Trial = 1 complete set of epochs - all rows and columns • Symbol = Alphanumeric characters or pictures ThinQ Innovation

11. BCI Competition 2003 • Provided pre-collected data for competition • P300 Spelling Paradigm: • Winners included Kaper et al. • Used Support Vector Machines • Achieved high transfer rate with real-time implementation possibilities ThinQ Innovation

12. System Operation • Steps • Training (approximate 1hr) • Provide visual stimuli (flashing of rows/columns) • Record data with known classification label • Run data through pattern recognition algorithm (SVM) • Create customized models for each individual • Spelling • Load customized model for individual • Provide visual stimuli (flashing of rows/columns) • Record data with unknown classification label • Run data through SVM classifier • Sum up decision values • Feedback most probable letter ThinQ Innovation

13. Display • Flexible matrix size • Flexible matrix contents • Alphanumeric Characters • Words • Symbols ThinQ Innovation

14. Display cont… • Random and exhaustive flashing of all of the rows and columns on display • Flashing cycle: 300ms • 100ms intensification period • 200ms de-intensification period • 10 second rest period at the end of each symbol ThinQ Innovation

15. Data Collection • Collect data from DAQ sampled at 240Hz • 600ms after intensification • Buffer overlap • Flexible data collection delay • Flexible data recording time ThinQ Innovation

16. Data Collection – cont. • 10 channels collected simultaneously • Data from each channel concatenated together • Data stored into program memory • Collected until end of a symbol • Converted to array • Memory cleared for next symbol • System is timing critical ThinQ Innovation

17. Timing Issue • Purpose • Process within 300ms window • Bottleneck • Online SVM processing • Old design = 340ms/Epoch • New design = 17.67ms/Epoch • Requirement • Pentium4 or equivalent is sufficient ThinQ Innovation

18. Matlab Interface • Why we use Matlab? • VB–Matlab interface using APIs • Common functions • Pass matrix array to Matlab workspace • Get matrix array from Matlab workspace • Execute command line or script ThinQ Innovation

19. Support Vector Machines • Pattern recognition Algorithm • SVM used for: • Creating models for different individuals (train) • Getting discriminant scores (spelling) • Detailed information covered later ThinQ Innovation

20. Score Matrix ThinQ Innovation

21. Word Prediction • Idea: predict intended words based on previous spelling. Similar to cellular phone ‘smart text’ • Extract top ranked words • SQL for fast searching • Dynamic database • Selection updated on the bottom of the display • Words chosen same way ThinQ Innovation

22. System Design • Modular Design Approach ThinQ Innovation

23. What is SVM? • Developed by Vapnik in 1992 at Bell Labs • Broad applications • Based on concept of ‘learn from examples’ • Key concepts: • Linear Decision Boundary with Margin • Nonlinear feature transformation ThinQ Innovation

24. Basic Concept • {x1, ..., xn} be our training data set • yiÎ {1,-1} be the class label of xi then, • Find a decision boundary • Make a decision on disjoint test data ThinQ Innovation

25. Decision Boundary (linear) • Infinite possibility Class 1 Class -1 ThinQ Innovation

26. Bad Decision Boundary Class 1 Class 1 Class -1 Class -1 ThinQ Innovation

27. Class 1 m Class -1 Good Decision Boundary • Want to maximize m • Boundary found using constrained optimization problem ThinQ Innovation

28. Optimization Problem • Optimization Problem ThinQ Innovation

29. After Training • xi’s on the decision boundary are calledSUPPORT VECTORS • Support vectors and b defines the decision boundary ThinQ Innovation

30. Class 1 a10=0 a8=0.6 a7=0 a2=0 a5=0 a1=0.8 a4=0 a6=1.4 a9=0 a3=0 Class -1 Geometrical Interpretation ThinQ Innovation

31. Non-separable Samples • Use of Soft Margin Separation • Kernel Transformation ThinQ Innovation

32. Class 1 Class -1 Soft Margin Separation ThinQ Innovation

33. Soft Margin Separation • Idea: simultaneous maximization of margin and minimization of training error ThinQ Innovation

34. Nonlinear Samples • Some Samples are inherently nonlinear in input space • No linear boundary is sufficiently accurate ThinQ Innovation

35. Solution? ThinQ Innovation

36. Kernel Transformation • Idea: map input space into feature space such that samples become linearly separable ThinQ Innovation

37. Gaussian Kernel ThinQ Innovation

38. SVM Implementation • Matlab interface to libsvm • Kernel: RBF with  = 6.6799e-4 • C parameter: 20.007 ThinQ Innovation

39. SVM Implementation • Average Method (61.538%) • Multi-Model Method (65.22%) • Concatenation Method (82.418%) • Weighted Concatenation Method (max. 86.264%) ThinQ Innovation

40. Possible Improvements • Weighted concatenation method • Customized Kernel Parameters ThinQ Innovation

41. Measure of Performance • Bit Rate • N: number of available symbols • p: prediction accuracy • t: number of seconds taken to choose one symbol • Letters per minute ThinQ Innovation

42. Cont… • Resulting Transfer Rates • Without using dictionary • With using dictionary ThinQ Innovation

43. More Accurate Measure • Resulting Transfer Rates • Without using dictionary • With using dictionary ThinQ Innovation

44. Cont… • Mechanism • Receives a chosen letter from control module • Appends the letter to current letters in the word • Searches SQL database • Return list of most probable target words based on ranking ThinQ Innovation

45. Result Analysis • Accuracy across subjects • Accuracy over time, same subject • Accuracy over number of trials • Accuracy versus model size ThinQ Innovation

46. Accuracy Across Subjects ThinQ Innovation

47. Accuracy Across Subjects ThinQ Innovation

48. Accuracy Over Time, Same Subject • Subject: Jack ThinQ Innovation

49. Accuracy Over Number of Trials • Subject: Jyh-Liang ThinQ Innovation

50. Accuracy Versus Model Size • Subject: Jyh-Liang ThinQ Innovation