Multimodal Bio-signal based Control of Intelligent Wheelchair

1. Multimodal Bio-signal based Control of Intelligent Wheelchair Professor Huosheng Hu Leader of Activity 3 University of Essex, U.K.

2. SYSIASS: Autonomous and Intelligent Healthcare System

3. Aim and Objectives • Multimodal Human-Machine Interface aims to make wheelchairs more user-friendly. • The key challenge is the understanding of • the user who interacts with the wheelchair • the system (the computer and sensor technology &their usability) • the interaction between the user and the wheelchair. • A proper balance between • Functionality that is defined by the set of actions or services that it provides to its users, and • Usability that is the range and degree by which the system can be used efficiently and adequately to accomplish certain goals for certain users.

4. System Configuration Multimodal HMI Voice, gesture, EMG, EEG

5. Detection of Human Intension • Inertial Sensing: to sense the position and motion of the human hand and other body parts for use in HMI. • Audio Sensing: Using microphones to sense the sound is to interpret speech, which is the most natural modality. • Visual Sensing: Using cameras to detect the human motion, such as gestures, lip motion, gaze, facial expressions, head & other body movements. • Touch and force: This is especially important for building a proper feel of “realism” in human intension. • Muscle and Brainwave: These bio-signals are obtained noninvasively from the surface of the scalp and skin; used for wheelchair control.

6. Wireless Voice based Control Video show • To use human voice commands; • Pre-defined control phrases: “forward”, “turn left”, “turn right”, “backwards”, “stop”; • Wireless wearable headphone device; • Fast voice recognition and no pre-training • Plug and play wireless and comfort control • Obstacle avoidance with laser scanner • Unlimited voice control commands available

7. Visual Head Gesture based Control • Visual detection of head gestures of users • Pre-defined commands (left, right, forward, backward, stop) for controlling awheelchair • Laser based obstacle detection & avoidance Video show • To help disabled and elderly who cannot use joystick • To realise hands-free control of the wheelchair • Obstacle avoidance to ensure user safety • Functionalities:

8. Facial Expression & Head Movement 3F+1H 2F+1H • Facial expressions are detected using the cognitiv suite of Emotiv. • Head movements are detected by the gyroscope of Emotiv. • Pre-defined commands (left, right, forward, backward, and stop) • A simple and safe hands-free control with flexible configurations for users. • An ideal alternative to the joystick control for users with severe disability.

9. Wireless IMU in a Baseball Cap Video show • Detect head motion for wheelchair control • Minimum user head motion required • Adapt to people with weak neck abilities • Obstacle avoidance with laser scanner • Totally replace joystick control interface

10. Minimally Invasive Intra-Oral Palate for wheelchair User • A minimally invasive way to control wheelchair • It is comfortable for users and allows prolonged use. • It is based on an intra-oral dental retainer clip with • 9 force sensitive resistors • 10 Bit 200Hz data capture • Opto-isolation is deployed. • Connectivity to GPSB

11. Conformal RFID Sensing for wheelchair control • Strain gauge RFID tag is designed to sense the stretch on skin (neck/eyebrow movement) • A tongue controlled RFID tag is designed to sense the tongue position in the mouth • To provide an on/off switch and control mechanism for controlling the wheelchair.

12. Conclusion • Activity 3 of SYSIASS has been conducted successfully. • Human-Machine Interaction (HMI) aims to achieve information full accessibility. • Human-Machine Interaction plays an key role in assistive technology. • Hands-free control of wheelchairs are extremely useful to general public without programming skills. • Multimodal HMI is necessary to overcome the limitations of single modality HMI.