1 / 32

BlueEyes Human-Operator Monitoring System

Team members. Paweł Kowalik Piotr Kubiaczyk Krystian Nowak Stanisław Osiński Tomasz Pużak. Team mentor. Jan Kniat, Ph. D. BlueEyes Human-Operator Monitoring System. Motivation What is BlueEyes ? Physiological foundations System design System implementation Future improvements

steadmanm
Download Presentation

BlueEyes Human-Operator Monitoring System

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Team members Paweł KowalikPiotr KubiaczykKrystian NowakStanisław OsińskiTomasz Pużak Team mentor Jan Kniat, Ph. D. BlueEyesHuman-Operator Monitoring System • Motivation • What isBlueEyes ? • Physiological foundations • Systemdesign • System implementation • Futureimprovements • Project worksummary • Systemdemonstration Poznań University of Technology, Poland http://www.put.poznan.pl

  2. Agenda • Motivation • What isBlueEyes ? • Physiological foundations • Systemdesign • System implementation • Futureimprovements • Project worksummary • Systemdemonstration • Motivation • What is BlueEyes (and what is not) ? • Physiological foundations • System design • System implementation • Future improvements • Project work summary • System demonstration

  3. temporal indisposition Motivation • Motivation • What isBlueEyes ? • Physiological foundations • Systemdesign • System implementation • Futureimprovements • Project worksummary • Systemdemonstration • Human error - a frequent reason of catastrophes and ecological disasters • weariness • oversight • Monitoring of consciousbrain involvement • Capable sensors available • Lack of complex solution exploiting wireless link technology, database systems and AI algorithms

  4. What is BlueEyes ? • Motivation • What isBlueEyes ? • Physiological foundations • Systemdesign • System implementation • Futureimprovements • Project worksummary • Systemdemonstration • The complex solution for human-operator monitoring: • conscious brain involvement monitoring • triggering user-defined actions • data recording What is BlueEyes not? • Doesn’t predict nor interfere with operator’s thoughts • Cannot force directly the operator to work

  5. Who is BlueEyes for ? • Motivation • What isBlueEyes ? • Physiological foundations • Systemdesign • System implementation • Futureimprovements • Project worksummary • Systemdemonstration • Generic control rooms • Power station • Captain bridge • Flight control centers • Operating theatres – anesthesiologists • Common application • A simpler system version for drivers

  6. Minimization of • ecological consequences • financial loss • a threat to a human life BlueEyes - benefits • Motivation • What isBlueEyes ? • Physiological foundations • Systemdesign • System implementation • Futureimprovements • Project worksummary • Systemdemonstration • Prevention from dangerous incidents • The reconstructionof the course ofoperator’s work

  7. Physiological foundations • Motivation • What isBlueEyes ? • Physiological foundations • Systemdesign • System implementation • Futureimprovements • Project worksummary • Systemdemonstration Preconscious brain Human brain Conscious brain • alogical • „automatic” • manages the knowledge • logical reasoning • makes decisions

  8. Conscious brain - operation modes • Motivation • What isBlueEyes ? • Physiological foundations • Systemdesign • System implementation • Futureimprovements • Project worksummary • Systemdemonstration Exploratory mode Planning mode

  9. Retina - central vision (macula) Eye movements • Motivation • What isBlueEyes ? • Physiological foundations • Systemdesign • System implementation • Futureimprovements • Project worksummary • Systemdemonstration • Central vision andthe conscious brain Saccade - a rapid eye jump to a new location assigned by the conscious attention process. Saccades presence  Exploratory mode

  10. Jazz Multisensor The eye movement sensor • Motivation • What isBlueEyes ? • Physiological foundations • Systemdesign • System implementation • Futureimprovements • Project worksummary • Systemdemonstration • Eye position measuring - direct infrared oculography • Oxy- and deoxyhemoglobin measurment • Two axialaccelerometer • Ambient lightsensor

  11. Wires, wires, wires... • Motivation • What isBlueEyes ? • Physiological foundations • Systemdesign • System implementation • Futureimprovements • Project worksummary • Systemdemonstration A serious limitation of: • mobility • comfort • safety

  12. A wireless Personal Area Network Personal Area Network ! • Motivation • What isBlueEyes ? • Physiological foundations • Systemdesign • System implementation • Futureimprovements • Project worksummary • Systemdemonstration Solution:

  13. Design objectives • Motivation • What isBlueEyes ? • Physiological foundations • Systemdesign • System implementation • Futureimprovements • Project worksummary • Systemdemonstration • A mobile data acquisition device • A reliable real-time buffering, processing and recording • A clear visualization interface • Mechanisms for creating custom data analyzers • To ensure a possibility of distributing the processing among a number of computers • Objectives • System overview • DAU • CSU

  14. Data Acquisition Unit Central System Unit Bluetooth Bluetooth Data Logger Bluetooth technology provides means for creating a Personal Area Network linking the operators and the central system. System overview • Motivation • What isBlueEyes ? • Physiological foundations • Systemdesign • System implementation • Futureimprovements • Project worksummary • Systemdemonstration Connection Manager Visualization Module Atmel 89C52 microcontroller • Objectives • System overview • DAU • CSU Data Analysis Jazz Multisensor

  15. Data Acquisition Unit DAU - features • Motivation • What isBlueEyes ? • Physiological foundations • Systemdesign • System implementation • Futureimprovements • Project worksummary • Systemdemonstration • Lightweight • Runs on batteries - low power consumption • Easy to use - does not disturb the operator working • ID cards for operator authorization • Voice transmission using hardware PCM codec Atmel 89C52 microcontroller Bluetooth • Objectives • System overview • DAU • CSU Jazz Multisensor

  16. Central System Unit Data Logger CSU - features • Motivation • What isBlueEyes ? • Physiological foundations • Systemdesign • System implementation • Futureimprovements • Project worksummary • Systemdemonstration • Connection management • Data processing • Visualization • Data recording Connection Manager Connection Manager Visualization Module Bluetooth • Objectives • System overview • DAU • CSU Data Analysis • Accessverification • System maintenance

  17. Differentiation Conscious brain involvement level saccade CSU - saccade detector • Motivation • What isBlueEyes ? • Physiological foundations • Systemdesign • System implementation • Futureimprovements • Project worksummary • Systemdemonstration • Saccade detector and conscious brain involvement level calculation Raw eye movement data • Objectives • System overview • DAU • CSU

  18. VU-meters • pie-charts • time series • audio and videodata playback CSU - Visualization Module • Motivation • What isBlueEyes ? • Physiological foundations • Systemdesign • System implementation • Futureimprovements • Project worksummary • Systemdemonstration • Raw and processed data visualization using: • Objectives • System overview • DAU • CSU

  19. Data security • Motivation • What isBlueEyes ? • Physiological foundations • Systemdesign • System implementation • Futureimprovements • Project worksummary • Systemdemonstration • Only registered mobile devicescan connect to the system • Bluetooth connection authentication • Bluetooth connection encryption • Access rights restrictions • Personal and physiological data encryption • Objectives • System overview • DAU • CSU

  20. Made from scratch by the team Implementation - DAU • Motivation • What isBlueEyes ? • Physiological foundations • Systemdesign • System implementation • Futureimprovements • Project worksummary • Systemdemonstration • Bluetooth module inside • Jazz sensorinterface • Microphone andearphone sockets • 5-key keyboard • Two lines, 8characters LCD • Powered from6 AA batteries • DAU • CSU

  21. Microcontroller board PCM codec board Batteries andvoltage level monitor Bluetooth module Implementation - DAU • Motivation • What isBlueEyes ? • Physiological foundations • Systemdesign • System implementation • Futureimprovements • Project worksummary • Systemdemonstration • DAU • CSU

  22. Standard MDI userinterface Implementation - CSU • Motivation • What isBlueEyes ? • Physiological foundations • Systemdesign • System implementation • Futureimprovements • Project worksummary • Systemdemonstration • Custom-builtBluetoothcommunicationroutines • Intel USB camera • ID card programmer • Developed using Visual C++ 6.0 • Deployment environment:Windows 2000,MS SQL Server • DAU • CSU

  23. Future improvements • Motivation • What isBlueEyes ? • Physiological foundations • Systemdesign • System implementation • Systemdemonstration • Futureimprovements • Project worksummary • DAU • small CMOS camera to monitor the operator’s point of gaze • single PCB (SMD technlogy) • low voltage ICs - LiIO batteries power • CSU • data mining algorithms • advanced database encryption using e.g. AES algorithm

  24. Project work summary • Motivation • What isBlueEyes ? • Physiological foundations • Systemdesign • System implementation • Systemdemonstration • Futureimprovements • Project worksummary • Bluetooth and PAN technology acquired • New skills on designing software and hardware systems gained • A mobile device implemented • 31 KLOC of C++ code and4 KLOC of 8051 assembler code written • All parts of the Project Kit efficiently used • New approach to system safety proposed and implemented

  25. Motivation • What isBlueEyes ? • Physiological foundations • Systemdesign • System implementation • Futureimprovements • Project worksummary • Systemdemonstration BlueEyesHuman-Operator Monitoring System System demonstration

  26. Thank you for your attention • Motivation • What isBlueEyes ? • Physiological foundations • Systemdesign • System implementation • Systemdemonstration • Futureimprovements • Project worksummary BlueEyesHuman-Operator Monitoring System Poznań University of Technology, Poland http://www.put.poznan.pl

  27. microphone earphone Bluetooth module BlueEyes Data Acquisition Unit DAU - components MC 145483 PCM codec Atmel 89C52 microcontroller Jazz Multisensor Beeper UART LCD display MAX232 UART LED indicators ID card interface ID card Simple keyboard

  28. Bluetooth ... ... Operator Manager Raw Data Buffer Bluetooth Alerts Alert buffer CSU - Connection Manager Operator Data Manager • Initializes Bluetooth modules • Inquires for new devices in range • Establishes connections • Supports operator roaming • Feeds Operator Manager with physiological data • Data demultiplexer • Data producer queues Bluetooth Connection Manager Operator Manager Raw Data Buffer Separated data streams Alerts Alert buffer ... ...

  29. Vertical Acceleration variance value high Y N Horizontal Acceleration variance high Horizontal Acceleration variance high N Y N Y CSU - Data Analysis • Custom data analyzers - decision tree based classifiers Example: Operator gesture recognition (simplified) No answer “YES” “NO”

  30. Assignedcategory Data samples “Yes” No answer Decision tree coefficients “Yes” “No” No answer CSU - Data Analysis • Custom data analyzers - decision tree based classifiers • Automatic decision tree coefficients generation - C4.5 based algorithm ... ...

  31. register at Processing Dataconsumer & producer Data queue Processing Dataconsumer & producer Data queue CSU - intermodule communication • Uses buffered thread-safe queues and producer-consumer scheme Data consumer Data queue creates Data producer Data consumer Data consumer Data consumer

  32. Connection Manager ... Operator manager (OM) OM OM CSU - intermodule communication Separated physiological data streams Alerts Data Analysis Visualization Module Data Logger Processed data Recorded (off-line) data

More Related