Non-invasive Techniques for Human Fatigue Monitoring

1. Non-invasive Techniques for Human Fatigue Monitoring Qiang Ji Dept. of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute qji@ecse.rpi.edu http://www.ecse.rpi.edu/homepages/qji Funded by AFOSR and Honda

3. Visual Behaviors • Visual behaviors that typically reflect a • person's level of fatigue include • Eyelid movement • Head movement • Gaze • Facial expressions

4. Eye Detection and Tracking

5. Eye Detection

6. Develop an eye tracking technique based on combining mean-shift and Kalman filtering tracking. It can robustly track eyes under different face orientations, illuminations, and large head movements. Eye Tracking

8. Eyelid Movements Characterization Eyelid movement parameters • Percentage of Eye Closure (PERCLOS) • Average Eye Closure/Open Speed (AECS)

10. Gaze (Pupil Movements) • Real time gaze tracking • Develop a real time gaze tracking technqiue. • No calibration is needed and allows natural head movements !.

11. Gaze is determined by Pupil location (local gaze) Local gaze is characterized by relative positions between glint and pupil. Head orientation (global gaze) Head orientation is estimated by pupil shape, pupil position, pupil orientation, and pupil size. Gaze Estimation

13. Gaze Parameters • Gaze spatial distribution over time • PERSAC-percentage of saccade eye movement over time

14. Gaze distribution over time while alert

15. Gaze distribution over time while fatigue

16. Gaze distribution over time for inattentive driving

17. Plot of PERSAC parameter over 30 seconds.

19. Head Movement • Real time head pose tracking • Perform 3D face pose estimation from a single uncalibrated camera. • Head movement parameters • Head tilt frequency over time (TiltFreq)

20. The flowchart of face pose tracking

21. Examples Face Model Acquisition

23. Head pitches (tilts) monitoring over time (seconds)

25. Facial Expressions • Tracking facial features • Recognize certain facial expressions related to fatigue like yawning and compute its frequency (YawnFreq) • Building a database of fatigue expressions for training

27. The plot of the openness of the mouth over time

28. Facial expression demo

29. Fatigue Modeling • Observations of fatigue is uncertain, incomplete, dynamic, and from different from perspectives • Fatigue represents the affective state of an individual, is not observable, and can only be inferred.

30. Overview of Our Approach Propose a probabilistic framework based on the Dynamic Bayesian Networks (DBN) to • systematically represent and integrate various sources of information related to fatigue over time. • infer and predict fatigue from the available observations and the relevant contextual information.

31. Bayesian Networks Construction • A DBN model consists of target hypothesis variables (hidden nodes) and information variables (information nodes). • Fatigue is the target hypothesis variable that we intend to infer. • Other contextual factors and visual cues are the information nodes.

32. Causes for Fatigue Major factors to cause fatigue include: • Sleep quality. • Circadian rhythm (time of day). • Physical conditions. • Working environment.

33. Bayesian Fatigue Model

34. Dynamic Fatigue Modeling

35. Bayesian Fatigue Model Demo

36. Interface with Vision Module • An interface has been developed to connect the output of the computer vision system with the information fusion engine. • The interface instantiates the evidences of the fatigue network, which then performs fatigue inference and displays the fatigue index in real time.

38. Conclusions • Developed non-intrusive real-time computer vision techniques to extract multiple fatigue parameters related to eyelid movements, gaze, head movement, and facial expressions. • Develop a probabilistic framework based on the Dynamic Bayesian networks to model and integrate contextual and visual cues information for fatigue detection over time.

39. Effective Fatigue Monitoring • The technology must be non-intrusive and in real time. • It should simultaneously extract multiple parameters and systematically combine them over time in order to obtain a robust and consistent fatigue characterization. • A fatigue model is needed that can represent uncertain and dynamic knowledge associated with fatigue and integrate them over time to infer and predict human fatigue.