ABSTRACT

1. Monitoring body movement during sleep through computer vision and analyzing the pattern between different stages of sleep Name: Jin U Bak, GagyungKim, Nikolas G Instructor: Dr. NikolaosMavridis Prof. Leonard Course: Inventions Spring 2012 ABSTRACT This project aims to find patterns in body movement during sleep and also the relationship between body movement and sleep stages using computer vision and EEG. Thermal imaging camera and Zeo Sleep Manager that identifies sleep stages have been used to develop the system that analyzes body movement during sleep and compares it to the stages of sleep determined by Zeo Sleep Manager. We have collected real data from Jin U Bak, the author of the article for few consecutive nights. Our method in analyzing the data was reliable since two different methods were tested and gave out very similar outputs. After processing and analyzing data that were collected for three consecutive nights, it is found that body movement mostly occurs during REM and light sleep stages. The system built can also be used to predict sleep stages since it is found that sleep stage diagram and movement diagram have similar patterns. INTRODUCTION Assuming that people sleep 8 hours a day on average, people spend about 33% of their lives sleeping. Studies on sleep have been done in many different ways such as EEG monitoring, body movement tracking using pressure mat [1] or bed temperature monitoring [2] to make that 33% of people’s lives more valuable and productive. Sleep monitoring could be crucial in detecting sleep disorders and treatment of sleep disorders [3]. In order to monitor one’s sleep, identifying the sleep stage the person is in could be useful since there is a relation between body movement and the depth of sleep [4]. Sleep stages mainly consist of Non-Rapid Eye Movement (NREM) and Rapid Eye Movement (REM) sleep. People go through what is called as ‘Sleep Cycle’ that consists of combination of periodic NREM and REM sleep. NREM sleep can be further divided into four stages; N1, N2, N3 and N4 [5]. Sleep stages are identified using EEG that collects signals from brain activity and it recognizes the stages of sleep depending on what wave form the brain sends out. Monitoring sleep through EEG involves direct contact between the user and the device which could discomfort user’s sleep. Studies on body movement during sleep have been done in relation to health condition of a body [1]. We develop a system using thermal imaging camera and computer vision to detect body movements of a person during sleep and find patterns in movement in relation to different stages of sleep. METHOD 1. Video recording from PathFindIR camera In order to be able to detect body movement during sleep, we have used thermal imaging camera called ‘PathFindIR’ manufactured by FLIR. This device enables us to record a video of a person sleeping in dark environment and makes it easier for us to differentiate a human body from the background which is one of the most important factors for computer vision processing. Unlike other studies that have been done in the past which use sensors that collect data from direct contact from a human body, we have used computer vision as a tool to analyze body movement from the video that has been recorded from the thermal imaging camera. Figure1:An image from the thermal imaging camera 2. Identification of sleep stages The sleep stage in which a person is in can be identified using a device called ‘Zeo Sleep Manager’ which is a wearable device that is put around forehead to collect brain signals. Collected brain signals are transferred to another device that writes this information into an SD memory card. ‘Zeo’ has an online application that processes the data and presents it in different forms which could be useful and handy in comparing it with the data processed from computer vision. Figure2: Zeo Sleep Manager used in the experiment 3. Movement detection and Video processing This part was one of the most important parts in this project since detecting body movement during sleep through computer vision required careful attention in design the algorithm to give out an accurate data set. We have used programming software called ‘MATLAB’ that had built-in computer vision functions that were useful in creating a motion detecting program. We had different methods to analyze the video such as optical flow, frame-by-frame comparison and background estimation. In this experiment, two methods were used; optical flow and frame-by-frame comparison. Then two different results from each method were compared and it has come to our attention that outputs obtained from each method were similar and therefore, we have decided to use frame-by-frame since it could process lengthy video in few minutes whereas optical flow required more time, almost the length of the video to process since it required the whole video to be played from the beginning to the end. Frame-by-frame method is simply implemented by taking one frame from each second of the video and comparing it to the previous frame. Frame taken from each second was converted into gray scale image that has two dimensions from RGB that has three dimensions. Then the chosen frame is compared to the frame taken from the previous second. The previous frame is subtracted from the next frame then the difference was compared to the threshold. Threshold is determined using trial and error method by analyzing the video that has an empty bed and no movement. Several values of threshold were tested and the optimal threshold was applied in analyzing the videos. When the difference is compared to the threshold, the output is given in terms of logical matrix that consists of 0s and 1s. Then the number of 1s was counted in the logical matrix and this represents body movement in the video. The thermal image made this process simple since the video consists of only black and white but different intensity depending on the temperature of the objects in the video. The above process was looped and in the end of the analysis, it was coded in such a way that it plots movement diagram using proper time scale. Figure3: Movement diagram using frame-by-frame analysis. The higher the peak, the bigger body movement RESULT Data from both Zeo Sleep Manager and thermal imaging camera were collected for 3 consecutive nights. The collected data sets were processed through Zeo online application and code that was written in MATLAB. Figure4: Analysis of data from 1st night Zeo app (top) and computer vision analysis (bottom) Figure5: Analysis of data from 2nd night Figure6: Analysis of data from 3rd night CONCLUSION/DISCUSSION AND FUTURE WORK After analyzing the data and processing them through different media, we were able to get the output the way we were expected it to be. The results were studied carefully by comparing two different graphs. We notice that most of body movement during sleep occurs during REM sleep stage and this is an expected outcome since people dream during this period and body movement increases during this period [7]. It is also clear that when a person is in light sleep or awake, body movement occurs as well. But when a person stays in deep sleep stage known as N4 NREM stage [5], the person rarely makes body movement. These results agree with the related work done in the past [7]. The other interesting finding from the results is that there is a tendency of body movement when there is a transition from one sleep stage to another. As it can be seen from the results, when there is a change in the stage of sleep, there is a peak at that particular point in the movement diagram. Using this information, one can predict the stage of sleep without EEG. This project can be further developed with better analyzing algorithm to predict one’s stage of sleep using non-invasive device. The system can also be turned into different application such as alarm clock that uses computer vision to detect a body on the bed and wake a person up and gets the person out of the bed to make sure that the alarm clock does its job. This project can be used in monitoring seniors at homes and also for security purposes that detects human bodies and informing the user if there is any abnormal pattern. ACKNOWLEDGEMENTS We would like to thank Prof. NikolaosMavridis and Prof. Leonard RetelHelmrich for giving us such a great opportunity to carry out such an interesting research and for all the support throughout the project. REFERENCE [1] Harada, T, T Sato, and T Mori. “Pressure distribution image based human motion tracking system using skeleton and surface integration model.” Proceedings 2001 ICRA IEEE International Conference on Robotics and Automation Cat No01CH37164. Ieee, 2001. 3201-3207. [2] T Tamura el al. “A bed temperature monitoring system for assessing body movement during sleep.” Clin. Phys. Physiol. Meas. 9 139. [3] Vangelis, M, G Galatas, A Papangelis, D Kosmopoulos, and F Makedon. “Recognition of sleep patterns using a bed pressure mat.” Proceedings of the 4th International Conference on Pervasive Technologies Related to Assistive Environments 2011. [4] E. Aserinsky and N. Kleitman. “A Mobility Cycle in Sleeping Infants as manifested by Ocular and Gross Bodily Activity.” Journal of Applied Physiology, Volume 8:11. The American Physiological Society Jul 1, 1955. [5] Kittredge, Clare. "Sleep Stages." EverydayHealth.com. 19 Jan. 2010. Web. 15 May 2012.<http://www.everydayhealth.com/sleep/101/stages-of-sleep.aspx>. [6] T Tamura, S Miyasako, T Fujimoto, and T Togawa. “Monitoring Bed Temperature in Elderly in the Home” 18th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Amsterdam 1996 1.1.5: Monitoring Instruments [7] William Dement, Nathaniel Kleitman, “Cyclic variations in EEG during sleep and their relation to eye movements, body motility, and dreaming.” Electroencephalography and Clinical Neurophysiology, Volume 9, Issue 4, November 1957, Pages 673-690, ISSN 0013-4694, 10.1016/0013-4694(57)90088-3.