Stay Kinected: A Home Monitoring System Combining Safety and Comfort

1. Stay Kinected: A Home Monitoring System Combining Safety and Comfort Purpose Abstract The purpose of this project is to use the Microsoft Kinect sensor to implement a home monitoring system. The target audience for this system are the elderly or those with physical disabilities who may be candidates for assisted living. The Stay Kinected system combines safety and the comfort of remaining at home. The purpose of this project was to develop a home monitoring system that used the Microsoft Kinect Sensor to track a user’s body and recognize different motions/positions that would suggest they are hurt or in danger. A part of developing the system was creating a phone application that would alert a caretaker or the local authorities when an accident occurred so that the user could receive immediate attention. The phone application would also allow a caretaker to view a live video feed from the Kinect senor so that they could check in on the user to ensure the user was safe during the day. The system was created using the traditional coding steps which include design, setting up the development environment, coding, debugging, and testing. After the functions of the program were established and the development environment and other necessary tools were acquired, the coding process was initiated. For testing purposes, many different situations were simulated to ensure the program could recognize all of the possible situations that could occur. By the end of testing, the Kinect Sensor was implemented successfully in a functional home monitoring system that promotes the monitoring of a safe living environment. Assistance Arrives Quickly Kinect Sensor Installed Sensor Detects Event Event Occurs Program Sends Text Alert Objectives • This project was created with future marketing and real-world application in mind. The Graphical User Interface (GUI) was made in such a way that would allow for an easy in-home setup. Part of the GUI includes installation steps for mounting the Kinect Sensor. • The goals of this project were to: • Implement the Kinect Sensor in a home monitoring system • Program the Kinect to recognize certain events (e.g., detecting if the user has fallen, if the user leaves the house, and if there is an intruder) • Have the program notify a predefined caregiver or contact when one of the events occurs • Create a simple Graphical User Interface for home installation in which the user decides which of the above features they would like the system to detect • When the program is started, the user is greeted with set up instructions for the in-home system. Set up for the system is quick and easy and only requires that the Kinect is mounted and the angle of elevation is adjusted for a clear view of the room. • First the software program is started and set up • The program continuously runs and monitors the user’s motions • If the program detects a certain event (e.g., the user has fallen, left the house, or there is another person in the house) a text message is sent to a caretaker’s phone • A message can also be sent to a local hospital so that an ambulance can be dispatched for assistance Program Methods The method used to track if the user has fallen down. To do this, the program compares the position of the Shoulder Center joint and the Hip Center joint. If these joints have the same Y value, the program recognizes the position and returns true. This Boolean method is used to determine if a text message should be sent. Development • Design • Included in the design was a general function that I wanted the program to perform, which was customized to specific behaviors • The general goal was to use the Windows Kinect Sensor in a home monitoring system • After more thought and contemplation, I decided on more specific functions for the program. These specific functions included: • Implementing the Kinect skeletal tracking in fall detection • Monitoring if more than one person was in the home (real-world application being if there was an intruder in the home) • Detecting if a user walks out the door (for those who have Alzheimers) • Setting Up Development Environment • I used the C# language in Microsoft’s Visual Studio Premium 2012 Development Environment • The language and development environment were chosen because their capabilities best fit what was needed for creating the software • One setback of using Visual Studio 2012 was that I had no prior experience using this development environment. Because of this, a large portion of time was spent teaching myself how to use Visual Studio 2012 • Coding/Debugging • Most of the time was used for programming and debugging • Some of the features of the Kinect that were pre-programmed • My program implements some of these capabilities, such as Skeletal Tracking, which were accessible though reference to other coding projects • Testing • Trials of 50 scenarios were ran in order to determine the success rate of each function of the program(50 trials for each function = 150 total) • The fall detection, intruder detection, and detection of leaving the home were tested separately C# The method used to detect if the user has left the house. In this method, the program compares the pre-defined location of the door (measured during set up) and the user’s position to recognize if they have left the house. The process used for creating Stay Kinected followed the normal steps of software development. These steps (diagramed above) were followed in order. The bulk of time used to create the software program was spent in the Coding/Debugging phase. The essential tools used to create the program were the Windows Kinect and Kinect programming SDK, Visual Studio Premium 2012 Development Environment, and the Twilio SMS Text Messaging Service. The method is used to detect if there is another person in the house. In this method, the program takes a count of the number of tracked people in the home and if the number of people is larger than some predefined number (2 is used here), the program will indicate that an alert should be sent out.

2. The Mathematics Behind Fall Detection Kinect Skeleton Joint Mapping And Detection • When the Kinect Sensor is tilted, a new set of X, Y, and Z planes are established. Because of this, the program can detect the difference in Y values of the Shoulder Center Joint and the Hip Center Joint. • As seen by the sensor, a person laying on the floor would still have some distance between these joints. • The Elevation Angle of the Kinect sensor has a range of -27° to 27°. Above The image above shows the different mechanics of the Kinect Sensor Left When the Kinect Sensor tracks a user, it maps a skeleton onto them by identifying the 20 different joint positions Below The Kinect Sensor can detect up to 6 users and can map skeletons onto 2 of them. • The blue dots indicate the Hip_Center joint and the Shoulder_Center joint • The red lines show the new X and Y axes created when the sensor is tilted • To find the distance between the Shoulder_Center joint and the Hip_Center joint (as seen by the Kinect) we define the difference in the Y values as “x”. • We also use the distance between the joints (in the regular X and Y planes), which is measured during system set up. We label this value “d” in the diagram. • Lastly, we use the angle between the regular X plane and the Kinect’s X plane. This angle is equal to the negative value of the Kinect’s tilt angle. • From these variables, we derived an equation for x. • The green line is the desired length which is labeled “x” • The red line is the distance between the joints labeled “d” Future Work Results • The program successfully implemented the desired functions set for the project. However, there are a vast number of possibilities for future expansion of this program’s capabilities. With more time and a better understanding of Visual Studio Premium 2012 and what all the Kinect Sensor is capable of doing, this home monitoring system could be enhanced in many ways to further provide more safety detection capabilities. • Some of the future work ideas that will be considered are: • Implementing detection to monitor if the user has taken their medication • Implementing detection to monitor if the user has left the oven or stove on • Implementing detection to monitor if the user has slept too long • Programming a specific (in-wall mounted) sensor to recognize certain actions that the user can do to indicate that they think there is a problem • As an example, if the user stands in front of the sensor and makes an “X” signal with their arms, the sensor would read this as “I don’t feel well and need assistance” and then the program would send out an alert. • Also, the program could make use of the Kinect Sensor’s microphone; thus, if the user says something like “Help”, the program will then send an alert. • Further enhancing the Graphical User Interface for easy in-home setup • Adding customization capabilities in the User Interface so that the system becomes specialized to cater to the needs of each individual user • Adding a recognition system so Stay Kinected recognizes what the user’s skeleton looks like. This will enhance the intruder detection. • One large expansion to Stay Kinected would be connecting different Kinect sensors throughout the house to “communicate” with each other in order to keep track of where the user is at all times. With this addition to the system, there are even more enhancements that could be added such as: • Programming the system to keep a log of the users activities as they move throughout the house • Providing an accessible location (i.e., phone application, website, etc.) in which a member of the users family can check this log. • Adding a live stream view on the phone application or website that a family member can login and see, in real-time, what the user is doing. The final version of the program achieved the goals set for the project. The Microsoft Kinect sensor was implemented successfully in a home monitoring system that provides protection and comfort to any person, especially the elderly or those with physical disabilities who may not have access to a nursing home or other assisted living facility. The program uses the skeletal joint mapping capabilities of the Kinect Sensor to analyze different body positions that the user makes. The program recognizes certain positions as indicators that an accident has occurred or the user may be in danger. The system is capable of detecting a fall, an intruder, and if the user leaves the house. The alerting ability was also successfully incorporated into the software program using the Twilio web messaging service. With the functional alerting system, a family member or friend can receive an SMS text message from the system whenever the user has fallen, left the house, or if there is an intruder. Another goal set for the program was to create a simple Graphical User Interface, with commercial capability in mind, that would allow simple home installation in the future. This goal was achieved and customization was incorporated into the setup interface. In the setup, the system takes measurements of the user, of the front door’s location, and allows the user to check-box the functions (fall detection, intruder detection, leaving the house detection) that they would like the system to use. Images of the User Interface are below. The final setup tab allows the user to choose which functions they would like to use. This adds the desired customization to fit every user’s distinct needs. Screenshot of the initial greeting screen that appears when the program is executed. The mounting tab provides pictures and simple instructions for easy installation.