Personal Assistant System: Use of Computer Technologies to Facilitate Elderly People with Assisted Living

1. Personal Assistant System: Use of Computer Technologies to Facilitate Elderly People with Assisted Living Presenter: Jennifer C. Hou‡ Collaboration with Linda Ball*, Stanley Birge*, Marco Caccamo‡, Carl Gunter‡, Karrie G. Karahalios‡, Narasimhan Nitya #, Lui Sha‡, Yang Yu# ‡ Dept of Computer Science University of Illinois at Urbana Champaign *College of Medicine, Washington University in Saint Louis #Pervasive Platforms and Architecture Lab, Motorola Labs jhou@cs.uiuc.edu, 217-265-6329

2. Population Aging • Table compiled by the U.S. Administration on Aging based on data from the U.S. Census Bureau. • Aging of the baby boomer has become a social and economical issue. • In the United States alone, the number of people over age 65 is expected to hit 70 million by 2030, almost doubling from 35 million in 2000.

3. Percentage of People of 65+ and 85+ • People over age 65 are expected to constitute 20% of the population in 2030. • Similar increases are expected worldwide. • Table compiled by the U.S. Administration on Aging based on data from the U.S. Census Bureau.

4. Similar Expenses Worldwide 2002 SOURCE: United Nations ▪ “Population Aging ▪ 2002”

5. Similar Expenses Worldwide 2030 SOURCE: United Nations ▪ “Population Aging ▪ 2002”

6. Consequences Are ….. • Along with the increase of elderly people population, the expenditures of the United States for health care will project to rise to 15.9% of the GDP ($2.6 trillion) by 2010. -- Health care industry study, Digital Foresight • Many elderly people will stay at home, rather than being consigned to expensive retirement homes. • Even today, only 10% of elderly people of age 65-85 and 25% of those of age >= 85 are institutionalized. • Many elderly people choose to stay at home also for privacy/dignity issues.

7. Overview of the Talk • Why Do We Care About Senior Care? • Overview of Personal Assistant System (PAS) • Underlying Research that Makes Things Work • Systems Architecture • Wireless communications • Tracking and localization • Security and privacy • Software safety, reliability, and availability • Human computer interface • Do Elderly People Accept This Technology? • Pilot studies • Where Do We Go From Here?

8. Why Do We Care About Senior Care • Because your parents are the next in line to be qualified as seniors, and you are next to the next in line. • Can advances in sensing, object localization, wireless communications technologies • enable elderly people to regain their capability of independent living? • make possible unobtrusive supervision of basic needs of frail elderly and thereby replicate services of on-site health care providers? We believe the answers are Yes!

9. We Believe Technology Can Help • Time Driven Reminders of Daily Activities: Home PC • Serves as the intelligence. • Sends reminder messages to wireless-enabled appliances. • Closes the loop with HCI/ localization techniques. • Takes action in the lack of response: A reminder can be sent more times, after which a designated on-site personnel or a healthcare provider is notified. Jennifer, It is 8:30am. Time to take your Insulin injection before breakfast.

10. Assisted Living Hub (ALH) Clinician ALSP Service Gateway Medical Device Overview of PAS • A drop-box architecture in which • A security-enhanced, assisted living device called Authentication Manager for You(AMY) co-exists with a home PC, called the Assisted-Living Hub (ALH). • The ALH is equipped with multiple wireless interfaces and serves as the local intelligence. It also communicates with the Assisted Living Service Provider (ALSP). • Through web interfaces, ALSP allows healthcare providers and clinicians to retrieve/analyze data and give instructions.

11. A More Technical View of PAS Home Environment ALSP ALH Monitoring IEEE 802.11 WLAN Internet . AMY Service Clinicians • Easy to deploy. • Few entities that are subject to security attacks. • Numerous applications can be built on top of it.

12. What Applications Are Most Critical Factors Contributing to Loss of Independence and Institutionalization* % of Residents % of Residents as Need With Need Primary Cause ________________________________________________________________________ Needs prompting to take medications 95 42 Risk of injury due to falls 42 17 Unable to get up after a fall 20 17 Monitoring of vital signs too labor intensive 12 20 Needs physical assistance with Activities of Daily Living 90 67 Needs prompting to toilet on a schedule 67 17 Needs prompting to go to meals 33 10 Needs prompting to bathe 75 0 Gets lost in apartment 17 0 May wander out of facility 12 10 Needs monitoring of blood sugar frequently 20 8 Needs monitoring of weight daily/weekly 25 0 * Respondents include 8 geriatricians, 10 nurse administrators of assisted living facilities, and 6 home health clinicians at Washington University in Saint Louis

13. Assisted Living Hub (ALH) Clinician ALSP Service Gateway Medical Device Application I: Time-based Reminder Services • PAS can help reminding residents of daily activities. • ALH obtains from the ALSP updated prescription and appointment records of a resident. • When it is time for the resident to carry out their time driven routines, the ALH locates active wireless-enable devices and sends reminder messages to one or more devices that are in the proximity of the resident. • Whether or not these routines are followed as advised is detected in a non-intrusive manner by exploiting sensor localization techniques.

14. Application II: Monitoring of Physiological Functions • A number of physiological functions critical to maintaining homeostasis for different medical conditions can be measured by Bluetooth-enabled medical devices, transmitted to the ALH and then to the ALSP to be evalauted by healthcare providers. • Measures will have a prescribed desired range and deviations from that range will generate an alert from the ALSP to the health care provider. • This enables prompt intervention before the situation deterioates to a point requiring hospitalization and may simply be in the form of additional instructions to the resident. ALSP Home Environment IEEE 802.11 WLAN Monitoring Internet . . Service AMY

15. Application III: Non-intrusive Monitoring of Daily Activities and locations • Detection of early warning signs for depression and/or other chronic diseases: • The location of a resident and the duration in which he/she stays at that location are profiled by Motes-based (or Ubisense-based) sensing and tracking in a privacy-preserving manner. • Movement profiles are transmitted to the ALSP and analyzed (if necessary) for • Early warning signs for severe depression (e.g., not taking medicine, not eating, staying in bed for very long time) • Preventive measures (behavior changes) for chronic diseases commonly seen in elderly people (such as Parkinson’s disease and/or Alzheimer’s disease).

16. Application IV: Fall Detection • Residents wear sensors equipped with accelerometers (with fall detection algorithms that detect falls with the combination of speed and orientation changes). • In the case of fall detection, the sensor device beeps and an alert message is sent to the ALSP and to a designated healthcare provider. • In the case of false alarm, the resident can press a button and disable the message sending.

17. Overview of the Talk • Why Do We Care About Senior Care? • Overview of Personal Assistant System (PAS) • Underlying Research that Makes Things Work • Systems Architecture • Wireless communications • Tracking and localization • Security and privacy • Software safety, reliability, and availability • Human computer interface • Do Elderly People Accept This Technology? • Pilot studies • Where Do We Go From Here?

18. Software Infrastructure Needed • A software infrastructure that integrates sensing, communication, and event/information management. • Understand, analyze, structure and control the complex interactions across the layers of computing, communication and sensing along the dimensions of robustness, reliability, QoS, security and privacy. Emergency event processing Telemedicine Reminders/social interaction Monitoring Interface mgmt Dependability Evolvability Real-time Security & Privacy Events/data management Communication Sensing

19. ServerSocket ss = new ServerSocket(90); ss.accept(); Networked Software Architecture Socket s = new Socket(myaslp.com:90) Active Application A ALSP Server Under Normal Situation When the Gateway is Available J2ME API for AccessingBT Nodes and Internet TCP/IPComm. API ALSPServerOS FixedALHOS TCP/IP Stack BT Stack TCP/IP Stack Bluetooth Cellphone Network + Internet Vital Sign Meters IP Routing Service 802.11 802.11MAC WirelineMAC Gateway

20. System Architecture of the ALH

21. OS Layer in the ALH Architecture • The OS layer • Is equipped with various communication stacks and corresponding platform-dependent APIs • Depending on the family of peripherals to support, other stacks and APIs may include Bluetooth, Zigbee, and Infrared.

22. Middleware Layer in the ALH Architecture • Middleware • Device monitoring daemons: monitor the join/leave of peripheral devices in the environment and registers/de-registers the devices in the Device Registry Services. • The register-de-register process includes creating/destroying proxy or singular proxy stub objects for the device. • The proxy/proxy stub allows applications to be built upon well-known device APIs instead of vendor specific APIs/semantics. It also allows off-the-shelf devices to be integrated into the PAS system, as long as the vendor provides the semantics specifications on how to communicate with the device.

23. Middleware Layer in the ALH Architecture • Device registry service: maintains a database of peripherals available in the environment. Each entry of the database is a proxy (or proxy stub) object created for a specific peripheral device. • Applications will query this service for appropriate proxies. When an appropriate proxy is found, a clone of the proxy migrates to the application.

24. Middleware Layer in the ALH Architecture • The Unified Peripheral Communication APIs • Abstracts different network stack programming interfaces into a consistent paradigm, which basically follows java.net.* APIs. • Allows users to specify (in XML) their QoS demands when establishing communication links, and the middleware returns approved QoS guarantees. • This allows proxies to be developed independently of specific network stack programming interfaces.

25. Middleware Layer in the ALH Architecture • Internet Heartbeat Daemon: • periodically checks the availability of Internet access through the gateway router. • When the gateway router fails/recovers, this daemon activates/deactivates the Bluetooth cell phone to access to the ALSP. • ALH Main Daemon: is in charge of managing (start, suspend, stop, restart etc.) all the application daemons and middleware daemons on the ALH.

26. Overview of the Talk • Why Do We Care About Senior Care? • Overview of Personal Assistant System (PAS) • Underlying Research that Makes Things Work • Systems Architecture • Wireless communications and networking • Tracking and localization • Security and privacy • Software safety, reliability, and availability • Human computer interface • Do Elderly People Accept This Technology? • Pilot studies • Where Do We Go From Here?

27. Networked Software Architecture TV Peripheral Bluetooth Network ALSP ALH earplug Internet Gateway WLAN(e.g. IEEE 802.11) Internet-capable Medical Meters

28. ServerSocket ss = new ServerSocket(90); ss.accept(); Networked Software Architecture Socket s = new Socket(myaslp.com:90) Active Application A ALSP Server Under Normal Situation When the Gateway is Available J2ME API for AccessingBT Nodes and Internet TCP/IPComm. API ALSPServerOS FixedALHOS TCP/IP Stack BT Stack TCP/IP Stack Bluetooth Internet Vital Sign Meters IP Routing Service 802.11 802.11MAC WirelineMAC Gateway

29. ServerSocket ss = new ServerSocket(90); ss.accept() Networked Software Architecture Socket s = new Socket(128.174.11.11:90) Passive Application A ALSP Server Under Normal Situation When the Gateway is Available J2ME API for AccessingBT Nodes and Internet TCP/IPComm. API ALSPServerOS TCP/IP Stack FixedALHOS BT Stack TCP/IP Stack PC IP: 128.174.11.11 Bluetooth Cellphone Network + Internet Dumb Dev IP Routing Service 802.11 802.11MAC WirelineMAC Gateway

30. What If the Internet Gateway Is Down? • One major deficiency of PAS is that all the traffic is transported through the gateway AMY, which becomes unavailable when it fails or when the resident is away from home. • To enhance robustness and ubiquity of PAS, we have used cell phones (transparent to users) as both a backup AMY and the local intelligence for data aggregation and acquisition. • we have leveraged the programming capability of Motorola EZX platforms (mainly the A780, E680 and E680i phones) to incorporate the following novel features that are not currently present in any cell phones: • Enabling cell phones to serve as local data storage/fusion intelligence. • Enabling cell phones as a delivery endpoint for reminder messages

31. Networked Software Architecture mobile medical devices TV Peripheral Bluetooth Network ALSP ALH cellphone earplug Internet WLAN(e.g. IEEE 802.11) GPRS network or WiFi Base Station Internet-capable Medical Meters When the Gatewayis Not Available

32. ServerSocket ss = new ServerSocket(90); ss.accept(); Socket s = new Socket(myaslp.com:90) The cellphone modem is activated Active Application A Cellphone DUN Modem(service provided by most BT enabled cellphones) ALSP Server J2ME + Bluetooth Dial-Up Networking Profile (BT-DUN) for Internet Access APIs (TAPI/NAPI, File system,etc.) TCP/IPComm. API ALSPServerOS Service Layer (Network, Connectivity, Telephony, etc) BT Stack TCP/IP Stack FixedALHOS BT Stack Cellphone OS GPRS Bluetooth Cellphone Network + Internet Bluetooth Medical Meter IP Routing Service 802.11 802.11MAC WirelineMAC A. L. Device

33. ServerSocket ss = new ServerSocket(90); ss.accept() The cellphone modem is activated Socket s = new Socket(128.174.11.12:90) Active Application A Cellphone DUN Modem(service provided by most BT enabled cellphones) ALSP Server J2ME + Bluetooth Dial-Up Networking Profile (BT-DUN) for Internet Access APIs (TAPI/NAPI, File system, etc.) TCP/IPComm. API ALSPServerOS Service Layer (Network, Connectivity, Telephony, etc) BT Stack TCP/IP Stack FixedButlerOS BT Stack Cellphone OS GPRS Bluetooth Cellphone Network + Internet Bluetooth Medical Meter IP Routing Service 802.11 802.11MAC WirelineMAC Gateway

34. Networked Software Architecture mobile medical devices Peripheral Bluetooth Network ALSP Cellphone earplug GPRSor WiFi Internet Base Station WLAN(e.g. IEEE 802.11) Internet-capable Medical Meters When cell phone takes the roll of ALH

35. ServerSocket ss = new ServerSocket(90); ss.accept(); Socket s = new Socket(myaslp.com:90) The cellphone is used as a smart device Application 2 (Sending data to ALSP Server) Active Application A Application 1 (Listening to PC) ALSP Server QoS based J2ME APIs (TAPI/NAPI, File system, etc.) TCP/IPComm. API ALSPServerOS Service Layer (Network, Connectivity, Telephony, etc) BT Stack TCP/IP Stack FixedButlerOS BT Stack Cellphone OS GPRS Or WiFi Bluetooth Cellphone Network + Internet Bluetooth Medical Meter IP Routing Service 802.11 802.11MAC WirelineMAC A. L. Device

36. ServerSocket ss = new ServerSocket(90); ss.accept(); Networked Software Architecture Socket s = new Socket(myaslp.com:90) Active Application A ALSP Server When cell phone takes the roll of ALH J2ME API for AccessingBT Nodes and Internet TCP/IPComm. API ALSPServerOS FixedALHOS GPRS Or WiFi BT Stack TCP/IP Stack Bluetooth Vital Sign Meters Internet IP Routing Service GPRS network or 802.11 Base Station

37. Overview of the Talk • Why Do We Care About Senior Care? • Overview of Personal Assistant System (PAS) • Underlying Research that Makes Things Work • Systems Architecture • Wireless communications and networking • Tracking and localization • Security and privacy • Software safety, reliability, and availability • Human computer interface • Do Elderly People Accept This Technology? • Pilot studies • Where Do We Go From Here?

38. Real-Time Tracking and Localization • We have built a system that uses a combination of ultrasonic and RFID technologies as the underlying sensing mechanism for real-time tracking of both human and objects. • Human Tracking: • Ultrasonic transmitters, called beacons and denoted by Bi, occasionally sends out an ultrasonic pulse and a short RF signal at the same time. • Due to the speed difference between an RF signal and an ultrasonic signal, the listener denoted by Lj on the resident can infer its distance from the beacon using Time Difference Of Arrivals of the two signals. • This distance measurement can be reported to the ALH for real-time tracking of the use (marked as (2) and (3)).

39. Real-Time Tracking and Localization • Object tracking • The RFID (radio frequency identification) reader attached on the wristband can read RFID tags of objects touched by the user. • The objects can be tracked, whenever the user touches and/or carries them. • The whereabouts of the objects are logged.

40. Overview of the Talk • Why Do We Care About Senior Care? • Overview of Personal Assistant System (PAS) • Underlying Research that Makes Things Work • Systems Architecture • Wireless communications and networking • Tracking and localization • Security and privacy • Software safety, reliability, and availability • Human computer interface • Do Elderly People Accept This Technology? • Pilot studies • Where Do We Go From Here?

41. SOAP Envelope Header Medical Information PubAS/PrivAS PubC/PrivC SOAP Envelope Header SOAP Envelope Medical Information Header Medical Information PubC PWAP PWAP PubAS KF Sign. on Msg. KF PWAP How Medical Data Can Be Securely Transmitted ? Sign. on Msg. Sign. on Key [Michael et al. 06]

42. SOAP Envelope Header Medical Information PubAS/PrivAS PubAS/PrivAS PubC/PrivC SOAP Envelope Header Medical Information PWAP PubC PWAP KF PubAS Sign. on Msg. Sign. on Key KF PWAP PWAP PWAP How Medical Data Can Be Securely Transmitted ? Sign. on Msg.

43. SOAP Envelope Header Medical Information PubAS/PrivAS PubAS/PrivAS PubC/PrivC SOAP Envelope Header Medical Information PrivAS PubC PrivAS PubC KF Sign. on Msg. Sign. on Msg. Sign. on Key KF PWAP PWAP PWAP How Medical Data Can Be Securely Transmitted ?

44. SOAP Envelope SOAP Envelope Header Header Medical Information Medical Information PubAS PubAS/PrivAS PubAS PubAS PubAS/PrivAS PubC/PrivC PubC/PrivC PubC/PrivC SOAP Envelope Header Medical Information PrivAS PubC PubC PrivAS PubC PubC KF KF Sign. on Msg. Sign. on Msg. Sign. on Key KF PWAP PWAP PWAP How Medical Data Can Be Securely Transmitted ? SOAP Envelope Header Medical Information Medical Information

45. Implementation

46. Implementation • H/W • Device: Digital Pulse Oximeter • AL Hub: A labtop, Windows • ALSP: A desktop, Linux • Clinician’s computer: A labtop, Windows • S/W • Language: Java • Database: MySQL • Web service: Apache web server, tomcat, axis 1.4 • Libraries: XML-SEC, SAAJ, avetanaBluetooth, etc.

47. Security tokens • The following security tokens are set up • Doctor’s public key certificate • The ALSP’s public key certificate • A secret key shared among AP and AP’s family members or friends • AP’s and family/friends’ password to ALSP • Access control matrix (ALSP), DataID table (AL Hub) GDoc, GAS, KAP, U*; V*, PPA , PFA , ACLAP , ΔAP • GDoc: Doctors office’s public key certificate. • GAS: The ALSP's public key certificate. • U*; V*: URL pointers to the ALSP's public key certificate. U*; V* are the ones which AP and clinician have, respectively. • KAP: A secret key shared the AP and her family members. • PPA; PFA: AP's and Family's secret passwords (respectively) shared with the ALSP. • ACLAP: An access control list for the AP's records at the ALSP. • ΔAP: A mapping table from a serial number of a medical device to DataID

48. Bootstrapping • Clinician sets up relationship with ALSP / Exchange URL pointers • Assisted person (AP) subscribes to an ISP / Set up a gateway for WPA network • AP visits clinician. Clinician recommends a list of devices, hubs, and ALSPs. AP buys devices and hub and adds them to home network • AP subscribes to ALSP. ALSP sends a URL pointer and a password token. AP loads them into the hub. • AP creates a shared “family key” for family members’ and friends’ access to data on the ALSP. Family members register passwords to ALSP. • Hub generates ΔAP , a set of DataID. A dataID is mapped to a device serial number and recipients’ URL pointers. Each entry is a tuple (serial number, dataID, URL pointer) • With the recipient information (clinicians and family members), ALSP creates access control list ACLAP.          Gdoc, GAS, KAP, U*, V*, PPA, PFA, ACLAP, ΔAP Clinician Family AP ALSP

49. Overview of the Talk • Why Do We Care About Senior Care? • Overview of Personal Assistant System (PAS) • Underlying Research that Makes Things Work • Systems Architecture • Wireless communications and networking • Tracking and localization • Security and privacy • Software safety, reliability, and availability • Human computer interface • Do Elderly People Accept This Technology? • Pilot studies • Where Do We Go From Here?

50. Pilot Study at Nazareth Living Center* • With the help of geriatricians at Washington University in Saint Louis, we have carried out a 3-week pilot study at the Nazareth Living Center for Assisted Living in June-July 2006. • This facility houses 110 well-educated, predominantly female residents, whose average age was 88. Of 30 residents who attended a presentation, 14 agreed to participate. • After administering a standard cognitive assessment, two residents were consented to test the PAS prototypes (application 1 and applicatoin2) and ten residents to carry/wear a placeholder device for a period of two weeks. * Nazareth Living Center is a Catholic, nonprofit skilled nursing and assisted living community, sponsored by the Sisters of St. Joseph of Carondelet.