Mobile Medication Assistant for the Elderly

1. Mobile MedicationAssistant for theElderly António Teixeira, Flávio Ferreira, Nuno Almeida, Ana Filipa Rosa Department of Electronics, Telecommunications and Informatics (DETI) Institute of Electronics Engineering and Telematics of Aveiro (IEETA) Health School (ESSUA) University of Aveiro Portugal S4S Presentation

2. Method • An iterative/cyclic method composed by • Requirements Analysis • Get Personas, Problem and Vision • Brainstorming (with people from different domains) • Identify Persona expectations • Construct Context Scenarios • Identify requirements • Developing the application • Testing the application • Evaluation with users S4S Presentation

3. Motivation • Theelderlypresenthighlevels of non-adherence to medication • High number of errors in takingmedication, concernsaboutsideeffects, lackofmotivation, memoryissues Goal: Mobile application for seniors to: • improve adherence to medication and minimize errors • serve as a testbed for research in the best uses of different modalities with a particular focusonspeech. S4S Presentation

4. Persona & Persona Expectations • Persona Expectations • Able to use thesameapplicationwitharthrisis • Allowspeechinteraction • Helpprevent gaps in medication • Reportthemedicationandsideeffects • Could use theAppevenwithpoorability to workwithelectronicdevices Persona: Primary Name: Emília Rodrigues Age: 62 years old Mrs. Emília lives in Coimbra with her husband Filipe Rodrigues. She’s housewife. Mrs.Emília doesn’t know very well to work with electronic devices. She is right handwriter. She is diabetic and has arthritis in her right superior member. Her health condition requires a regular and diary medication. Mrs. Emília has the habit to call her daughter during the dinner preparation. However she has some difficulty in doing the two activities at the same time due to her limitation in the right superior member. She has weekly appointments in her local health centre for surveillance. Mrs. Emília would like to buy equipment that facilitates the contact with her daughter. Something that allow her to control her medication that she forgets to take. S4S Presentation

5. Requirements • Data Requirements • When to take a medication • Why to take a medication • Medicationsideeffects • Medicationexpiration date • Maximum dose • Distributed data • Userprofile • Clinicalinformation • UserRequirements • Inform in a casual way • Use speech for everything • Credibleandreliable • Avoidcontentoverloadandsmallicons • Personalization • Adaptation • Extra carewithlanguage, dimensionandtouch S4S Presentation

6. SystemArchitecture • Windows Phone 7 • Web servicesto overweightfeatures • Several modules to controlmodalitiesandintelligence User Face Detection S4S Presentation

7. SpeechService • AddGrammar: allowthe input of a newgrammar for theservice in GRXML or ARPA format • GetGrammar: getsthelistofallgrammars in theserviceandtheirnames • ASR: translates na audiostream to text, usingthemicrosoftspeechplatform • TTS: convertstext to speech SpeechWS Add Grammar Grammar Repository Get Grammars Microsoft Speech Platform ASR TTS S4S Presentation

8. ImageService • Aims to generate a synthesizedimageof a pill/packages • Inesteadofhaving a databasewithalltheimages • Basedonpill/package discriptionitreturnanimage • Informationneeded: • Pill/Package shape • Pill/Package pattern • Pill/Package colors • Pill/Package name S4S Presentation

9. User, NLG andFaceDetectorService • TheUserservicethatmanagetheusermedication data andplan • TheUserserviceprovidethepushnotificationservice to display alerts • The NLG (natural language generation) service that automatically generates sentences with the provided information • TheFace Detectorservicethatdetects faces in a imageandreturnsthesizeofthecloser face S4S Presentation

10. MainFeatures • Alerts • to remember users about medication schedules • Help • on common medication related questions • I forgot to take my medication? • What are the side effects of this medication? • Medication insertion and management by third parties. • Configurable interaction • How to use application guide and tips • User knowledge adaption Alerts Help Guide Options S4S Presentation

11. Multimodal Input What to do if a medicationintakewasforgotten? What are thesideeffectsofthismedication? • Touch • Accepted for alloperations • Speech • MSFT SpeechPlat. (MSP) • Canbe more intuitiveandfasterthantouch Choose: 1 ) “I forgotmyMedication” 2) “I forgot a medication" 3) Select the forgotten medicine Choose: 1 ) Selecttherightoption TOUCH TOUCH Usersays: “I forgot to take X” Usersays: “What are thesideeffects?” VOICE VOICE S4S Presentation

12. Multimodal Output • GraphicsandText • Panoramicview • Medicationimagesgeneration • Medicationdetails • Speech • TTS (e.g. “Thesideeffectsofmedication are…”) • Use of Natural Lang Generation Medicationname Quantity Applicationmethod Intakeinterval “YoushouldtakeonetabletofAspirin” MOBACC 2013

13. Adaption to Context • Userdistanceto mobile phone • Relevantdisplayelementsresizedifuser moves phonecloser/farther • Dealwithvisionproblems (itwilldependofuser– myopiaorastigmatism) • Ambientnoiselevel • Soundsampledwhileuserisnotspeaking • Volume leveladjustedaccordingly MOBACC 2013

14. Furtherwork • Improve zoom adaption algorithm and add it to more views • Implement the dark mode of the application • Improve the auto voice detect algorithm • Improve the speech interaction • Improve “how to use” features • Add features • Auto adaptable help system • Advice for more daily problems about medication intake S4S Presentation

15. Thankyou for yourattention. S4S Presentation