Speech Matrix® Systems Automated Speech Recognition for Self-reports in Health Research

1. Speech Matrix® SystemsAutomated Speech Recognition for Self-reports in Health Research Alex Levin, Spacegate, Inc. alex@spacegate.com Esther Levin Spacegate, Inc City College of New York esther@spacegate.com

2. Acknowledgement Speech Matrix Research & Development is supported by NIH, NCI SBIR Phase I program Confidential – Spacegate, Inc.

3. Background Health Data Collection • Traditional methods • Paper based diaries • Observations • Modern methods – Electronic Data Capture (EDC) • PDA • Web based • IVR (touch–tone) • Next Step - EDC • ASR (automated speech recognition) Confidential – Spacegate, Inc.

4. Electronic Data Capture – Goals & Challenges • Real data in real-time • Data validation • Regulatory compliance • Increase clinical trial efficiency • Reduce time-to-market • Cost • Patient/Subject burden Confidential – Spacegate, Inc.

5. Automated SpeechTechnology – Science & Art • Voice to Data - voice data entry via automated natural dialogue • Automated Dialogue – Voice Forms • Voice Interface Design • Real-time Monitoring & Reporting • Privacy & Security • Regulatory Compliance Confidential – Spacegate, Inc.

6. ASR System for Data Collection Confidential – Spacegate, Inc.

7. Why Speech? • Speech is a natural modality of interactions • Phone is user friendly and ubiquitous and no special training for its use is required. • Dynamic Dialogue Flow • personalization of both content and style based on the profile and history. • Real time feedback and monitoring • real-time reports of captured data • Automated Compliance monitoring • Flexible and extensive scheduling • Inbound/outbound call sessions • the calls can be initiated by a system following a prescribed protocol • Overall ASR based system offers an extensive and practical tool to facilitate efficient and convenient real-time, two-way communications. Confidential – Spacegate, Inc.

8. Speech Matrix®-VDC™ Systems • Extensive data collection system for health, clinical, life science and behavioral research. • Another branch of EDC • Real-time data capture in a participant native environment Confidential – Spacegate, Inc.

9. Applications • Pain Monitoring Diary (PMD™) prototype application - based on questions in standard pain questioners (BPI) • Drug use Diary (VDMD™) : Prototype application - based on questions in standard Drug Use Diary (DR. Linda Sobell) • Implements Dynamic Questionnaires • Interactivity & Communications • Reporting and Management Confidential – Spacegate, Inc.

10. Dialog Design • Task characteristics: • Need to guarantee data validity, accuracy and integrity, taking into account speech recognition errors • improve the overall accuracy using dialog actions such as re-prompts, confirmations, error handling, and, if necessary, recording and flagging the unrecognized utterances for later transcription • The system should accommodate both novice and experienced callers • enough information and help to guarantee question understanding and successful session completion for novice. • short and effective call flow for experienced caller • Subjects identify themselves in the beginning of the each session. • Opportunity to use the knowledge accumulated across sessions for personalization. • Subjects may receive some training on the use of the spoken dialog system during the enrollment session. • Dialog Design Issues: • controlling the captured data accuracy. • adaptive level of user support Confidential – Spacegate, Inc.

11. Controlling the Accuracy of Data Capture:ASR 101 • Speech Recognition Grammar Design • Example: yes/no grammar {yes, no} • Caller utterance is matched compared to the possibilities described by the grammar • The output of ASR is the best matching ‘sentence’, and a score • If the score is too low => rejection • Out-of-vocabulary utterances cannot be recognized correctly • Design tradeoff: minimize out-of-vocabulary and minimize grammar size Confidential – Spacegate, Inc.

12. Controlling the Accuracy of Data Capture. • Improved rejection mechanisms to deal with out-of-vocabulary utterances. • System prompt: Was that your left shoulder? • User: no, left elbow • System prompt: I didn’t get that.Was that your left shoulder? Please say ‘yes’ or ‘no’ . • Reliable confirmation recognition. • Using confirmations as the way to control the larger grammar’s accuracy. • Using recording to capture the out-of-grammar answers and problematic user inputs. • System prompt: “Was that your left shoulder?” • User: “No” • System prompt: “Sorry about that. Let’s try it this way. Please choose carefully a body part from the following list that best describes the location of your pain, and just say it. If none of the locations match, please say ‘none of those’. Here is the list: abdomen <pause>, ankles …” • User (barges in) ”none of those” • System prompt: “Ok. Let me just record your answer. Please describe the location of your pain in your own words.” • User: <……> • System prompt (after recording is finished): “Thanks, I got that. Let’s move on.” • recorded utterance is captured and flagged as “transcription is needed” for later processing. • same mechanism of fall-back to recording instead of recognition is used after several repeated recognition failures. Confidential – Spacegate, Inc.

13. Adaptive Level of User Support-I. • Prompt Design. • “Where does it hurt? <pause>. For example, your head stomach or back? <pause>. Remember, if you don’t know how to answer this question, just say ‘I need help’ “. • Context sensitive help. • help information describes and clarifies the current question, • Provides examples of possible answers • Example: help for the “where does it hurt” question: “Okay. Here is the help information. At this point I need to find out the part of your body that hurts the most. Please choose carefully a body part from the following list that best describes the location of your pain, and just say it. If none of them matches, please say ‘none of those’. Here is the list: abdomen <pause>, ankles <pause>, back <pause>,...( list continues) …, toes <pause>. Which one is it?" Confidential – Spacegate, Inc.

14. Adaptive Level of User Support-II • Detecting speech recognition failures. • The re-prompts are designed as an escalating list, providing increasingly more information and progressively constraining the caller. • “Where does it hurt? <pause> For example, your head, stomach or back? <pause>. Remember, if you don’t know how to answer this question, just say ‘I need help“., • “I didn’t get that. Please tell me the part of your body that hurts the most, Remember, you could always say ‘I need help’ ”, • Detecting Misunderstandings. • the user says “no” to a confirmation question as in: • System prompt: Was that your left shoulder? • User: No. • System prompt: Sorry about that. Let’s try it this way. Please choose carefully a body part from the following list that best describes the location of your pain, and just say it. If none of them matches, please say ‘none of those’. Here is the list: abdomen <pause>, … (list continues). Which one is it? Confidential – Spacegate, Inc.

15. AdaptiveLevel of User Support-III • Dialog Personalization. • Taking advantage of a larger inter-session context by designing two types data collection sessions: normal and follow up. • Both content and style are different. • Reminding prompts: “Is the pain still in your left shoulder?”. Confidential – Spacegate, Inc.

16. System: Your pin please. Caller: **** System: Ok, *****, lets begin. Are you in pain now? Caller: yes System: Where does it hurt?<pause> For example, your head, stom... Caller: my left shoulder System: Was that your left shoulder? Caller: yes System: Please rate your pain on a scale of zero to ten, with zero being no pain and ten being the worst possible pain Caller: it is seven System: as that 7?<pause> Please say yes or no. Caller: yeah System: How does the pain feel? Caller: I don't know System: Okay, Here is the help information. At this point I need to find out the type of your pain. Please choose carefully a word from the following list that best describes your pain and just say it. If nothing matches, please say ‘none of those’. Here is the list: aching, <pause> burning, <pause> dull… NORMAL SESSION Caller: it feels sharp System: Was that ‘sharp’? Caller: yeah System: Is the pain constant or does it come and go? Caller: comes and goes System: Do you have any other symptoms? Caller: yeah System: Please tell me your most disturbing symptom. For... Caller: I have fever System: I heard ‘fever’. Is that right? Caller: yes System: Would you like to leave a message for the staff? … Caller: no System: Thanks for your cooperation. I've got all the information that I need. I'll call you again in 4 hours. Goodbye. Confidential – Spacegate, Inc.

17. System: Is the pain still in your left shoulder? Caller: yes System: Does the pain still feels sharp? Caller: no System: How does the pain feel? Caller: It’s dull System: Was that ‘dull’’? Caller: yeah System: Is the pain constant or does it come and go? Caller: It’s constant System: Do you have any other symptoms? Caller: no System: Would you like to leave a message for the staff? … Caller: no System: Thanks for your cooperation. I've got all the information that I need. I'll call you again tomorrow. Goodbye. FOLLOW-UP SESSION System: Your pin please. Caller: **** System: Hi, Esther, This is your follow-up call … Are you still having pain? Caller: yeh System: Did you take your pain medication today? Caller: yes System: What time did you take the medication? Caller: Two hours ago System: Was that around 9:45 am? Caller: yeh System: Did the medicine diminish your pain?... Please Caller: yeh System: Please rate your pain on a scale of zero to ten, with zero being no pain and ten being the worst possible pain. Caller: four System: Was that 4? Caller: yeah Confidential – Spacegate, Inc.

18. Normal session report Confidential – Spacegate, Inc.

19. Follow-up session report Confidential – Spacegate, Inc.

20. Usability Test • 24 subjects • 118 dialog sessions • 113 completed • 5 hang-ups • 42 follow-up • 1766 dialog turns • 98% automatic data capture – the rest flagged for transcription Confidential – Spacegate, Inc.

21. Results Confidential – Spacegate, Inc.

22. Summary • ASR & Spoken Dialog Methodology for data capture can provide: • Additional real-time data collection tool • Flexible protocol design • Improves data validation and compliance • Centralized collection and monitoring • Telephone as ubiquitous device • System design needs to take into account the specificities of the task and the limitations of the technology • Flexible level of user support • Controlled accuracy of the captured data Confidential – Spacegate, Inc.