200 likes | 332 Views
Accuracy of Data Collection on Mobile Phones: A Study of Forms, SMS, and Voice. Somani Patnaik 1 , Emma Brunskill 1 , William Thies 2. Massachusetts Institute of Technology. 1. Microsoft Research India. 2. Mobile Data Collection . • Broad applications in . –.
E N D
Accuracy of Data Collection on Mobile Phones: A Study of Forms, SMS, and Voice Somani Patnaik1, Emma Brunskill1, William Thies2 Massachusetts Institute of Technology 1 Microsoft Research India 2
Mobile Data Collection • Broad applications in – Mobile banking – Microfinance – Healthcare – Environmental monitoring • •Benefits • Immediate digitization • Fast and cheap • Environment friendly • Flexible questionnaires
Study of data collection interfaces • •Comparison of three interfaces for health data collection
1. Electronic Forms Interface • • Consist of numeric fields and multiple-choice menus. • Can be implemented in Java or a native phone platform. • • General Strengths • Easy patient identification • Ongoing cost is low (SMS or data plan) • Can store visits when connectivity is poor • • General Weaknesses • Requires programmable phones • Requires basic literacy skills • Hard to alter survey questions • Hard to enter in free-form notes • Application can be deleted by user
2. SMS Interface • • Sending a structured SMS messages to a server • Logical fields separated by delimiters in the message • • General Strengths • Can be used with any phone • Ongoing cost is low (SMS or data plan) • Many workers familiar with SMS • • General Weaknesses • Requires basic literacy skills • Changing survey requires new cue card • Quite easy to fake visits (copy old SMS) • Hard to enter in free-form notes
3. Live Operator Interface • • A normal telephone call • Live human operator that enters the data into a centralized • spreadsheet • • General Strengths • No literacy required of workers • Can be used with any phone • Hard to fake a visit: operator can ask new questions • • General Weaknesses • Ongoing cost of operator salary • Voice plans often higher cost than SMS • Awkward 3-way social interaction
Operations • Setup time - Electronic forms require application, • which requires either an Internet-enabled phone or an • external computer • Training time – Voice interface requires least amount of • education and background • System coverage and reliability - voice most reliable but • require sufficient number of operators. • Flexibility - Ability to modify the data collection interface, • fix an error, improve usability etc
Effectiveness • •Characterized by two factors- • Data should not be intentionally faked by the user • The data should be accurate(not intentionally faked) • Unfortunately, quite easy to fake data in SMS systems, just • requires copying and pasting prior messages • Faking data on electronic systems slightly harder and most • difficult in voice. • Voice also allows correcting previous visits
Cost Comparison • • Electronic phone requires a programmable phone(such as a • Java enable phone or windows phone) • Voice has the ongoing cost of the operator • Cost of live-operator in India proves to be cost-effective • Decreased cost of voice-only handsets, training time and • literacy requirements of health workers compensate the cost
Context: Rural Tuberculosis Treatment • With local partners, working to improve tuberculosis treatment in rural Bihar. • Monitoring the patients symptoms remotely by collecting data remotely.
Study Participants • 13 health workers and hospital staff (Gujarat, India) Age Education Cell Phone Experience (Median) • • Training • Health workers: big groups, 6-8 hours • Hospital staff: small groups, 1-2 hours • Every user made two error-free reports on each interface
Testing and Results • •Testing • Tested in pairs, one patient on data entry, and the other being • the fake patient • Two complete patient–worker interactions for electronic forms • and SMS interfaces • Results
Results • • Error rate higher in health workers as compared to hospital staff • Two possible reasons: • Hospital staff more educated and old • Difference in training
Sources of Error • Types of errors • Misplacement of decimal point in the temperature entry • Non-revision of cue cards for the SMS interface • Putting wrong patient identity when using SMS • Usability barriers • small keys • scrolling/selection • SMS encoding
Conclusions • •Accuracy of mobile data collection demands attention • 5% error rates for those lacking experience • There exist cases where a live operator makes sense • Can be cost effective, esp. for short calls • Study has limitations • Small sample size • Varied education, phone experience, training of participants • Future work • Distinguish factors responsible for error rates • Compare to paper forms, Interactive Voice Response