Doctoral Thesis Proposal

1. Doctoral Thesis Proposal Viral Entertainment as a Vehicle for Disseminating Core Development Services Agha Ali Raza Thesis Committee: RoniRosenfeld, LTI, CMU (Chair) Jaime Carbonell, LTI, CMU Bhiksha Raj, LTI, CMU Umar Saif, ITU, Pakistan Bill Thies, MSRI, India Language Technologies Institute

2. Contributors • Babajob.com (Polly-Babajob) • Sean Blagsvedt • Maya Chandrasekaran • Archna Bhatia • MSRI (Polly-MSRI) • Bill Thies • IndraniMedhi • SpandanaGella • IIT Delhi (Polly-JMV) • ZahirKoradia • Aaditeshwar Seth • CMU • Christina Milo • JehanzebSherwani • Guy Alster • Yibin Lin • Haohan Wang • Rita Singh • Christos Faloutsos • DanaiKoutra • Jay Yoon • LUMS/ITU (Polly-Lahore) • Umar Saif • MansoorPervaiz • SamiaRazaq • FarhanUlHaq • Zain Tariq Intro Viral Ent. Conduit Test-bed Reproducible

3. ICT4D • Information and Communication Technologies for Development • More and better information and communication furthers the development of a society. • Can lead to better management of available resources, improved monitoring & reporting of corruption and more connectivity among people • To achieve impact at a massive scale: • Robust solutions to reach the masses using available means with minimum resource expenditures Intro Viral Ent. Conduit Test-bed Reproducible

4. How do you reach low-literate masses? • TV, radio are non-interactive • PCs are not feasible • Smart phones are not always feasible • Text is problematic for the non-literate • Speech over simple phones is a viable way to reach • low-literate masses Intro Viral Ent. Conduit Test-bed Reproducible

5. Long Term Vision • Speech-based information access • for low-literate people • in developing countries • Examples of information services: • speech-based Craig's List • speech-based citizen journalism • speech-based message boards/blogs • health, agriculture, jobs, education,… • … Intro Viral Ent. Conduit Test-bed Reproducible

6. Problem 1: User Training & Motivation • Speech Interfacesusually require user training • e.g. Healthline(Sherwani et al 2009), AvaajOtalo (Patel et al 2010) • Explicit training is not a scalable solution • Solution: Incentivize people to train themselves • “Where there's a will there's a way…” [Smyth et al 2010] • “Entertainment turns UI Barriers into mere speed bumps” Intro Viral Ent. Conduit Test-bed Reproducible

7. Problem 2: Mass Dissemination How to advertise / communicate to masses who don’t read? Solution: Viral Spread Intro Viral Ent. Conduit Test-bed Reproducible

8. Goals • Develop Viral Entertainmentas a vehicle for disseminating Development relatedtelephone based services • Research Objectives • Find the right kind of Entertainment • Use it to Introduce and Popularize speech interfaces • Use entertainment as a Viral Conduit for delivering Core Development Services • Setup an Experimental Test-bed for testing speech interface choices Intro Viral Ent. Conduit Test-bed Reproducible

9. Research Questions • Can we find a: • simple-to-understand, • non-controversial, • quick-to-engage-and-spread form of entertainment that is suitable for low-literate telephone users formerly inexperienced with automated dialog systems? • Could it become Viral? Intro Viral Ent. Conduit Test-bed Reproducible

10. Our First Attempt: Songline • A telephone-based, voice-based application which allowed users to listen to songs recorded by others, as well as to record their own songs and to forward them to friends • User Feedback (Pakistan) • About the idea: • Songs and music are considered controversial • Privacy concerns • About the Interface: • Entering phone numbers is tedious • Busy tone is confusing • Detailed call tree and numerous options are confusing Intro Viral Ent. Conduit Test-bed Reproducible

11. Polly • Polly is a telephone-based, voice-based application which allows users to make a short recording of their voice, modify it and send the modified version to friends. • Features • Non-controversial entertainment • Peer-to-peer • Simple & Easy-to-understand (few options) Intro Viral Ent. Conduit Test-bed Reproducible

12. 2011-Pilot Launch • Seeded with the 32 users on March 31, 2011 • Remained up for 22 days • Amassed 2,032 users who took part in 10,629 interactions Intro Viral Ent. Conduit Test-bed Reproducible

13. Research Questions • Can we use Entertainmentas a delivery vehicle for core development services? Intro Viral Ent. Conduit Test-bed Reproducible

14. 2012 – Large Scale Deployment • Deployed locally in Lahore • Call charges went down to $0.023/minute • Increased capacity to 30 phone lines • Increased number of voice manipulations to six • Added Job Audio-Browser Intro Viral Ent. Conduit Test-bed Reproducible

15. First Information Service: Job Audio-Browser • Scan Pakistani newspapers for jobs for low-skilled workers • Record these ads • Invite Polly’s users to audio-browse them Ref: paperpk.com Intro Viral Ent. Conduit Test-bed Reproducible

16. Intro Viral Ent. Conduit Test-bed Reproducible

17. 2012 – Large Scale Deployment • Launched on May 09, 2012: • Cold seeded with 5 most-frequent users of the pilot • After 141 days (mid-September 2012): • 495,000 calls • 85,000 users • Spreading to 1,000 new people daily • 27,000people used the job search service • listened 279,000 times to job ads • and forwarded them 22,000 times to their friends. Intro Viral Ent. Conduit Test-bed Reproducible

18. 2012 – Large Scale Deployment Intro Viral Ent. Conduit Test-bed Reproducible

19. 2012 – Large Scale Deployment Intro Viral Ent. Conduit Test-bed Reproducible

20. After a Year… • Polly had accumulated: • 636,000 calls • 165,000 users • 34,000people used the job search service • listened 386,199 times to 728 job ads • and 19,000 users forwarded them 34,000times to their friends. Intro Viral Ent. Conduit Test-bed Reproducible

21. Research Question • Can we use the virally spreading entertainment service as an Experimental test-bed for performing randomized controlled trials and demographic studies? Intro Viral Ent. Conduit Test-bed Reproducible

22. User Retention • Enthusiasm is Lost Quickly • A few people continue long term Intro Viral Ent. Conduit Test-bed Reproducible

23. Randomized Controlled Trials • So far we had been paying all airtime charges using “missed call” mechanism • Goal: Reduce our airtime charges while maintaining system spread among the poor • Question:How will various quotas affect user behavior? • Introduced a caller-paid line • Calls picked up, caller pays airtime • Impose various quotas on the number of toll-free calls per day (for each user). • When quota exceeded, direct caller to caller-paid line Intro Viral Ent. Conduit Test-bed Reproducible

24. Caller-paid Line Significantly less activity as compared to the toll-free line Intro Viral Ent. Conduit Test-bed Reproducible

25. RCT: Effect of Daily Quota of 7 Calls No appreciable difference after a week Intro Viral Ent. Conduit Test-bed Reproducible

26. User Demographics • Determined by listening to a sample of recordings: • Used mostly by Punjabi speaking men… Intro Viral Ent. Conduit Test-bed Reproducible

27. User DemographicsEstimated from 207 survey calls Socio-Economic Status Low SES (No Education) Primarily used by low-educated, low-income people Intro Viral Ent. Conduit Test-bed Reproducible

28. Geographical Spread • - Seeded in Lahore and Okara • - Reached all parts of Pakistan. • - And also a handful of calls from: • India • Belgium • Oman • Saudi Arabia • UAE Intro Viral Ent. Conduit Test-bed Reproducible

29. Analysis of User Behavior • Does more experience using Polly lead to: • an improvement in users’ interaction skills? • Use of more advanced features, • Fewer mistakes. • 2. any change in usage preferences? • (Analyses are based on: 50,414 users, 292,951 calls, 934,742 menu interactions) Intro Viral Ent. Conduit Test-bed Reproducible

30. Barge-in Behavior in Main Menu Clear increase in the use of barge-in with experience Number of times user previously encountered main menu Intro Viral Ent. Conduit Test-bed Reproducible

31. Research Question • Reproducibility: Can our setup and results be reproduced in a different country? Intro Viral Ent. Conduit Test-bed Reproducible

32. The BabajobCollaboration(Sean Blagsvedt, Maya Chandrasekaran) • Babajob.com is one of India’s largest informal and entry level job portal that connects registered job seekers and employers using their website (primarily), voice services as well as SMS • (Another benefit: A limitation of our previous study was our inability to track employment outcomes) • Goal: Advertise Babajob to low-literate job seekers through Polly and connect them to Babajobs’ voice services Intro Viral Ent. Conduit Test-bed Reproducible

33. Setup • A local phone number in Bangalore to receive the missed calls • Polly calls back from the US • Support for Kannada (Archna) and Hindi (Rita) voice prompts • Call transfer to Babajob IVR system from Polly’s main menu. • We added a feature: directory of previously-called friends Intro Viral Ent. Conduit Test-bed Reproducible

34. Seeding Attempts (via Babajob) • Automated calls to 84 active Babajob users in the Driver category followed by an SMS • Staff members at Babajob volunteered to pass Polly’s number on to 5 of their friends • Results • Very little activity that declined within a few weeks • Very few users; mostly listening to job ads (not much playing or forwarding) • We suspect these users to be JobData brokers Intro Viral Ent. Conduit Test-bed Reproducible

35. Seeding Attempts (via MSRI)(Bill Thies, IndraniMedhi, SpandanaGella) • On Jan 9, 2014 a set of undergraduate students were asked to spread word about Polly. They posted on their university Facebook pages and message boards • In the last week of Jan, 2014 Spandana gave Polly’s demo to a security guard in her building and asked him to spread the word • Results • Over the last 106 days: • 5,051successful calls • 2,793 call-back requests (from 441 users) • 2,258 message delivery calls (to 699 users) Intro Viral Ent. Conduit Test-bed Reproducible

36. MSRI Seeding: Viral, but Not Exponential Still Ongoing… • Neither exponential growing nor decaying! • Steady stream of new users • Being used primarily by low-literate users in West Bengal and Delhi Intro Viral Ent. Conduit Test-bed Reproducible

37. Why isn’t Polly Spreading Exponentially in India? Intro Viral Ent. Conduit Test-bed Reproducible

38. Back-Channel Spread • Observation: Polly is often introduced to new users not via a Polly message • Indication of this: some users place their first call without ever receiving a Polly message (“Word of Mouth” users) • 47.5% of call-initiators in Polly-LHR pilot • 15% of call-initiators in Polly-LHR full deployment • 18% of call-initiators in Polly-BLR • In person introduction? +demonstration? • Explained in a phone conversation? Intro Viral Ent. Conduit Test-bed Reproducible

39. Why is Spread Non-Exponential? (Hypotheses) • Introduced as a “free voice messaging system” • Different perception perpetuated by “back-channel spread”? • Introduced to college undergraduates • Different behavior in this SES? • Indians have bad past experience with IVR systems? • SPAM, scams • Complex billing mechanisms due to telecom circles (=“area codes”)? • Linguistic complexity? • “Dead-Skunk” hypothesis: something subtle is broken in the user experience • Cultural differences: • Users expressed less interest in voice mods Intro Viral Ent. Conduit Test-bed Reproducible

40. Remaining Work • JMV-collaboration (ZahirKoradia, Aaditeshwar Seth) • Jharkhand Mobile Vaani is a citizen radio-over-phone platform that makes recorded radio content accessible through phone calls • Goal: Cross-spread: Polly and JMV • Benefits: • JMV operates in the Hindi-Belt • They already have a trusting user-base • More flexibility of seeding • They have a vested interest in this venture • Questions: • Can Polly grow exponentially in India? • Will Polly-JMV cross-spread?

41. Remaining Work • Launch Polly-JMV (May 2014) • Complete analysis of Polly-BLR data and remaining experiments (May, June 2014) • Analyze data from Polly-JMV (June, July 2014) • Write it up! (July 2014) • Thesis Defence (end of July, 2014) Thank you! Questions?

42. Hypotheses regarding Non-Virality • Limitations • I am not on-ground • I do not understand the culture/speak the language • Either some inherent cultural/sociological difference that we do not appreciate • Wrong system image (a voice SMS system) due to incorrect seeding? (Back-channel hypothesis). • Peoples' past experience with IVR systems has left a bad taste? In Pakistan it was an untouched population. • Phobia of pressing keys lest thy may lose money? • Fear of calling/picking up calls from another telecom circle? • More entertainment options? More free messaging options? Is the entertainment void situation among low-SES Indians similar to their Pakistani counterparts? • Busy people, less time.

43. Hypotheses regarding Non-Virality • Some system bug that we have not been able to isolate • Are we getting all the missed call requests from all the regions? • Is the SMS okay. (No!) • Is the caller ID okay? (it was not. Just fixed.) • Are there pauses? Delays? buggy IVR? (None that we are aware of) • Is the language okay? (we believe so)

44. BLR Lessons Learned • Microphone vs telephone based prompt recordings • Translation of prompts. Formal vs informal. • Outgoing caller-id problem. • Quick detection of bugs. The silent voice prompt. • Problems in seeding. • On ground presence and understanding the local culture, norms, language.

45. Caveat: Confounding User Types • This analysis lumps together long-term and short-term users. • Hence, it confounds true learning by any one user with differences between the different user types. • Long-term users may be more adept at using IVR systems to start with.

46. Changes in User Preferences:(now controlling for user type) • We define 3 user sets and explore changes in their usage patterns as a function of their experience(number of prior Polly calls): • Short-term users: 2,701 users who interacted with Polly exactly5 times. • Intermediate-term users: 1,862 users who interacted with Polly exactly 10 or 11 times. • Long-term users: 1,523 users who interacted with Polly 30+ times.

47. Menu Choices vs. ExperienceShort-term Users (made exactly 5 calls each)

48. Menu Choices vs. ExperienceIntermediate-term Users (made exactly 10 or 11 calls) Compared to short-term users; activity starts at a higher level and climbs higher!

49. Menu Choices vs. ExperienceLong-term Users (made 30+ calls) Compared to intermediate-term users; activity starts at a higher level and climbs higher!

50. Common Trends among all user sets • - The Tendency to: • 1. Press 2 (forward) starts off at a high value and stabilizes • “Back Channel” • System’s popular image as a messaging system • 2. Press 3 (next effect) increases with experience • System Exploration • Hunting for an effect of choice • 3. Press 0 (re-record) and 1 (repeat) decreases with experience • Adapting to 15 seconds recording limit • Playing with the system, alone or with friends • - Invalid button presses are rare