1 / 27

Big Data, Global Development & Complex Social Systems: Why Engineers Matter

This talk explores the rise of big data and its impact on global development and complex social systems. It discusses the challenges and possibilities of utilizing big data to prevent diseases, combat crime, and hold governments accountable. The talk also highlights the role of engineers in making sense of big data and creating meaningful interventions.

lcarlson
Download Presentation

Big Data, Global Development & Complex Social Systems: Why Engineers Matter

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Big Data, Global Development & Complex Social Systems:Inference & Behavioral Sensor Data: Why Engineers Matter. CSSS ‘10 June 10, 2010 Nathan Eagle Omidyar Fellow / Visiting Assistant Professor The Santa Fe Institute / MIT nathan@mit.edu

  2. The Rise of Big Data • New types of Data • Sloan Digital Sky Survey (140 TB over 10 years) • GenBank Sequences (100b base pairs) • Wal-Mart (1m transactions / hr, 2.5 PB) • Facebook (40b photos, 15PB) • New possibilities • Prevent Diseases • Combat Crime • Hold Governments to Account • Enables Scientific Revolutions

  3. Big Data = Big Problems ? • Our Hard-drives are Full • Data Security and Protecting Privacy is Harder • The practitioners are not trained to make sense of it. T. Cukier (2010). ‘Data, data, everywhere’, The Economist.

  4. Big Data = New Discipline? D. Lazeret al., (2009) “Computational Social Science",Science. 323: 5915, pp. 721 – 723.

  5. Big Data = New Discipline? Scale (N)

  6. The Physics of Society: “People are Particles” • Towards “Universal Laws” of Human Behavior? M. Gonzales et al., (2008) “Understanding individual human mobility patterns", Nature. 779-782.

  7. Engineering Social Systems: From Observations to Action • http://ess.mit.edu Scale (N)

  8. Big Data in the Developing World • Majority of ‘behavioral data exhaust’ comes from the developing world T. Standage (2009).‘Mobile Marvels’, The Economist.

  9. Modeling Contagion Spread:Proximity as a Proxy for Disease • How does RSV spread through a local community? • 50 Households in Kilfi, Kenya – Phones and RFIDs • Location, Bluetooth, Communication, Close Proximity (RFIDs) • Can we link proximity patterns with infection? Moses Kiti, Nathan Eagle, James Nokes.

  10. Engineering Automated Interventions:12 Sex Workers in Mtwapa • Can we identify the onset of a high-risk situation? • Can we use mobiles as a tool for education and the dissemination of context-specific information? Nathan Eagle and Eduard Sanders.

  11. Big Data from Small Samples:150 Smokers in New York City • 150 Undergraduate Smokers / Recent Quitters • Location, Bluetooth, Communication, Context-Driven Surveys • Question: Is there a behavioral signature associated with relapse? Yuelin Li and Nathan Eagle.

  12. Reactions to Regional Shocks: Natural, Disease, Economic • Spatial Dynamic Bayesian Anomaly Detection • Quantify tower-level behavioral priors A. Kapoor, N. Eagle and E. Horvitz (2010), "People, Quakes, and Communications: Inferences from Call Dynamics about a Seismic Event and its Influences on a Population", Proceedings of AAAI Artificial Intelligence for Development (AI-D'10).

  13. Reactions to Regional Shocks: Disease Surveillance Nathan Eagle, Leon Danon

  14. Reactions to Regional Shocks: Disease Surveillance Martin Lajous, Nathan Eagle, Leon Danon

  15. Mobile Phones and Malaria: Malaria Eradication on the Kenyan Coast Caroline Buckee, Nathan Eagle, Bob Snow

  16. Using Big Data as a LensComparing Surveys with Call Records J. Blumenstock, N. Eagle. (2010).‘Mobile Divides: Gender, Socioeconomic Status, and Mobile Phone Use in Rwanda’, (in submission)

  17. Socioeconomic Status Inference Coupling Surveys with CDR J. Blumenstock, N. Eagle. (2010).‘Mobile Divides: Gender, Socioeconomic Status, and Mobile Phone Use in Rwanda’, (in submission)

  18. Are Slums Good? Economic Springboards and Resource Allocation A. Wesolowski and N. Eagle (2010), "Parameterizing the Dynamics of Slums", Proceedings of AAAI Artificial Intelligence for Development (AI-D'10).

  19. Socioeconomic Status and StabilityQuantifying the Evolution of Communities • How do communities evolve over time? • What makes a community stable? Y. Montjoye, N. Eagle, A. Clauset (2010).‘The Stability of Society’, (in prep).

  20. Getting Structure in Big DataThe Outsourcing Industry • Only 5% of data is ‘structured’ • Data Structure Tasks • Forms Processing • Image Tagging • Audio Transcription • Huge Operational and Capital Expenses • Worker salaries < 30% of costs

  21. Big Data and OutsourcingMobile Crowdsourcing? • Mobile Money • 7M users in Kenya • Virtual Outsourcing • No OpEx • No CapEx • No Management • (just math)

  22. Accuracy InferenceBinomial Model N. Eagle (2009). ‘txteagle: Mobile Crowdsourcing’, Human-Computer Interaction International 2009.

  23. Expertise InferenceExpectation Maximization N. Eagle (2009). ‘txteagle: Mobile Crowdsourcing’, Human-Computer Interaction International 2009.

  24. Virtual OutsourcingDistributed Human Input to Big Data • 15 million East Africans can do work and earn money on their phones. • Got Big (unstructured) Data?

  25. N = BILLIONS… Bolivia, Dominican Republic, United States, Japan, Belgium, Thailand, Rwanda, Mexico, United Kingdom, Kenya, Uganda, Saudi Arabia, Kuwait, India, Burkina Faso, Chad, Bahrain, Jordan, Lebanon, Brazil, Spain, Saudi Arabia, Indonesia, DRC, Gabon, Ghana, Ireland, Madagascar, Malawi, Niger, Nigeria, Sierra Leone, Argentina, Peru, Sudan, Tanzania, Uganda, Indonesia, Zambia N. Eagle, K. Green. (2010). Reality Mining: Big Data to Engineer a Better World. MIT Press.(in prep).

  26. thanks. Aaron Clauset (SFI / CU Boulder) 
Stephen Guerin (RedFish) 
David Lazer (Harvard / Northeastern) 
Mika Raento (Google) 
HannuVerkasalo (HUT) 
Peter Wagacha (University of Nairobi) John Quinn (Makerere) 
CosmaShalizi (CMU) 
Robert Snow (Oxford / KEMRI) 
Raja Hafiz (CMU) 
Caroline Buckee (SFI / Oxford / Harvard)
Sune Lehmann (Northeastern) 
Yan Ji (MIT) 
Yves-Alexandre de Montjoye (SFI / MIT) 
 Marta Gonzales (MIT) 
Dirk Brockmann (Northwestern)
MarcelFafchamps (Oxford) 
Edo Airoldi (Harvard) 
Neil Ferguson (Imperial) 
Joshua Blumenstock (Berkeley) Rob Claxton (British Telecom) 
Michael Macy (Cornell) Alex (Sandy) Pentland (MIT) Ben Olding (txteagle / Harvard) Eric Horvitz (MSR) Luis Bettencourt (SFI / LANL) Jameson Toole (SFI / MIT) Amy Wesolowski (SFI / CMU) nathan@santafe.edu

More Related