Michael Bernstein mit computer science and artificial intelligence laboratory

1. DARPA’s Network Challenges Michael Bernstein mit computer science and artificial intelligence laboratory mit human-computer interaction

2. …the roles the Internet and social networking play in the timely communication, wide-area team-building, and urgent mobilization required to solve broad-scope, time-critical problems. - DARPA Network Challenge …new technologies to rapidly create theoretically-informed, data-driven models of complex human, social, cultural, and behavioral dynamics that are instantiated in near-realtimesimulations… - DARPA TASC …harness the unique cognitive and creative abilities of large numbers of people to enhance dramatically the knowledge derived from ISR systems. - Deep ISR Processing by Crowds

3. Crowdsourcing is not simple. THE PRESIDENT:  Three point five million people voted.  I have to say that there was one question that was voted on that ranked fairly high and that was whether legalizing marijuana would improve the economy -- (laughter) -- and job creation.  And I don't know what this says about the online audience -- (laughter) -- but I just want -- I don't want people to think that -- this was a fairly popular question; we want to make sure that it was answered.  The answer is, no, I don't think that is a good strategy -- (laughter) -- to grow our economy.  (Applause.) Digital Town Hall A 'teacher' told my child in class that dolphins were mammals and not fish! And the same thing about whales! We need TRADITIONAL VALUES in all areas of education. If it swims in the water, it is a FISH. Period! End of Story. Republican Crowdsourced Policy Suggestions

4. data isready andwaiting

5. Realtime Social Media Analytics • Countries are made of people, and people produce social media.

6. Realtime Social Media Analytics • Microblogging produces an up-to-the moment, publicly minable snapshot of people in crisis.

7. Realtime Social Media Analytics • The challenge is to identify: • Who can we trust? • What’s happening, where? • Which information is the most important? • …in seconds or minutes.

8. Red River Floods Tweets Post-hoc analytics of a crisis scenario Red River at East Grand Forks is 48.70 feet, +20.7 feet of flood stage, -5.65 feet of 1997 crest. #flood09 there's an emergency animal shelter set up at the Fargo fairgrounds we are on the western central edge of town, so we are a fair distance from any water for now. From MWCFD: Eastwood addition, a lot of damage. Part has power, part doesn't. Won't open it back up • [Starbird et al. CSCW ‘10, Vieweg et al. CHI ‘10]

9. Red River Floods White dots: tweets on the flood topic Geolocation information in tweets • [Starbird et al. CSCW ‘10, Vieweg et al. CHI ‘10]

10. Tweet the Debates Post-hoc analytics of McCain-Obama Financial Crisis Tweet Sentiment Valence Tweet Volume McCain War Story Terrorism • [Diakopoulos and Shamma CHI ‘10]

11. Approach and Success Metrics • Archive datasets from global events. • Can we mine the data that would help us aid the situation, within seconds or minutes of the news breaking? Can we detect spammers? • Apply the techniques to new onset events.

12. people areready andwaiting

13. The Problem • UAV pilot interfaces are held back by AI-hard problems like computer vision. MIT professor Missy Cummings used to fly F/A-18 Hornet fighters for the Navy. “I spent whole time complaining — who was the moron who designed this thing?” she recalled. If you’ve ever peeked inside a fighter cockpit, you’ll understand her gripe. Dials, displays and controls pack every nook and cranny. It’s the farthest thing from ergonomic. - Wired

14. Crowd-Powered Interfaces • Can these cockpit interfaces: • Learn from each others’ expertise? • Benefit from outsourcing tasks to non-experts?

15. The Interface User as a Crowd • Gathering pilot data and then mining it can lead to user interfaces that: • Prevent error (“This is an uncommon operation!”) • Speed up laborious tasks • Provide trainee pilots with realtime feedback on what expert pilots would do in a given situation

16. [Hartmann et al. CHI ‘10]

17. Outsourcing • Non-pilots can solve AI-hard problems in the background to support the user interface of the pilot. • There are ~25 million veterans and 850,000 reservists in the United States who could help from home.

18. Soylent: A Word Processor with a Crowd Inside • Amazon Mechanical Turk workers will solve basic cognition tasks for cents. We can embed these workers inside the user interface: • [Bernstein et al. UIST ‘10]

20. Approach and Success Metrics • Design and build novel interfaces. • Demonstrate that they improve performance and reduce error with complex military or intelligence user interfaces • Deploy the interfaces and work through personnel training/scaling issues

22. Research Problems • Integrating social media into our on-the-groundsituational knowledge in real-time. • Teaching user interfaces to learn from other users and to outsource AI-hard problems to humans who can solve them quickly. • Why: nations are increasingly producing social media, which breaks news faster than any other known method • If we don’t: we won’t be able to solve burgeoning problems before they become huge “there's an emergency animal shelter set up at the Fargo fairgrounds” “From MWCFD: Eastwood addition, a lot of damage. Part has power, part doesn't.” • Why: highly trained individuals are still performing basic interface actions • If we don’t: our interfaces will continue to be more of a hindrance than a help to personnel

23. Related Research • Social data mining • Vieweg, S., Hughes, A. L., Starbird, K., and Palen, L. 2010. Microblogging during two natural hazards events: what twitter may contribute to situational awareness. CHI '10. ACM Press. • Starbird, K., Palen, L., Hughes, A. L., and Vieweg, S. 2010. Chatter on the red: what hazards threat reveals about the social life of microblogged information. CSCW '10. ACM Press. • Diakopoulos, N. A. and Shamma, D. A. 2010. Characterizing debate performance via aggregated twitter sentiment. CHI '10. ACM Press. • Crowd-powered interfaces • Bernstein, M., Little, G., Miller, R.C., Hartmann, B., Ackerman, M.S., Karger, D.R., Crowell, D., Panovich, K. Soylent: A Word Processor with a Crowd Inside. UIST ‘10. ACM Press. • Hartmann, B., MacDougall, D., Brandt, J., and Klemmer, S. R. 2010. What would other programmers do: suggesting solutions to error messages. CHI '10. ACM Press. What these researchers need: data sets, and the ability to roll out testbed interfaces to small groups.

24. How to Engage • Create TREK-style datasets and competitions for social media data analysis: can we extract the data we need to, quickly, and avoid spammers? • Encourage the development of novel crowdsourced interfaces for complex sensemaking scenarios

25. Success Metrics • Interfaces demonstrably improve efficiency and accuracy, and lessen the learning curve. • Embedded, distributed human help becomes a critical part of complex user interfaces. • We can use the Twitter Firehose to detect, track, and send support to crises when they occur

26. Proposed DARPA Initiatives • Archive all tweets about #iranelection and #peruearthquake, make them publicly available, and hold a competition to produce a time-dependent model of the crises as they unfold • Call for interfaces that can integrate latent human knowledge and expertise (e.g., in veterans and reservists) to support the main user.