Passive Capture & Ensuing Issue for a Personal Lifetime Store

1. Passive Capture & Ensuing Issue for a Personal Lifetime Store Jim Gemmell, Lyndsay Williams, Ken Wood, Roger Lueder & Gordon BellCARPE Workshop Oct 15, 2004

2. The Price of Active Capture “When I had my first child, I bought a camera and took many pictures. But eventually I realized I was living behind the camera and no longer taking part in special events. I gave that up – now I don’t have nearly as many pictures of my second child.”

3. Passive Capture • Make capture passive • Let the user experience! • Save a rich record of personal experience • Possible application as memory aid • The challenge: we were already swamped in media… won’t this just make it worse?

4. Outline • What is SenseCam • What is MyLifeBits • Demo • Conclusion

5. SenseCam

6. sensecama photographic memory for everyone Ken Wood & Lyndsay Williams Microsoft Research Cambridge 15 Oct 2004

7. sensecam • Essentially a “Black Box” data and image recorder for the human body • Current wearable prototype can be worn for a day and captures up to 2000 images in 12 hours • Image capture triggered by sensors, e.g. motion, light, temperature, people in field of view, ... • Sensor data is also recorded for later presentation, analysis, and correlation

8. a short walk through Cambridge

9. image stabilisation • Accelerometer measures motion of the device • When any sensor indicates image capture, we wait a few tens of milliseconds to capture image when movement is less than 10 degrees/second if possible • Greatly reduces the number of blurred images Before After

10. ultra wide-angle lens • Captures everything regardless of exact camera orientation

11. dealing with sensecam data: MyLifeBits Sensor readings Corresponding images in rapid serial viewer

12. status and plans • 12 devices built; adding audio trigger and GPS • But hardware is just the beginning • Trials and studies are the interesting bit • domestic probe • give it to people and see what they do: personal memory enhancement? family fun? • clinical trials with memory-loss patients • ethics approval for 20 Cambridge patients over 3 years • tourism trials • Edinburgh Festival, museums, ... • cross-cultural privacy study • UK, China, India, Egypt, ...

13. fridgecam

14. MyLifeBits

15. MyLifeBits is: • An experiment in lifetime storage • Digitizing Gordon Bell’s past • Capturing more of his future • A software system • Capture • Storage & retrieval • Organization & annotation • Minimum requirement: fulfill Vannevar Bush’s 1945 “Memex” vision

16. The guinea pig • Has now scanned virtually all: • Books written (and read when possible) • Personal documents (correspondence including memos and email, bills, legal documents, papers written, …) • Photos • Posters, paintings, photo of things (artifacts, …medals, plaques) • Home movies and videos • CD collection • And, of course, all PC files • Now recording: phone, radio, TV (movies), web pages… conversations and meetings to come • Paperless throughout 2002. 12” scanned, 12’ discarded. • Only 44 GB, incl. 10 wma, 14 SQL!!! Video: o(100) + 500 mov

17. So you’ve got it – now what do you do with it? “A record if it is to be useful … must be continuously extended, it must be stored, and above all it must be consulted” “The difficulty seems to be, not so much that we publish unduly … but rather that publication has been extended far beyond our present ability to make real use of the record” - Vannevar Bush

18. Everything goes in a database • You need all the features of a database(Consistency, Indexing, Pivoting, Queries, Speed/scalability, Backup, replication) • If you don’t use one, you will find yourself creating one! • Files as blobs, also sync with file system for legacy apps SQL

19. Entities & Links Photo of Event Caller in Phone Call Annotates Transcludes

20. SenseCam Telephone capture tool PocketPC transfer tool PocketRadio player TV capture tool GPS import & Map display Radio capture & EPG TV EPG download tool MAPI interface Legacy email client Browser tool Internet files Screen saver Legacy applications MyLifeBits Shell IM capture Voice annotation tool Text annotation tool Import files MyLifeBits Software MyLifeBits store database

21. DEMO

22. A Storocratic Oath • Do no harm to dates(File creation, Photo taken) • Do no harm to device created & other meta-data. • Camera data & location data are sacred. • Support & aid the creation of critical meta-data. • When/how the user feels like it • Auto-magically! • Maintain user confidentiality

23. www.MyLifeBits.com research.microsoft.com/hwsystems

24. BONUS SLIDES

25. Collaborators • Other MSR Collaborators: Kentaro Toyama, Ron Logan, Steve Drucker, Curtis Wong, Mary Czerwinski, Brian Meyers • Interns: Josh Blumenstock, Evan Salomon, Aleks Aris

26. MemexAs We May Think, Vannevar Bush, 1945 “A memex is a device in which an individual stores all his books, records, and communications, and which is mechanized so that it may be consulted with exceeding speed and flexibility” • Full-text search, text & audio annotations, and hyperlinks

27. The AI Challenge:Content analysis & Data Mining “Creative thought and essentially repetitive thought are very different things. For the latter there are, and may be, powerful mechanical aids” – Vannevar Bush • Is MyLifeBits just enough rope to hang yourself with? • MyLifeBits must become MyPersonalAssistant • Content analysis and data mining • Doc similarity & “clean living” • Document meta-data extraction

28. Moving forward:new capture modes/devices Deja View Body Media SenseCam Quindi

29. Future work: UI • Focus to date on functionality – now we need to make it friendlier • User studies with SFSU

30. Web Scout Future work: Visualizations Don't give me a little card image and say, "That's all you've got, because that's what I thought you should want for your virtual shoebox." There have got to be multiple modalities and the designers have to be able to deal with that. … don't metaphor me in, don't give me only one way of looking at things. -Andy van Dam, Hypertext '87 Keynote Address U. Maryland IN-SPIRE Next Media

31. By Gordon Bellhttp://research.microsoft.com/~gbell Dear Appy, How committed are you? Signed, Lost and Forgotten Data Dear Appy, I'm having trouble with long-term commitment -- not on my end, heaven knows, but from the apps that created me and with whom I like to associate. Over time, these pesky apps evolve and they simply don't recognize the data that they once helped create! But, we data progeny -- and there are lots of us -- feel that as our creators, these apps should be responsible for eternal support. But the little problem with recognition isn't the worst of it – sometimes the apps even disappear altogether. I ask you, is it expecting too much for 20-something year old data like me to be interpretable by my app (e.g. Acrobat, DB2, Draw, Eudora, Office, Quicken, or RealNetworks), or am I just associating with irresponsible apps? If things continue on their current path, it seems I will be completely un-interpretable within 20 to 50 years! My apps will move to other platforms, or evolve to be more Internet- or Next-Big-Thing-centric...

32. Classification wish list • Download classifications rather than build them • Definitions & synonyms should help find what I want • Today it is too expensive to manually classify my scanned paper. E.g. “right time” meta-data is critical! • Next year I hope “the system” can classify papers and other documents e.g. bills • In 10 years I expect all documents to appear electronically & classified with a little help from me

33. Personal Search is notProfessional or Web search • System sees every entry & access • Everything, not just a professional life • Limited to SIS, not an infinite amount, covers a profession & personal life Professional user Depth e.g. information item types & coverage MyLifeBits Web as seen by search engines Knowledge breadth e.g. Dewey classification