MyLifeBits: Realizing the Memex Vision Santa Clara University 13 May 2004

1. MyLifeBits: Realizing the Memex VisionSanta Clara University13 May 2004 Gordon Bell, Jim Gemmell & Roger Lueder www.MyLifeBits.com www.research.microsoft.com/~gbell

2. Mylifebits collage

3. Outline … MyLifeBits • Background…fulfilling the Memex vision • Cyberizing everything • File to database transition • Use…beyond search • Working with Media Center for home use • Long-term agenda and outlook • Archiving persons and things.

4. MemexAs We May Think, Vannevar Bush, 1945 “A memex is a device in which an individual stores all his books, records, and communications, and which is mechanized so that it may be consulted with exceeding speed and flexibility” • Full-text search, text & audio annotations, and hyperlinks

5. Capturing what you see

6. I am data

7. The guinea pig • Gordon Bell is digitizing his life • Has now scanned virtually all: • Books written (and read when possible) • Personal documents (correspondence including memos and email, bills, legal documents, papers written, …) • Photos • Posters, paintings, photo of things (artifacts, …medals, plaques) • Home movies and videos • CD collection • And, of course, all PC files • Now recording: phone, radio, TV (movies), web pages… conversations and meetings to come • Paperless throughout 2002. 12” scanned, 12’ discarded. • Only 30 GB!!!

8. Capture and encoding

9. Quindi conference capture

10. I mean everything

11. Wearable & interactive jewellery LEDs flash according to sensor type triggered

12. Potentially useful trivia – but not normally photographed

13. GPS: tells where and when

14. Kentaro Toyama wwmx.org

15. gbell wag: 67 yr, 25Kday life

16. MyLifeBits organization: time and space Archival (time) Working Timeline/ Context(space) Personal (some $s) GB Co.(angel, etc.) Professional ACM, etc., … @Microsoft.com, New co’s.

17. MyLifeBits: Some Lives(t) • CGB@ Microsoft • MLB • Clusters • Telepresence • WWW presence • Computer History Museum • BOD member • Fund-raising • CyberMuseum • Startups & boards • Bell-Mason Director • Diamond & Vanguard Brds. • Personal • Parents, children, grandkids • CGB himself • GKB • Close friends • GB $s • Personal incl. several legal structures • Properties: autos, real estate, • Investments & contracts • Past prof. companies/organiz’ns • DEC • Carnegie-Mellon U. • DEC, NSF, Encore, Ardent, Me Inc.,

18. Bell Lives timeline 1900 1910 1920 1930 1940 1950 1960 1970 1980 1990 2000 2010 C,L m d d CGB... GB SR mB,L KF SB Where KvMO B ABosP B WCa 6-year --GS-HS---MIT DEC---+++++.+++---++++ Education KV-----mit,F cmu Work Bell Elec DECcmuDEC E,NSF MSFT ComputerMuseum M B SiValley Books BN SBN HiTechVent Computers 4-6 11 VAX E A

19. Personal LifeLog Applications Diary/Journal Self Tutor Mentor Advisor Meeting Prep Personal Assistant Babysitter Financial Manager Companion Caretaker Medical Manager Parole Officer Photo Album Assistant for Elderly Application used by: Pers Flight Recorder Autobiography Captain’s Log Conservator Biography Baby Book Trustee Obituary Personal Proxy Executor Others Application controlled by: Others Self

20. Radio capture tool Telephone capture tool PocketPC transfer tool PocketRadio player TV capture tool Radio EPG tool TV EPG download tool MAPI interface Legacy email client Browser tool Internet files Legacy applications MyLifeBits Shell IM capture Voice annotation tool Text annotation tool Import files MyLifeBits Software MyLifeBits store database

21. MyLifeBits is: • Memex and more (audio and video) • Universal store for all personal stuff • Guiding principles for the system: • Full text search & collections (> than hierarchy) • Visualizations for search, display, insight • Annotations and links add value and essential • Increase search ability and value of information. • So make many kinds and them easy to create! • Stories are the ultimate annotation • Keep the links when you author: “transclusion”

22. MLB database: size and content? • Database features are essential: Consistency, Indexing, Pivoting, Queries, Speed/scalability, Backup, replication. • Folders &Files were the starting point >> database into sets aka “collections” that are identical to the folder structure • Outlook (msgs, attachments, calendar, contacts) • Web trails including voice message annotation • Journal (Outlook), trails: every document use & transaction • What about? • Money (transactions, payees, etc.)…is their lifelog/trail • Streets and trips to cross-index to all docs • Attributes for photos for retrieval? Location, time, settings • Presentations as a report or trail. Each slide an object!

23. Why bother? An existence proof.The following exist in abundance: • Shoeboxes full of photos • Photo albums & framed photos • Creative Memories is a thriving business selling resources for created high-end photo albums that are well laid out and highly annotated, using long-lasting materials. • Home videos • Bookshelves and filing cabinets • Old bundles of letters • Professional video/photo companies do capture at kids’ sports events and sell content like hotcakes • Probably not accessed very often but TREASURED (what’s the one thing you would save in a fire?)

24. Why bother? ..more reasons • To eliminate physical storage (paper, CDs…) • It costs more (in time) to delete than the cost the storage • You may only want to retrieve one of many items in the future, but cannot predict which one (which is why you file many things now) • For posterity and nostalgia • For memory enhancement & faster search(search your LifeBits rather than the web … a single source to look for anything you have ever seen) • Let content analysis and data mining discover trends and correlations in your life

25. application specific data application specific data application specific data application specific data people user infrastructure system application specific data • Extensible XML schemas • Logical views • Programmatic relationships • Synchronization service • Information agents

26. Annotation like this… Voice Annotation

27. Pivot to look at all of MLB(t) Call, contact, pivot by time to find web page

28. Find brig, image, and look for 80

29. Here are the photos

30. Timeline view tells a story

31. Interface to xls

32. Statistics of use

33. Value of media depends on annotations • “Its just bits until it is annotated”

34. Getting the user to tell a story is the ultimate in media value • A story is a “layout” in time and space • Most valuable content (by selection, and by being well annotated) • Stories must include links to any media they use (for future navigation/search – “transclusion”). • Cf: MovieMaker; Creative Memories PhotoAlbums We took him to lunch at our favorite Dim Sum place to say farewell Dapeng was an intern at BARC for the summer of 2000 At table L-R: Dapeng, Gordon, Tom, Jim, Don, Vicky, Patrick, Jim

35. Value of media depends on annotations “Its just bits until it is annotated” • Auto-annotate whenever possible e.g. GPS cameras • Make manual annotation as easy as possible. XP photo capture, voice, photos with voice, etc • Support gang annotation • Make stories easy

36. Web Scout Future work: Visualizations Don't give me a little card image and say, "That's all you've got, because that's what I thought you should want for your virtual shoebox." There have got to be multiple modalities and the designers have to be able to deal with that. … don't metaphor me in, don't give me only one way of looking at things. -Andy van Dam, Hypertext '87 Keynote Address U. Maryland IN-SPIRE Next Media

37. LifeLines (Plaisant et al.) www.cs.umd.edu/hcil/lifelines University of Maryland

38. Rethinking collections & files • Date collections (“summer 99”) • Much better as a query • By Person (“Photos of Bill”) • Better as links of type “photo of” to person “Bill” • By Event (“Trip to UCLA”) • Better as links to event in calendar • Working set • Better as query that figures it out for me so I don’t need to maintain it

39. Facets and people • Time (& stage of life). Events… • Location (lat/long vs home, vacation) • Institution (relations including family, work, clubs,…) • Role (student, professional, parent, owner, etc.) • Content type • Audio, graphics, photo, video aka moving picture • Document t type o(200) plus profession specificad, bill…will, cards (calling, credit, grade, greeting), certificate (birth…death), correspondence, diary, essay, forms, legal (6), instructions, lists, resume, reservation, scrapbook, transcript, • Dissemination • Book, electronic, serial, unpublished, • Special collections (e.g. geology, stamps, species, places)

40. Facet Lists

41. Certificate facets

42. “By region” and “by time” should be facets!

43. Telephone, Television, and Radio in theHome of the Future

44. Evolution of media in the home Tomorrow: Today: Yesterday: • Analog storage and transmission on separate networks • Physical space limitations • Tedious management and manual search • Digital storage (CDs, DVDs, PVRs, MPEG & WMA/V) • Digital cable, internet radio, but phone is mostly analog • Still limitations on what we can store • Different stores for different stuff • All digital • Everything connected • Unlimited storage • Everything in a database SQL

45. Legacy Legacy Legacy Redundant stereo Cassette Receiver Wfr Spkr Cables/links Speaker 5+1 Plasma 2 or 3 Cable/Enet 2 IR 8 Stereo 4 5.1 digital 2 Comp./S-video 3 Plasma panel 1 Power 10 Kbd/mse 2 Monitor II (opt.) 4 Camera 2 Total 42 – 46 Things 18+remotes stereo CD 5 speakers Spkr IR stereo Video* VCR 5.1 digital comp. DVD stereo Video* Set top Set top Cable/ Satellite Video* 5.1 digital Plasma Panel Media Center Computer Ethernet SVHS-wide Camera Mic *Video = composite or S-video Kbd Mse

47. The Agenda for the Tbyte(s), Lifetime, PC:The killer app after office and mail. • Guarantee that data will live forever! “dear appy” problem • Cheap, easy, and data-rich (e.g. time, place) capture: GPS and time everywhere Paper capture has to be as easy as discarding (scanner/shredder) Personal meeting capture... E-book…e-magazines & journals need to have critical mass! Telephony and audio capture with indexing Media Center compatible for entertainment (photos, video, TV, radio) • Content analysis (critical for photo & video!) • Information control: privacy, security, expunge/deniability,… • Having to be schizophrenic or have a lobotomy when leaving a “life” • One dbase for everything (articles, books, conversations, ... financial transactions) …vs. long-term use of hierarchical files. Is dbase intuitive? • Annotations/meta-information add every-increasing valueEasy annotation for aiding search and it becomes the content • The “killer apps”: Alzheimer, immortality, surrogate memory? • GUI’s to improve use (e.g. time to learn, use, retention)

48. The “dear appy” problem Dear Appy, How committed are you? Please come back to me, Lost and forgotten data • Who’s responsible? • media • platform, file, and databases • evolving standards and formats • evolving and/or disappearing apps

49. Problems: “Amnesia” control & deleting corporate “life” bits • Full sharing of bits that are mine • I created them, OK to copy and distribute • DRM: purchased for my own use • “OK to look at, but I only own half the bits” • Controlling forgetfulness • Private, do not “demo” • Expunge forever... “this never happened” • The bits “belong” to a corporation or org.

50. The Content Analysis Problem • “Cliplets”: Automatic segmentation of a pile of documents and video into individual documents and scenes. • Item typing: Would like a minimal Dublin Core for each item: date, creator, title, source, abstract, and type • “Type” classification: articles, letters, memos, etc. • Ontology creation for collections