Seattle, WA 22 October 2003 Gordon Bell Microsoft Research

1. Vis 2003 MyLifeBits Challenge for VisualizationVisualization: “Form follows function” Visualization: Science? Engineering? Or Art? Seattle, WA 22 October 2003Gordon Bell Microsoft Research

2. My affair with Visualization • 1958 wrote programs that displayed on the 32x32 scrolling display of Deuce, a machine that came from NPL that Turing worked on. Interactive Marriage proposal program. • 1959 MIT Speech Lab and TX-0 10” 1K x 1K points, basis for DEC display. Displayed sliders, sonograms, traditional engineering graphs. Made a Land use map of Boston for Gwen Bell, a student at Harvard School of Design. They later formed the Graphics lab or such displays. • 1960 DEC. Type 30 . Spaceware was built on it, but at one of the joint computer conferences, Ed Jacks of GM watched the display the whole time and got IBM to build them displays for car design. • 1961 saw the difference of logic diagrams vs. logic equations; i.e. east vs west coast design • 1986-1989 Dana>Ardent>Stardent “the graphics supercomputer”. • 1987 Commissioned the NSF Study for visualization. • 1987 FCCSET report for the Internet proposal claimed we need the net for visualization. • 1990 Keynoted first visualization conference in SF. Met the Fakespace team and became an investor. They produced the boom for VR, large scale displays, tables, etc.

3. Outline • Now I find I was doing interactive vis in 1958. • Review of Vis 1990 keynote • Visualization (Google) 7M; 4.4M data • Vis 1990 then and now • Shneiderman, Tufte, & examples • Can you help MyLifeBits? • Timelines • Document … knowledge management

4. 1987 FCCSET Report for NREN

5. Titan Graphics Supercomputer c1988 25Kpolygons/sec

6. Keynote from Visualization I.23 October 1990 Or how I saw it then… before the web, but after NREN (Internet.)

7. How I "See" saw Visualization 13 years ago* The Turing TEST for Visualization: 3D, dynamic computer graphics and computer imaging are aimed at photo realism, ... i.e. the standard of comparison is limited by a view or photo of the phenomenon itself (reality). The insight comes from within the user.Technology helps somewhat in the viewing. Few of today's vizs provide the insight that compare with the creative representations that Tufte collected. Challenge: Make the computer do what the artists did! Users need Vizicalcs (…otherwise everything is a creation) Training and integration into the computing mainstream is still the bottleneck to use *form mental image of something not visible

8. Computation is the Driving Force for Visualization 4. Analysis-by-synthesis in a real time environment Real, experimental science that simulates and simultaneously compares observations. Courtesy of Web Services 3. Simulation* is the Third Paradigm of Science & Engineering, after 2. Experimentation (taking and plotting data) and 1. Theory 0. Talk is for philosophers *The best way to advance a science is by building a computer (i.e. program) for that phenomenon in an "analysis-by-synthesis" fashion. Viz has come about through integrated computation and graphics, with a push from supercomputing.

9. The NSF Visualization Report Domain & Needs: Steer simulation and computation W/S driven use of supers Graphics-oriented programming environments Higher-order viz of scalar, vectors ... tensor fields Dynamic viz of fields and flows Massive data sets for signals and images Algorithms and archs for parallel graphics & IP Framework for international Viz standards

10. The NSF Visualization Report (Cited) Benefits Integrated set of portable tools Scientific progress and leadership Scientific productivity Standardization of research tools and the cross-discipline flow of knowledge Safeguard American Industrial Competitiveness Making the Advanced Supercomputing facilities useful

11. Any Effect of the NSF Visualization Report? Raised awareness of the opportunity for Viz by creating a word and focus Got funding at various centers (e.g. UC/SD) Galvanized a collection of people working in the area ... Ultimately stimulated this conference* a year or two before it might have otherwise happened *Reaction to size and lack of applications focus at SIGGRAPH

12. Searching for Vizicalc:"have data, need a picture" Users wanted to visualize their data:how much and how fast can it be displayed? how long to get the first picture? Applications done on ad hoc case at a time basis New tools such as AVS came about because of limited resources that had to apply the first graphics supercomputers.

13. Progress: 1990 - 2003 • The big bang: Internet and the web • 1987 FCCSET Report set fiber goal by 2000 • Moore’s Law • 10s of Mips >> 1-4 Gips; Custom rendering… • 10s of Mbytes >> 100 Gbytes • 25 K polygons/sec >> 25 M polygons/sec • Mpixels = Mpixels; mono; but lower cost • 1.5 Mbps > 1-2 Gbps (peak) • Progress in Vis • Realistic rendering… see complete virtual scenes/sets • Expensive, large screens and caves: • Using “2001, A Space Odyssey” progress in: • Visualization was much, much greater • AI, including speech was much, much less

14. Princeton 6K x 3K+ pixel; 18’x8’; $100K

15. Vis 2003 Problem and opportunity: have many nouns… Need:create verbs

16. Taxonomy • Pseudo phenomena (ICs, molecules, colored physical parameters e.g. T, p, fields for flow, stress • Photo realism • Real world: movies, VR, Augmented Reality, • People and digital immortality • Telepresence… ever emerging, potentially important • Data visualization… Tufte gallery candidates? • Small no. of dimensions • Large dimensionality information maps • Ordinary text and user visualization beyond UI

17. Shneiderman’s insights “exploratory data analysis and visualization process is …characterized by mantra for visual information seeking: “Overview, zoom and filter, details-on-demand” “The future … larger, higher resolution screens, that present perceptually-rich & information-abundant displays. … the flood of information can be turned into a river of knowledge...Human perceptual skills are quite remarkable and largely underutilized”

18. Tufte on Graphic Excellence

19. Tufte

20. Tufte Napoleon Map Paris-Moscow

21. Data maps… near and dear

22. Network flow of trade

24. Hillis Taxonomy

25. Axon

26. Kiviat/radar of car parameters

27. Evolution of the “Ideal State” at Each Stage: Entrepreneurials Stage IV. Market Development Business Plan Stage III. Manufacturing Marketing Product Development Product Stage II. Development Seed Sales Stage I. Concept Technology/ CEO Engineering Team Control Fundability Board of Directors Cash

28. Many dimensions.. Inverse pattern recognition

29. Tufte: On PowerPoint

30. I am data

31. The guinea pig • Gordon Bell is digitizing his life • Has scanned/encoded virtually all: • All computer files • Personal documents (memos and email, bills, legal documents, papers written, …) • Books written (and read when possible) • Photos • Posters, paintings, photo of things (artifacts, …medals, plaques) • CD collection • Home movies and videos • Now recording real time: phone, radio, TV (movies), web pages… started meetings, conversations to come • Paperless throughout 2002. 12” scanned, 12’ discarded. • Only 30 GB!!!

32. The MyLifeBits Challenge • Files, photos, documents • Challenge is viz relevant for organizing, searching, and utilizing your shoebox… data problem • Tufte vs. Visioneers: • Visualization is an art. • Visualization is science and it has to be automatic! • Challenge: • 1. What are the questions viz can help answer? • 2. What viz can help make a life or after life interesting? • We have many nouns. You make verbs.

33. Statistics of use

34. Visualization Needs & Possibilities • Time lines • Video cliplets, photos searching • Automatic story creation from content • Related document maps • Concept mapping of entire space • Collections & taxonomies • Synthesis of talking heads “in their words” • Tweening from photos to recreate entire movies of the experience.

35. Personal LifeLog Applications Diary/Journal Self Tutor Mentor Advisor Meeting Prep Personal Assistant Babysitter Financial Manager Companion Caretaker Medical Manager Parole Officer Photo Album Assistant for Elderly Application used by: Pers Flight Recorder Autobiography Captain’s Log Conservator Biography Baby Book Trustee Obituary Personal Proxy Executor Others Application controlled by: Others Self

36. Radio capture tool Telephone capture tool PocketPC transfer tool PocketRadio player TV capture tool Internet MyLifeBits store Radio EPG tool TV EPG download tool MAPI interface Legacy email client database Browser tool files Legacy applications MyLifeBits Shell Voice annotation tool Text annotation tool

37. MyLifeBits organization: time and space Archival (time) Working Timeline/ Context(space) Personal (some $s) GB Co.(angel, etc.) Professional ACM, etc., … @Microsoft.com, New co’s.

38. 1900 1910 1920 1930 1940 1950 1960 1970 1980 1990 2000 2010 C,L m d d CGB... GB SR mB,L KF SB Where KvMO B ABos P B WCa 6-year --GS-HS--MIT DEC---+++++.+++---++++ Education KV-----mit,F cmu Work Bell Elec DECcmuDEC E,NSF MSFT ComputerMuseum M B SiValley Books BN SBN HiTechVent Computers 4-6 11 VAX E T Awards..

39. Time-machine ComputingRekimoto

40. Lifestreams: View by timeDavid Gelernter et al

41. Rock & Roll History graph

42. Timeline view tells a story

43. MyLifeBit v2 clustered time view

44. …Information Visualization within a Digital Video Library • Colored notches show where matches occur

45. …Information Visualization within a Digital Video Library • Keyword color matches color in ranking

46. Here are the photos

47. Getting the user to tell a story is the ultimate in media value • A story is a “layout” in time and space • Most valuable content (by selection, and by being well annotated) • Stories must include links to any media they use (for future navigation/search – “transclusion”). • Cf: MovieMaker; Creative Memories PhotoAlbums We took him to lunch at our favorite Dim Sum place to say farewell Dapeng was an intern at BARC for the summer of 2000 At table L-R: Dapeng, Gordon, Tom, Jim, Don, Vicky, Patrick, Jim

48. Digital Immortality: CMU Interactive Einstein:

49. U of MD display of health data

50. http://www.research.ibm.com/history/gallery.htmDocument Changes History Flow