Visual Lexicons: The Quest for Data- Driven Decision Making

1. Visual Lexicons: The Quest for Data- Driven Decision Making Charles H. House Director Research Collaboratory Intel Corporation (Here, let me show you what I mean . . . ) MLMI 2nd Annual Conference

2. Visual Lexicons are one topic – Effective Communication is the goal • I applaud researchers at this conference for your bold investigations of meetings • I will argue for Visual Lexicon inclusion in such meetings, using dramatically better Computer-aided tools • A more fundamental point is that meetings are undergoing radical character change • Understanding more about the character of meetings (today) and the pressures for change could prove enormously fruitful

3. “Communications” means many things • so pervasive that it is seldom “taught” or “studied” in a holistic fashion • Communications “schools” are usually Journalism or Creative Writing(the literary side), or Advertising or Film-making (the graphic arts side), or else MEDIA-heavy • Political Science and Sociology programs usually emphasize the Leadership and Crowd Behavior side • Communications “technologies” usually include Signal Processing, Network Queuing, Switching Systems, Topology, or Bit Rate / Bandwidth / Frequency Spectrum studies -- these are EE and CS depts

4. Enhanced Communications • Effective Communication is the basis of most ORGANIZED ACTIVITY, and is ESPECIALLY CRITICAL TO MODERN BUSINESS • There has been but modest study of the specific character of effective Business communications methodologies from the holistic point of view • Small focused laboratories, seeking to answer the “right” questions, have produced great results • Digital Communications Technologies radically impact our ABILITY and METHODOLOGIES to communicate Both some GREAT OPPORTUNITIES and some SIGNIFICANT CHALLENGES await solution

5. This research requires a multi-disciplinary approach • Just as XeroxPARC tried to understand the “flow of paper” in an office, and used cultural anthropologists and linguists to develop the “Office Document” metaphor, • and THEN they used Computer Scientists to create better solutions • we believe that the various threads of Communications, from “Personal” abilities and perceptions, to Institutional opportunities, to Societal “technological solutions” -- all merit serious inter-acting contextual study

6. How do we learn about “the future”? ‘99 • By ALREADY HAVING LIVED IT • according to Bill Buxton of Alias Research: you can’t imagine the future while living in the past • according to Laurence Wilkinson (GBN): If you think things are REALLY CHANGING. Write new rules . Invest in learning new rules Invest in companies who use the new rules • according to Chuck House (HP)“I never thought of it as insubordination”

7. Structured meetings at Intel ~ 4 million hrs of Classroom instruction / yr ~ 3 million hrs of scheduled meeting rms / yr ~ 5.7 million Audio Bridge Conferences held / yr ~ 56,000 hrs “Effective Meetings” taught / yr We have sizable operations in 243 centers in 22 states, 49 nations, and 5 continents ~ 88,000 “Blue Badge” employees, 145,000 “Green Badge” contractor employees, and 270,000 vendor/client inter-actors (Firewall “perimeter”?) EFFECTIVE COMMUNICATION is VITAL

8. STAFF EXTENDED TEAMS • Full • Attendance • Attention • Perception • Participation • Agreement • Adequate • Skills • Resources • Time • Alignment • Desire MEETING EFFECTIVENESS TRAINING also PRESUMES RATIONALITY, LOGIC, and COMPANY INTEREST ahead of SELF-INTEREST Meeting Effectiveness presumes . . . ALIGNMENT? LEADER • Correct • Goals • Issues • Assignments • Momentum Other things Horn = WICKED, ILL-STRUCTURED PROBLEMS

9. M1 M2 Q1 Q2 H2 Y2 M1 M2 Q1 Q2 H2 Y2 M1 M2 Q1 Q2 H2 Y2 + = “Their Lab” “The Program” (“our” program?) CO-ORD LINKAGE DIRECTED Separate, EQUAL Communitiesin Centralized vs Distributed Development “Our Lab” • Issues • A. Does each Lab: • 1. “know” its role? • 2. “agree to” its role? • B. The Program is: • 1. Designed where? • 2. Managed where? • C. Communication is: • 1. Structured • 2. Ad hoc

10. Purpose of collaboration • Similar to all other communications • To share information with others • To co-solve problems • To build trust, camaraderie • Different than many communications • Problem-solving is paramount • TEAM SOLUTION is most effective • Implicit inhibitors are the most insidious • NASCENT SKILLS • Team decision-making • Data-driven decision making • Virtual teams have many “unnatural” inhibitors

11. Many of you might know the Edward Tufte books Envisioning Information Horn’s work is far more useful for most business applications

12. Many, nay MOST, employees cite DATA OVERLOAD as their 1st or 2nd biggest problem

13. Wicked, ill-structured Problems Abound for Teams – SELDOM CITED, however

14. A TRULY EFFECTIVE Colab capability will deal strongly with this class of issues These are the true hobblers of effectiveness, not speed of Spreadsheet transfer

15. Intel Research Collaboratory • New idea at Intel – more experimentalists than true researchers • Born of frustration w company IT backbone • Loose federation of individuals • Created a Virtuality Index, identified some key problems, and created a “Concept Car” for the company with a Flash Demo • Obtained some (modest) funding • “Underway” a few months now

16. Virtuality Index: What we found • Trend toward being distributed • Across locations, time zone, business units, cultural diversity and different ways of working – GROWING • 75% of Intel folk work on Multi-site teams weekly+ • Multi-teaming: 2/3 of workforce on 3+ projects simultaneously • Overall a 2/3 “virtual” organization and trending • Constant adaptation • ~ 20% are NOT co-located with their supervisor • Good corporate culture holding up • highest rated value: “GREAT PLACE TO WORK” • lowest rated values: communication & timeliness across distance* • Some problems with • Different software tools • Different cultures BIGGEST PROBLEM is that no COLLABORATION TOOLS EXIST that deal with “MY REALITY”

17. A “Big Thought” Problem Statement • No team can be presumed to be co-located • Multiple team membership must be presumed • Data-Driven Decision Making must be enhanced • Turbulent environment, esp. re Investments and re “Politically correct” must be presumed • Infrastructure gap must be presumed • ************************************************** Effective Virtual Collaboration capability is the single biggest Innovation and Productivity enhancer for 21st century workforces  BETTER THAN BEING THERE

18. Better than “Being There”? • Collaboration Tools (especially ROI analyses) are often defended on two bases • Saving of Travel $$$ • Saving of Travel Time • Seldom defended on basis that it’s TRULY BETTER • 30 years experimentation with Stanford  BETTER GRADES (by a WHOLE LOT) Gibbons, JSB, House • 2 years experimentation at Dialogic  BETTER DECISIONS (by a WHOLE LOT) House • 4 years study at Intel  MORE EFFECTIVE ORGANIZATION (by a WHOLE LOT) Bless / Wynn

19. 4.0 = A 3.0 = B 2.0 = C 1.0 Legit tapes, on-site TA First m wave-link Industry “students” MS xE median Bootleg Tapes, off-line office hrs Adding in an “audio back-channel” Factors in Distance Learning 30 year experiment with 15,000 students ~ ½ million student hours Stanford Instructional Television Network This bothered the Dean This irked the professors This worried the administration This angered the on-campus students Faculty revolution overthrew this “experiment 4 years ago – the administration “killed the golden goose” with “class size and class support issues”

20. Virtual Meeting structures REMOTE PARTICIPANT TEAM/ CLASS SPEAKER GRAPHICS Interaction, involvement Immediate feedback Archive, ability to review Tutor, coach What does the Stanford example tell us? • Demands “zero latency” or a “queuing mechanism” • Interaction with others produced better outcome • A Replay of a Staff Meeting has enormous benefit • Semi-synchronous “interaction” is distracting • Question/Answer environment crucial • Videotape allowed “replay” = archived “event” There’s a message here for managers who want to build teams And companies who want to facilitate better communications

21. Video Presence The HP Halo Collaboration Studio Not announced yet, but see Thomas Friedman, The World is Flat An immersive collaboration environment that provides the customer with a connection experience with people in remote locations that is “better than being there.” It is a simple, walk-in ready experience that dramatically changes how people work w/others in distant locations. It literally improves how people work: - more frequent involvement - greater effectiveness - multiple places at one day - higher productivity - reduced travel - better time mgmt & personal efficiency - lower travel costs The growing HVEN network enables expanded communications with a range of external contacts and, in turn, increases the value of the solution. Undetectable LATENCY, incredible REALISM, can do six simultaneous conversations We are jointly exploring with HP re converged products for Halo

22. Video Presence Our hope is to do research with Users of these stations First room is $600,000 (gulp); Volume discounts are available

23. All in main room can see the screen, and be synchronized to its view* Requires NetMeeting or Xweb, and remote attendees to have Desktops, plus pre-thought by all members to have materials NetMeeting-ready All in main room can read the hand-outs However, for Remote attendees . . . where the meeting is distributed Now, imagine one staff “team” in a meeting The meeting is (A) status reports; 11 vs. 9 B occasionally happen in parallel; fewer, but more likely, since 5 of 11 A reports are via phone C, the # of possible conversational relationships during a full meeting goes down dramatically with almost half the group “on the phone” D is the number of parallel visual/body language cues possible at one time A. Number of “primal” 1-on-1 conversations = (n-1) = 11 B. Number of “easy” 1-on-1 conversations in “background” ~ 3 or 4 C & D. Number of 1-on-1 communications able to be conducted = nlocal(nlocal-1) = 42+ Better with dual-remote steerable cameras and “HDTV”

24. What happens for the remote attendee? During 90% of the meeting, listening to presentations • Irritation, since you typically can only hear audio but don’t have access to handouts or to projected slides • Frustration, since the audio is terrible for most “presenters” • Rage,since you cannot interrupt due to Half-duplex lines or High Latency The result = RESENTMENT, ANGER, then INATTENTION During 10% of the meeting, when presenting • Frustration, since you cannot see the “body-language” of the listeners and tune your presentation to meet the audience need • Irritation, when you hear murmuring in the background, without a clue who is saying what, but clear that sidebar conversations are going on • Rage, if you have to request something like support or funding, since you can’t make an impact coming from the remote site The result = DISEMPOWERMENT During the meeting, a tremendous feeling of ALONE-NESS, of being a SPECTATOR in a PARTICIPATIVE SPORT

25. Asking for clarification while remote +84% +54% Voting Affirmative while remote +64% +62% Voting Negative while remote +77% +543% CHANGING THE DYNAMICS OF A NASDAQ 100 COMPANY STAFF: with WebeX, Full Duplex Confer’c’g Phones Percentage Change (Yr 2/Yr1) for Folk based in  Remote HQ Questioning speaker while remote +130% +148%

26. Transforming the Practice of Health Care • Transforming the Nature of Commerce • Transforming the Nature of Work • Transforming How We Design and Build • Transforming Our Understanding of Envir • Transforming Government PITAC1 Vision -- IT Transforming our Society • Transforming the Way We Communicate • Transforming the Way We Deal w Info • Transforming the Way We Learn And we know about Cell Phones, IM, and eLearning 1 Presidential Info Technology Advisory Committee, Feb 99

27. What are the important Research issues? 1 • Software -- Nation (and the world) needs FAR more usable, reliable, powerful SW • Scalable Information Infrastructure -- Learn how to build and use large, complex, highly-reliable and secure systems • High-End computing -- drive research by trying to attain sustained petaflops on Real Applications by 2010 • Socio-economic impact – ITresearch on socio-economic and policy issues. Accelerate and expand education in IT at all levels (incl President of USA) 1 Presidential Info Technology Advisory Committee, Feb 99

28. PITAC “got it right” mostly, except . . . • Collaboration – to labor “with the enemy” was understated, altho the group had earlier coined the word “Collaboratory” • “Rich data” acquisition, coupled with significant mathematics both at “the node” and topologically, is the key to much system understanding • We lack a visual lexicon, or lingua franca, to “represent” this rich data • PITAC research is “unfunded”

29. Virtual Meeting structures “What might be done?” To answer that question, we need to consider four areas: • The technology of creating “presence” • The technology of creating “archives” • The act of creating “context” • The skills and training of participants/leaders • Audio and video systems, network bandwidth, room arrangement, shared ‘whiteboards’ can all be invoked to increasing degree. FOR ASYNC, trust, participation, acknowledgment, affirmation, regularity are all key • Audio and video-taped recordings, archived handouts and slide presentations, and meeting notes. FOR ASYNC, ARCHIVAL FILES, esp. with auto-indexed audio/video tracks and text word structures is imperative • The meeting moderator or support aide appends related material from group input or other sources, including URLs and archive files. KEY to INCLUDE mathematical manipulators, dynamic graphics, et al • Multi-year research and experience with Virtual Teams has resulted in books, seminars, and training courses teaching the ‘rules’ of Small Group and Team Communication with special Virtual attention Even BOB KAHN said that HP’s new Project HALO CBD NONE TEACH “VISUAL LEXICONS” – today’s CALCULUS

30. Three simple wishes • If only . . . the Computer Science community could work more closely with the Social Science community • If only . . . the AI / IA community could work more closely with the CSCW community • If only . . . users could be more rational

31. CS and Social Science • Internet II Sociotech Conference 9/99 UMA • Bring one each from CS / SS (100 Univ) • Who knew whom? • CS folk could “design and build” something REALLY useful • But it wouldn’t generate new PhD theses • And it would have to WORK • SS cannot even define, let alone design, a really interesting system • So they study IM, and “Democracy via e-Mail” • And they get excited about yesterday’s Video Conferencing technologies Oh, let me tell you now Ask me later about Project ARGUS

32. When DID scientists build anything? • Radar / sonar / oscilloscopes in WWII • Transistors at Bell Labs • ARPAnet • Berkeley UNIX • XEROX Star

33. ACM Turing Award in 2005, only 43 years after the fact Best “product of the year” • TCP/IP (Kahn, Cerf) • In 1982 . . . BPY In 1997, at ACM 1, Cerf: “Internet happened ‘so fast’, if you think 35 years is fast”

34. For IEEE 100th Anniversary • I preceded the Dean of EE at MIT, each to give a pithy ten minute talk on “Information Transfer” • He held a hand-scribbled set of notes • I did as well, for the first 2½ minutes, • followed by a 2½ minute overhead set of graphs • followed by a 2½ minute set of 35mm pictures • followed by a 2½ minute videotape • He wasn’t very happy, walking on stage • Let’s “do a simple simulation” of that right now

35. You’re a geographer/sociologist, really a 20th century urbanologist • You want to describe to your students the shifting tide of American cities away from the East Coast • You describe the decay of the “inner city”, and the rise of the “new” Western city • You focus on the difference in cities designed for horse and buggy vs. those built around the auto • You build a “word-image” of Detroit as mecca, and then bitterness, for automotive industry jobs And the students yawn, and look at the ceiling . . . Worse, some are IM’ing; suddenly one points at you, and seven students break into hysterical laughter

36. The 3rd, 4th, and 5th largest in 1950 had a very different next 50 years 1 1 1 11 13 13 2 2 3 +0.0% -1.3% 2 3 3 5 4 4 -0.4% -0.6% 5 6 +1.3% +0.0% 10 ±1.3%/yr is big over 50 years Some small Western cities had a tumultuous 50 years Some famous Eastern cities had a tough 50 years 6 7 8 6 7 10 10 11 -0.8% 17 18 +5.2% 20 21 8 9 -0.6% 33 +2.6% 26 17 -1.3% 49 +4.6% 31 -1.3% >100 >100 Armed with some graphing skills, though, you show them some PowerPoint / Excel graphs of 50 years at America’s largest cities Some, fabled in American lore, are pretty constant in size This is an exceptional Geographer / Sociologist Most are not adept with Logarithmic calculations (nor are they much good with Kalman filters, Bayesian networks, or Markov chains) Baltimore was 2nd, Boston 3rd in 1850 St. Louis, “Gateway to the West” was America’s 4th largest city in 1900; only 49th a century later

37. And THEN, armed with a “cool” 4-D plotting package, you show them a multi-variate dynamic graph (which this really isn’t . . . )of the population of seventeen cities in 1950 7Cities, 4 are very large, 3 are large 10 Cities, 8 are small

38. And THEN, you overlay a multi-variate dynamic graph (which this really isn’t . . . ) of the population of the same seventeen cities in 2000 7 Cities, 2 are very large, 2 are large 3 are small 10 Cities, 1 is very large 3 are large Only 2 are small

39. We communicate lots of thingsWe collaborate about HARD PROBLEMS Edward Tufte’s famous books Envisioning Information, or this one, Visual Display of Quantitative Data e.g. Comparative Colon Cancers by: Gender, Race, Locale, Employment Applicable to: Epidemiology Climatology Mineral / Gas exploration Crime patterns Infrastructure weaknesses Difficulties are manifest: Graphical interpretive “literacy” Statistical “data” selection Application packages missing Sensor nets deployment Semantic net understanding

40. Computer Generated Graphic images The Visual Display of Quantitative InformationEdward R. Tufte, 1983, Graphics Press, Cheshire, CN All types of cancer, white females, Age-adjusted rate by county, 1950-1969

41. Here’s another Graphic image The Visual Display of Quantitative InformationEdward R. Tufte, 1983, Graphics Press, Cheshire, CN Trachea, bronchus, and lung cancer; white females; age-adjusted rate by county, 1950-1969

42. All types of cancer; white females; age-adjusted rate by county, 1950-1969 These plots were shown on Page 17 in the First Edition; Printings 1-4 All types of cancer; white males; age-adjusted rate by county, 1950-1969

43. All types of cancer; white females; age-adjusted rate by county, 1950-1969 These plots were shown on Page 17 in the First Edition; from printing 5+ All types of cancer; white males; age-adjusted rate by county, 1950-1969

44. This data had been compiled for multiple diseases With multi-variate discrimination ( > 17 variables) And it has taken AMA 25 years to BEGIN to learn how to use it 1500% more likely to die of emphysema if you lived in a high mountain valley in 1940s-1960s. AMA denied it Trachea, bronchus, and lung cancer; white females; age-adjusted rate by county, 1950-69 My assertion  Images are compelling We are NOT practiced in “reading them” These plots are shown on Page 18 in the First Edition from Printing 5 on Trachea, bronchus, and lung cancer; white males; age-adjusted rate by county, 1950-1969

45. Recognition from vastly different views and scales: Accurate 3D structure can be deduced from such view tracking Recent Advances in Vision Finding, gathering, scaling, stitching together similar images for automatic free-form panoramas: Images from: M. Brown and D. G. Lowe. Recognising Panoramas. In Proceedings of the 9th International Conference on Computer Vision (ICCV2003) back

46. Video Security Application Landscape 26 million surveillance cameras have been installed worldwide, 11 m US IBM hires 3000 consultants for customers to incorporate digital video security into existing IT operations The remote digital surveillance camera market is growing 40-50%/year Back

47. Gene Regulatory Network: Full Body Scan or even a Foreshortened Heart Scan Speed up on clusters Application Landscape: Bioinformatics THE RACE TO COMPUTERISE BIOLOGY; DECEMBER 12TH 2002, Economist Bioinformatics: In life-sciences establishments around the world, the laboratory rat is giving way to the computer mouse--as computing joins forces with biology to create a bioinformatics market that is expected to be worth nearly $40 billion within three years 2500% more likely to die of a sudden mid-life heart attack if you have both markers. AMA denied it for 62 years Models can be learned via structure learning. My Brother, 2 years ago Low Body Fat, Mod Cholesterol 1 hour Workout sessions 4 days/week “Perfect” EKGs Dead fr Sudden Cardiac Arrest 95% blockage, Coronary Artery 70%+ in other three Homocysteine Markers Simple Blood Test Two “common” Mutant Genes HE HAD BOTH THIS HAS A “SIMPLE FIX”

48. Video is often a “turn-off” Great Video can be compelling Stored, malleable Video is awesome ALL e-Learning Video Systems show the Teacher’s face to the Students – never the other way ‘round The power of video is evolving: “Talking Heads” are “never” compelling N-way™ multicast provides great video imaging with “low-bandwidth” ip v.6 Archiving compelling video enriches collaboration How many of you have seen the “dual skiers” on World Cup or Olympics TV? http://www.dartfish.com

49. If a Picture = 1000 words, what’s a video stream worth? http://www.dartfish.com "If a picture is worth a thousand words then Dartfish is worth a thousand pictures! I believe that The Dartfish Motion Analysis Software Program has completely redefined the future of all athletic training environments. It decreased the time required for our skiers to internalize their skill levels and move rapidly forward toward enhancing them. The time spent with their coaches has become significantly more productive and efficient. . . . simply placing athletes in front of a television for 'video' could be compared to coaching in the stone age. Tony Nunnikhoven Steamboat Springs Winter Sports Club, Alpine Program Dir. 85 of America’s 103 medal winners in the 2004 Summer Olympics trained with Dartfish SW

50. The point? • Computer-augmented data EXISTS in copious quantities in field after field. It is often graphic, visual, dynamic. It is usually NOT understood. • People, even well-intentioned smart educated leaders, don’t avail of data very well for a wide variety of reasons – politics, ignorance, apathy • Systems could approach this problem 3 ways • Designing “answers” in graphical/visual formats • Providing “intuitive training” for erstwhile users • Automating “answers”, removing people from data • If people are to be included in the equation, graphical interpretive dataset presentation – visual lexicons, if you will – are fundamental