“ Semantics ” for Innovation in Visualization and Multimedia: Smarter Information Science

1. “Semantics” for Innovation in Visualization and Multimedia: Smarter Information Science ICSTI Workshop February 8, 2011, Redmond WA Peter Fox (RPI) pfox@cs.rpi.edu Tetherless World Constellation http://tw.rpi.edu

2. Please buckle your seatbelt • Working premise and the burden • Opportunity – new means • Linked open data (LOD) • Open-source software (Field) • Science conduct • Semiotics of portrayal • Includes semantics • Representing e.g. • Uncertainty, quality, bias • Speculation Tetherless World Constellation

3. Working premise Scientists – actually ANYONE - should be able to access and use a global, distributed knowledge base of scientific data that: • appears to be integrated • appears to be locally available But… data and information is obtained by multiple means (instruments, models, analysis) using various (often opaque) protocols, in differing vocabularies, using (sometimes unstated) assumptions, with inconsistent (or non-existent) meta-data. It may be inconsistent, incomplete, evolving, and distributed AND created in a form that facilitates generation, not use (except by accident) And … significant levels of semantic heterogeneity, large-scale data, complex data types, legacy systems, inflexible and unsustainable implementation technology… Uh-oh

4. Changing the equation • “Changing the Equation for Scientific Data Visualization” – Fox and Hendler (Feb 11, 2011) Science (Perspectives), in press (embargoed, sorry) • Three important points • Unlocked data (and it’s big, really, really…) • Visualization for the masses throughout the ‘life-cycle’ but scale-free (!) • Smarter data, smarter visualization

5. .. Data has Lots of Audiences More Strategic Less Strategic From “Why EPO?”, a NASA internal report on science education, 2005 Science too!

6. Shift the Burden from the Userto the Provider – for Viz. too! Fox Informatics and Semantics, © 2008

7. Too many diagrams

8. Visualizing Linked Open Data (logd.tw.rpi.edu)

9. Linked open data • Simply put: data is in RDF and has a URI and/ or it’s behind a query-able ‘triple-store’ interface ‘convert’ ‘load’ ‘query’ ‘render’

10. New means – artists to the rescue • Digital artists, they needed good creative visual tools, art at the speed of creative thought, feeling, intuition, mental representation and they love programming • And, RPI has EMPAC – Experimental Media and Performing Arts Center

11. From flat screen to black box - EMPAC

12. Field – rapid visualizing

13. What we are doing • Field meets Linux! • Linked data meets Field! • Feed the current LOD graphics into Field for manipulation • Then…. • Unscrew the Google graphics • Unscrew the JSON feed • Query / consume raw RDF • Visualizing at the speed of thought/ typing.. • From the laptop to scale • So this is where the semantics re-enter, especially for portrayal

14. Linked open data • Field consumes JSON (webify it) ‘convert’ ‘load’ ‘query’ ‘render’

15. Linked open visualization • Field queries ‘triple stores’ (semantic webify it) ‘query’/‘render’ ‘convert’ ‘load’

16. Linked open visualization • Field queries ‘triple stores’ (semantic webify it) ‘query’/‘render’ ‘dynamic’ ‘access’ ‘load’

17. Science - Means of conduct

18. So what about abduction? • No, not the criminal meaning… • Is a method of logical inference introduced by Peirce which comes prior to induction and deduction for which the colloquial name is to have a "hunch". • Abductive reasoning • starts when an inquirer considers of a set of seemingly unrelated facts, • armed with an intuition that they are somehow connected • oh, wait, good job for visualization!!! • Leverage open world, semantics, too… on the web

19. Information theory • Semiotics, also called semiotic studies or semiology, is the study of sign processes (semiosis), or signification and communication, signs and symbols, into three branches: • Syntactics: Relation of signs to each other in formal structures • Semantics: Relation between signs and the things to which they refer; their denotata • Pragmatics: Relation of signs to their impacts on those who use them

20. Semiotic model

21. Semiotics of portrayal • But we are talking about a digital world increasingly more than an analog one • Beyond the separation of content from presentation • We have means for content (context and structure) semantics but pragmatics? • Portrayal (not just ‘maps’ or ‘graphs’) • How – representation of content, context and structure, capture visualization provenance • Graphs, points, lines polygons, titles, axes, color, shade, dimensions, … and their relation to each other!

22. For science viz. • We leave the untold things untold – big (really, really big) problem(s) – like: • Uncertainty • Quality • Bias • Need evidence • An example?

23. MODIS Terra & Aqua vs. AIRS Cloud Top Pressure AIRS vs. MODIS Terra AIRS vs. MODIS Aqua Correlation maps for Jan 1 – 16, 2008 Impact:Throw your hands up in the air and just walk away, silently… MODIS Aqua vs. MODIS Terra

24. Known Issues: The difference of EQCT and Day Time Node, modulated by data-day definition, caused the included overpass time difference, which makes the artifact difference. See sample images: BUT WHY ARE WE SAYING THIS IN WORDS? Included Overpass time Difference MODIS Terra vs. MODIS Aqua AOD Correlation

25. Abductive Information System? • What would this look like in application tools? How to explore ‘hunches’ (hints)? • If you consent that induction is fundamentally part of how an information system is developed, then how to allow for abduction before induction may be possible? • Open world, integrative • Design factors? Architecture factors? Library factors? Cognitive factors?

26. Speculation • But back to big data and the need to turn the visualization ‘walls’ into exhibits, 4-dimensions – installations – i.e. not immersion but experience • Synesthesia – why only one sense? • Rapid

27. Speculation • At scale – why? • Stereo – why? • Linked to the live data – minimal curation! • Goal: restore abductive reasoning to the conduct of science for specialists and non-specialists • And this has to be informatics-based not some ad-hoc techies making stuff up… • Collaboration – wanna play?

28. So long and … • pfox@cs.rpi.edu • http://tw.rpi.edu • http://openendedgroup.com • http://logd.tw.rpi.edu • http://empac.rpi.edu

29. Back shed

30. Need to be here Curation stages 20080602 Fox VSTO et al.

31. Mind the Gap! • There is/ was still a gap between science and the underlying infrastructure and technology that is available • Informatics - information science includes the science of (data and) information, the practice of information processing, and the engineering of information systems. Informatics studies the structure, behavior, and interactions of natural and artificial systems that store, process and communicate (data and) information. It also develops its own conceptual and theoretical foundations. Since computers, individuals and organizations all process information, informatics has computational, cognitive and social aspects, including study of the social impact of information technologies. Wikipedia. • Cyberinfrastructure is the new research environment(s) that support advanced data acquisition, data storage, data management, data integration, data mining, data visualization and other computing and information processing services over the Internet.

32. Modern informatics enables a new scale-free** framework approach • Use cases • requirements • Stakeholders • Distributed authority • Access control • Ontologies • Maintaining Identity

33. Multi-tiered interoperability used by

34. Tetherless World Constellationtw.rpi.edu • Future Web • Web Science • Policy • Social Hendler Themes • Xinformatics • Data Science • Semantic eScience • Data Frameworks Fox McGuinness • Semantic Foundations • Knowledge Provenance • Ontology Engineering Environments • Inference, Trust Multiple depts/schools/programs ~ 35 (Post-doc, Staff, Grad, Ugrad)