The IST-059 Framework for Network Visualisation

1. The IST-059 Framework forNetwork Visualisation M. Martin Taylor (Secretary, IST-059) Martin Taylor Consulting mmt@mmtaylor.net Presentation to Visualisation Network of Experts Workshop: 4-6 November, 2008, Malvern, UK

2. C2 Network Visualisation Approaches (From Vernik-Bouchard Presentation at the IST-063/RWS-010 Workshop, Copenhagen October 2006) Networks can be displayed in many ways

3. Why another Framework for Network Visualisation? • IST-059 considered that the very concept of a real world network is ill-defined. A network is more than a graph that can be described in a matrix, even a matrix of N dimensions. It exists in a context that gives it meaning. • Brilliant displays for many tasks involving networks have been devised, but • IST-059 knew of no way tasks, networks and contexts could be coherently described, which makes it hard to link user requirements with potentially useful tools or applications. • Consistent description of tasks, networks, and display types might help in designing novel displays useful for new problems. The IST-059 Framework incorporates description, function, and a process for using it in support of users, researchers, and designers.

4. The form of a Framework for Network Visualisation • A Framework for network visualisation should include: • A structured approach to describing user needs • A structured set of displayable properties of networks • A structured way of describing display techniques • A structured was of describing display interactions • A process to help the user match needs to displayable properties using the appropriate display techniques.

5. What should a Framework for network visualisation do? The Framework should support users with ad-hoc needs, and should support system designers and researchers by highlighting aspects of network properties that are poorly supported by existing technology.

6. The Framework project introduced the concept of “Embedding Fields”, which provide real-world meaning to real-world networks (“network pragmatics”), and bring the idea of inheritance to the structure of networks (“network semantics”). A “Unified Theory of Networks”? • Although no such grandiose objective was intended, the Framework development turned out to include what looks like a first pass at a “Unified Theory of Networks.” • The overt objective was to develop a descriptive structure in which the different facets of real networks in a real world could be characterized. This effort brought together work that had been done in several different domains, including to a greater or lesser degree: • Social Network Analysis (SNA) • Graph Theory • Complexity Theory • System Dynamics • Object-oriented Programming • Information Theory

7. Framework for Network Visualisation • Discussion Plan • Ancestor frameworks (VisTG Reference Model, RM-Vis Framework) • Kinds of networks and of network properties • Embedding fields and context (of networks and of displays) • Data, Display, and Modes of Perception • Using the Framework • Summary and future work

8. How to develop a Framework for Network Visualisation? • The IST-059 Framework was developed in parallel with a Survey of existing applications and tools for network visualisation. • From the RM-Vis model developed by TTCP C3I AG2, the first requirement was to consider how to describe the important attributes of • User needs and capabilities • Displayable properties of networks • Display and interaction techniques • Only when these and related characteristics had been adequately described could the process of using the Framework be designed. • The Framework was planned in part as a front-end to the Survey. They should use compatible descriptions of what is needed and available. But the main use of the Framework is to help the user understand what kind of display might be well suited to the immediate task.

9. The Framework Concept — 1 Display allows user to visualise real-world state Selected properties are arranged for display User wants to solve a real-world problem Analysis gives network properties Selected data are analysed Real-world data are abstracted into computer representation

10. The Framework Concept — 2 Survey

11. How can the properties of networks be described so that the questions become answerable? The Framework From the User’s viewpoint • The Approach • Start with what the user wants to know or achieve • Consider what the user already knows and how it differs • What new information does the user need? • What means are available to provide the user the desired information? • What might prevent the user from getting the required information? Framework A Framework for network visualisation should make it reasonably easy to answer the foregoing questions — for any particular task, and for a variety of users of different abilities and knowledge.

12. The Framework From the Researcher/Developer’s viewpoint • The Framework shows • A set of dimensions of description for networks • A (start on) a taxonomy of user task types • Implications of display usage for display structure • Modes of perception and their implications for display (interactivity and structure) • Networks in their real-world context Framework Using the Framework, researchers should be able to see areas that lack good displays to show users what they might want to visualise, and developers should be able to see where available technology could be improved to serve user need. Can these promises be realized in practice?

13. Framework roots: Visualisation Reference Models • RM-Vis Reference Model • A descriptive model developed initially by TTCP C3I AG-3 • Separable dimensions of description for application domain, content to be displayed, and display approaches • Defines descriptive dimensions in the Framework • VisTG Reference Model • A functional model developed initially by predecessor groups of IST-059/RTG-025 and reported in the HAT Report • User’s purposes determine the representation characteristics • Separate interaction loop levels for primary tasks, algorithms and engines, and interface • Defines process in the Framework

14. The descriptive RM-Vis Reference Model developed by TTCP C31 AG-3

15. The VisTG Reference Model has 3 loops, the outer acting through the inner: (1) The user understanding and acting on the data in the dataspace, which involves... (2) The user visualising the data provided by and massaged by the analytic and presentation Engines, under the control of the user, who works through... (3) The Input-Output devices that interact with the user’s sensors and muscles. The VisTG Reference Model But the user “really” wants to understand and influence the outer world!

16. The VisTG Reference Model • The VisTG reference Model starts with what the User wants to achieve and how that differs from the current state. (The “WHY” loop) • The User achieves the goal through visualisation (the “WHAT”), and analysis (a different “WHAT”, not shown). These are supported by the computer Engines (the “HOW”). • The Engines • translate the data into displays on the output devices through which the user visualises, and • interact with the User through the input devices. RM-Vis is mainly concerned with the outer “Why” loop and the outer world

17. Framework — Disciplines needed Cognitive Psychology Human factors Engineering Controllers Views Computer Science (Model-View-Controller Design) Model Network Analysis

18. Framework for Network Visualisation • Discussion plan: • Nature of a Framework, and Ancestor frameworks (VisTG Reference Model, RM-Vis Framework) • Kinds of networks and of network properties • Embedding fields and context (of networks and of displays) • Data, display, and Modes of Perception • Using the Framework • Summary and future work

19. Framework roots: Network properties • Network types: • Point-to-point, broadcast, striped, stigmergic, fuzzy or crisp, with or without traffic • Mathematical relations and functions in abstract networks • Many important representable properties (e.g. SNA) • Dynamical properties of real networks (e.g. Feedback loops) • Transformational properties of nodes and links of real networks • Inputs may be of different nature to outputs • Embedding fields of real networks and of displays • Set context and and constrain potentialities of the network • Data Source: static or streaming, and other properties • Is the network changing while the user watches?

20. Network Types • Point to point:The classic network. Nodes are defined and each node is or is not linked to each other node by a link with some “weight” and structure. • Striped or Coloured:Nodes of type A can be linked only to nodes of type B and vice-versa (e.g. malaria transmission from mosquitos to humans and back). • Broadcast:A transmitting node cannot know which of many eligible receiving nodes may receive the traffic (e.g. airborne infection). • Ephemeral:Traffic not received at the time of transmission is lost. • Stigmergic:“Traffic” is left in the environment and may be received at an indeterminate later time by an indeterminate number of receivers (e.g. ruts that tend to guide later traffic, etc., or the clues to a crime left by the criminal that may be read by a detective.) • Fuzzy:Nodes and/or links are not well defined. Places may be more or less node-like, and indefinitely linked to other nodes. The membership of an element in class “node” or “link” may depend on the user’s purpose.

21. A B Road between two towns A and B is a link Fuzzy Nodes and Links Fuzzy link membership should not be confused with link weight. Here’s a simple example 2 1 Original situation A Farmhouse is built near the road Farm A B Is the road between A and B a link? Yes, Pretty much. Is the farm a node? Hardly. 3 More facilities are built to service travellers 4 The cluster becomes a new town Farm X A B A B Gas Hotel Is the building cluster a node? Somewhat, but not really. Is the road between A and B a link or a two-link path? A bit of each! Road between A and B is no longer a link though it remains a path. Roads A-X and B-X are links, and the expanded cluster at X has clearly become a node.

22. Varieties of Link “Strength” – 1: Basic • In many displays of networks, “strong” links are shown more vividly than are “weak” links. However, links have several independent parameters that might be called “strength.” Here are a few: • Traffic-related • Utilization — How much traffic does the link carry? • Capacity — How much traffic could the link carry? • Availability — What is the probability the link is open for traffic? • Coherence or Similarity • How tight is the relationship between the terminal nodes? (sibling is tighter than second cousin; “see” is closer to”view” than to “grow”) • Fuzzy membership — How much like a link is the connection? How should these different kinds of link strength or weight be distinguished in displays?

23. Varieties of Link “Strength” – 2: Complexity • A link may be simple, carrying one kind of traffic or representing one relationship, but what seems to be a single link might actually be a bundle of elementary links of different kinds. To view the network this way is different from viewing it as a layered set of networks of different character. • For example, person A might at the same time • be the father of person B, • lend money to B, • enjoy B’s company, • telephone B frequently. • The complexity of a link bundle implies that the nodes it links are themselves complex, each perhaps including a whole processing network that interconnects the elementary links of the bundle. This is almost certainly true if a subnet has been compressed in order to represent it as a single node (as in Bjørke’s Hypernode process). How should a “bundle” link be distinguished in displays? Is the number of elementary links another dimension of link “strength”?

24. Transformational properties of nodes and links • In an abstract mathematical network, even a multimodal one, a node might be only a place where traffic enters and is distributed to outgoing links. • In a real network, the nature of the traffic and its timing are determined by processes that occur in the node and in the links. • Example: a person (a node) may receive messages from a variety of sources over a period of time, may interpret the messages, and may take action that affects other people, but not by sending similar messages. • Point-to-point gossip about the evil effects of immunization may cause a parent not to immunize a child, who then catches and propagates a serious disease; • Public broadcast messages may induce sufficient people to get immunized that a potential pandemic is avoided. • This network contains both broadcast and point-to-point elements, the links are of different kinds, some fuzzy, and the nodes significantly transform their inputs in generating their outputs. How might all this be displayed?

25. Mathematical Properties • Most of the mathematical properties of networks have been developed in connection with crisp point-to-point networks. A few examples: • Network topology: random, scale-free, tree. • Centrality: distribution of linkage degree over the nodes • Directivity: Whether links are unidirectional or two-way • Cyclicity: Can traffic go from A through other nodes and back to A? • Diameter: The longest geodesic between any pair of nodes • etc….. • The mathematical properties of fuzzy networks are less well developed, but should reduce to those of crisp networks in the limit of binary membership functions (only zero or unity allowed). Mathematical properties often are important in interpreting the implications of network structure in the real world, and should be displayed when needed. How?

26. Information and Uncertainty • What a user can learn about a network is limited by the information available. In the real world, there are many possible kinds of uncertainty. • Links may exist in the real world that are not known in the dataspace, and these unknown links may be critical to the network dynamics. • Locations of nodes may be geographically ill-defined. • Situations may change faster than accurate data can be obtained and interpreted. • The nature of known links or their strength may not be well defined. • Users can take advantage of only a limited amount of what might be displayed. • User’s time to interpret a display may be limited. • Etc… Some of these uncertainties may be measurable in information-theoretic terms, and the results used to create effective displays. In other cases, the user needs to know the uncertainty itself, and the display of uncertainty is a hard problem.

27. Dynamic Properties of real networks • Network traffic changes over time, and networks themselves change. • If a network contains cycles, as most do, the traffic can vary regularly or chaotically, independently of any effects from outside the network. • The passage of traffic can alter the network stigmergically • e.g., in an infection network, the structure of the network changes when a node (person) moves from susceptible to infective to immune (or dead). • Because of such stigmergic effects, cycles are not possible in an infection network in which persons become immune after being infected, even though the static structure of the network and its “social contact network” embedding field suggest that cycles should exist. Epidemic pulses must come from outside the network – a larger network. Network Dynamics is the object of a field called “System Dynamics”

28. Framework for Network Visualisation • Discussion plan: • Nature of a Framework, and Ancestor frameworks (VisTG Reference Model, RM-Vis Framework) • Kinds of networks and of network properties • Embedding fields and context (of networks and of displays) • Data, display, and Modes of Perception • Using the Framework • Summary and future work

29. Real Networks • Are not mathematical abstractions. • They are messy. • They are embedded in a complicated environment • They are not well-defined or completely known • They are what real users have to deal with.

30. Embedding fields of real networks – 1 • A network in the real world consists of physical or conceptual entities connected by relationships that may be • physically embodied (e.g. roads, wires) or • purely conceptual (family tree, social influence, etc.) • The network is embedded in a physical or conceptual substrate, but what determines a relevant “embedding field” is the set of contextual attributes in which changes make a difference to the network for the user’s current purpose. The effectiveembedding fieldcan be thought of as the currently relevant context. Left: Road network in embedding field of map showing directions, distances, and political, cultural and landscape features. Right: Subway network with embedding topology that includes river and rail lines. In any display, the focal element will be best understood if it is shown in a relevant context, and without irrelevant context. Networks are no different.

31. Embedding fields of real networks – 2 Networks are often displayed along with some aspect of an embedding field to supply context. But not always: Two representations of part of the Internet.

32. Embedding fields of real networks – 3 • The embedding field for a network may or may not be another network • e.g. for a contagious disease, the network of infections is embedded in the network of social contacts, but for an airborne disease or one with an insect vector it is not. • Networks can inherit properties from their embedding fields • e.g. location for a geographic embedding field, potentially infectious contacts for a social contact network embedding field. • The embedding field constrains the properties of the embedded network, but new attributes can be developed, as in a class inheritance hierarchy • e.g. contacts are limited to those of the embedding social network, but contact type – casual, intimate, telephonic, etc. – is a new dimension of description.

33. Syntax, Semantics, and Pragmatics are all required to make sense out of language. Graph theory, SNA, and Embedding Fields seem all to be required to make sense out of real life networks. Embedding fields of real networks – 4 • In Linguistics, three different kinds of structural relationship are recognized: syntactic, semantic, and pragmatic. The concept of Embedding Fields provides an analogous conceptual framework for networks. • Syntax is the set of rules that determine permissible word selection and ordering within a sentence and among sentences. In networks, graph theory has this role. • Semantics refers to the meaningful relationships among the words: “Colourless green ideas sleep furiously” is syntactically valid, but semantically invalid. To some extent, Social Network Analysis extends into the analogous network terrain, but the inheritance hierarchy of embedding fields develops the analogy further. • Pragmatics refers to the way the concepts expressed relate to the real world represented in the mind of the speaker and listener. Embedding Fields similarly relate the abstract networks to their effects on and reactions to the real world in which the networks exist.

34. Embedding fields and network display • Embedding fields are the context in which the network exists. • Not all aspects of the context are relevant to the user’s task. • Not only the network, but also the display medium can be considered as a hierarchy of embedding fields, the root of which is, say, the set of pixels of the display screen, intermediate levels might be 2-D and then 3-D spaces containing lines and objects, while the leaves might consist of the coloured lines and objects used to show the network attributes of concern. • (Speculation) The immediately ancestral embedding field for the display of the network may well be the appropriate environment in which to display the user-relevant contextual embedding field of the network. The concept of Embedding Field hierarchy for displays needs more study

35. Framework for Network Visualisation • Discussion plan: • Nature of a Framework, and Ancestor frameworks (VisTG Reference Model, RM-Vis Framework) • Kinds of networks and of network properties • Embedding fields and context (of networks and of displays) • Data, display, and Modes of Perception • Using the Framework • Summary and future work

36. Framework: Categorizing Data Types Six Descriptive Dimensions from the Final Report of IST-013/RTG-002 (The HAT Report, RTO-TR-030) Uncertainty may apply to most of these dimensions, but especially to “Values”.

37. Framework: Categorizing Display Techniques Four Descriptive Dimensions from the Final Report of IST-013/RTG-002 (The HAT Report) Data Values for display are supplied by the engines from the Dataspace, and are not uncertain. “Uncertainty” is simply a kind of datum for possible display.

38. VisTG Framework: Perceptual Modes • We consider four distinct modes or purposes of perception. They suggest approaches to information display, and can help categorize user tasks. • Perceptual Modes • Monitoring/Controlling: Keeping track of a changing situation and possibly acting to alter it. • Searching: Looking for something wanted right now for Monitoring or Controlling • Exploring:Building understanding of slowly varying aspects that could still be useful for later search or control. • Alerting: Noting that a prespecified condition has occurred in a datastream or exists within a dataspace too large to be seen all at one time. Alerting is usually an automated process.

39. Framework: Perceptual mode implications for display The four modes often have implications for display:e.g. in an anti-terrorist scenario Exploring involves the discovery of networks, and might benefit from a fisheye display of the portions of the network so far discovered. Monitoring implies continuing observation of network changes and traffic dynamics, and often requires interactive display control. Searching concerns the attributes of specific nodes, to discover their potentialities when matched with those of linked nodes, and hence requires both wide range and closely focused display representations. Alerting is a programmed background activity that suggests the requirement to display relevant aspects of the network in context, when any of the prespecified patterns is detected.

40. Framework: Perceptual mode implications for display

41. Framework: Interaction modes The display may be for a single user or for multiple users Interactive: A single user may interact directly with the display Coordinated: Multiple users cannot interact freely with the display, but can work together to Coordinate their interactions with one or more displays. Coordination may use the displays or may use communication side-channels. Mediated: Single or multiple users may use a Mediated interaction with the display, in which an operator manipulates the display for viewing by the end user(s). Briefing is usually done by Mediated interaction with multiple end-users; senior officers usually interact with their displays as single end-users mediated by an operator. Passive: In passive viewing, the user has no influence on the content or manner of the display. Any number of users can view passively a display such as in a book or on a Web site.

42. Almost all displays presented in demonstration and all presented in books are viewed passively, and are used mainly in Explore mode for investigating network structure or historical dynamic behaviour. Framework: Interaction modes – 2 Interaction Modes affect which perceptual modes are more likely to be used. (Table from draft Final Report of IST-059)

43. Interactive:Informal, unstructured, pragmatic Mediated or Coordinated:semi-formal, structured, semantic Passive:formal, culturally appropriate, structured, syntactic. Framework: Interaction modes – Display Implications Interaction Modes affect the need for formal display syntax In language, the syntax of conversational interaction is less formal than that of a spoken lecture, which in turn is simpler than the syntax of written text, because in conversational interaction, the parties can interact to query poorly understood elements, whereas the author of a written text must supply syntactic clues to reduce the likelihood of misunderstandings. Likewise, in displaying complex material, a single user interacting with a display can build a representation of the dataspace in an informal and unstructured way, as one might when using a blackboard, whereas a display created for later viewing by other people must contain culturally appropriate syntactic clues that aid viewers to interpret it as the designer intended.

44. Framework for Network Visualisation • Discussion plan: • Nature of a Framework, and Ancestor frameworks (VisTG Reference Model, RM-Vis Framework) • Kinds of networks and of network properties • Embedding fields and context (of networks and of displays) • Data, display, and Modes of Perception • Using the Framework • Summary and future work

45. Survey The Framework Concept

46. Framework Worksheet Concept – 1 The Framework process starts with a worksheet of questions that define what one is trying to achieve. The answers to these questions should serve to generate queries to a database of potentially relevant applications, software tools, or display techniques. • A first draft worksheet was tried out for some diverse use-cases. • An analogue to the current anti-terrorist intelligence situation, in Elizabethan England (1570-1600), seeking evidence of any possible assassination plot against Elizabeth • The spread of avian flu on farms • The social network analysis of a terrorist network • Protection of a computer network • Not all the questions were easy to answer for all use-cases, and the worksheet will be reviewed and revised. • The questionnaire worksheet is intended to provide answers that could be used to develop queries into a database of presentation techniques and available applications or software tools. The results of these queries should lead either to suggestions for the immediate user, or to the identification of gaps in the armoury of tools for network visualisation. The next few slides show the questions used in the first draft worksheet.

47. Framework Worksheet Concept – 1 The worksheet progresses in stages, starting with the overt problem definition, followed by questions of the network properties, the dataspace, any dynamic issues, context, measures, resources, and so on. It will be developed and the stages and questions modified as a result of experience with use-cases. This is a first draft.

48. Framework Worksheet Concept – 2

49. Framework Worksheet Concept – 3

50. Framework Worksheet Concept – 4