Shared Knowledge and Information Flow in Systems Engineering

1. NASA Goddard Space Flight Center Systems Engineering Seminar March 3, 2009 Shared Knowledge and Information Flow in Systems Engineering Socio-Cognitive Analysis of the GSFC Mission Design Laboratory Mark S. Avnet Ph.D. Candidate Engineering Systems Division Massachusetts Institute of Technology

2. Who Am I and Why Am I Here? • S.B. in Physics, MIT, 2001; M.A. in Space Policy, GWU, 2005 • Software Engineer, 2001 – 2003; NASA HQ, 2004 – 2005 • Observed an Interesting Phenomenon • Decisions in space systems development require integration of perspectives: policy, scientific, engineering, public, etc. • Systems engineering takes into account the unique views of each, but the engineer is taken to be outside of the stakeholder framework. • Ph.D. in Engineering Systems, MIT, 2009 • Research addressing this issue • Focus of this talk: contributions to SE here at GSFC

3. Perspectives on Space Systems Design Source: Robinson, G.L., “Systems Engineering Initiatives at NASA,” Goddard/SMA-D Education Series, 25 Sept 2008.

4. Structure of the Presentation 1 Overview of the Mission Design Lab Integrated Analysis: People and Process A Model of Shared Knowledge Analysis of the Design Process 2 3 4

5. Overview of the Mission Design Lab Part 1

6. GSFC Integrated Design Center Integrated Design Center (IDC) Focus of this Talk Mission Design Lab (MDL) Instrument Design Lab (IDL)

7. The Mission Design Lab

8. The MDL: Structure and Products http://idc.nasa.gov/mdl/products.cfm http://idc.nasa.gov/idc/services.cfm

9. The MDL: Roles and Facility

10. MDL Design Study Observations . “Typical” Studies

11. Analysis of the Design Process Part 2

12. The Design Structure Matrix (DSM) Task A depends on information from Task G Tasks D and E must be done concurrently

13. Design Process Analysis Series Series Coupled Coupled Phases of the Design Life Cycle Starting Assumptions Parallel

14. Modeling the MDL Design Process

15. Partitioning the DSM: The Conceptual Design Lifecycle Requirements Definition Phase Engineering Design Phase Integration Phase Maintenance and Support Phase Costing Phase

16. Critical Design Trades and Interdependent Disciplines 13 Core Loop Types

17. Tearing the DSM: Indentification of Starting Assumptions Tear the Design Budgets Power Budget Mass Budget Reliability Budget

18. The Torn DSM: MDL Process with Starting Assumptions Made Requirements and Assumptions Phase Orbit Determination Phase Sequential Engineering Design Phases Iterate Integration Phase Costing Phase

19. The Core of Interdependent Disciplines Flight Dynamics Location Mission Operations Avionics Location Communications Electrical Power Location Mechanical Thermal Ground Segment Loop Data Loop

20. Insights from DSM-Based Analysis The Design Structure Matrix is a powerful tool for describing and analyzing the space systems design process. (Results for your system may vary.)

21. A Model of Shared Knowledge Part 3

22. Mental Models of the System Mental Models “Mechanisms whereby humans are able to generate descriptions of system purpose and form, explanations of system functioning and observed system states, and predictions of future system states”* * Rouse, W.B. and N.M. Morris (1986). “On Looking Into the Black Box: Prospects and Limits in the Search for Mental Models.” Psychological Bulletin 100(3): 349–363. Shared Mental Model (SMM) SMM Condition in which two people utilize the same underlying mechanisms or at least utilize mechanisms that lead to similar descriptions, explanations, and predictions Team Member Team Member

23. Measuring Mental Models • Survey Question on Major Design Drivers

24. Measuring Shared Mental Models Mental Model Sharedness, Sx,y, is defined as: Ratio of common choices to total choices Dx = # of drivers selected by person x Dy = # of drivers selected by person y Dx,y = # of drivers selected by both x and y Sx,y Team Member y Team Member x

25. Social Network Analysis A set of tools and techniques for analyzing a large group of entities (nodes) and the structure of interactions and/or relationships among them (edges). Node Edge Node = Design Team Member x or y Edge = Shared Mental Model between x and y • Edgeweight = Value of Sharedness, Sx,y

26. Dynamics of Shared Knowledge Post-Session Pre-Session CSMM = structural similarity (edge-by-edge correlation) Change in Shared Knowledge 

27. Dynamics of Shared Knowledge: Relationship to System Attributes

28. Integrated Analysis: People and Process Part 4

29. Content of Shared Knowledge: Perceived Importance of Drivers IP,Comm = proportion of team checking Communications

30. The Communications Subsystem: An Indicator of Shared Knowledge Recall the Central Role of Communications in the Design Process

31. Measuring Team Coordination • Expected Interaction Matrix • Based on Core Loop Types in • the Partitioned DSM • Actual Interaction Matrix • Based on Survey Data of • Interactions for Each Study • (Study 3 Shown Here)

32. Socio-Technical Congruence • Congruence Matrix • Overlay of Expected and • Actual Interactions • N#=number of # cells • Nb=number of blank cells • N =total number of cells

33. Dynamics of Shared Knowledge: Relationship to Team Coordination

34. The Typical MDL Process: Recommendations in Discussion People Process Tools Period of learning and consensus building Resolve orbit determination trades DSM-based process automation software Determine starting assumptions Facility Sub-teams based on interdependent disciplines Lab layout based on interdependent disciplines Design sequentially… then iterate Proposed Standard Design Process Model under Development in Conjunction with the MDL

36. The People Behind This Work Annalisa Weigel, MIT, Thesis Advisor NASA Graduate Student Researchers Program (GSRP) Deborah Amato, Former IDC Systems Engineer Jennifer Bracken, IDC Systems Engineer Tammy Brown, IDL Team Lead Bruce Campbell, IDC Manager Anel Flores, MDL Systems Engineer Gabriel Karpati, Former IDC Systems Engineer John Martin, MDL Team Lead Mark Steiner, SESAC Branch Head IDC Support Staff: Felicia Buchanan-Jones, Dawn Daelemans, Elfrieda Harris, Erica Robinson, Ed Young 12 MDL Customer Teams And, of course, the MDL engineers, whose sustained participation made this work possible.

37. Thank You

38. Backup

39. Building the DSM for the MDL Although collocation accelerates the pace of design activity, it also presents an obstacle to formal analysis and process improvement. DSM construction must account for this. • Parameter-Based DSM • Steps of DSM Construction in the MDL • Preliminary Interviews • Surveys on Design Sessions • Structured Interviews • Verification and Validation • Guiding Principles for DSM Construction in the MDL • Document maximal flow for a typical design session • Include only deliberate and purposeful information flow • Abstract two-way negotiation-type interactions

40. Data Collection on Mental Models • Survey Data on Major Design Drivers • Team members indicate whether each of a set of issues drives the ultimate design. • Simple Example with Only Four Possible Drivers • Cost • Schedule • Performance • Science 24 = 16 Possible Mental Models

41. Filtering Out Random Responses: A Cutoff For Shared Mental Models SMMx,y= 0 SMMx,y ≥ 1 x and y do not share mental models to any greater extent than two people with no prior knowledge of the task answering at random 35 Possible SMMs

42. Quantifying Shared Knowledge: Edge Weights in a Social Network • Surveys Distributed: 20 Drivers and 1,771 Possible SMMs • Network Edge Weights on a 1-4 Scale • Time Dependence of Shared Knowledge • 12 Design Sessions Observed • Pre- and Post-Session Data Collected for Each .

43. Dynamics of Shared Knowledge: Relationship to System Attributes

44. Propulsion Subsystem and Mission Type

45. Team Coordination and Shared Knowledge in the Team

46. Proposed Standard Design Process Model

47. Contributions to the Research • Product Development • Guiding Principles for Building a Design Structure Matrix in a Rapid Collaborative Design Environment • Method for Converting a Parameter- to a Team-Based DSM • Cross-Functional Teams and Shared Mental Models • Scalable Network Model of Shared Knowledge in Engineering Design • Metric that Captures Dynamics of Shared Knowledge • Systems Engineering and Space Systems Design • System-Level Representation of the Entire Design Process • Analysis of the Role of People in the Process • Standardized Design Process Based on Both of the Above • Explicit Connection between Organizational/Social Psychology and Systems Engineering Best Practices

48. Future Work Apply Methods to the Instrument Design Laboratory and to Other Similar Design Centers; Apply Both DSM and SMM Work to Longer Development Programs Build DSM with Types and Strengths of Dependencies Time Series Analysis – 1 to 2 Surveys Each Day Tracking the Evolution of SMMs Over Time Measure SMMs Based on Other Forms of Knowledge in Addition to Task – Team, Process, Context, Competence Network Analysis of Design Sessions Experimental Approach with a Learning Period Structured in Various Ways and Several Combinations of Number and Length of Design Iterations