Planetary Science Decadal Survey Mission Studies Lessons Learned Presentation to NASA HQ

1. Planetary Science Decadal Survey Mission Studies Lessons LearnedPresentation to NASA HQ 9/28/2011 Kim Reh, Chet Borden, Tom Spilker, John Elliott, Nathan Strange

2. Planetary Science Decadal Survey Mission Studies Lessons Learned: Introduction The Planetary Science Decadal Survey (PSDS) was conducted by the National Research Council to develop a comprehensive science and mission strategy for planetary science  A major element of this strategy was the conduct of a broad set of mission and technology studies to assess the feasibility, implementation cost, and science return from missions designed to address high priority science at destinations throughout the solar system Mission destinations included Inner Planets, Mars, Primitive Bodies, Outer Planets and Outer Planet Satellites With the support of NASA HQ, the NRC assigned each study and one of its NRC science panel members to a NASA center for study Study team results were reviewed by the relevant NASA HQ point of contact (POC), subjected to independent cost evaluations, and then passed on to the PSDS Steering Committee As a result of this work, many lessons were learned along the way that can benefit future efforts that require similar design team study support This PSDS Lessons Learned task was undertaken to identify those lessons learned and make them available for future similar efforts

3. Planetary Science Decadal Survey Mission Studies Lessons Learned: Study Approach • A questionnaire was developed to solicit inputs from study participants pertaining to the mission, technology, and architecture trade study process • The questionnaire focused on four areas associated with the PSDS mission studies: • Approach, Process, Results, and Products • A fifth area solicited more general comments • The questionnaire was sent to a broad set of study participants, including: • PSDS science panel study team members • science champions • study leads • center leads • center participants • NASA points of contact

4. Planetary Science Decadal Survey Mission Studies Lessons Learned: Study Approach (2) • Responses to the questionnaire were collected and results synthesized • In a few cases, respondents were individually interviewed and their input documented by a task team member • The broad range of recipients had different points of view of the process (e.g., panel members, study leads, NASA HQ) resulting in a wide range of responses • This was seen as a strength, allowing evaluation of the mission study process from multiple stakeholder perspectives • The team sent questionnaires to a total of 73 study participants at 27 different institutions • 24 participants responded, including a representative mix of respondents from all critical areas of the study process: team members from the primary lead study centers that performed studies (JPL, GSFC, and APL), science champions, PSDS panel leads, and NASA HQ POCs

5. Planetary Science Decadal Survey Mission Studies Lessons Learned: Responders to Questionnaire • 24 participants in the PSDS studies responded to the mission and trade studies lessons learned questionnaire • Responses received from participants representing the NRC science panels, NASA centers (primary study centers were JPL, GSFC, and APL) and NASA HQ • 7 responses received from PSDS science panel members • At least 3 responses from each NASA center and 2 from NASA HQ * includes science champions, one panel lead and one steering committee member

6. Planetary Science Decadal Survey Mission Studies Lessons Learned: Summary Findings Listing • Study implementation focused findings: • Mission point design studies were high quality • Architecture trade studies demonstrated high value to the science Panels • Panel science reps (and other scientists) working directly with the study teams was very effective • Schedule for mission studies was extremely tight and impacted scientists and studies • Panels desired additional mission studies • Technology approach was inconsistent between study centers and studies • Study process focused findings: • Rigid separation of the NASA Center and CATE study teams resulted in miscommunications that impacted the consistency of the CATE results with the center mission studies results • CML concept was useful, but it is not yet well understood among the scientific, engineering, and management communities, leading to non-uniformity of application • NASA Centers, NASA HQ, and the panels could have been better prepared for performing the PSDS mission concept design, technology, and architecture trade studies • Study guidance documentation findings: • The formal “Study Questionnaire” document was very helpful • NASA Study Ground Rules were viewed as helpful, though there were issues with specific aspects • Standard report formats were useful, but different report formats should be allowed • Study interface with NASA HQ findings: • NASA HQ planning for study support to the Decadal Survey should be more timely and more robust • NASA HQ’s contributions to mission studies were essential, but some questioned the extent of the oversight role • It appears that NASA HQ did not have a consistent and robust mission studies review process

7. Planetary Science Decadal Survey Mission Studies Lessons Learned: Summary Findings (1) 1. Science panels perceived the quality of the mission point design studies to be high among all the NASA study centers • Study center and some science panel participants identified differences in study center products, focusing on various aspects such as level of fidelity of the results, appearance of the final report document, and risk analysis • Perception of “appropriate level of fidelity” varies among centers • Some report documentation appeared consistent with a mission proposal effort • Capture and documentation of risks varied among center studies • Have all the study centers better understand the level of fidelity associated with each Concept Maturity Level (CML), and how it applies to the different types of studies 2. Architecture Trade Studies, based on JPL’s RMA process, were conducted for the first time as a formal part of the PSDS and they demonstrated high value to the science Panels • In several cases, the architecture trade studies identified promising architectures that led to point design studies • Trade space efforts varied considerably among NASA study centers, spanning scope and depth of the studies, products generated, and use of results • Quantitative analysis plays a critical role in trade space assessments; among the centers, the amount of analysis performed varied markedly • Since the science Panels and Steering Committee were highly supportive of the architectural trade space approach and results; continue using trade space evaluations in the next Decadal Survey • Prior to the next round of multi-center mission studies, NASA should educate those involved about the reasons for, general conduct of, and products and use of an architectural trade space evaluation

8. Planetary Science Decadal Survey Mission Studies Lessons Learned: Summary Findings (2) 3. Panel science representatives working directly with the study teams, and center science representatives working with the Panel science leads, was a very effective strategy • Scientists stated study results accurately reflected their Panels’ science priorities and inputs • Participants on all sides of studies thought the Panel/study team interface worked well • Retain the broadly-based science capability in future mission studies programs • Encourage participation from science Panel members beyond the science champion to better capture the trade space, including science options 4. The schedule for the mission study program was extremely tight, which impacted both the science panels’ deliberations and mission study center staffing due to heavy peak loads, and might have reduced the number of studies that could be conducted • Schedule compression was exacerbated by a late start without a concomitant delay in completion date • Some science teams were driven to prioritize mission discussions over science discussions • Some scientists wanted more time for a period of Panel science deliberations before the start of the mission studies program • Early briefings of “actionable items” were useful to the panels • Address scheduling needs early in the mission study process • More time in the schedule would provide sufficient time for NASA HQ review and comment on results of mission studies

9. Planetary Science Decadal Survey Mission Studies Lessons Learned: Summary Findings (3) • The desire for additional mission studies was common among almost all of the Panel respondents, especially for panels dealing with a host of potential destinations (e.g., the Satellites of the Giant Planets Panel and the Primitive Bodies Panel) • Panel members indicated that an insufficient number of architectural trades and mission point design studies were conducted • This shortfall impacted the PSDS results and long-term mission options • There were multiple suggestions that an NRC mid-term reassessment will be critical to address PSDS shortcomings resulting from too few mission studies • Provide the Survey with additional mission study resources and support for the range of studies needed • The primary concern about the technology approach was the lack of a consistent, systematic approach for incorporating technology push and pull into the studies • Technology assessment process was non-uniform among and even within centers, and depended mainly on the skills and knowledge base of the study participants • Limited infusion of available technologies for cost reduction, performance enhancement, etc. • Providing clear, consistent guidance on the role of technology in the mission studies would be helpful • Prior to the start of the study program the level of technology allowed in the studies should be agreed upon, and then applied uniformly • To aid in this effort, the NASA Office of the Chief Technologist could provide a database of technologies as input to the process

10. Planetary Science Decadal Survey Mission Studies Lessons Learned: Summary Findings (4) • Rigid separation of the NASA Center and CATE study teams resulted in miscommunications that impacted the consistency of the CATE results with the center mission studies results • There was a perception of inflated CATE “threat” costs • Although the CATE team could communicate with the NRC panels, the panels were not always cognizant of many of the study technical details and assumptions • Since CATE cost estimates were systematically higher than study team cost estimates, despite the conservatism of the NASA ground rules, this supports the notion of likely miscommunication between the CATE team and the study teams • PSDS may have placed too much faith in the veracity of ICE assumptions and accuracy of CATE results • The next Planetary Science Decadal Survey should include an independent cost estimate and technical evaluation reconciliation with the center study teams • Center mission study teams and the independent cost estimate and technical evaluation teams might use a similar and consistent cost estimating process in the future • Although there was a general opinion that the CML concept was useful, multiple comments suggest that it is not yet well understood among the scientific, engineering, and management communities, leading to non-uniformity of application • Consensus was that the CML concept should be a part of future Decadal Surveys, but that the concept needs further maturation and dissemination to the study centers by NASA HQ so it is more uniformly applied • NASA could standardize CML just as it has TRL • NASA HQ should take steps to ensure that future study participants understand the CML concept and how to apply it to future mission studies efforts

11. Planetary Science Decadal Survey Mission Studies Lessons Learned: Summary Findings (5) • NASA Centers, NASA HQ, and the panels could have been better prepared for performing the PSDS mission concept design, technology, and architecture trade studies • Examples of aspects of the study process that could have benefitted from better preparation include: • Study ground rules were not available early enough in the study effort • Study output expectations were not clearly communicated to the centers • Study results (i.e., fidelity and consistency with CML level) varied from center to center • Extent to which technology options were evaluated was not consistent between studies • Limited ability to run parallel studies at Centers • Better prepare NASA Centers and the panels for effectively performing future DS mission concept design, technology, and architecture trade studies • Examples of better preparation approaches include: • Do these studies every year to provide a database of mission concepts and a group of trained personnel at each center • Conduct pre-DS workshops – include NASA HQ, NASA Centers, DS Science panels. Individual workshops could focus on trade studies; DS study objectives, approach, constraints, products; etc. • The formal “Study Questionnaire” document was very helpful • Early definition of the science objectives and goals greatly helped with science traceability when evaluating mission options • Questionnaire was an excellent reference document which helped make the studies more efficient • Have Panels iterate with the study team when they generate the questionnaire to better aid study preparations and reduce rework PSPS MMR Aug 2011

12. Planetary Science Decadal Survey Mission Studies Lessons Learned: Summary Findings (6) 11. The NASA Study Ground Rules were overwhelmingly viewed as generally helpful to the study process and provided a uniform basis that helped foster consistency among mission studies, though there were issues with specific aspects • The ground rules were intentionally conservative (e.g., requiring a 50% cost reserve), provoking a largely negative response from study participants, who felt that some parts of the Ground Rules should be clarified or revised in the future • Launch vehicle descriptions too generic and did not accommodate the wide range of launch C3 • Clearer guidelines on cost caps were needed and costing guidelines and rules (such as launch timing constraints) prevented considering potential cost-saving options • Mass and power margin guidelines were inconsistent with Discovery and New Frontiers guidelines • Planetary Protection requirements went beyond the scope of typical pre-Phase A studies • NASA HQ planning for the ground rules should have started earlier • Retain the use of ground rules as a mechanism for providing a consistent basis for the studies but consult with centers and science panel leads to develop a clear and consistent set of ground rules

13. Planetary Science Decadal Survey Mission Studies Lessons Learned: Summary Findings (7) 12. Standard report formats are very useful for consistency among mission studies, but different report formats should be allowed for different study types • Though there were several different study types (i.e., point design mission studies, architecture trade studies, technology studies, and detailed studies), there was a single report format that each study team was directed to use • For many of the point design studies, the template requested information exceeding the depth of study requested by the science panels, and was inconsistent with the resources available • For the architecture trade studies and technology studies, there were significant inconsistencies between the structure and content of the study and the report template, requiring study teams to resort to ad hoc formatting, intended to be consistent with the format provided, but possibly resulting in gaps in reporting • Provide different templates for the different study types or, alternatively, provide standard tables and figures (as the minimum required) for the different study types • To allow for unique characteristics of a particular study, allow flexibility in the report formats, with approval from the NASA POC • NASA HQ planning for study support to the PSDS should be more timely and more robust • Some NASA HQ POCs felt that they needed to improve their planning for the next decadal survey • Scope of NASA planning could cover: scheduling, cost, agreement on rules of engagement for NASA centers and study teams, mission concept prioritization, NASA review of study products, timing of placing independent cost estimator on contract, and relationship and communications with NRC • NASA HQ should define and implement a better up-front planning process prior to initiating a mission studies program

14. Planetary Science Decadal Survey Mission Studies Lessons Learned: Summary Findings (8) • NASA HQ’s contributions to the mission studies were essential, but some questioned the extent of the oversight role • NASA HQ’s funding for the mission studies program enabled well-informed assessment by the PSDS Science Panels and Steering Committee, leading to the PSDS final report and recommendations • NASA HQ’s review process required a significant amount of time, which impacted the timely transmission of final report products to the science Panels as well as the number of studies that could be conducted in the constrained schedule • Some respondents felt that the science Panels should be fully independent, without the need for NASA HQ oversight • NASA HQ should continue funding mission study programs in future strategic planning efforts. This should include an appropriate mix of architecture trade studies, point design studies, technology studies and detailed studies • It appears that NASA HQ did not have a consistent and robust mission studies review process • NASA HQ review process was lengthy and not consistent between NASA POCs • Need advanced planning at HQ with clear direction to the study teams as to expectations

15. Planetary Science Decadal Survey Mission Studies Lessons Learned: Final Thoughts The results of this study identify a number of observations and potentially valuable recommendations for the study centers, NASA HQ, and possibly the NRC to consider in the conduct of future mission studies and Decadal Surveys.