Effective Use of Lessons Learned: A Key Element for Mission Success Presented by Paul S. Gill Manager, NASA Technical St

1. Effective Use of Lessons Learned:A Key Element for Mission SuccessPresented byPaul S. GillManager,NASA Technical Standards Program Office June 18, 2003 “Try to make original mistakes rather than repeating those of your predecessor.” Secretary of Defense Donald Rumsfeld

2. Background • Lessons Learned Are Crucial To The Agency’s Continued Success • They Capture The Knowledge And Experiences From NASA’s Programs And Projects • A Principal Component Of Any Organizational Culture Committed To Excellence • NASA Recognizes The Need To Systematically Document And Utilize The Knowledge Gained From Past Experiences. • NASA Policy And Guideline (NPG) 7120.5B, “NASA Program And Project Management Processes And Requirements” • NASA Policy Directive (NPD) 8730.3, “NASA’s Quality Management System Policy” • GAO-02-195 Revealed Fundamental Weaknesses In How NASA Performs The Collection And Sharing Of Lessons Learned Information. • Lessons Learned Not Routinely Identified, Collected Or Shared By Program And Project Managers • Lack Of Proper Review For Lessons Learned • Need For More Positive Lessons Learned (Generated Not As a Failure)

3. Approach Proposed Initiative Leverages State-of-the-art Information Technologies and Organizational Processes to Optimize the Capture, Management, and Dissemination of Lessons Learned • Develop and Initiate a Process to Facilitate the Identification and Input of Lessons Learned Information • Through the utilization of Chartered Technical Discipline Working Groups, Review and Categorize Submitted Lessons Learned Information • Consolidate and Link Lessons Learned Databases That May Exist at Different NASA Facilities and NASA Contractors • Utilize Automated Workflow Capability to Manage and Monitor Tasks • Facilitate the Dissemination of This Information Across a Broad Spectrum Via a Portal Within the Agency’s Knowledge Management Initiative, Thus Providing Programs and Projects With Advanced Search and Retrieval Mechanisms • Obtain Current Performance Metrics to Evaluate System Effectiveness

4. Generation and Communication of Knowledge Begins With Lessons Time Project 1 Project N Projects Generate and Acquire Lessons Approval Process Lessons Update/Create Standards Approval Process Communicate Knowledge – Infuse into Practice Standards and Practices Update/Create Training Gather Lessons at Control Gates Training

5. Lessons Learned Implementation Process PROGRAMS AND PROJECTS (NPG 7120.5) FORMULATION IMPLEMENTATION Lessons Learned Databases Generate and Communicate Lessons Learned Proposed Lesson ATP SRR PDR CDR DCR Post Flight FRR Screened And Approved Lessons CONCEPT DEVELOPMENT DESIGN PHASE DEVELOPMENT & OPERATIONS SYSTEMS ENGINEERING (NPG 71xx.x and SP-6105) Technical Discipline Working Group Review

6. Lessons Learned Closed-Loop Process Programs and Projects Generate & Identify Generate & Identify Lessons Learned Establish Feedback System Feedbacks Needed to Provide User Based Enhancements Document Lessons Learned Document Infuse the Lessons Learned Into the Engineering Process Categorize Categorize Into Discipline Categories Infuse Into Engineering Process GENERATE KNOWLEDGE Characterize into Programmatic Versus Technical Link the Lessons Learned to System Development Lifecycle Requirements COMMUNICATE KNOWLEDGE Characterize Link To Requirements Lessons Learned Validity is Determined by the Working Group Evaluate

7. Key Agency Roles Success of a Lessons Learned Program Will Require the Support and Participation of the Following: • Office of the Chief Engineer (Code AE) • Overall Management and Oversight Responsibility (i.e. NPD, NPG, and Program Plan) • Charter the Technical Discipline Working Groups • Chief Information Officer (CIO) • Responsible for Agency Information Technology Infrastructure • Safety and Mission Assurance Office (Code Q) • Responsible for the Collection and Archiving of Mishap Reports and Failure Investigation Reports • Enterprises/NASA Centers/Agency Contractors • Source and Users of the Lessons Learned Information • Membership in Technical Discipline Working Groups • Lesson Learned Program Office • Agency Lessons Learned Working Group • Lessons Learned Information System • Engineering Discipline Working Groups • Responsible for the Collection, Analysis, and Dissemination of Lessons Learned Information • Training (Code FT) • Advanced Program and Project Leadership (APPL) • NASA Engineering Training (NET)

8. Programs and Projects Generate and Use Knowledge NASA Technical Discipline Working Groups Acquire Lessons (Develop, Review, Screen) Place Lesson in Discipline Taxonomy Update/create Standards to Incorporate Lessons NASA Lessons Learned Program Facilitate Capture and Use of Lessons Learned Provide On-line Access to Lessons Learned System NASA Technical Standards Program Identify Lessons for Potential Conversion Into Technical Standard Link Lesson to Relevant NASA Preferred Technical Standards NASA Training Incorporate Lessons and Standards Into Training Materials Provide Training on Use of Lessons Learned System Key Program Roles

9. Lesson Learned Program Proposed Organizational Structure Office of the NASA Chief Engineer Theron Bradley Lessons Learned Executive Gregory Robinson NASA Engineering Management Board Lessons Learned Program Office NASA Lessons Learned Working Group (Senior Representative From Each Center/JPL) NASA Engineering Discipline Working Groups Lessons Learned Systems

10. Proposed Tasks Plans Initiated in FY03: • Enhance Existing Taxonomy of Categories/Subcategories (Technical and Programmatic) • Categorize Existing Lessons Learned Databases or Raster Images (PDF, etc.) • Identify NASA Failure Mishap Reports • Initiate the Development of a Process for Formulating Discipline Working Groups for Screening, Validating and Categorizing Existing Lessons Learned and to Facilitate Searching, and Identifying New Lessons Learned • Initiate and Develop a Common Lessons Learned Process and Format • Lessons Learned To be Infused Into the Agency's Programs and Projects at the Appropriate Stages During the Lifecycle • Inputs From Program and Project Managers Will Help Define the Requirements to Implement Lessons Learned Into Product Lifecycle • Define Agencywide System Architecture

11. Proposed Tasks (Cont’d) Plans for FY04 Through FY08: • Establish an Agency Lessons Learned Working Group • Comprised of Senior Representatives From All NASA Field Facilities and Enterprises • Establish Agencywide Technical Discipline Working Groups • Catalyst for Infusion of Lessons Learned into Product Life Cycle • Working Groups Defined By The Ten Discipline Categories Established by the NASA Technical Standards Program • Develop, Review, and Screen Lessons Learned for Endorsement by the Agency • Working Groups Provide Agencywide Expertise in Specific Technical Areas DISCIPLINE WORKING GROUPS ARE THE KEY TO SUCCESS

12. Proposed Tasks (Cont’d) Plans for FY04 Through FY08 (Cont’d) • Implementation and Maintenance of the Information System Architecture • Including Advanced Search Engine That Meets System Requirements • Promote the Agency Lessons Learned Initiative Through an Agencywide Awareness and Training Initiative • Designed to Reach All NASA Employees and Supporting Contractors • Ensure Lessons Learned Are Available From a User Friendly Website • Develop Metrics to Ascertain the Effectiveness of the System Usage at the Agency Level • Establish a Feedback System for Inputs Needed to Provide User Based Enhancements

13. Deliverables The Anticipated Products of the Proposed Lessons Learned Initiative Are Contained in the Following List: • NASA Lessons Learned Working Group • NASA Engineering Discipline Working Groups • NASA Lessons Learned Program Website for Agency Wide Use With Timely Feedback Response System • Including Integration of Legacy Lessons Learned Systems • Common Lessons Learned Format • Awareness Initiative • Metrics on Usage of Lessons Learned • A Process Whereby Lessons Learned May Be Linked to the Applicable Period/Phase in the Lifecycle of a Program or Project • Periodic Brief to the NASA Chief Engineer’s Office

14. Challenges to Overcome • Establishment of Technical Discipline Working Groups • Chartered by Code AE • Identify, Categorize, and Promote Incorporation and Use of Lessons Learned • Resources Required (Dollars and Manpower) • Standardization • System Requirements of Standardized Electronic Templates (Data Format Standards) to Facilitate the Capture of Metadata and Integration of Legacy Lessons Learned • Development of Large–Scale Advanced Technology Information System • Get It To the People Agencywide • Education and Awareness on Value of Generating and Using Lessons Learned • Develop Personalization Profile • Infuse Into Program/Project Execution • Oversight of Use in Programs/Projects

15. Accomplishments To Date • Lessons Learned /Best Practices Identified • Over 1,700 Lessons Learned/Best Practices Accessible on the NASA Technical Standards Program Website • Over 245 Preferred Technical Standards Are Linked to 477 Unique Lessons Learned • Lessons Learned Sources Have Been Identified From Databases, Technical Papers and Memorandums, Mishap Reports, NASA Special Publications (Monographs) • Lessons Learned Search Screen Capability on NASA Technical Standards Program Website • Over 1 Million Keywords Contained in the Database • Enhanced Search Permits Users to Search by • Multiple Keywords • Phrases • Functional Categories • Organizations • Project • Date

16. Accomplishments To Date (Cont’d) Correlation of Lessons Learned With NASA Technical Standards Taxonomy

17. Summary • Lessons Learned Are Necessary to Preserve and Enhance the Agency’s Corporate Knowledge • The Effort Must Be Optimized to Meet Users Needs and Fully Take Advantage of Available Information Technology Assets • Success of This Effort Will Be Highly Dependent on Buy-in From All Enterprises/centers • Agency Level Initiative, Including NASA Contractors • NASA Technical Standards Program Office Is Best Positioned to Launch and Lead Effort • Has Effectively Implemented Agency-wide Working Groups • Very Successful Linking Lessons Learned to Technical Standards • Has Understanding of Task/Challenges – Similar to Technical Standards Program, But of Greater Magnitude • A Must Do Task

18. This Is Our Opportunity To Make Lessons Learned A Viable Element of the NASA Culture!

19. Backup

20. Interactive, Web-Based Documents NPG 71xx.x Systems Engineering Processes and Requirements Concept Development Prod’n Oper’n Retire NPG 7120.5B Program/Project Mgmt. Processes and Requirements • Capture and Deliver Lessons Learned in their Context of Use • Advanced Search Based on Discipline, Activity, Phase, Etc. Standards Lessons Learned

21. Lesson Learned Work Flow Start Observation of Lesson Learned/Best Practice (LL/BP) Submit LL/BP to Lessons Learned (LL) Program Office LL Program Office Initial Screening • Insufficient Information • Not Technically Accurate • Duplication • Proprietary Information • ITAR/EAR Information Lesson May Be Sent Back to Author for Editing or Removed From System No Valid? Metrics, Feedback, and Analysis Yes Submit to Technical Discipline Working Group (TDWG) for Review of Value to NASA Users Search for Lessons Learned No Approved and Recommend Development/Change of a NASA Standard based on LL/.BP? Lesson May Be Sent Back to Author for Editing or Removed From System Disseminate the Lessons through the Application Server NASA Lessons Learned Databases Yes Review Existing Standards for Applicability

22. “Vee” Model Of System Development Concept Identification and Integration of Technical Standards and Lessons Learned into Appropriate Stages of the Product Life Cycle User Requirements (ICD) & Concept of Operations System Demonstration, Verification,and Validation System Requirements & Architecture System Integration and Test Refine Design (SRR, PDR, CDR) Component Integration and Test Build and Deliver Increments

23. A Vision Enabled by Integrated Shared Knowledge • Integrated Access to NASA Knowledge • Knowledge-based Search Through Lessons and Standards • Discover and Link Related Knowledge • Prompt Potential Users That Knowledge Is Available • Generate New Knowledge as Part of Project Life Cycle • Knowledge Metadata • Document Content and Structure • Taxonomy and Ontology • Technical Discipline and Discipline Ontology • Etc. • Interactive, Web-based Documents • Deliver Relevant Standards and Lessons Learned in the Context of Use • Gather New Lessons at Critical Project Gates and Engineering Milestones Advanced Common Interface XML-Based Data Exchange In A Common Format Legacy Lessons Learned Legacy Lessons Learned Agency- wide Lessons Learned NASA Standards and Practices External Standards