Surveillance Process for MRDOC

1. Surveillance Process for MRDOC Presentation to Project Managers’ Working Group August 7, 2000 Dennis Rohn (E&TS) and Gary Kelm (OSAT)

2. Agenda • Introduction • MSD Contract Management Approach • Surveillance Organization and Roles • Proposed Surveillance Process • Surveillance Classification Guidelines • OSAT/E&TS Organization and Estimated Staffing • Feedback

3. Introduction • MRDOC is a Performance Based Contract, but the government is still responsible for ensuring that contract and delivery order requirements are met. • The process of doing this is called “surveillance”, which involves monitoring and communicating with the contractor to assure requirements are properly interpreted and verifying they are satisfied. • A draft Surveillance Plan is currently part of MRDOC, but a more specific “Implementation Plan” was needed. • A team comprising Tom St. Onge, Joyce Wanhainen (MRDOC SEB Chair), Dennis Rohn and Gary Kelm developed a proposed surveillance process. • The team presented this process to Steve Simons (MRDOC COTR) and is now presenting it to the MSD Project Managers for feedback.

4. Project Surveillance Teams • Generate/Closeout SRFAs • Evaluate Deliverables • Interpret Requirements • Identify Risk Areas • Support ERB/FRB • Attend Maj Milestone Rev MSD Contract Management Approach • Microgravity Science Program • (Directorate Level) • Workforce Planning • Make-Buy Decisions • Inter-directorate Issue Resolution • MSD Management Team (MMT) • Information Exchange • Personnel Actions • Programmatic Strategy • Make-Buy Recommendations • ISO Management • Issue Resolution • MSD Change Board (MCB) • Approve Contract Changes • Define DO Allocations • Approve Delivery Orders • Approve Fee Requests • Issue Technical Direction • Approve Deliverables • MSD Surveillance Board (MSB) • SRFA Review & Closeouts • Identify Trends/Estab Metrics • Initiate Special Studies • Monitor/Assess System Reqts • & Program Risk • Recommend Actions to MMT • Support Project Surveil Teams

5. Surveillance Board • Participants: • COTR or Designee (1 or 2) • Exhibit Representatives (5) • E&TS System Engineer Representative (1) • OSAT Representative (1) • Business Management Representative (1) • Secretary (1) • Caucuses monthly (or as required). • Evaluates problems, issues and concerns (e.g., Surveillance RFAs); requests additional surveillance activities, as needed. • Reviews contractor metrics, trends and performance indicators and takes corrective or preventative action as required to reduce risk or bring contract into compliance.

6. Surveillance Teams Primary Goal: Ensure what the contractor proposes and builds meets contract requirements and customer needs. Teams are led by Project Managers. Team members support the PM and need to: • Become familiar with MRDOC, including SOWs, Data Item Deliverables and technical requirements (i.e., science, safety, integration). • Become thoroughly familiar with the MRDOC Exhibits and/or Delivery Orders (DOs) to be monitored. • Work closely with the Project Manager. • Create mutual understanding and agreement on what is needed and how requirements should be interpreted. • Conduct independent analyses, as needed or directed, to assure validity of contract deliverables. • Identify areas of risk or concern and report to the Project Manager. • Conduct audits or other assessments as directed by the Project Manager or Surveillance Board. • Evaluate contractor performance against established metrics. Suggest new metrics that might promote contractor performance. • Work closely with the contractor team. • Help contractor interpret and understand requirements. • Offer suggestions, where appropriate, to help contractor understand requirements. • Serve as a consultant: propose potential options or alternatives for contractor consideration, without explicitly telling the contractor how to do the job. • Participate in contract reviews, meetings, telecons, etc. • Monitor or witness contractor fabrication, assembly, test or other activities to assess progress and problems. • Utilize personal expertise or recruit other NASA personnel with relevant skills and experience to evaluate/approve DIDs, investigate problems, etc.

7. Surveillance Team Roles • What are our roles? • Extremely important! • Independent Verification and Validation • Requirements Interpretation (We are the Customer) • Providing Added-Value to the Contractor (they would like to know if they may not meet a requirement as early as possible to avoid loss of P-R-O-F-I-T) • Do’s • Attend Contractor meetings (you have the right!) that pertain to your area • Provide “advice”, add-value to their process (“consultant” model – they don’t have to follow our advice) • Report to Project Manager on any identified problems (generate Surveillance RFAs, when necessary) • Perform analysis and review contractor analysis for independent validation • Don’ts • Argue with the contractor • Provide technical direction (we have the right to provide Technical Direction, but this needs to be done through MRDOC Change Board and could involve $$$) • Perform contractor’s work – they are responsible – could be a problem later!

8. Surveillance is as Easy as ABC 3) The customer confirms the pre-coordinated design decision Customer Role 2) The builder formally nominates options, also known as design decision options, that they are willing to stand behind Works for Associate Role 1) The customer invites the builder to nominate options Works with Builder Role The associate (or surveillance team member) works here to make sure that the options offered up will meet the needs of the customer. All ideas only enter here - never from above Informally operated option generation and assessment {formally organized by the builder role] In which any and all may participate The three meeting purpose dimensions: (1) To generate and assess design options [formally organized but informally operated--no boss, tables-of-three, efforts]; (2); To present the results [by information briefings, lectures, conferences, and etc.] of the formally organized but informally operated option generation and assessment efforts [that include formally organized but informally operated "independent" assessments and reviews]; and (3) To confirm pre-published results and planned decisions

9. Project Team Project Issue Lead Project Contractor Microgravity Change Board (MCB) Microgravity Surveillance Board (MSB) GRC MSD Management Team (MMT) START Issue identified by team member Make Issue Known to contractor Clarification provided, issue goes away PM reviews for clarity return to originator for update and distribution Draft issue on RFA format, forward to PM for review Initiates investigation of issue and assesses risk (comparison to risk database). Reports status at monthly reviews. Update RFA and submit original to PM. Copies to PAM, CSE, Functional Mgr, contractor PM, MSB Review Monthly Issue reports for trends Cognizant of issues through monthly program discipline reviews Review contractor position and PM or designee make recommendation to issue. Continue discussions with contractor to try to resolve. Define contractor position and make recommendation to issue A MSD Surveillance Process

10. Project Team Project Issue Lead Project Contractor Microgravity Change Board (MCB) Microgravity Surveillance Board (MSB) GRC MSD Management Team (MMT) A Agreeable solution to all parties, resolution reached YES NO Supports in-house study. PM requests in-house alternative study, prepare results PM requests ERB/FRB to assess options and recommend approach ERB/FRB reports recommendation from review. B MSD Surveillance Process

11. Project Team Project Issue Lead Project Contractor Microgravity Change Board (MCB) Microgravity Surveillance Board (MSB) GRC MSD Management Team (MMT) B Is the Issue dismissed with no further action YES NO Is there a minority view, decision outside scope of project, increased risk, or resource constraints? Resolved by MMT YES NO Is there a cost, schedule, or performance impact requiring a contract mod., wavier, or task close-out? YES Submit RFA form to MCB for resolution Resolved by MCB. NO PM documents agreement on RFA and distributes copies to CAM, CSE, originator, functional Mangers, contractor PM, MSB Updates risk database as needed. Review closeout reports. END MSD Surveillance Process

12. SRFA Form

13. SRFA Form

14. Surveillance Activities Classification Guidelines • Class 1 (typical for FCF): >5 FTE E&TS/OSAT Support • Engineering review of contractor deliverables • Independent engineering analysis and assessments • System requirements assessments • Support PM/contractor TIMs • Major review support • Change board support • Class 2 (typical for Shuttle, Single-PI Experiments, Multi-PI Experiments, Exhibit 2, SAMS, Sounding Rockets): 2-5 FTE E&TS/OSAT Support • Engineering review of contractor deliverables • System requirements assessments • Support PM/contractor TIMs • Major review support • Change board support • Class 3 (typical for reflights, Gloveboxes, and PI Experiments for Multi-PI Experimental Apparatus): <1 FTE E&TS/OSAT Support • System requirements assessments • Major review support • Change board support

15. OSAT MSD EDAD/SED

16. OSAT Staffing

17. MSD FCF Project Projects OSAT One person from the team will be assigned to provide primary surveillance, and coordinate team surveillance activities for each project EDAD/SED Organization Supporting Surveillance

18. EDAD/SED Surveillance Staffing • Level of effort dictated by Make/Buy Committee at about 21-22 Civil Servants for FY01

19. EDAD/SED Work Authorization • BMS requirement is to have an Engineering Request • EDAD/SED responds with two additional forms that have to be filled out, one of which captures the monthly budget estimate. • Proposal is to have a single Engineering Request (see form) to cover all surveillance needs – one JO, one set of BMS forms • A Surveillance Engineering Support Request (see form) form has been developed to define what support is requested and when, for each project, including any special tasks • Information for the form is developed between the Project Manager/Requester and the Engineering Surveillance Lead (or Chief Systems Engineer – Microgravity)