Date: 25 Sep 12

1. UNCLASSIFIED Presented to: MRL Government / Industry Workshop MRL Lessons Learned Distribution authorized to DoD and DoD contractors only; Critical Technology; August 2001. Other requests for this document shall be referred to the U.S. Army RDECOM, ATTN: RDMR-SEP. Presented by: Steve Watts Team Leader Aviation and Missile Research, Development and Engineering Center Date: 25 Sep 12 UNCLASSIFIED

2. Task • Develop an initial list of lessons learned for implementing MRLs to be posted on the dodmrl.org website (Assigned at the 22 May 12 MRL WG meeting) • Assigned to the Army, Air Force, and Navy manufacturing representatives. • Army – Steve Watts • Air Force – Dave Karr • Navy – Brent Gordon • Presented results at MRL WG meeting (24 Jul 12) Note: special recognition to John Bibel, Navy for documenting his experience on a recent review.

3. Lessons LearnedSummary Prior to Review • Include SOW requirement for MRL assessment • Select SMEs • Educate Team • Assign Team Members responsibility • Coordinate with Project Office • Coordinate with Contractor • Offer MRL training to the contractor and their suppliers • Develop MRA Plan • Have Contractor provide Self-assessment prior to review During the Review • Kick-Off Briefing • Keep Meeting focused • Take control of Splinter Groups • Schedule Government Only end of day meetings • Keep Contractor informed of key issues • Have Team members document findings • Data Collection during review • Control the Exit Briefing After the Review • Schedule Team meeting • Develop Back-up Data File • Have Team review Final Report • Have Team present in briefing to Project Office

4. Lessons LearnedDetailed Prior to Review • Include SOW requirement for MRL assessment • To ensure adequate contractor support and avoid contractual issues, insert a requirement in the Statement of Work to conduct MRL assessments. Use Section 6 of the MRL Deskbook as a guide for what requirements to include. • Select SMEs • Each program has different technologies. Make sure you have personnel knowledgeable of these specific technologies. Also, ensure you have manufacturing, quality, and industrial base expertise. It is better if they have previous MRA experience and are from different organizations (broader perspectives). • Educate Team • Schedule a pre-review team meeting to explain MRL concepts and purpose. Have team read the MRL Deskbook and Matrix (you would be surprised how many have not). Explain team member roles & responsibilities, discuss the review schedule, and emphasize the need to document their findings. Go over the review process and expectations. Emphasize that guidance is a supplement and framework within which to apply their experience and knowledge during the assessment. Try to avoid the “absolute” in applying the criteria. Assessment is not a problem-solving event or a forum for issue abatement. • Assign Team Members responsibility • Assign the team members to evaluate specific Threads and/or Sub-Threads. You want to make sure each team member is aware of their responsibilities prior to the review. Share gov’t assignments with contractor to facilitate alignment with contractor team POCs. Also Involve local DCMA representatives and use them as a resource. Note: don’t allow individual agendas to crepe into the assessment. • Coordinate with Project Office • Need to have Program Manager (PM) buy-in and support. Ensure PM understands the purpose of the review. Discuss key issues that need to be addressed at the review. It is good to have some Project Office personnel on the team because they have history and internal knowledge of the program (but don’t overload the team with Project Office – you want some independent standpoint in the review).

5. Lessons LearnedDetailed Prior to Review (continued) • Coordinate with Contractor • Contractor should establish a POC responsible for supporting these reviews. Ensure they have adequately prepared for the review (schedule, agenda, overview briefing, plant tour, documentation, and space for splinter groups). Conduct regularly scheduled coordination telecoms with contractor POC as a run-up to the actual event to help with process education/clarification and to resolve/avoid issues. Helps build the “teamwork” relationships. • Offer MRL training to the contractor and their suppliers • Sometimes, contractors may not be familiar with MRLs and MRAs. The program office may provide them a one hour overview of MRLs and MRAs. A little training goes a long way towards fostering understanding and overcoming initial reluctance. Training can be done in person or even by phone or VTC. Contractors are also eligible to participate in AFIT SYS 213 class. • Develop MRA Plan • Develop a documented plan that describes purpose, approach, schedule, and criteria for the review. Also define the roles and responsibilities for the Team Lead, Team members, and contractor. Share draft with contractor POC for clarification and buy-in. • Have Contractor provide Self-assessment prior to review • This helps the contractor prepare and ensures the type of data needed at the review will be available. It also gives the team members information ahead of time so they are not seeing it for the first time at the review. This should be of same framework as described in www.dodmrl.org website.

6. Lessons LearnedDetailed During the Review • Kick-Off Briefing • Government should provide a short Kick-off briefing (approximately 15 minutes) that provides a brief introduction, purpose, and procedure for the review. Government should state the limitations of this review – that proprietary information will be protected, and that findings and recommendations resulting from this review are not considered contractual direction. • Keep Meeting focused • A program overview briefing and plant tour are necessary to orient the team. However, these should be limited to 1-2 hours each. It is good for the entire team to hear the same overview data, but this session should not prevent or limit detailed discussions in splinter groups. Tours should only address the products being evaluated, not an overall company tour. • Take control of Splinter Groups • Team members should control the splinter group sessions (working group vs. briefing format). Ask detailed in-depth questions and follow through (not just Yes/No questions). Ask for documentation or results of claims (trust but verify). Revisit the manufacturing floor as needed after the general tour. But at all times, maintain professional courtesy, remember we are invited guests in a contractor’s facility. • Schedule Government Only end of day meetings • Schedule team meetings at the end of each day to get status from team members, identify issues being discovered, and/or discuss issues with receiving adequate information. These should be Government only to allow team members to discuss concerns freely.

7. Lessons LearnedDetailed During the Review (continued) • Keep Contractor informed of key issues • Identify issues and risks to the contractor as you go. This will prevent surprises and discontent at the exit briefing. It also will allow the contractor the opportunity to provide additional information or clear up misunderstandings. • Have Team members document findings (real-time) • Team members need to document, elaborate, and justify their risks so they are defendable. Also, many reviews involve a number of site visits over a short period of time. Team members need to document their findings so this information is not lost or confused with other site visits. • Data Collection during the review • Collection of data during the review is important as you prepare for the exit briefing. However, since thumb drives were outlawed, the transfer of data at the contractor's facility is difficult. Past teams have tried two methods to address this. Either hook into the contractors e-mail system and transfer files that way, or use writable CDs. There's some pain to both methods. Either way, this needs to be worked out before the visits. Also note: some facilities require permission to take in laptops. • Control the Exit Briefing • This should be a short briefing. Inform the contractor that the purpose is to provide them the preliminary risks, but that these will not be final until the Final Report. This briefing should not turn into a debate over the risk or rating level. Inform the contractor that you will accept additional information if available. Some teams have taken the approach of only identifying the risk, stating that the rating level will be determined later.

8. Lessons LearnedDetailed After the Review • Schedule Team meeting • Schedule a team meeting at your home office after completing the site visits (may need to consider VTC if you used team members from other locations). Often there is limited time during the reviews for detailed discussions. Use this meeting to further discuss and understand the risks, receive feedback from other team members, and organize the data for preparation of the Final Report. • Develop Back-up Data File • A lot of data is evaluated during the review, and much of it is in isolated splinter groups. It is important for team members to adequately document their findings. For practical reasons, the Final Report will focus on highlights and the risks. The back file can be used to address questions (such as “did your team evaluate “xyz”?), or to provide further detail on identified risks. Have the team members provide a detailed write-up of their notes after returning from the site visit. Some have used the “MRL Sheets” to require members to address how a particular sub-thread does or does not support a particular rating. • Have Team review Final Report • Allow the team to review the Final Report for buy-in and to ensure that the risks identified are accurately explained. • Have Team present in briefing to Project Office • The review was performed by a team effort, and the briefing to the PM should also be a team effort. The team leader will need the support of team members to fully explain risks identified. Team members should be responsible for their findings and expert opinions.