APMP TASK FORCE

1. APMP TASK FORCE QUESTIONS AND RESPONSES By Industry and Government 26 FEBRUARY 2010 Chair – Quentin Redman qredman@raytheon.com

2. Results to Date • Identified focus area: 4-Step Process • Clarify • Define • Price • Substantiate • Completed first 2 steps • Plan Forward includes completion of final 2 steps • Drafted written report to Industry/Government Committee • Identified Next Steps Successful Capture Strategy: The Art of Winning Bruce Morton

3. 4-Step Process to Analyze • Clarify • Government and industry need to understand request and submittal requirements • Define • Industry needs to ensure that cost development team understands what is needed for submission. • Government needs to ensure that cost evaluation team understands what will be submitted and how to review it. • Price • Industry develops pricing. Government prepares internal estimate. • Substantiate • Industry and Government should provide detailed assumptions to support costs Successful Capture Strategy: The Art of Winning Bruce Morton

4. 1. Clarify • Issued examples of instructions from various government agencies to compare and contrast clarity of requirements • Provided input on comments focused on: • Provide clear instruction to bidders and obtain better substantiation on pricing • Ensure that bidders understand and provide what is required Successful Capture Strategy: The Art of Winning Bruce Morton

5. 2. Define • Industry representatives • Provide feedback as to what is being requested (e.g., what would you anticipate providing based on the Government’s request? How much information and in what format?) • Government representatives • Provide input as to what is expected based on guidance provided (what are you expecting to see submitted? How much detail and in what format?) • Is further definition needed? What questions should industry ask to obtain better definition? Successful Capture Strategy: The Art of Winning Bruce Morton

6. 2. Define (cont’d) • Complexity of Request for Proposal/Bid • To what level of a Work Breakdown Structure (WBS) does the customer want the pricing? • What type of rationale is requested? • How much detail? (e.g., historical data, estimating methodologies) • Is an outline provided? If not, do we have one to understand the level of detail? • Do we have a sample template? Something that has worked before? Successful Capture Strategy: The Art of Winning Bruce Morton

7. 2. Define (cont’d) • Cost Proposal Response • Involve your cost proposal team at the start of the proposal process • Have cost team read/review the technical approach often • Are you going to be using subcontractors? • Recognize the time it takes to obtain pricing and justification from subs • Allow ample time for your team to review their submission, ask clarifying questions, and iterate/modify to provide most efficient price Successful Capture Strategy: The Art of Winning Bruce Morton

8. 3. Price - Path Forward • Industry • How do you determine the price you will propose? • What type of estimate do you prepare? • How far down in the WBS do you prepare it? • Government • When preparing your internal estimate, what is your technical baseline? • How detailed are your estimates, e.g., level of WBS? What are your estimating methodologies? What is your historical data? • Would you consider providing this information to industry as part of the solicitation request, especially if there is an incumbent? • Why discrepancies might exist • What might industry do to provide a realistic cost proposal? • What might government do to receive a realistic cost proposal? Successful Capture Strategy: The Art of Winning Bruce Morton

9. 4. Substantiate - Path Forward • Industry • What types of assumptions do you make to support your estimate? • Do you clearly define these as assumptions? • Do you desire more information from the Government? • How much detail do you provide to substantiate your price? Do you prepare detailed justifications? • Government • How much detail is needed to substantiate bidders’ cost? • Is there a format that you might like to see this information in so that you can compare apples-to-apples? • Cost Proposal Substantiation • What might industry do to provide more consistent substantiation? • What might the Government ask for to ensure that it gets the level of detail it is seeking? Successful Capture Strategy: The Art of Winning Bruce Morton

10. Next Steps • Complete Steps 3 and 4 of the 4-Step Process • Clarify • Define • Price • Substantiate • Probe further into causes for differences between Industry proposed costs and Government evaluated most probable costs • Understand differences in the approaches behind developing Industry cost proposals and performing Government cost evaluations • Identify contradictory assumptions Successful Capture Strategy: The Art of Winning Bruce Morton

11. INDUSTRY RESPONSES TO APMP TASK FORCEDEFINED QUESTIONS Provide feedback as to what is being requested – general overall feedback • It ia recommended that the customer should allow the bidder to prepare their BOEs and pricing in whatever manner they are comfortable, but require bidders to submit a data-driven estimated price, and then explain and substantiate any variances between the two. The data-driven estimated price should be at the same WBS level that the customer and bidder have data, typically at the CCDR-level (<=Level 4). • This recommendation is based on the best procurement practices that have observed over the last 30 years. Industry needs to achieve bottoms-up buy-in with top-down pricing targets. Therefore, they need to perform BOEs in the manner in which their management is comfortable. Industry is often “betting the company” on the bid, so they need to be comfortable with the method. At the same time, the Government customer has pricing expectations based on past experience. The industry bidder should be required to reconcile and credibly explain any differences between the two. In this way, practices that cause cost overruns such as pricing based on buy-in, over-optimism, and heroic assumptions can be identified and addressed early-on.

12. INDUSTRY RESPONSES TO APMP TASK FORCECOMPLEXITY OF REQUEST FOR PROPOSAL/BID 1. Complexity of Request for Proposal/Bid • –To what level of a Work Breakdown Structure (WBS) does the customer want the pricing? Less than or equal to WBS level 4, or the same level as the commodity types CCDR submissions • –What type of rationale is requested? Bidder BOE and data-driven analogous or parametric estimate, and reconciliation of the two • –How much detail? (e.g., historical data, estimating methodologies) Detailed assumptions into both estimating methods, description of historical data for data-driven estimates, detailed description of variances • –Is an outline provided? If not, do we have one to understand the level of detail? Government customer should provide outline for data-driven estimate should be provided: Product, Work, and Resource Breakdown structure. Reconciliation between estimates should be at the lowest level of data-driven estimate.-Do we have a sample template? Something that has worked before? No response

13. INDUSTRY RESPONSES TO APMP TASK FORCEPRICE – PATH FORWARD How do you determine the price you will propose? • A technical baseline based on engineering team’s initial proposed solution to meet the customer’s minimum set of requirements, and then the changes and excursions start. Technical solution excursions continue to the eleventh hour of the proposal due date. Typically the lead time to complete the estimate is the factor that eventually freezes the technical solution. Management will decide on the proposed solution to bid based on a judgment of the price-to-win target and customer flexibility – strategy changes with each bid, but a risk analysis of some type will be conducted so that management has a sense of how much they may be flexing submitted departmental estimates with price-to-win. Bid is often prepared parametrically and bottom-up BOEs follow. Reconciled estimates of labor hours and material dollars are then priced with an application of approved forward pricing rates. What type of estimate do you prepare? •  Parametric • Detailed bottoms up with validated history How far down in the WBS do you prepare it? • Hardware LRU level, typically L4 Why discrepancies might exist: What might industry do to provide a realistic cost proposal? • Temper over-optimism with unbiased, data-driven estimates. Use as many “should cost” estimating methods as possible, including analogy and parametrics. Acknowledge and highlight risks. What might government do to receive a realistic cost proposal? • Require at two estimates (bottom-up and data-driven parametrics or analogy) to be performed and differences reconciled.

14. GOVERNMENT RESPONSES TO APMP TASK FORCE DEFINED QUESTIONS Provide input as to what is expected based on guidance provided (what are you expecting to see submitted? How much detail and in what format?) • Evaluators would like to see estimates traceable back to a broad base of actual cost data from similar efforts. All too often, we see information provided as a basis of estimate (BOE) that has no context with regard to the overall data and performance of past programs. Analogies are offered up from multiple past programs at low levels of the WBS with limited justification. Evaluators would like to see the whole picture when it comes to cost history and actual data. • For example, if a subset of actual performance is used, e.g. “good” software development efforts, evaluators want to see rationale why the subset is better than the whole. They don’t want to see dozens of BOEs with identical or near-identical text simply because a data-set was used for a high-level estimate then apportioned down to a lower level WBS. Likewise, they don’t want to see multiple BOEs for a particular WBS that cites staffing levels based on the experience of a manager who worked similar efforts. In the end, evaluators want to see all of the data from programs that are used as a BOE. It provides context to the individual BOEs that allow evaluators to judge the “goodness” of an analogy. • The Joint Space Cost Council (JSCC) has been addressing acquisition in the satellite and launch industry. One of the guiding principles the have been working is "Improving BOE Guidelines". This initiative suggests that proposal BOEs should GENERALLY be prepared at the same level that program historical data (actual costs) are routinely provided to the government through contract data deliverables (i.e., Contractor Cost Data Reports). This puts the evaluator in a position where they can evaluate proposal BOEs against reliable historical data that tracks to actual total program costs. The data’s reliability is improved through the opportunity to validate estimates and data outside the source selection. This is beneficial because constraints in the source selection environment on discussions potentially limit the ability to achieve a mutual understanding of the data.

15. GOVERNMENT RESPONSES TO APMP TASK FORCE DEFINED QUESTIONS Is further definition needed? What questions should industry ask to obtain better definition? • All source selections for major systems acquisition have opportunities for contractor questions, clarifications, and comments. The draft RFP is made available to industry for comment well in advance of the RFPs release. Most Systems Centers begin with generic templates for Section L and tailor it to the acquisition program. Each question or comment from industry is addressed in writing. In addition, prior to release, there is pre proposal or bidders conference where industry and the evaluation team may discuss the RFP. The lead for the cost evaluation will usually step through Section L and Section M of the RFP for cost. They will explain the process, go over formats and templates, and answer questions related to the evaluation. • We recognize this is not always a perfect process. We recommend industry clearly document technical assumptions and risks driving the cost estimate within the cost section of the proposal. From an evaluation standpoint, if there is a disconnect in understanding and definition needs to be clarified, the evaluation team can issue an evaluation notice (EN). Also, if the government enters into discussion, the ENs can be addressed to provide further clarification.

16. GOVERNMENT RESPONSES TO APMP TASK FORCECOMPLEXITY OF REQUEST FOR PROPOSAL/BID To what level of a Work Breakdown Structure (WBS) does the customer want the pricing? • The government does not typically dictate a WBS level to industry. We look to industry provided an estimate at an appropriate WBS level that allows for an estimate that is justified with and traceable to a broad base of actual cost performance from similar efforts (See the answer to question 1). Ideally a contractor would be providing estimates at a WBS level consistent with a company’s Cost and Software Data Reporting (CSDR) level for similar commodities. In the end, the WBS will need to be Mil-Hdbk-881 Level 3 at a minimum. Though, it may require level 4 or lower depending on the WBS element and commodity be acquired. • It is in the interest of both the Government and Industry to build estimates at the highest level of WBS possible. This reduces the number of BOEs required to support a proposal and reduce the effort for both sides. Unfortunately, it seems industry feels the lower they can go into the WBS the more credible their estimate becomes. As the contractors get into lower WBSs, it becomes more difficult for evaluators to relate each BOE to historic cost data. However, credibility is only increased when an estimate is justified with and traceable to a broad base of actual cost performance for similar efforts.

17. GOVERNMENT RESPONSES TO APMP TASK FORCECOMPLEXITY OF REQUEST FOR PROPOSAL/BID What type of rationale is requested? • Section L of the RFP provides guidance for the supporting rational for each estimate. The specifics of what is requested vary throughout DoD. Overall however, evaluators want estimates to be justified with a broad base of actual cost data from similar programs or projects. Good estimates are data driven and can be justified by a company’s performance at any level of the WBS. Engineering judgment and management experience should be avoided as rational. If they are necessary, then an explanation of the logic used to make the judgment is required and at some level it should be justified with data. • In addition, material and sub contracted efforts make-up a large percentage of the overall effort for major DoD acquisitions. Quotes and fixed price contracts do not constitute substantiation for the cost of these items or efforts. Major sub contract efforts are held to the same justification standards as the prime contractor. While quotes and negotiated prices may be part of the cost analysis, it is incumbent upon the prime contractor to demonstrate that the individual prices and overall material costs are valid based on their performance in executing and managing similar procurement efforts. The expectation that a broad base of actual cost data from similar programs or projects will be used to justify a proposed price applies to material and sub-contracts.

18. GOVERNMENT RESPONSES TO APMP TASK FORCECOMPLEXITY OF REQUEST FOR PROPOSAL/BID How much detail? (e.g., historical data, estimating methodologies) • Contractors need to provide enough detail to allow evaluators to understand how the estimate was developed and to justify why the estimate is realistic and reasonable. Section L will give guidelines as to format, content, and what constitutes justification for the cost volumes. However, section L is not prescriptive as to how much detail is required. The burden of proof is on industry and they need to provide enough detail to justify their proposal consistent with the instructions in section L. • Industry should focus on the quality of the BOEs and ensure that they justify their proposed costs based a broad base of actual cost data from similar programs or projects. All too often, it appears industry takes a “volume is quality” approach the generating BOEs and breaks the estimate down into hundreds or thousands of small estimates. As stated previously (see question 1), this leaves evaluators with a lot of information to review that is at a level well below their ability to make a comparison to historic data. Do we have a sample template? Something that has worked before? • Section L will give guidelines as to format and may contain attachments that have templates that the contractor may choose to use for BOEs and number presentation. Often at the pre-proposal or bidders conference, the evaluation team will provide examples based on the templates provided in section L.

19. GOVERNMENT RESPONSES TO APMP TASK FORCE QUESTIONSPRICE-PATH FORWARD When preparing your internal estimate, what is your technical baseline? • The technical baseline typically comes from the government program office’s technical team. Ideally, this is the form of a Cost Analysis Requirements Document (CARD). However, the cost estimators often have to meet with the technical team to develop a complete understating of the system adequate to develop an estimate. It may be less formal for smaller acquisitions, but it still will come from interaction between the program office and cost analysts. Throughout the evaluation the technical descriptions are updated based on the proposal from each offeror. How detailed are your estimates, e.g., level of WBS? What are your estimating methodologies? • This is very dependent on the commodity, but WBS Level 2 is probably a minimum and levels 3 to 5 are common depending on element. The methods are primarily going to be analogies to similar efforts or parametric. Ideally, we are using a broad base of actual cost data from similar programs or projects. What is your historical data? • From an overall DoD perspective, we have access to Contract Cost Data Reports (CCDR) and Contractor Performance Report (CPR) form major (ACAT 1) acquisitions. At the System Center level, we have access to cost information from other estimating efforts and on-going projects in the form of contractor accounting data, contractor engineering data, and CPRs. Ideally, we would receive usable contractor specific data as part of the offeror’s proposal. Would you consider providing this information to industry as part of the solicitation request, especially if there is an incumbent? • Proprietary issues prohibit the sharing of cost and technical data as part of the solicitation. Besides, it is incumbent upon the offeror to justify their cost based on their demonstrated experience in developing and building products. • However, the Air Force and Navy Cost agencies have considered forming government and industry cost integrated product teams to meet regularly to discuss cost methodologies and share data. The idea of the CIPT would be to have general sessions with government and all industry reps, and then have side sessions to show specific companies the normalized data that the government has for that company and resolve any inconsistencies in normalization or understanding of data for that company. This could go a long way to ensure proposals and government evaluations use consistent and common data.

20. GOVERNMENT RESPONSES TO APMP TASK FORCE QUESTIONSPRICE-PATH FORWARD Why discrepancies might exist: What might industry do to provide a realistic cost proposal? • The theme of the answers for most of the previous questions would suggest that industry should endeavor to use a broad base of actual cost data from similar programs or projects to justify their proposed price as a means to provide realistic cost proposals. Not previously addressed, however, is industry’s motivation to provide “realistic” cost proposals. It appears that industry is attempting to put forward the lowest possible cost proposal vice a realist cost proposal. The defense industry’s performance on competitively awarded cost plus development efforts is terrible. • If industry were to move away from the “price to win” approach to cost proposals and endeavor to put forward genuinely “realistic” cost proposals, much of the previous discussion would be moot. It is amazing how much cost data is available and how “realistic” proposals become in a sole source fixed price environment. Industry knows how to put together a “realistic” cost proposal, but the questions is whether they really want to put one forward. What might government do to receive a realistic cost proposal? • The quest for realistic cost proposals and the discussion of the previous questions is not new. All DoD acquisition organizations have struggled to get realistic cost proposals out of industry. Most of these organizations have processes and templates for building solicitations that attempt to get realistic cost proposals. These processes and templates are born of the experience of hundreds of evaluations. Each organization feels it is clear in identifying what industry needs to do to produce a “realistic” cost proposal. • That being said, DoD could look across the acquisition organizations to be more consistent as a community in the evaluation of cost proposals. The adoption of a DoD wide standard for the evaluation process based on the community’s best of breed approach would give industry consistent solicitations for all DoD business. Consistency in evaluations across the services and systems centers would help industry develop proposal preparation processes that are acceptable in all DoD acquisitions. It would also focus process improvement discussions such as this one. • Finally, DoD needs to recognize that industry’s motivation is “price to win” and not “realism” in a competitive environment. An examination of the contracting process and incentives for industry to produce “realistic” proposals needs to be undertaken. Industry will only conduct business in a manner that is beneficial to their bottom line. Until it is beneficial to industry to be “realistic” they will continue to “price to win”.