Tradeoff: Proposal Evaluations for Task/Delivery Orders

1. Tradeoff: Proposal Evaluations for Task/Delivery Orders

2. Agenda • Importance of Proposal Evaluation Process - Guiding Principles • - Tailored to the Procurement • - Highlight Important Evaluation Criteria of the Acquisition • - Demonstrate Difference Between Contractors • - Allow Government to Achieve Overall Objective • Keys to the Evaluation of Proposals • - Rating Scale • - Definitions • - Selecting Evaluators • How to Evaluate Proposals • - Initial Proposal Evaluation • - Final Consolidation & Recommendations • - Tradeoff Discussions • - Justification • Questions & Answers - TMA Guide - TMA Templates - Regulatory Guidance/Best Practices • References

3. Importance of Proposal Evaluation Process • Provides a standardized process to determine which offers submitted in response to a solicitation best meet the Government’s stated needs. • Proposal evaluation results in an assessment of the offeror’s ability to successfully accomplish the requirement. • Source selection decision is based on the proposal evaluation; therefore, it is important that the evaluation criteria: - Reflect the Government’s need and facilitate preparation of proposals that best satisfy the requirement as stated in the Performance Work Statement (PWS). - Provide for an accurate evaluation of an offeror’s proposal. - Represent key areas of importance and emphasis to be considered in the source selection decision. - Support meaningful discrimination and comparison among competing proposals.

4. Impact Statement • Failure to take the Source Selection Process seriously leads to: - An inaccurate assessment of an offeror’s ability to meet the solicitation requirements. - Inability to differentiate between offerors due to having too many evaluation factors which can lead to a leveling of ratings or having narrowly defined evaluation criteria which eliminates the evaluator’s flexibility and decision- making capabilities. - Increase the Government’s risk of not awarding the most capable offeror, as well as, not receiving the best value. - Contractors may appeal to a task/delivery order ombudsman to review their complaints and ensure they were afforded a fair opportunity to be considered, consistent with the procedures in the contract. - No protest under Subpart 33.1 is authorized in connection with the issuance or proposed issuance of an order under a task-order contract or delivery-order contract, except for a protest on the grounds that the order increases the scope, period, or maximum value of the contract.

5. Guiding Principles • Evaluation Criteria must be: - Tailored to each procurement. - Developed to highlight the important components of the acquisition. - Designed to demonstrate difference between contractors. - Developed to allow the government to achieve its overall objective. • Can’t cookie cut the evaluation criteria. • May “plug and play” evaluation criteria to include, but are not • limited to, Management Approach, Technical Approach, Key • Personnel, Experience, & Past Performance.

6. Tailored to the Procurement • Relevant to the specific requirement. • Consistent with the requirement. • Measurable qualitatively or quantitatively. • Limited to the number of factors that are most likely to reveal relevant, substantive difference between offeror proposals. • TMA has templates, however, use what is important to you.

7. Highlight Important Evaluation Criteria of the Acquisition If Technical Approach is important, then that evaluation factor becomes your most highly rated evaluation factor and cost/price is not, then…Technical Approach is significantly more important than the quality factors. When combined, Management Approach, Cost/Price, Past Performance, and Key Personnel are less important to Technical Approach. If Management Approach is important, then that evaluation factor becomes your most highly rated evaluation factor and cost/price is not, then…Management Approach is significantly more important than the quality factors. When combined, Technical Approach, Cost/Price, Past Performance, and Key Personnel are less important to Management Approach. If Key Personnel are important, then that evaluation factor becomes your most highly rated evaluation factor and cost/price is not, then…Key Personnel are significantly more important than the quality factors. When combined, Technical Approach, Management Approach, Cost/Price and Past Performance are less important to Key Personnel. In a tradeoff process, all evaluation factors and significant subfactors that will affect contract award and their relative importance shall be clearly stated in the solicitation; and the solicitation shall state whether all evaluation factors other than cost or price, when combined, are significantly more important than, approximately equal to, or significantly less important than cost or price.

8. Demonstrate Difference Between Contractors • Evaluation factors and subfactors must be discriminating to be effective. • Evaluation factors and subfactors should result in different ratings for competing proposals. • Evaluation factors, subfactors, and the resulting evaluation should reflect those areas that best demonstrate offerors’ understanding of, and compliance with, the solicitation.

9. Allow Government to Achieve Overall Objective • In a tradeoff evaluation, Best Value means the award of an acquisition that provides the greatest overall benefit in response to the requirement. • Minimizes risk. • Awarding to the most advantageous offeror who meets the Government’s requirements and objectives as identified in the solicitation. • Best Value can be determined by using one of two distinct processes: Tradeoff: the Government will consider award to other than lowest priced offeror or other than the highest technically rated offer. • Lowest Price, Technically Acceptable: • The award will be made to the offeror whose price is lowest • among all proposals that were deemed to be technically acceptable. The goal of source selection is to award an order to the offeror whose proposal represents the best value to the Government.

10. Evaluation Rating Scale(Example) • In some instances, evaluators of a proposal will use a rating scale as • depicted below to evaluate how well each proposal meets the evaluation criteria: Acceptable is sufficient per the PWS for the average contractor.

11. Evaluation Definitions Regardless of the approach selected, narrative documentation must be developed to explain the basis for each rating, and identify strengths, weaknesses, and deficiencies.

12. Selecting Evaluators • A key to a successful proposal evaluation process is the selection of the evaluators. • Inherent in the Source Selection Process, the evaluators must have the requisite knowledge, skills, and abilities, to apply sound business judgment when evaluating offeror’s proposals. • Evaluators may be analysts, technical experts, contracting personnel, resource managers, and of the like. • Evaluators will rate the proposal contents only, any personal opinions must be removed, and any outside knowledge must be documented in past performance. Role of Contractors in the Evaluation Process: - Contractors are not authorized to be voting members of the evaluation panel. - Contractors may be used to support the panel. - Any contractor participants must sign the Non-Disclosure Agreement for Contractor Employees And Subcontractors. - Whenever the Requiring Activity plans to use Contractors in any capacity during the evaluation, the acquisition package must specify that they will be used and must identify the specific company name and the specific role that the Contractor will perform. Failure to include this information in the acquisition package will preclude the Contractor from any exposure to the proposal or evaluation deliberations.

13. Initial Proposal Evaluations:Evaluate Proposals Against PWS Requirements • Evaluate proposals against the PWS requirements, not against one another, to avoid technical transfusion. • Ensure that it is clear how your comment relates to the evaluation factor and your evaluation rating. • Example: What NOT to write! • Offeror #5’s approach is not nearly as effective as Offeror #2. • Offeror #4 is equally as capable as Offeror #3. • This proposal was significantly stronger than all the others. • Example: What TO write! • Offeror #5’s approach does not meet CMM Level 2 technical requirements as defined in Section C.4.2. • Offeror #2 offers their employees 2 weeks of paid time off to attend training sessions every year. This encourages retention and staff growth.

14. Documenting the Findings: Do’s and Don’ts • DO ensure that it is clear how your comment relates to the evaluation factor. • Examples: What TO write! • Offeror #2, page 7, para 3. The • Offeror does not appear to • understand the direction of the • program nor the intent of the • PWS and has specified a CMM • approach which has been • proven unsuccessful to this • program in the past because… • Examples: What NOT to write! • Offeror #2, page 7, not where • we are going. So what? What is the impact? Ensure that your comments address whether or not the offeror’s proposal adds value or will meet the Government’s concerns, even those unspecified in the solicitation.

15. Documenting the Findings: Do’s and Don’ts • DO provide detailed comments and accurate references. • Examples: What TO write! • The Key Personnel resume • presented by Offeror #3 offers • the Government a strong • technical staff member with 13 • years of healthcare IT • experience and 25 years of • management experience. • Examples: What NOT to write! • Offeror #3 offers great • resumes.

16. Initial Proposal Evaluation:Conduct Independent Evaluations • Proposals should be evaluated independently by each evaluator: • All voting members of the Technical Evaluation Panel (TEP) are considered equal members of the team. • Independence reduces the possibility of “group think” and improves the quality of the evaluation. • Do not discuss the merits of a proposal amongst each other until the reconciliation meeting. • After the individual evaluations are complete the TEP will be able to discuss their thoughts and opinions on each proposal during the reconciliation sessions.

17. Initial Proposal Evaluation:Be Considerate of Each Offeror’s Proposal • Each initial proposal is given the same consideration up front. • The name of the offeror should not influence (positively or negatively) the evaluator’s comments or ratings. - The evaluators will rate the proposal contents only, any personal opinions must be removed, and any outside knowledge must be documented in past performance. • Evaluate each offer as a standalone document.

18. Initial Proposal Evaluation:Ensure Comments Provide Brevity & Clarity • Provide detailed comments and accurate references. • Comments should be detailed enough that the Contract Specialist and KO are able to determine the evaluators intent without having to contact them. (See Example #1). • Qualify your evaluation and ensure consistency between the ratings and narrative statement. (See Example #1). • Comments should cross-map to the PWS and the offeror’s proposal. (See Example #2). Example #1: Incorrect - Offeror #3 offers great resumes’ (Too vague – How are they great?) Correct – The Key Personnel resume’ presented by Offeror #3 offers the Government a strong technical staff member with 13 years of TRICARE experience and 25 years of management experience. Example #2: Incorrect - Offeror #2’s approach to MCSE training is overly burdensome for the Government. Offeror #2 states, “XYZ Inc. proposes to examine the market to determine if the requirement for an MCSE training session is required. Once they have determine that their might be a need….” (Too wordy – use references) Correct – The approach presented in section 4.3.5 of Offeror #2’s proposal is too time consuming for the Government.

19. Documenting the Findings: Do’s and Don’ts • DO provide comments that are clear and “plainly” written. - Avoid using language that may be too “technical.” - Avoid using language that is jargon-laden and legalistic. - Avoid undefined or overused abbreviations and acronyms. The SQL scripts are non compliant with the RTM, and incompatible with Cold Fusion requirements. ABCs QSP is not stuff to meet rqmts of CMMI CMP or spt DPM or DPL.

20. Documenting the Findings: Do’s and Don’ts • DO cross-map comments to the PWS and the offeror’s proposal. • Examples: What TO write! • The training approach presented • in section 4.3.5 of Offeror #2’s • proposal relies on the • Government participants rather • than contractor responses as • required by section 4.2.7 of the • PWS. • Examples: What NOT to write! • Offeror #2’s approach to MCSE • training is overly burdensome • for the Government. Offeror #2 • states, “XYZ Inc. proposes to • examine the market to • determine if the requirement for • an MCSE training session is • required. Once they have • determined that there might be • a need…”

21. Initial Proposal Evaluation:Evaluate proposals fairly & consistently • Be fair and consistent in proposal evaluation. • If an item is a strength/weakness for one proposal it should be equivalently noted as a strength/weakness for other proposals. • Do not “take it easy” or be overly harsh. Fairly evaluate all proposals against the requirements of the PWS. • Assume nothing, evaluate each offer based upon what is actually contained in the written proposal. • Make sure the proposal details how the offeror plans to meet the stated requirements.

22. Documenting the Findings: Do’s and Don’ts • DONOT make assumptions. For instance, a proposal that states, “I will comply” with the requirements does not explain how they intend to ensure compliance with the requirements; it’s a marginal effort at best. • Describes “How” • “We will apply Microsoft excel based tools • to build a budget tracking tool for the • PMO. Once the budget is accurately • reflected in the tool, we will coordinate at • least weekly with the PMO budget • manager and key designated PMO staff to • identify planned and actual expenditures, • as well as obligations, commitments, and • document tracking numbers of related • PRWs, DRAMs, MIPRs, etc.” • Insufficient detail • We will assist the PMO in • conducting budget • management and tracking.”

23. Initial Proposal Evaluation:Ask Questions Throughout the Evaluation Process • Always ask yourself: How are they going to meet the requirements of the PWS? • Always ask yourself: If I was present at the debrief, would I be able to defend this assessment? - Ensure that your comments are concise, clear and professionally stated. - Antagonistic and inflammatory comments can lead to a protest.

24. Initial Proposal Evaluation:Documenting Findings & Disclosure of Documents • Do not write on the technical proposals. • All comments need to be entered into the Evaluation Tool. • You may take notes on scrap paper, but the notes should be discarded to the appropriate “proprietary” trash bins immediately upon completing evaluation of each proposal. • Note that in the case of a protest, all documentation, including the technical proposals and email messages, will be disclosed.

25. Documenting the Findings: Do’s and Don’ts ONLY rate evaluation factors based on criteria included in the PWS. If you do not, this mistake can lead to a protest.

26. Videotaping and Audiotaping Videotaping and Audiotaping Under NO CIRCUMSTANCES should evaluator discussions or deliberations be videotaped or audiotaped!

27. Final Consolidation and Recommendations • Following the detailed evaluation process the TEP will meet to finalize their consensus comments and ratings for each proposal. • Based upon the final consensus comments and ratings the TEP will meet to develop a list of recommended awardees. • Final recommendations will include both tech recommendations and cost tradeoff analysis. • The TEP will develop a document to justify these recommendations to the KO.

28. Tradeoff Discussions • Discuss the rationale for recommendation based on the factors • selected (strengths) and any price/cost trade-offs made if the lowest • price proposal was not recommended. When recommending a • proposal that is not lowest cost, describe WHY the added technical • capability outweighs the cost differential. • Example #1:“Vendor A was recommended over Vendors B and C, even though Vendor B and C offered prices lower than Vendor A, because the approach proposed by Vendor A will significantly reduce risk of performance impact to the system.” • Example #2:“Vendor A was recommended over Vendor B, even though Vendor B’s price was $X lower than Vendor A, because the skills and experience of the staff proposed by Vendor A were directly related to this project, while those proposed by Vendor B have never worked with the MHS.”

29. Justification • If only one offer is received, provided below is an example and/or • recommendation to fully support the narrative justification. “Vendor A’s technical proposal was deemed to be adequate because it describes a proven approach. The price proposal was in line with the IGCE. Therefore, award to Vendor A is recommended.” Vendor A’s technical proposal was deemed to be adequate because it describes a proven approach. However, the price proposal was 25% more than the IGCE. The TEP must provide negotiation points to the KO on level of effort (LOE), rates, non-value added scope.

30. Questions & Answers

31. BACKUP SLIDES

32. Evaluation Guide & Template • AM&S COD-FC has developed a guide to assist the requiring • activities in planning and conducting proposal • evaluations. • - Available: • http://tricare.mil/tma/ams/downloads/Eval_Guide.htm • - Many elements have been integrated into contract • user’s guides and templates.

33. Regulatory Guidance/Best Practices • Evaluation Guides, such as the one TMA has created, contain clear guidance regarding evaluation ratings and standards. • In accordance with the FAR, Army Material Command Contracting for Best Value – Best Practices Guide to Source Selection, and the Defense Procurement and Acquisition Policy, the adjectival ratings system is an effective method for proposal evaluation, allowing appropriate flexibility for evaluator discretion. • GAO Report B-296334, Matter of Trajen, Inc & Maytag Aircraft Corporation cites, “When considering the ratings assigned by an agency to an offeror’s proposal, we have consistently taken the position that evaluation ratings, be they adjectival, numerical or color, are merely guides for intelligent decision-making in the procurement process.” • The Department of Energy Office of Procurement & Assistance Management Acquisition Guide states, “What is key in using a rating system in proposal evaluations, is not the method or combination of methods used, but rather the consistency with which the selected method is applied to all competing proposals and the adequacy of the narrative used to support the rating.”

34. References Slide 3: Acquisition Guide, Chapter 15.3, Establishing Evaluation Criteria. http://professionals.pr.doe.gov/ma5/MA- Web.nsf/WebAttachments/AcqGuide15pt3/$File/AcqGuide15pt3.doc Slide 4: Acquisition Guide, Chapter 15.3, Establishing Evaluation Criteria. http://professionals.pr.doe.gov/ma5/MA- Web.nsf/WebAttachments/AcqGuide15pt3/$File/AcqGuide15pt3.doc Slide 5: HQ, USAMC, January 2003, page 14; Material Acquisition Management, ALM 31-0319-RB2, 16 June 2000. Slide 6: Material Acquisition Management, ALM 31-0319-RB2, 16 June 2000. Slide 7: FAR 15.101-1, Tradeoff Process. Slide 8: Material Acquisition Management, ALM 31-0319-RB2. Slide 9: TMA’s Proposal Evaluation Guide and FAR 15.101-1. Slide 11: TMA Seminar Briefing, Proposal Evaluation, October 2004. Slide 12: FAR 15.001, Definitions and TMA Seminar Briefing, Proposal Evaluation, October 2004. Slide 13: Acquisition Guide, Chapter 15.3; Army Source Selection Guide, June 2001, page 17. http://professionals.pr.doe.gov/ma5/MA- Web.nsf/WebAttachments/AcqGuide15pt3/$File/AcqGuide15pt3.doc Slides 13-26: TMA Source Selection Evaluation Board Briefing Slides and TMA Proposal Evaluation Guide, http://tricare.mil/tma/ams/downloads/Eval_Guide.htm Slide 27: TMA Source Selection Evaluation Briefing Slides Slide 28-29: TMA Proposal Evaluation Guide, http://tricare.mil/tma/ams/downloads/Eval_Guide.htm Slide 32 TMA Guide & Templates Slide 33: TMA Evaluation Guide, http://tricare.mil/tma/ams/downloads/Eval_Guide.htm FAR 15.304, Evaluation Factors and Significant Subfactors. Army Material Command Contracting for Best Value – Best Practices Guide to Source Selection Defense Procurement and Acquisition Policy GAO Report B-296334, Matter of Trajen, Inc & Maytag Aircraft Corporation The Department of Energy Office of Procurement & Assistance Management Acquisition Guide.