Breakout Session #: G02 Presented by: Jacqueline Owens Lancaster Date: 24 July 2013

2. A Top Ten You Can Use:Ten Key Fixes that Avoid Big Problems in Procurements and Proposals Breakout Session #: G02 Presented by: Jacqueline Owens Lancaster Date: 24 July 2013 Time: 9:45 a.m.

3. About Your Speaker • Jacqueline Owens Lancaster (Jackie O.) • CBDO/VP, ASI Government, since 2011 • BD/Capture/Proposal executive roles in major Government contracting firms since 1996 • Worked on first proposal in 1985 • Started career supporting “Big A” acquisition for U.S. Navy (NAVAIR) • MS, Management, University of Maryland • BS, Mass Communications, Virginia Commonwealth University

4. Who’s in the audience? More and more of us have been on both sides, so let’s face it… I have written or contributed to developing RFPs/ RFQs for the Government I have written proposals in response to Government RFPs/RFQs …we need to stop hurting each other! Right hand up I’ve done both! Left hand up Both hands up World Congress 2013

5. Little things can lead to big trouble Government Industry Failure to allot adequate market research time Failure to respond with good pricing data Generic proposal pricing requirements Multiple questions on pricing requirements Delays caused by need to amend RFP Confusing proposals based on RFP changes Discussions required to complete evaluation Further confusion in Final Proposal Revisions Winner selected, but paper trail very messy Everyone but winner angry and dissatisfied Limited debriefs to prevent protest Protest based on conflicting requirements World Congress 2013 4

6. How do we recognize, avoid, or -- as a last resort –fix these “little things” before they become big problems? So here’s the multi-million-dollar question: World Congress 2013 5

7. Start by learning from the past …with a little help from late night TV Top 10 List: Ten Key Fixes that Avoid Big Problems in Procurements and Proposals World Congress 2013 6

8. (First person to match one on my list gets a prize!) But before we get started… Let me ask you…What do you do to avoid problems in acquisitions before they occur? World Congress 2013 7

9. Number 10: Conduct market research to enable selection of effective, differentiating evaluation criteria • If you don’t do it, what happens? • No understanding of “realm of the possible” • No picture of support requirements • Unawareness of commercial products • Failure to bundle for efficiency • Limited understanding of potential risks • No insight on industry standards/benchmarks • Inadequate number of qualified vendors • Unfamiliarity with small business providers • No visibility into potential incentives • Inclusion of unnecessary constraints • Few criteria to evaluate good/not so good • Unreasonably high prices Example In 2011-2012, an in-depth review of the Joint Tactical Radio System (JTRS) program revealed that inadequate market research had been conducted as changes in commercial items and user needs occurred. Upon conduct of market research that revealed user needs had dramatically changed and non-developmental items were available to address them, the existing contract was modified to end all remaining development efforts. - GAO Report, “Defense Acquisitions: Assessments of Selected Weapons Programs,” March 2013 World Congress 2013 8

10. Number 10:Conduct market research to enable selection of effective, differentiating evaluation criteria Do:  Research existing research – don’t start from scratch  Conduct market research across all acquisition phases  Maximize contact with industry – don’t fall victim to myths!  Keep your mind open – don’t assume you know the answer  Focus research on identifying differentiating characteristics  Target what you need to know – avoid “free form” research  Contact the right people from industry – multiple disciplines  Iterate – it’s generally not a “one-and-done” approach World Congress 2013 9

11. Number 9: Take the time required to develop a comprehensive, integrated Acquisition Plan (AP) • If you don’t do it, what happens? • Lack of input from some stakeholders • Lack of market research to support strategy • Inability to integrate (logically) inputs received • Inconsistency among requirements • Failure to address security requirements • Failure to consider support needs • Inadequate or unfounded IGCE • Discovery that funding is unavailable • No use of performance-based techniques • Discouragement of competition • Requirements that don’t yield desired result Example Since 2009, DoD has cancelled three MDA programs due to concerns about technical issues, schedule delays, and cost-effectiveness and operational roles. A GAO investigation documented in 2013 revealed that the issues that ultimately caused these cancellations would most likely have been discovered before the programs began if more robust Analyses of Alternatives (AoAs) had been conducted as part of acquisition planning. - GAO Report, “Missile Defense: Opportunity to Refocus on Strengthening Acquisition Management,” April 2013 World Congress 2013 10

12. Number 9:Take the time required to develop a comprehensive, integrated Acquisition Plan (AP) Do:  Maximize up-front, cross-disciplinary stakeholder input to AP  Use iterative market research to make AP outcomes-focused  Consider full realm of needs (e.g., support, security, IT)  Reach out to varied experts to get the IGCE right  Use the “So what?” test – is the AP going to improve results?  Consider criteria early – what separates good from better?  Use cross-disciplinary reviews to check for integration  Make senior level sign-off/supervisory approval mandatory World Congress 2013 11

13. Establish and enforce accountability for the acquisition Number 8: • If you don’t do it, what happens? • Rushed, “thrown over” requirements • No ownership of requirements integration • Failure to consider connection to strategy • Loss of perspective on foundational need • No consideration of past approaches to need • Failure to consider impact on competition • Loss of connection between needs and costs • Inadequate knowledge base for acquisition • Loss of focus on supportive services • No consequences for unsatisfactory input • Unbalanced acquisition that is hard to defend Example In 2005, GAO sustained protests from two companies challenging the Air Force's award of a sole-source contract based on an unusual and compelling urgency J&A. GAO agreed with the protestors’ claim that the agency's circumstances did not fulfill the criteria for an unusual and compelling urgency; that the urgency of the requirements was the result of the agency's inadequate procurement planning; that the agency unreasonably concluded that the awardee was the only firm capable of meeting the requirements; and that the agency failed to obtain competition to the maximum extent practicable. - GAO Protest Decision, Worldwide Language Resources; SOS International Limited, November 2005 World Congress 2013 12

14. Number 8:Establish and enforce accountability for the acquisition Do:  Identify ultimate acquisition authority early  Define key milestones at which authority must sign off  Make sure COR is identified early and involved throughout  Use methods that build stakeholder engagement while cutting cycle time (e.g., Acquisition Acceleration Centers)  Evaluate and provide feedback on quality of input  Use systems that provide visibility of status (e.g., SharePoint)  Employ “color review” approach to build in quality World Congress 2013 13

15. Intensify up-front stakeholder engagement to build consensus on requirements Number 7: • If you don’t do it, what happens? • Inconsistent internal view of requirement • No ID of “must haves” versus “nice-to-haves” • Many inconsistencies/contradictions in terms • Limited understanding of overall timeline • Delays caused by over-the-transom approach • Unbalanced view based on involvement level • Failure to consider novel approaches • Loss of focus on supportive services • Many difficult questions after RFP release • Inconsistent proposals • Failure to meet objectives with acquisition Example In 2010, GAO sustained a protest challenging the elimination of company’s proposal from competition for a Federal Transit Administration PMO contract. GAO agreed with the protest claim that FTA’s RFP criteria were internally in conflict. Specifically, the RFP did not require the submission of cost or pricing data, but the protesting firm was evaluated against, and excluded for, not meeting the requirements of FAR Section 15.408, Table 15-2, which only apply when cost or pricing data are required. - GAO Protest Decision, PMO Partnership Joint Venture, October 2010 World Congress 2013 14

16. Number 7:Intensify up-front stakeholder engagement to build consensus on requirements Do:  Identify key stakeholders early and bring them together  Consider every discipline that will be involved in the program  Identify Authoritative, Responsible, Consulted, and Informed (ARCI) stakeholders by task  Establish an acquisition schedule that promotes collaboration  Use performance-based methods to focus on outcomes  Identify approaches that are collaborative versus sequential  Use tools that improve visibility of status (e.g., SharePoint)  Integrate efforts through the ultimate acquisition authority World Congress 2013 15

17. Number 6: Select a proposal type and process that aligns with overarching objectives • If you don’t do it, what happens? • Too few bidders based on overly limiting reqs • Unqualified bidders based on overly loose reqs • Too time-consuming considering need • Unnecessarily paper-intensive • Too lengthy considering scope of requirement • Misalignment between prop & execution needs • No ability to meet key staff during bid process • Inappropriate re-use of existing materials • Interpretation that “it’s locked” for someone • Failure to consider oral presentation approach • Inability to select the best possible solution Example • In 2013, DOE issued a Policy Flash that identified the importance of DRFPs in helping match the proposal and selection process to the requirement. Specifically, DOE noted that using DRFPs to guide selection of the proposal type can lead to significant cost savings and productivity enhancements; reduced proposal preparation and evaluation time; reduced need for solicitation amendments that disrupt timely completion; and better proposals, end products, and services. • DOE Policy Flash 2013-45, • April 2013 World Congress 2013 16

18. Number 6:Select a proposal type and process that meets overarching objectives Do:  Relate the proposal requirements to the work requirements  Use market research to gain specific RFP recommendations  Put DRFPs out early; include as much content as possible  Directly ask for specific RFP recommendations from industry  Consider schedule in selecting response methods (e.g., orals)  For small bids, keep it simple (complexity drives out bidders)  Promote electronic submittal to greatest extent possible  Carefully review re-used artifacts – are they appropriate? World Congress 2013 17

19. Number 5: Use critical thinking skills when selecting evaluation criteria • If you don’t do it, what happens? • Many questions based on evaluation criteria • Industry self-elimination for the wrong reasons • Inappropriate focus on less significant aspects • Uneven, difficult-to-evaluate proposals • Unintended conflicts with other RFP elements • Misalignment with Acquisition Plan • Loss of confidence in fair and open process • Loss of focus on overarching objectives • Potential protest before proposal submittal • Poor props that inhibit selection of best option Example • In 2009, GAO sustained a protest claiming that an RFP from the Army Medical Research Acquisition Activity unreasonably required certification of proposed equipment by the Joint Interoperability Test Command at the time of proposal submittal. “The Army has simply not explained why the apparent purpose behind the certification requirement -- to ensure that a system be certified prior to the time it must be fielded -- requires that evidence of certification be provided as early in the process as the time at which quotations are submitted.” • GAO Protest Decision, SMARTnet ,Inc., January 2009 World Congress 2013 18

20. Number 5:Use critical thinking skills when selecting evaluation criteria Do:  Make criteria specific to needs/objectives (limit re-use!)  Limit the number of criteria to focus on key discriminators  Define to greatest extent possible what makes an offer great  Make sure proposal instructions align with selected criteria  Define key expectations (schedules, processes, templates)  Provide adequate time for offerors to meet expectations  Consider use of sample tasks to promote realism  Think about how you would choose if it was your money World Congress 2013 19

21. Number 4: Challenge ambiguity in all aspects of process • If you don’t do it, what happens? • Annoyed, contentious offerors • Perception that ambiguity is deliberate • High potential for unintended conflicts • Multiple questions focused on gaining clarity • Difficulty in tracking back to AP for answers • Many different but reasonable interpretations • Uneven, difficult-to-evaluate proposals • Increased need for lengthy discussions • Delays in completing acquisition process • High potential for successful protests Example • In 2012, GAO partially sustained a protest claiming that an RFP from the Defense Intelligence Agency did not clearly state that use of uncleared personnel by the awardee demonstrated a lack of requirements understanding and should have been considered in a price realism evaluation. GAO also upheld the protestor’s claim that a cost realism evaluation must be conducted if it is defined in the evaluation criteria, even if the evaluation criteria do not specifically state by name that a cost realism evaluation will be conducted. • GAO Protest Decision, Science Applications International Corporation, November 2012 World Congress 2013 20

22. Number 4:Challenge ambiguity in all aspects of process Do:  Remember: if it seems ambiguous to you, it is ambiguous  If you find ambiguity, enlist fresh perspective to help fix it  Use industry to identify ambiguities and propose solutions  Write short, direct statements – less is often more (clear)  Use an iterative approach that considers holistic alignment  Use a color team type approach to build in clarity in phases  Assign at least one person to search for ambiguity issues  Document your thinking so it is defensible downstream World Congress 2013 21

23. Number 3: Review all clauses for currency, accuracy, and applicability • If you don’t do it, what happens? • Use of outdated guidance/regulatory artifacts • Conflicts among accurate/inaccurate clauses • Improper application of clauses • Multiple questions from annoyed bidders • Loss of bidder confidence in process • Automatic necessity for RFP revision • Need (often) for re-review of entire solicitation • Delays in completing acquisition process • “Technicalities” with potential to derail process • High potential for successful protests Example • In 2011, the U.S. Court of Federal Claims determined that GSA’s inclusion of a 15-year fixed-pricing schedule violated customary commercial practice and therefore also violated FAR 12.301(a)(2). The court found that GSA’s market research failed to show that the solicitation requirement for providing fixed prices at contract outset was consistent with commercial practice, and it therefore found the pricing requirement to be in violation of FAR 12.301(a)(2). • COFC Decision, CW Government Travel v. the United States, August 2011 World Congress 2013 22

24. Number 3:Review all clauses for currency, accuracy, and applicability Do:  Use automation for generation, but review manually  Apply tools and references that focus on clause updates  Provide resources/training on clause changes/updates  Conduct research and seek guidance if you identify an issue  Make an individual responsible for clause accuracy  Schedule a final clause review immediately before release  Document identified issues/inconsistencies and resolutions World Congress 2013 23

25. Number 2: Make Section L (Instructions) and Section M (Evaluation Criteria) the same • If you don’t do it, what happens? • Confusion over what to cover/to what extent • Varied proposals that are hard to evaluate • High potential for conflicts • Multiple questions from annoyed bidders • “Everything is equally important” mentality • Loss of focus on key acquisition objectives • Delays in the proposal development timeline • Challenging evaluation process • Confusing, overly complicated proposals Example • In 2009, GAO sustained a protest claiming that TRICARE failed to apply its evaluation criteria properly in the areas of technical approach, past performance, and cost realism in selecting the initial awardee for a contract. In its decision, GAO noted that the instructions and evaluation criteria were complex and could be interpreted differently. The initial awardee followed all RFP instructions, but GAO held that TRICARE failed to apply its evaluation criteria appropriately. GAO upheld the protest. • GAO Decision, Health Net Federal Services, LLC, • November 2009 World Congress 2013 24

26. Number 2:Make Section L (Instructions) and Section M (Evaluation Criteria) the same Do:  Write Sections L and M concurrently with SOO/SOW  Complete evaluation plan before RFP release; map L/M to it  Limit the information required to what will be evaluated  Match what you will evaluate with what you need in proposals  Consider offerors’ reactions to differing Sections L and M  Use past RFPs as references – What worked? What didn’t?  Consider pass/fail for some elements of the proposal  If Sections L and M differ, document your rationale World Congress 2013 25

27. Number 1: Allow sufficient time to conduct iterative reviews and to respond to flaws they disclose • If you don’t do it, what happens? • No “red-threading” of changes to solicitation • No “fresh eyes” to find errors/inconsistencies • Missed errors/inconsistencies • Misalignment with AP and Evaluation Plan • Disappointment from requirement initiator • “Amend it later” mentality • Delays from questions/amendments • Confusion caused by multiple changes • Increased potential for successful protests Example • In 2007, GAO sustained a protest claiming that the Army improperly evaluated proposals with regard to each of the solicitation’s evaluation factors (fill rate, experience, and transition). Specifically, the evaluations of offerors’ responses to the evaluation factors could not be aligned to specific criteria in the evaluation factors. It was therefore determined that there was inadequate support for the Army’s source selection determination. • GAO Decision, L-3 Communications Titan Corporation, • March 2007 World Congress 2013 26

28. Number 1:Allow sufficient time to conduct iterative reviews and respond to flaws they disclose Do:  Establish a timeline that supports an iterative review cycle  Create teams that combine historic and fresh perspectives  Remember: it is always better to fix it when you find it  Consider and correct ambiguities – even if they’re not “wrong”  Use tools to promote rapid, effective, timely collaboration  Use review results as learning opportunities  Consider process changes and training based on reviews  Document, document, document – maintain your file World Congress 2013 27

29. Little things can lead to big successes Government Industry Market research identifies key criteria Feedback sharpens focus on discriminators Schedule/Acquisition Plan/DRFP reflect input DRFP yields good suggestions from industry Unambiguous, direct RFP includes best ideas Well-prepared offerors have fewer questions Fewer amendments keep timeline intact Offerors submit clear, consistent proposals Winner selected based on clear eval plan Offerors receive notification/request debriefs Debriefs clearly explain logical rationale Program awarded! Protests avoided! World Congress 2013 28

30. Success begins with the basics Significant problems are generally avoidable or resolvable by focusing on simple-to-implement acquisition best practices World Congress 2013 29

31. Questions? World Congress 2013 30