1 / 34

Industry Engagement Request Questionnaire 1 Assessment Methodology

Industry Engagement Request Questionnaire 1 Assessment Methodology. Meeting with Companies May 03, 2013. Purpose. To provide companies with details on the assessment methodology that will be used in assessing your responses to Questionnaire 1

alden-farmer
Download Presentation

Industry Engagement Request Questionnaire 1 Assessment Methodology

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Industry Engagement RequestQuestionnaire 1Assessment Methodology Meeting with Companies May 03, 2013

  2. Purpose • To provide companies with details on the assessment methodology that will be used in assessing your responses to Questionnaire 1 • To provide clarifications and provide answers to questions • To provide companies with information on next steps • We are committed to an open, fair and transparent process

  3. Industry Engagement RequestQuestionnaire 1 • Canadian fighter mission and threat information across two time horizons (2020-2030 and 2030+). • Responses to include defined mission configurations: • Air-to-Air; • Air-to-Ground; and • Air-to-Surface. • Responses to include capability developments (planned upgrades). • Responses to include descriptions of capability, production and supportability.

  4. PRINCIPLES The following Principles will guide all participants involved in the assessment of the responses: • Consistency • Team leads will ensure that the assessment is well documented, including all qualitative and professional military judgement aspects. • Individual aircraft are being assessed against each Canada First Defense Strategy (CFDS) mission. • No aircraft will be screened out. All options will remain on the table. Final product will be a high level risk assessment. • Objective • The National Fighter Procurement Secretariat will provide oversight through all steps of the assessment process • Use of Third Parties • Responses will be shared with the Independent Review Panel (i.e. Panel) members and any significant differences of opinion between assessment teams will be brought to the attention of the Panel.

  5. Overview of Q1 Assessment • The “Capabilities” questions of the Industry Engagement Request will be assessed using a 3-step assessment process: • Step 1 – Assessment of Measures of Performance (MOP); • Step 2 – Assessment of Measures of Effectiveness (MOE); and • Step 3 – Assessment of Aerospace Capabilities weighted against Canada First Defense Strategy (CFDS) Mission and criticality to result in “Operational Risk” assessment of each aircraft for each of the missions. • The “Production and Supportability” questions will be assessed using a 2-step assessment process: • Step 1 – Assessment of responses by technical Subject Matter Experts teams on Military Strategic Assessment Factors – Impact Statements; and • Step 2 – Assessment of responses by the Royal Canadian Air Force Senior leadership level from a missions perspective to result in “Strategic Risk” assessment of each aircraft for each of the missions. • Both the “Operational Risk” and “Strategic Risk” will then be consolidated into an overall risk profile for each aircraft against each mission.

  6. Step 1 of CapabilitiesAssessment • Assessments being performed using responses to the 17 areas of « Capabilities » contained in the Industry Engagement Request. • National Defence teams assessing specific areas, with oversight from the National FighterProcurementSecretariat. • Qualitative assessment - independent of missions. • Results in Measures of Performance assessment for each aircraft.

  7. Evaluation Grid for Step 1 • Performance rating of individual systems • Rating scale from 0-10 to allow sufficient fidelity • Classified systems information represented by a number (supported by classified references) 7

  8. Step 2 of Capabilities Assessment • Three Royal Canadian Air Force teams (four members each) performing this assessment, with oversight from the National Fighter Procurement Secretariat: • Including fighter pilots and at least one Air Force Air Weapon Controller on each team. • Significant differences of opinion between assessment teams will be brought to the attention of the Panel. • The following inputs will be used in this assessment: • Scores and input from the results of Step 1; • Companies responses to the Industry Engagement Request, including the mission configuration systems; • Vignettes detailed in Appendix “A” of the Industry Engagement Request; • Detailed threat capabilities from Canadian Forces Threat Assessment; • Detailed fighter task lists; and • Other information available to Government. • Fighter Measures of Effectiveness areas of consideration detailed in the Industry Engagement Request.

  9. Information on Missions as per the Canada First Defence Strategy (CFDS) 1 – Conduct daily domestic and continental operations, including in the Arctic and through NORAD (Vignette 1) 2 – Support a major international event in Canada, such as the 2010 Olympics (Vignette 2) 3 – Respond to a major terrorist attack (9/11-type scenario under CFDS Mission 1 through NORAD) (Vignette 5) 4 – Support civilian authorities during a crisis in Canada such as a natural disaster (No Vignette) 5 – Lead and/or conduct a major international operation for an extended period [including complex peace enforcement operation, and coalition state-on-state war-fighting] (Vignettes 3 & 4) 6 – Deploy forces in response to crises elsewhere in the world for shorter periods (Vignette 6) Of the 6 Canada First Defense Strategy missions above, only 5 relate to the 6 vignettes. Mission 4 (Support civilian authorities during a crisis in Canada such as a natural disaster) will be analyzed together with Mission 2, understanding the fighter’s role in this mission is generally minor.

  10. Information on Aerospace Capabilities • Defensive Counter Air (DCA) • Offensive Counter Air (OCA) • Strategic Attack • Close Air Support (CAS) • Land Strike • Tactical Support to Maritime Operations (TASMO) • Intelligence, Surveillance, and Reconnaissance (ISR)

  11. Information on Fighter Measures of Effectiveness • Awareness • Survivability • Reach and Persistence • Responsiveness • Lethality • Interoperability

  12. Step 2 of Capabilities Assessment Overview • Step 2 will first generate “raw” scores for Measures of Effectiveness (MOEs). • Raw results will then be weighted against the applicable mission to come up with an Operational Level Aerospace Capability Score for each aircraft against each mission over the 2 timeframes. • Mission weightings are sensitive and not included as part of this presentation.

  13. Evaluation Grid for Step 2 Raw Scores • Effectiveness rating of contributing system performances. • Multiple systems will contribute to each measure of effectiveness. • Will vary across all missions. • Rating scale from 1-10 to allow sufficient fidelity. • Qualitative, and thus requires Subject Matter Expert input. • Will result in “Raw” scores for Step 2.

  14. Step 2 Example 14

  15. Step 2 Example – Assessment of Raw Scores 15

  16. Step 2 Example – CapabilityRaw Scores 5.8 Example Only 16

  17. Step 2 Example – CapabilityRaw Scores 7.66 Example Only 17

  18. Step 2 - Capability (Weighted) Applying weighting function to each measure of effect allows capability to be properly quantified within that Mission 6.7 Example Only 18

  19. Step 2 - Capability (Weighted) Applying weighting function to each measure of effect allows capability to be properly quantified within that Mission 7.1 Example Only 19

  20. Step 3 of Capabilities Assessment • This last step of the Capabilities Assessment takes into account: a) the weighting of each Fighter Aerospace Capability within a particular mission and b) criticalityas follows: • DCA – XX%, OCA – XX%, TASMO – XX%, ISR – XX%; • Mission Critical (MC), Mission Essential (ME), or Mission Routine (MR); • The Operational Level Score obtained at the end of Step 2 is aggregately weighted and assessed by Royal Canadian Air Force against the mission criticality matrix. • Aerospace capability limitations will carry risk forward dependant on their mission criticality (translation matrix on slide 22 expands). 20

  21. CFDS Mission 1 (example) DCA OCA Strat Atk. CAS Ld Strk. TASMO ISR Weighting roles VS CFDS X% X% X% Mission criticality (C/E/R) C C E Step 3a – Operational Mission Weighting CFDS Mission 1 (example) DCA OCA Strategic Attack CAS Land Strikes TASMO ISR Aircraft A Aircraft B Aircraft C Aircraft D Aircraft E Aircraft Y 7.1 6.1 7.3 Weighting of Fighter Roles/ CFDS missions + Mission Criticality 21

  22. Weighted Capability Assessment of each Maximum Aerospace Operational Capability Criticality of Aerospace Capability Assessment + = Mission Critical Mission Essential + = Mission Routine + = Step 3b - Mission Criticality Translation Matrix Definitions Mission Critical: A capability that delivers a direct effect as part of its primary function. This effect Is evaluated as critical to mission success and will pose risk to mission success if it is not employed. Mission Essential: A capability that is an essential enabler to Mission Critical Capabilities; the lack of one or more Mission Essential Capabilities will pose risk to mission success. Mission Routine: A capability that is required for the mission but either as a routine supporting function or a very low likelihood of employment. Only in cases where multiple Mission Routine Capabilities constitute a systems failure will any significant risk be posed to the mission.

  23. Step 3 – “Operational Risk” Assessment Step 3 - End State “Operational Risk” Mission Criticality Translation Matrix 23

  24. Assessment of the “Production and Supportability” Questions • Step 1 – Review of responses by technical Subject Matter Expert teams – no scoring. • Step 2 – Assessment of responses by the Royal Canadian Air Force Senior leadership level from a Canada First Defense Strategy missions perspective to result in “Military Strategic Risk” assessment of each aircraft against the missions. • Both Steps will be overseen by the National Fighter Procurement Secretariat. • Significant differences of opinion between assessment teams will be brought to the attention of the Panel. 24

  25. Factor Analysis through Deductive Reasoning Where does the Industry response fit inside the framework? Factors Considerations Deductions (step 1) (step 2) Likelihood Possible Impacts Required mitigations Strategic Risk Assessment Strategic Assessment Definition: There is a possible risk that… So What? So What? The mission need is… The threat is assessed as… CFDS Source Documentation Industry responses to Industry Engagement Request Open source study Government-to-Government Information Other products from the Evaluation of Options (used during step 2 only)

  26. Acquisition • This Strategic Assessment Factor (SAF) is an assessment of factors, other than cost, that would affect acquisition of aircraft types. This includes: • planned production periods of aircraft type; • the manufacturer’s ability to continue or re-establish production periods; • the ability of the manufacturer to offer a complete “cradle to grave” program, including; training, in-service / life cycle support and disposal; • the manufacturer’s successful completion of any required developmental work before the aircraft are ready for acquisition; and • the ability of aircraft type to be certified for airworthiness under Canadian Forces regulations.

  27. Supportability and Force Management • An assessment of the overall / long-term supportability and aspects related to management of the fighter force. This includes factors such as: • the required quantity of aircraft; • managed readiness postures and sustainment rations; • the training system and production of appropriately trained pilots and maintenance / support personnel (including use of simulators); • training considerations for maintenance / aircrew personnel; • aircraft mission availability rates and aircrew / technician workload; • long-term availability of components needed for aircraft maintenance; • implementation and sustainment of supporting infrastructure; • the ability to operate from required locations; • weapons compatibility and support; and • the suitability of a reprogramming capability including its flexibility and responsiveness for new threats and/or theatres of operation or new capabilities.

  28. Integration • This Strategic Assessment Factor assesses broad interoperability within the Canadian Forces and with allied forces. This includes interoperability with air-to-air refuelling services (other than Canadian Forces), common ground/spares support with allies and their supply lines, training systems, data sharing, communications, STANAGs, and the ability to feed data into Canadian Forces and Government of Canada networks taking into consideration national security requirements.

  29. Growth Potential • This Strategic Assessment Factor assesses the growth potential and technological flexibility to respond to unforeseen future advances in threat capabilities, to implement required enhancements to fighter technology, and to evolve as needed to meet the Canadian Forces’ needs. Factors include analysis of the architecture of aircraft types, power and cooling capability for new systems.

  30. Supportability & Force Management Integration Acquisition GrowthPotential Response to IER • Step 1 • Aircraft - SAF Assessment • Technical SME level working groups • Problem space defined and scoped by the use of a “possible risk” based questionnaire • SME comment on likelihood; then • Comment on possible impacts and required mitigations to the overall fighter program • Information to be collated and considered in Step 2. Strategic Assessment Factors Other reports resulting from evaluation of options Future Security Env’t Open Source/Other Info and Gov-to-Gov Info End State - Strategic Risk • Step 2 • SAF - CFDS assessment • Senior leadership level Working group (fighter heavy) • Problem space defined and scoped by the Task 2 report • - “Considering the likelihood of possible impacts to the overall Fighter Program, and based on your Professional Military Judgment, how would you rate the strategic risk to the RCAF ability to successfully fulfill the fighter contribution to each CFDS mission?” 30

  31. Final Stage – Overall Mission Risk Assessment End State: Mission Risk Strategic Risk OperationalRisk

  32. End State Mission Risk Mission Risk (fusion of Operational and Strategic risk) RCAF to apply professional military judgement C Air Force endorsement of Final Report 32

  33. Mission Risk • Mission Risk will be an outcome of the fusion of Strategic and Operational risk by the same RCAF senior leaders assessment teams who took part in step 2 of the Strategic Assessments. • Decision support tools, such as but not limited to a mission risk translation matrix and the DND/CF Integrated Risk Management Guidelines will be used to determine a rating. • Final assessment will be informed by, but not limited to the decision support tools provided. Assessors will be allowed to deviate from the mission risk translation matrix used by + or – one risk level. • Rationale behind a final assessment that deviates from the mission risk translation matrix used will be provided to the Independent Review Panel for review. 33

  34. Summing Up • We are committed to an open, fair and transparent process • IER and methodology posted publicly on Government website • All questions and answers during IER shared simultaneously with all companies • Details of assessment methodology shared with companies in advance of IER response deadline • NFPS overseeing entire assessment process • Significant differences of opinion between assessment teams will be brought to the attention of the Panel • The process will result in an assessment of benefits and risks associated with each aircraft against each mission. Canada will release a summary report that respects commercial sensitivities and classified information restrictions. 34

More Related