1 / 74

WEAG – THALES – JP 11.20

WEAG – THALES – JP 11.20. REVVA. Presentation Outline. Project facts ; the consortium Background and rationale The REVVA concepts The REVVA generic process The REVVA products Techniques and support REVVA technical achievements REVVA future.

Thomas
Download Presentation

WEAG – THALES – JP 11.20

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. WEAG – THALES – JP 11.20 REVVA WE 0201 – PKIT2 Date : November2004 Slide : 1 / 74

  2. Presentation Outline Project facts ; the consortium Background and rationale The REVVA concepts The REVVA generic process The REVVA products Techniques and support REVVA technical achievements REVVA future WE 0201 – PKIT2 Date : November2004 Slide : 2 / 74

  3. Common Validation, Verification and Accreditation Framework for Simulation (REVVA) Project Facts Purpose: The THALES JP 11.20 is a Western European Union / WEAG research program funded by 5 nations: France (lead nation), Denmark, Italy, the Netherlands, Sweden. Objectives: To develop a basis for a common methodological framework for the Verification, Validation and Accreditation (VV&A) of data, models and simulations. To be a starting point for a draft standard defining VV&A taxonomy, process and methodology. Project timeline: March 2003 – September 2004 WE 0201 – PKIT2 Date : November2004 Slide : 3 / 74

  4. The consortium The THALES JP 11.20 is conducted by a multi-national consortium led by ONERA (France). FOI – FMV (SW) UNI-C (DK) TNO (NL) DATAMAT (IT) ONERA – ALEKTO (FR) WE 0201 – PKIT2 Date : November2004 Slide : 4 / 74

  5. REVVA background and rationale WE 0201 – PKIT2 Date : November2004 Slide : 5 / 74

  6. DMSO ITOP Exchange SISO Capitalization Information JP 11.20 REVVA Capitalization NATO Exchange AFDRG RTP 11.13 REVVA Background (1) WE 0201 – PKIT2 Date : November2004 Slide : 6 / 74

  7. REVVA Background (2) • VV&A state-of-the-art: • DMSO, SISO, NATO, ITOP. • M&S scientific contributions: • Systems Theory (B. Zeigler) • Contributions on the concept of validation coming from the epistemology of simulation WE 0201 – PKIT2 Date : November2004 Slide : 7 / 74

  8. VV&A rationale (1) Doing (only) what makes sense for customers and users in their search for trust. • Trust: • identification of trust components, • support (based on project knowledge, consolidation by shared domain knowledge) WE 0201 – PKIT2 Date : November2004 Slide : 8 / 74

  9. VV&A Informal Formal VV&A rationale (2) M&S supplier skillsand reputation Audit trail of experiments and uses Formal analysis ofproducts, processes andorganisation Trust WE 0201 – PKIT2 Date : November2004 Slide : 9 / 74

  10. Needs Purpose Organisation Products Processes VV&A rationale (3) Validation Verification FormalAnalysis Acceptance WE 0201 – PKIT2 Date : November2004 Slide : 10 / 74

  11. VV&A rationale (4) Activities driven and prioritised by: • Requirements (acceptance requirements) • Identification / perception of risks • Organisational needs, risks and benefits • Costs WE 0201 – PKIT2 Date : November2004 Slide : 11 / 74

  12. REVVA Concepts WE 0201 – PKIT2 Date : November2004 Slide : 12 / 74

  13. REVVA Terminology (1) • Decomposition of VV&A in Verification, Validation and Acceptance; (temporal) independence of each concept. • Verification: Model Verification. • Validation: Model Validation within an Experimental Frame (EF) • Acceptance: Before use / After use Focus on Acceptance and Validation WE 0201 – PKIT2 Date : November2004 Slide : 13 / 74

  14. REVVA Terminology (2) • Correctness: • The property of a simulation model to comply with formal rules and bodies of reference information for its representation and for the transformation of its representation into another one. • Verification: • The process which is used to construct, under a set of time, cost, skills, and organisational constraints a justified belief about model correctness. WE 0201 – PKIT2 Date : November2004 Slide : 14 / 74

  15. REVVA Terminology (3) • Validity: • The property of a simulation model to have, within a specific experimental frame, a behaviour which is indistinguishable under a set of validation criteria from the behaviour of the System of Interest. • Validation: • The process which is used to construct, under a set of time, cost, skills, and organisational constraints a justified belief about model validity. WE 0201 – PKIT2 Date : November2004 Slide : 15 / 74

  16. REVVA Terminology (4) • Acceptance: • The informed decision of a M&S customer to use the results of a simulation for a specific purpose. • The informed decision of a M&S customer to use a simulation, within an experimental frame, for a specific purpose WE 0201 – PKIT2 Date : November2004 Slide : 16 / 74

  17. REVVA Assumptions • Acceptance (and Validation) should: • be purpose oriented, • be development neutral, • be technology neutral, • recognise the impact of organisations, • recognise the impact of timeframe of M&S and VV&A, • rely on data and knowledge management. WE 0201 – PKIT2 Date : November2004 Slide : 17 / 74

  18. A comprehensive view The "M&S Product World" The "Real World" The "Problem World" The "M&S World" System Requirements M&S Questions Objectives Needs Problem Formalisation Decisions Actions M&S Enabling Products Problem solution M&S Answers WE 0201 – PKIT2 Date : November2004 Slide : 18 / 74

  19. Methodological hints • Independence • Incremental nature of activities and results • Focus on the intended purpose • Objective assessment (Acceptance Criteria) • Careful construction of a justified belief • Tailoring WE 0201 – PKIT2 Date : November2004 Slide : 19 / 74

  20. EF M Tailoring Question Answer The EF used for Acceptance (and for Validation) is derived from the Acceptance Requirements • All the activities are planed, designed and controlled by them. • The Acceptance Requirements are not (necessarily) the M&S • Requirements ; they are specific of each use and purpose WE 0201 – PKIT2 Date : November2004 Slide : 20 / 74

  21. Before M&S use1 AfterM&S use1 BeforeM&S use2 | | | t t1 t2 t3 Validation t1 Validation t2 Validation t3 Impact of time • Verification is (mostly/hopefully) a one-shot process • Validation is a permanent/recurrent issue • Acceptance and Validation are purpose oriented and tailored WE 0201 – PKIT2 Date : November2004 Slide : 21 / 74

  22. The REVVA Generic Process WE 0201 – PKIT2 Date : November2004 Slide : 22 / 74

  23. Intended purpose Acceptance Recommendation Develop ToA Evaluate V&V Report The REVVA Generic Process ToA Acquire Information Product flow Model information and system knowledge V&V report Back step Develop ToVV Assess Evidence Integration Phase Assessed items of evidence Product ToVV ToA: Target of Acceptance Assess Evidence Conduct V&V ToVV: Target of Verification and Validation V&V Items WE 0201 – PKIT2 Date : November2004 Slide : 23 / 74

  24. Phase Overlaps Develop ToA Acquire Information Develop ToVV Conduct V&V Assess Items of Evidence Assess Evidence Integration Evaluate V&V Report Main effort VV&A Endeavour Duration WE 0201 – PKIT2 Date : November2004 Slide : 24 / 74

  25. Develop ToA • Performed by the Acceptance Agent in close cooperation with the user • Develop Acceptability Criteria (AC), including Validity Criteria (VC) • Document rationale for development WE 0201 – PKIT2 Date : November2004 Slide : 25 / 74

  26. Develop ToA (2) Intended Purpose Objectives and Sub-Objectives (”Questions”) (Fictive) Experimental Frames Precision Requirements for the Behavior Characteristics WE 0201 – PKIT2 Date : November2004 Slide : 26 / 74

  27. Intended Purpose Sub-Objective 1 Sub-Objective 2 Sub-Objective n ToA SO1.1 SO1.1 SO1.1 AC2 AC3 Acceptability Criterion m AC1 WE 0201 – PKIT2 Date : November2004 Slide : 27 / 74

  28. Acquire Information • Led by the V&V Agent, performed by the V&V Team • Get overview over the data, information, and knowledge (likely to become) available • about the Simulation Executable Model • about the System of Interest • Create data, information, and knowledge repository • Identify additional sources WE 0201 – PKIT2 Date : November2004 Slide : 28 / 74

  29. Develop ToVV • Performed by the V&V Agent, supported by the V&V Team • Develop an approach to demonstrate that the Validation Criteria are passed or failed, with the data, information, and knowledge available • Identify data, information, and knowledge desirable for more efficient V&V WE 0201 – PKIT2 Date : November2004 Slide : 29 / 74

  30. Intended Purpose Sub-Objective 1 Sub-Objective 2 Sub-Objective n ToA SO1.1 SO1.1 SO1.1 AC2 AC3 Acceptability Criterion m AC1 Sub-Criterion m.1 SC m.1.1 SC m.1.2 SC m.1.3 Task ToVV Task Task WE 0201 – PKIT2 Date : November2004 Slide : 30 / 74

  31. Conduct V&V • Led by the V&V Agent, performed by the V&V Team • Choose techniques and tools • Run the technique, e.g., • Use 2-test to measure deviation • Use SME to assess structural decomposition • Record technique, referrenced information, and result as V&V item • Modify ToVV, if creation of V&V item fails WE 0201 – PKIT2 Date : November2004 Slide : 31 / 74

  32. Assess Evidence • Performed by the V&V Agent • Review documented V&V items • Assess the possibility of an erroneous result • Accept V&V item as assessed item of evidence, or reject it • Direct feedback loop WE 0201 – PKIT2 Date : November2004 Slide : 32 / 74

  33. Assess Evidence Integration • Performed by the Acceptance Agent • Assemble items of evidence according to most recent ToVV • Derive which VC are passed or failed • Intuitively, subjectivly • Based on methods for dealing with uncertainty • Assess possibility of misperception WE 0201 – PKIT2 Date : November2004 Slide : 33 / 74

  34. Intended Purpose Sub-Objective 1 Sub-Objective 2 Sub-Objective n ToA SO1.1 SO1.1 SO1.1 AC2 AC3 Acceptability Criterion m AC1 Sub-Criterion m.1 ToVV SC m.1.1 SC m.1.2 SC m.1.3 Task Task Task Item of Evidence IoE IoE WE 0201 – PKIT2 Date : November2004 Slide : 34 / 74

  35. Evaluate V&V Report • Performed by the Acceptance Agent • Review expressiveness and completeness of AC with newly gained knowledge • Weigh impact of misperception according to possibility of misperception • Make acceptance recommendation WE 0201 – PKIT2 Date : November2004 Slide : 35 / 74

  36. The REVVA Products WE 0201 – PKIT2 Date : November2004 Slide : 36 / 74

  37. Intended purpose Acceptance Recommendation Develop ToA Evaluate V&V Report The REVVA Generic Process ToA Acquire Information Model information and system knowledge V&V report Product flow Back step Develop ToVV Assess Evidence Integration Phase Assessed items of evidence ToVV Product Assess Evidence Conduct V&V V&V Items WE 0201 – PKIT2 Date : November2004 Slide : 37 / 74

  38. Target of Acceptance (ToA) Intended purpose • Precise specification of the Acceptability Criteria (coming from validity related functional requirements) • Stakeholder oriented • “What exactly needs to be assessed?” • Important elements in the ToA nodes: requirement, MoE, constraint, level of impact, decomposition argument Develop ToA ToA Acquire Information Develop ToVV WE 0201 – PKIT2 Date : November2004 Slide : 38 / 74

  39. ToA (2) • Building • Getting the requirements • Deriving MoE and constraints • Determining the level of impact • Validation of the ToA • Reuse • Using • Part of contract • Basis for assessment report • Maintaining • Due to changes in science, stakeholder changing mind, corrections in ToA found during V&V work, V&V findings may indicate more limited validity range WE 0201 – PKIT2 Date : November2004 Slide : 39 / 74

  40. Model information and system knowledge • Information used to determine the V&V strategy for demonstrating the model’s correctness and validity • Identifies all sources of information and knowledge • Important elements: • design documents, code, modeller’s perception of the real world, behaviour data (of model and sub-models), conceptual model. • Real world data (referent): data on (related or similar) systems,expert opinions Acquire Information Model information and system knowledge Develop ToVV Assess Evidence Integration Conduct V&V WE 0201 – PKIT2 Date : November2004 Slide : 40 / 74

  41. Target of Verification and Validation (ToVV) Develop ToVV Assess Evidence Integration • V&V items: evidence required to substantiate the ToA Acceptability Criteria • When necessary: further decomposition of criteria (extension of ToA) ToVV Conduct V&V • Important additional elements in the ToVV: method to use for collecting evidence, required rigor, cost estimate, aggregation instructions • Tasks WE 0201 – PKIT2 Date : November2004 Slide : 41 / 74

  42. V&V Items • A V&V item consists of: • relevant information, • the evaluation objective, • reference information, • an evaluation technique, • evaluation result. • Must be reproducible Assess Evidence Conduct V&V V&V Items WE 0201 – PKIT2 Date : November2004 Slide : 42 / 74

  43. Assessed items of evidence • Assessed items of evidence: • V&V agent assessment of the V&V result, • the acceptance or rejection of each V&V item, • probative force (depending on method or technique and reference information) Assess Evidence Integration Assessed items of evidence Assess Evidence WE 0201 – PKIT2 Date : November2004 Slide : 43 / 74

  44. V&V report • The assessed items of evidence assembled and integrated based on the ToVV • Links the assessed items of evidence to the Acceptability Criteria • Indicates if • evidence is considered sufficient to pass an acceptability criterion • V&V items for a criterion are missing or too weak, • disproving evidence was found Evaluate V&V Report V&V report Assess Evidence Integration WE 0201 – PKIT2 Date : November2004 Slide : 44 / 74

  45. Acceptance Recommendation • The final recommendation on acceptance • Based on the ToA • Residual uncertainty after V&V • from probative forces, • use level of impact to determine recommendation • The recommendation states (with consequences for M&S product’s utility): • if required evidence is missing or too weak, • if a weakness in the ToA was detected Acceptance Recommendation Evaluate V&V Report WE 0201 – PKIT2 Date : November2004 Slide : 45 / 74

  46. A VV&A project management view The "3 pillars" concept Purpose Organisation Products Objectives Process WE 0201 – PKIT2 Date : November2004 Slide : 46 / 74

  47. The M&S and VV&A roles • M&S stakeholders • M&S suppliers • VV&A agents • Experts WE 0201 – PKIT2 Date : November2004 Slide : 47 / 74

  48. Levels of independence • DV&V: • dependent V&V; V&V performed by suppliers and accepted as it is • IA: • independent assessment; V&V performed by supplier and assessed by the V&V agent • IV&V: • independentV&V; V&V performed under the control of the V&V agent WE 0201 – PKIT2 Date : November2004 Slide : 48 / 74

  49. Feasibility assumptions • High-level organisational decision • Standards of V&V documentation • Incremental accumulation of knowledge: • project vs corporate knowledge management • Process for knowledge revision WE 0201 – PKIT2 Date : November2004 Slide : 49 / 74

  50. The REVVA Techniques and support WE 0201 – PKIT2 Date : November2004 Slide : 50 / 74

More Related