A Mathematical Logic Approach To Risk Based VV&A

1. A Mathematical Logic Approach To Risk Based VV&A Dr. Peter P. Pandolfini (443) 778-6193 peter.pandolfini@jhuapl.edu September 22, 2010

2. I. Defining the goals of Risk Based VV&A (RBA) SITUATION: A MODEL/SIMULATION (M/S) HAS A STATED LIST OF CAPABILITIES TO BE USED FOR STATED INTENDED PURPOSE(S) DESIRED: A RECOMMENDATION FOR EACH STATED CAPABILITY ON WHETHER: “THE M/S CAN BE USED FOR THE INTENDED PURPOSE” OR “THE M/S SHOULD NOT BE USED FOR THE INTENDED PURPOSE”AND THE CONFIDENCE IN THAT RECOMMENDATION DESIRED: A V&V PLAN WITH A CONFIDENCE IN EACH STATED CAPABILITY’S RECOMMENDATION, THEREBY UNDERSTANDING THE RISK OF BEING INCORRECT IN THE RECOMMENDATIONS & FACILITATING RESOURCE TRADE-OFFS

3. ELEMENTS OF METHODOLOGY COVERAGE: Required capabilities assembled and rated by importance to intended use ACTIVITIES: Depth of evidence determined as a function of four types of tasks TECHNIQUE: Examination techniques that are influenced by a capability’s level-of-importance RESULT: A scorecard containing, for each capability, its importance, its recommendation, and the level-of-confidence in the recommendation

4. Risk Based VV&A Scorecard

5. II. Forming the V&V Plan ACTIVITIES BASED ON COMPOSITE MODEL Each individual Task provides evidence of value in the order shown (i.e., CM V&V < DP verification < historical record analysis < results validation) TASKS w = Conceptual Model Verification & Validation (CM V&V) x = Development Product Verification (DP verification) y = Historical Record Analysis z = Results Validation Activity sets ordered by increasing confidence A. CM V&V B. DP verification C. historical record analysis or (CM V&V + DP verification) D. historical record analysis + CM V&V E. historical record analysis + DP verification F. historical record analysis + CM V&V + DP verification G. results validation H. CM V&V + results validation I. (historical record analysis + results validation) or (DP verification + results validation) J. historical record analysis + CM V&V + results validation K. historical record analysis + DP verification + results validation L. CM V&V + DP verification + results validation M. historical record analysis + CM V&V + DP verification + results validation

6. Factor Table for Activities Selection Factor (a) For each capability

7. Factor Table for Techniques Selection Factor (b) For each capability

8. III. Evaluating confidence in a Recommendation Forming a Weighted Raw Score: WRS = function[(a), (b)] For each capability, activities set, & selected examination technique Equations for WEIGHTED RAW SCORE:

9. Weighted Raw Score Interpretation “Confidence”, by quintiles WRS Percentiles (based on cumulative distribution) Assign to a capability’s Recommendation

10. An Example Portfolio of Capabilities & their examination techniques

11. Summary Coverage, Activities, Techniques • Assemble the relevant capabilities • Determine importance for each capability • Compute WRS based on Factor (a) state & Factor (b) states • Compute percentile for recommendation confidence • Assemble RBA Scorecard & perform V&V planning tradeoffs • Execute plan and develop recommendation for each capability