Towards Appropriate Selection of Analysis Tools and Methods

1. Towards Appropriate Selection of Analysis Tools and Methods

2. Overview • Research to determine the appropriate tools and methods for effective application within a broad range of analysis categories. • Longer term goal to complete whole activity • Summarised later • This paper focuses on • categorisation of study and problem types • Initial activities description • Discussion on derivation of metrics • for assessing the value of study methods and tools vs problem and study types.

3. Aim • Overall Aim and Introduction • The aim of the whole research is to determine the appropriate tools and methods for effective application within a broad range of analysis categories. • The work will • investigate and analyse a broad range of differing analysis study types and • research optimal methods and tools for utilisation within these types. • The analysis study types • will be example operational and system analysis studies • investigated to determine • the approaches used, • the constraints and issues associated with them and • the overall success and outcomes.

4. Activities to date • Initial Literature Review • OR Society • College Library • MORS Material • Methods Handbook • Model Taxonomy • SME Contact • No evidence of complete approach • But • Problem Structuring Methods • Paradigms

5. Problem Types • Obeng’s Problem types • Fog – Open • Movie – Semi Open • Quest – Semi Closed • Paint by Numbers – Closed • Mike Pidd • Problems • Puzzles • Messes • Obeng preferred • More refined

6. Proposed Factors affecting the problem • Organisational Level • Corps • Div • Bn • Combat Type • High Intensity • PK • Hybrid • Timescale of Battle • Hour • Day • Days

7. Proposed Factors affecting the problem II • The Question • Cost Effectiveness Comparison • Force Mix • Battlefield Effectiveness • Capability Gap • Tactics comparison • Numbers • The Customer • Soft Issues

8. Factors affecting the solution • Skill set • Analysis Timescales • Analysis Budget • Model Types • Hard • Soft • etc

9. Factors affecting the solution • Skill set • Defence Environment • Analysis within Acquisition and Systems Process • Uses • Capability Analysis • Acquisition Processes • Business Case • BOI • COEIA • Operational Support • Wargaming • SeBA • Training • Systems Approach • Validation

10. Factors affecting the solution – Skill set • Tools and Methods • OR Techniques • Soft Methods • Spreadsheets • Mathematical Modelling • Engineering Modelling • Modelling and Simulation • Virtual • Live • Constructive • Stochastic • Deterministic etc • Games • Simulations • Analytical methods – lanchester etc • SEs • Model Development Process

11. Factors affecting the solution – Skill set • Design of Experiments • Analysis and interpretation of results • Design of Experimental Studies • Confidence • Verification • Communication and Understanding • Model Components • Networking/ Distribution of Simulation • HCI • Protocols • Behavioural Representation/Intelligent Systems • Data • Data Analysis & Databases • Combat Sys, Terrain • Scenarios • Software Development

12. Skill set • Level of Skill required for differing roles • Senior Decision Maker • Problem Owner • Study Leader • Analyst • Developer • Player • Project Manager • Technical Manager

13. Success Criteria - Measuring how well the method(s) fits the problem • A. Answer obtained • Within time/budget/other constraints • B Answer accepted • Validated • C Answer Used • D Repeat work

14. Success CriteriaMeasuring how well the method(s) fits the problem • A. Answer obtained • Within time/budget/other constraints • Scale • On budget/time • Ahead of budget/time • 20% slip • 50% slip • 100% slip

15. Success CriteriaMeasuring how well the method(s) fits the problem • B Answer accepted • Validated • By peer/QA review • By acceptance • Accepted after changes

16. Success CriteriaMeasuring how well the method(s) fits the problem • C Answer Used • By one user • By several • Citations

17. Success CriteriaMeasuring how well the method(s) fits the problem • D Repeat work • Once • Extension • Frequent

18. Next Steps Study Examples - • Drawn from industry and Dstl where possible. • For each study • What was done, • what techniques used • resources used • Output measured Study Analysis • Studies categorised according to the range of criteria or dimensions, • Problem type • Effort levels available • Timescales • Tools available • Skills available • Audience/Customer • Other Soft issues – political Comparative Analysis of UK MoD Guidelines for conduct of Analysis

19. Outline • Evaluation of Study Approaches • Identification of a full range of approaches including tools and methods for potential use within these types of analyses. • Identification of the approaches and the associated tools and methods used within each example study. • Derivation of metrics for assessing the value of each of approaches and the tools and methods in achieving successful outcomes in the studies. • Derivation of Optimising Approaches • Derivation of theoretical/initial rule sets for identifying the optimal approaches and tools and methods for use within study categories.

20. Outline • Analytical Review of Example Study • Testing of the initial theoretical rule sets against the example studies to determine where non-optimal methods were applied and identification of the improved approaches which might have been employed and the impact these would have had on the study outcome. • Test phase using New Studies • Experimental phase using real study examples to develop approaches and test the hypothesised rule sets. • Conclusions • Identified optimised approaches including the risk areas and constraints and the ‘region of validity’ of the conclusions.

21. Towards Appropriate Selection of Analysis Tools and Methods