NCW/NEC Metrics Workshop 17-19 December, 2002

1. NCW/NEC Metrics Workshop17-19 December, 2002 Report from Work Group 4 Evidence and Assessment

2. Work Group 4 - Objectives • To discuss ways to develop a knowledge base relevant to NCW/NEC metrics. • Key tasks: • Identify and evaluate the utility of sources of evidence; • Identify the “meta-data” needed to organize and integrate the knowledge base.

3. Work Group 4 - Strategy • Journalist’s questions • Who? Why? What?When? Where? How? • Who? - Identified range of knowledge base user groups. • Why? - For each, identified why they need the knowledge... • What? - …and what types of knowledge they need. • Used NCW Framework to identify NCW-specific knowledge needs. • When?Considered qualities of evidence for different needs. • Where? Considered qualities provided by different type of evidence source. • How?Considered constraints on format/management of KB.

4. Themes for Evidence Workgroup • Source of evidence • history, experiment, model, judgement,.. • Multi-national availability • What you want evidence for? • persuade participation/commitment, • inform investment (BOI, individual cases, substantiate/test claims), • inform planning (strategic, campaign, mission), • generate usable heuristics. • What are the claims of NCW/NEC? • Nature of evidence • Strength,credibility, reliability, fidelity, specific/ generic, cost to gather • Risk of deciding the “wrong” thing • Types of evidence • What is this “knowledge base” (and why do we need it)? • What type of knowledge? (claims, evidence, …) - part of a process • What characteristics does the KB have. • Who, What, When, Where, Why?

5. Who? and Why? • Overall WHY? - Justify decisions, generate of commitment to action. • Policy and Force Structure folks • WHY? - Mission/Context drivers for capabilities. Asymmetric threats. Policy effectiveness. Economics. • Materiel folks • WHY? - Where are the new needs? What technologies? Trade-off assessment, effectiveness - against baseline of current/legacy. Efficiency/cost. • Doctrine developers • WHY? - Limits of capability. What is effective? Areas for promising experimentation. • Operators • WHY? - What have we discovered? ( heuristics). Under what conditions was this knowledge derived? (reliability). What doesn’t work? • Trainers • WHY? - What technologies are in-service or emerging? Awareness of current capabilities, concepts of use, emerging doctrine. [National differences here?]

6. What? • Evidence about the nature of NCW/NEC itself. • Evidence to: • Inform the generation of ideas, goals, concepts. • Justify decision and persuade others to endorse/participate. • Enable bringinginto being, engineering, production.

7. Policy and Force Structure • WHY - Mission/Context drivers for capabilities. Asymmetric threats. Policy effectiveness. Economics. • WHAT - Evidence to inform, justify, enable... • Mission requirements - What missions? Where/when? Constraints? • Capability requirements • Desired force structure attributes, e.g. • Ability to fight for decision superiority • Speed of command • Information access + shared awareness • Dispersed forces in non-contiguous/continuous operations • “De-massification” • Elimination of process lines (boundaries and comm chains) • Elimination of structural lines • Dynamic, self-synchronisation of forces • Compressed levels of operation (collapse of the levels)

8. Materiel folks • WHY - Where are the new needs? What technologies? Trade-off assessment, effectiveness - against baseline of current/legacy. Efficiency/cost. • WHAT - Evidence to inform, justify, enable… • Operational requirements (implies goals, threats, contexts, constraints) • Systemic requirements (addressable accessible entities, level of sophistication of information exchange allowed) • Capability gaps (implies baseline capability and capability needs, value gap) • Technologies, etc. to close gaps (maturity, costs, risks, potential synergies) • People issues (manning, skills, link to training and doctrine/processes) • Networking concepts (synergies, concepts for exploitation, potential to enable operations/activities, net-readiness requirements)

9. Doctrine developers • WHY - Limits of capability. What is effective? Areas for promising experimentation. • WHAT Thing itself, and Evidence to inform, justify, enable… • National Military strategy • Operational concept/concept of operations; joint doctrine • Organisational structure • Specific processes: effects based planning, information sharing, subscription lists, post and pull • Coevolution of doctrine/materiale by experimentation • TTPs

10. Operators • WHY - What have we discovered? ( heuristics). Under what conditions was this knowledge derived? (reliability). What doesn’t work? • WHAT - Evidence to inform, justify, enable… • Effectiveness of different NCW methods under different circumstances • Range of applicability--circumstances under which types of equipment should/should not be used • Robustness-- • Resource and information requirements for different kinds of actions • Rules of thumb/stories/heuristics for suggesting methods and judging applicability; execution level of detail

11. Trainers • WHY - What technologies are in-service or emerging? Awareness of current capabilities, concepts of use, emerging doctrine. [National differences here?] • WHAT - Evidence to inform, justify, enable… • Doctrine, TTP • Technologies; when to apply, how to use, human impact

13. Nothing much NCW knowledge needs for Trainers

14. Nothing much Nothing much Nothing much When to apply, how to use different info sources • When/how to exploit sharing capabilities • Consistent ontology, epistemology, context NCW knowledge needs for Trainers

15. Nothing much Nothing much Nothing much When to apply, how to use different info sources When/how to exploit sharing capabilities • Factors for“Net-ready people” • Frames of mind, heuristics • Sharing behaviour modes • Knowledge of networks and how to exploit them • Holistic habits, supra-task views NCW knowledge needs for Trainers

16. Nothing much Nothing much Nothing much When to apply, how to use different info sources When/how to exploit sharing capabilities Factors for“Net-ready people” Factors for“Net-ready people” • Factors for effective interaction • Trust, confidence, comfort, … (esp. in computer-mediated interaction) • Ownership of enterprise goals • Feedback and motivation, altruism? • Holistic habits, Basic business rules • Generational changes? NCW knowledge needs for Trainers

17. Nothing much Nothing much Nothing much When to apply, how to use different info sources When/how to exploit sharing capabilities Factors for effective interaction Factors for“Net-ready people” Factors for“Net-ready people” • Factors for shared sense-making • Consistent mental constructs • Cultural affinity • Prior interaction experience • Consistent Doctrines • Factors for collaborative DM • Consistent Doctrines, TTP • Consistent goals/objectives • Will to collaborate • Requisite explicitness • Feedback, confirmation NCW knowledge needs for Trainers

18. Nothing much Nothing much Nothing much When to apply, how to use different info sources When/how to exploit sharing capabilities Factors for effective interaction Factors for“Net-ready people” Factors for shared sense-making Factors for“Net-ready people” Factors for collaborative DM NCW knowledge needs for Trainers • “Gearing” from decision quality to synchronization • What quality factors and how much of each are needed to achieve the behaviour goal

19. NCW knowledge needed for Policy & Force Structure folks

20. Doctrine developers

21. Operators

22. Materiel

23. State of legacy network • Info systems/comms • Human co-location • Equipment capability gaps (implies requirements) • Emerging/available technologies • Cost/capability trade-offs • Nothing much • Individual sensor information that is not shared NCW knowledge needs for Materiel

24. Information service requirements • Driven by TTP? Driving TTP? • Emerging service concepts • e.g. addressable accessible entities • Criteria for “net-readiness” • …and for “net-dependency” • speaks to quality of service and priority for net-readiness • Information user groups • Security architecture requirements • Interoperation doctrine/TTP • Organisation agility requirements NCW knowledge needs for Material

25. Human factors/design • Non-organic information requirements • Cognitive process requirements NCW knowledge needs for Material

26. Legacy collaboration capabilities • Human factors/design for collaboration • Collaboration Technologies • Information requirements for collaboration • Collaborative cognitive process requirements NCW knowledge needs for Material

27. … ? NCW knowledge needs for Material

28. When? • Qualities of evidence for different user needs. • Need for good evidence • Manage the risk of reaching incorrect conclusion or knowledge claims. • Dimensions of the evidence: • Strength - the extent to which the evidence supports the hypothesis claim (the probability of the hypothesis given the evidence). • Credibility – the extent to which the decision-maker is predisposed to believe the evidence based in the source and prior experience with evidence. The extent to which it speaks to his understanding or is coherent with his criteria (or comes from sources he trusts). • Reliability – reproducibility under insignificantly context changes (e.g., different observer) • Fidelity – depth of detail or fineness of focus • Specific/ generic – relevance of evidence to wide range of context, robustness across variable conditions, applicability to the future context • Case to case comparability • Cost to gather evidence traded with quality • minimum acceptable quality • maximum effective quality

29. Qualities of evidence required

30. Where? • ALL evidence, of whatever quality, can be used provided one knows the quality • Sources of evidence • Historical records, personal experiences, synthetic experiences, expert opinion, controlled experiments, “forensic” analysis.

31. Qualities of evidence sources

32. How? • Network-centric knowledge representation • Knowledge distributed and self-synchronised? • Maintenance policy by central dictat or enlightened self-interest? • Using NCW principles in relation to creating/exploiting evidence • Dissemination • National differences ?

33. Summary • Knowledge relevant to NCW serves diverse user communities (e.g., policy makers, force structure, operators, trainers, materiel people, doctrine developers) • Information in the knowledge base can be mapped on to the NCW framework • Information needs to be organized to support each user community • There are technical and national policy issues to build a knowledge base • Evidence (meta-data) to substantiate the information in the knowledge base is available from multiple sources • Historical experiences, personal experience, synthetic, expert opinion, controlled experiments, analysis & synthesis • Information from multiple sources can increase strength • Required strength of evidence varies with how the information is going to be used