570 likes | 713 Views
C2 Maturity Model Experimental Validation. Statistical Analyses of ELICIT Experimentation Data Dr. David S. Alberts. Agenda. Background Available Data Analysis Plan Results of Analyses Conclusions. Background. SAS-065 has developed a C2 Maturity Model for NATO Network Enabled Capability
E N D
C2 Maturity Model Experimental Validation Statistical Analyses of ELICIT Experimentation Data Dr. David S. Alberts
Agenda • Background • Available Data • Analysis Plan • Results of Analyses • Conclusions
Background • SAS-065 has developed a C2 Maturity Model for NATO Network Enabled Capability • SAS-065 is engaged in a variety of activities to “validate” this model • Case studies: Analysis of historical complex endeavors to establish construct validity • Analysis of Experimentation Data to test a set of maturity model hypotheses • ELICIT is an experimentation platform that instruments the actions of a group of participants engaged in a situational awareness problem • ELICIT experiments have explored differences between “edge” and “hierarchical” organizations
ELICIT Scenario Hierarchy Edge • The goal of each set of participants is to build situational awareness and identify the who, what, when, and where of a pending attack • Participants can share factoids directly with each other or post factoids to websites • Participants build awareness by gathering and analyzing factoids and interacting with one another • No one is given sufficient information to solve their assigned problem without receiving information from others • The receiving, sharing, and posting of factoids and the nature of the interactions between and among participants can be constrained • C2 approach for this series of experiments were designated prior to the start of the run asHierarchy or Edge
Hypotheses • “Hierarchical” organizations as instantiated in ELICIT experiments correspond to De-Conflicted C2 in the NATO C2 Maturity Model • “Edge” organizations as instantiated in ELICIT experiments correspond to a higher level of maturity in the NATO C2 Maturity Model • Hypotheses: • Edge organizations exhibit the behaviors associated with Coordinated and Collaborative levels of C2 Maturity • Hierarchical organizations exhibit the behaviors associated with the De-conflicted level of C2 Maturity • Edge organizations exhibit higher levels of effectiveness and efficiency than Hierarchical organizations
ELICIT Data Set • Includes data from 37 ELICIT experimentation trials • Venues • Boston Univ. (2 runs) • NPS (16 runs) • Portugal (6 runs) • USMA (3 runs) • Singapore (10 runs) • Organization Types • 18 Edge • 19 Hierarchy
Agenda • Background • Available Data • Analysis Plan • Results of Analyses • Conclusions
Variables of Interest C2 Maturity Level Allocation of Decision Rights Patterns of Interaction Network Characteristics & Performance Distribution of Information Shared Understanding Shared Information Shared Awareness Task Performance Info Sharing & Collaborative Behaviors Quality of Information Quality of Awareness Quality of Understanding Measures of Merit Task Difficulty Individual & Team Characteristics Legend Controllable Quality of Information Sources Partially Controllable Culture
Dependent Variables • MOE = Quality of Awareness and Shared Awareness • Correctness (Authorized Correct IDs) • Timeliness (Person-Minutes with Correct IDs) • Accuracy rate (Correct IDs/Total IDs) • Efficiency, Given Effectiveness • Productivity (Correct IDs/Total Actions; Correct IDs/Person-Minutes Available) • Speed (Time of Earliest Correct ID) • Agility • Effectiveness over problem difficulty
C2 Approach Independent Variables • Hierarchy v. Edge • We expect Hierarchy to map to De-conflicted and Edge to map to a more mature level • Each run will be mapped to a point in the C2 Approach Space based on observed behaviors • Rules of Interaction • Website access • Sharing permissions • Initial Distribution of Factoids • Invariant in existing runs
C2 Approach Intervening Variables • Patterns of Interaction • Characteristic path length • Clustering coefficient • Connectedness • Distribution of Information • The average number of unique facts to which each participant has access as a function of time
Measures of C2 Effectiveness (MOCE) • Quality of Information Position • Percentage of relevant facts for the assigned task that a participant can access as a function of time • Percentage of key facts for the assigned task that a participant can access as a function of time • Extent of Shared Information • The average number of participants that have access to each fact as a function of time • The average number of participants that have access to each key fact as a function of time
Intervening Behavioral Variables • Activity over time (sharing, website posts, website pulls, ID attempts) • Sharing • Peer-to-peer sharing • Posting • Information Seeking • Pulling • Identification Attempts
Other Independent Variables • IDs allowed • ELICIT experience of the player • Factoid set (problem difficulty) • Translated factoids v. original • Native Language (English v. Other) • Communications media • Postcards • Chat • Time available • Degree of Education (Graduate, Undergraduate) • Seniority (Rank) • Subcultures (Military, Civilian, Special Forces, Civil Servants)
Agenda • Background • Available Data • Analysis Plan • Results of Analyses • Conclusions
Dependent Variables • MOE = Quality of Awareness and Shared Awareness • Correctness (Authorized Correct IDs) • Timeliness (Person-Minutes with Correct IDs) • Accuracy rate (Correct IDs/Total IDs) • Efficiency, Given Effectiveness • Productivity (Correct IDs/Total Actions; Correct IDs/Person-Minutes Available) • Speed (Time of Earliest Correct ID) • Agility • Effectiveness over problem difficulty
MOE: CorrectnessFraction of Participants with Correct IDs Edge Hierarchy
MOE: CorrectnessFraction of Participants with Correct IDs Edge Better Than HierarchySignificant at 0.95 level
MOE: CorrectnessFraction of Participants with Authorized Correct IDs Edge Hierarchy Note: Half credit given for partially correct answers in authorized areas
MOE: CorrectnessFraction of Authorized Participants with Correct ID Edge Better Than HierarchySignificant at 0.95 level |t-Ratio| = 7.46
Measuring Timeliness:Person-Minutes with Correct ID • In the ELICIT scenario, operational value is associated both with the number of participants achieving the correct answer, as well as when they achieved it • Person minutes with correct ID captures the proportion of possible situational understanding achieved by trial participants over the time of the trial • For each minute, each participant’s level of understanding is assessed by the score assigned to his/her most recent identification attempt • This value is summed over the duration of the trial to arrive at the person-minutes correct for each participant • Person-minutes correct for the trial is calculated by summing over all participants • A ratio of the person minutes correct for the trial to the total person minutes available in the trial adjusts for varying trial lengths
MOE: TimelinessFraction of Person-Minutes Correct Edge Hierarchy
MOE: TimelinessFraction of Person-Minutes Correct Edge Better Than HierarchySignificant at 0.95 level
MOE: Accuracy RateFraction of ID Attempts that are Correct Edge Hierarchy
MOE: Accuracy RateFraction of ID Attempts that are Correct Edge Better Than HierarchySignificant at 0.95 level
Efficiency: Productivity (Actions)Correct ID / Total Actions Edge Hierarchy
Efficiency: Productivity (Actions)Correct ID / Total Actions Difference not Significantat 0.95 level |t-Ratio| = 1.41
Efficiency: Productivity (Person-Minutes)Correct IDs / Person-Minutes Available Edge Hierarchy
Efficiency: Productivity (Person-Minutes)Correct IDs / Person-Minutes Available Edge Better Than HierarchySignificant at 0.95 level
Efficiency: SpeedTime of Earliest Correct ID Edge Hierarchy
Efficiency: SpeedTime of Earliest Correct ID Difference not Significant at 0.95 level |t-Ratio| = 1.62
AgilityMOE: Timeliness over Problem Difficulty Fraction of Person-Minutes Correct Fraction of Person-Minutes Correct DifficultFactoidSets Fraction of Person-Minutes Correct Fraction of Person-Minutes Correct StandardFactoidSets Edge Hierarchy Edge Effectiveness is Degraded Less Than Hierarchy
C2 Approach Intervening Variables • Patterns of Interaction • Characteristic path length • Clustering coefficient • Connectedness • Distribution of Information • The average number of unique facts to which each participant has access as a function of time
Distribution of InformationThe cumulative number of unique facts to which each participant has access over time Time (minutes)
Distribution of InformationThe cumulative number of unique facts to which each participant has access over time Edge Better Than HierarchySignificant at 0.95 level Time (minutes)
Measures of C2 Effectiveness • Quality of Information Position • Percentage of relevant facts for the assigned task that a participant can access as a function of time • Percentage of key facts for the assigned task that a participant can access as a function of time • Extent of Shared Information • The average number of participants that have access to each fact as a function of time • The average number of participants that have access to each key fact as a function of time
MOCE: Quality of Information PositionPercentage of relevant facts for the assigned task that a participant can access as a function of time Time (minutes)
MOCE: Quality of Information PositionPercentage of relevant facts for the assigned task that a participant can access as a function of time Edge Better Than HierarchySignificant at 0.95 level Time (minutes)
MOCE: Quality of Information PositionPercentage of key facts for the assigned task that a participant can access as a function of time Time (minutes) Key fact: Factoid Labeled as “Key” or “Key-Expertise”
MOCE: Quality of Information PositionPercentage of key facts for the assigned task that a participant can access as a function of time Edge Better Than HierarchySignificant at 0.95 level Time (minutes) Key fact: Factoid Labeled as “Key” or “Key-Expertise”
MOCE: Extent of Shared InformationThe average number of participants that have access to each fact as a function of time Time (minutes)
MOCE: Extent of Shared InformationThe average number of participants that have access to each fact as a function of time Edge Better Than HierarchySignificant at 0.95 level Time (minutes)
MOCE: Extent of Shared InformationThe average number of participants that have access to each key fact as a function of time Time (minutes) Key fact: Factoid Labeled as “Key” or “Key-Expertise”
MOCE: Extent of Shared InformationThe average number of participants that have access to each key fact as a function of time Edge Better Than HierarchySignificant at 0.95 level Time (minutes) Key fact: Factoid Labeled as “Key” or “Key-Expertise”
Intervening Behavioral Variables • Activity over time (sharing, website posts, website pulls, ID) • Sharing • Peer-to-peer • Posting • Information Seeking • Pulling • Identification
Activity over Time Peer to Peer Sharing Time (minutes)
Activity over Time Peer to Peer Sharing Hierarchy Better Than Edge at 10 minNo Significant Difference at 45 min(at 0.95 level) Time (minutes)
Activity over Time Posts Time (minutes)
Activity over Time Posts No Significant Differenceat 0.95 level Time (minutes)
Activity over Time Information Seeking (Pulls) Time (minutes)