Example – ACMREQ Processing

1. Example – ACMREQ Processing For Internal GILA Kick-Off Meeting May 30-31, 2006 Martin Hofmann, Ph.D. Manager, AI Programs Advanced Technology Laboratories

2. Starting from the Main Goal – Ace the Evaluation: From the PIP The metrics that will be used for evaluation are • Coverage percentile – how much was it able to correctly learn? For instance, if it is shown 40 steps in a plan, was it able to learn 39 correct steps? Sample instances of the coverage metric include (others might include testing for iteration, conditionals, etc.): (1) step coverage percentile: SCP = # steps learned correctly / # steps shown; (2) choice coverage percentile: CCP = # choices learned correctly / # choices shown. • Error rate – how many errors occurred and what was learned? Here we tabulate errors. Sample instances include (1) step error percentile: SEP = # steps learned incorrectly / # steps shown; (2) choice error percentile: CEP = # choices learned incorrectly / # choices shown. • Goal achievement – can the learner achieve the desired goal? If so, how well? Goal achievement cannot be directly determined by the above two metrics. A learner might correctly learn 40 out of 40 steps but then follow those 40 steps with 12 errors in which the goal is undone. There are also degrees of goal achievement, e.g., in a physical domain one can easily imagine putting together an object and having a few parts left over or having the object be functional but wobbly. The same ideas apply in computational domains, e.g., data analysis can be less complete than desired. To measure how well a goal is achieved, we will categorize goal achievement levels and score 0-10. GILA needs to learn correct steps in a process

3. Evaluation Plans and Metrics Program of System Evaluation and Research Experimentation • System Evaluation: Measure • system against human performance • Evaluations by Lockheed Martin, supported by research partners • Specific ATO domain problems • Results used as: • Go/no-go criteria • Feedback to support definition of research goals • Support for transition planning and implementation • Go/No-Go testing is done in this • military-relevant domain That’s what we told Tom • Research Experimentation: • Conducted using Wargus strategy • application to provide a platform for • controlled, intensive experimentation. • Experiments performed by individual researchers and the integrators, with the results used to: • Support ongoing research • Serve as regression testing • Validate individual technology components • Domain easily shared with other IL • teams for cross-project comparisons Lockheed Martin Proprietary Information

4. System Evaluation Readable from the back… Lockheed Martin Proprietary Information

5. Initial Process Description • In WEBAD tool, check if new ACMREQ airspace appears in the “Imported” group (displayed as a list) of airspaces • In WEBAD Tool, run conflict checking function (via button push) • Iterate over the listed conflicts with existing airspaces (airspace control measures (ACM)) • Select conflicting ACM to review its detailed description • If the new requested ACM is an exclusion ACM, compare its priority (derive from usage) to the conflicting ACM(s) • If the requested ACM has higher priority, request (via email, chat, or phone call – the POC is listed in the ACMREQ) to replan the lower priority ACM and any sorties already planned in the ATO that use this ACM • If the conflicting ACM is only a procedural boundary, then • In WEBAD tool, approve the ACMREQ • Check the time and space separation of the ACMs and any planned missions • Decide if the overlap can be removed, simply by shrinking an ACM. • In TBMCS, retrieve elements of the ATO that fall into the 4D volume of the conflicting ACM • Decide if there is • Time/space in the ACM that is not used by any mission • Sufficient separation within the missions in each ACM • If so, decide which ACM to alter, and specifically, which part to reduce, then notify the requester POC. • If the airspace is only “on standby” during the conflict space-time: • On the GCCS Common Operational Picture (COP), list any aircraft currently occupying the conflicting ACM space • If none, approve the ACMREQ • If the “Standby airspace” (e.g. a missile engagement zone) becomes active (this is driven by enemy actions, thus not predicatble), then change any mission that conflicts (airspaces and missions) • Otherwise, contact the ACMREQ POC to revise the request to eliminate the conflict, using chat, email, or phone. • Note: May do this in multiple levels of detail: – start with “super-box”, if this is ok, don’t worry about finer detail airspace deconfliction. Could you process an ACMREQ correctly based on this alone?

6. For example 1 • If the conflicting ACM is only a procedural boundary, then • In WEBAD tool, approve the ACMREQ

7. Example - Options 1 Using myself as an example novice: • We do not specify this condition in the basic process • GILA observes the expert – the expert approves ACMREQ without further checking • GILA figures out the condition – one of the airspaces is a procedural boundary • We specify this condition • GILA observes the expert approving the ACMREQ • GILA learns the kinds of airspace types that are procedural boundaries I have learned this already I could not tell you this right now – I’d have to learn this from the expert!

8. For example 2 How does GILA know this? • Check the time and space separation of the ACMs and any planned missions • Decide if the overlap can be removed, simply by shrinking an ACM. • In TBMCS, retrieve elements of the ATO that fall into the 4D volume of the conflicting ACM • Decide if there is • Time/space in the ACM that is not used by any mission • Sufficient separation within the missions in each ACM • If so, decide which ACM to alter, and specifically, which part to reduce, then notify the requester POC. I later learned that the ATO is not available until late afternoon – can’t do this step until it is published! Instead – might ask the requester Maybe learn what is sufficient separation for various missions?