Test & Evaluation/Science &Technology Net-Centric Systems Test (NST) Focus Area Overview

Test & Evaluation/Science &TechnologyNet-Centric Systems Test (NST)Focus AreaOverview NAVAIR Pt. Mugu Executing Agent Gil Torres

T&E/S&T Program Overview • The T&E/S&T program: • Exploits new technologies and processes to meet important T&E requirements • Expedites the transition of new technologies from the laboratory environment to the T&E community. • Leverages commercial equipment, modeling and simulation, and networking innovations to support T&E • Examines emerging requirements derived from joint service initiatives • Identifies needed technology areas and develops a long-range roadmap for technology insertion • Leverages research efforts from the highly developed technology base in the DoD labs, test centers, other government agencies, industry, and academia

InterTEC Netcentric Test Components Net-Centric Systems Test Focus Area Net-Centric Technology Feed TRMC OrganizationOSD Test Resource Management Center Director Deputy Director Joint Investment, Policy, & Planning Deputy Director Budget & Resources Deputy Director Strategic Planning Policy JMETC CTEIP Program S&T Program

Technology Gaps Technology Gaps BuildCharacterize & replicate JNO Emulate Networks & Services Construct NST Infrastructure Sim/Stim Battlespace Components • MeasureMeasure & Verify NR-KPP compliance • Assess Joint Mission Effectiveness • Automate analysis and visualization TechnologyGaps Net-Centric Systems Test (NST) Focus Area Overview GIG SERVICES & DATA TACTICAL DATA Netcentric Test Environment Recreate Net-Centric Test Battlespace Measure & Analyze Net-Centric Test Environment. Manage:Dynamically manage & monitor Test Env. Automated test planning & reporting Access & interface with the Netcentric Env. Manage Net-Centric Test Environment

NST Vision Solutions NST Reference Architecture Warfighter SME COITechnology LabJoint Architectures Netcentrtic Standards Net-Centric Policy GAP GAP GAP Build Manage Measure Accelerate the delivery of Mission Ready Net-Centric capabilities to the warfighter Net-Centric Tomorrows Plug & Test Net-Centric Environment Today’s Netcentric Environment • Stove-Piped Information• • Centralized Control• • Unique Software Solutions• • Data Not Shared• • Inefficiency• • Net-Centricity• • Decentralized Control• • Enterprise Services• • Shared Data• • Autonomous Agent• • Enterprise Architecture• • Web 2.0 NST PortfolioNST T&E Advanced Technologies/Capabilities Broad Agency Announcement NST T&E Needs Technology Gaps

NST T&E Needs(1 of 2) Recreate the Net-Centric Test Battlespace Emulate tactical edges Test mission threads with limited participants Integration/Responsiveness of Virtual Component with Live Component Integration of Cyber/Information Operations with Kinetic Simulations Representation of the Network Infrastructure and Network Environment Representation of Cyber threat to NSUT Simulate/Stimulate Irregular Warfare for urban environment Verification & Validation (V&V) of the Synthetic Battle Environment (SBE)

NST T&E Needs(2 of 2) Measure & Analyze Net-Centric Test Environment Monitor/Analyze Critical Chaining Issues Improving Near Real Time Analysis Improve Instrumentation & Analyzing Cyber/IO Manage Net-Centric Test Battlespace Quality of Service to Critical Nodes Synchronize Synthetic Environments Manage Test Environment for Net Based Systems Manage Tactically Dynamically Configured Networks

Recreate Net-Centric Test Battlespace Flexible, Scalable LVC Environment For JMe Testing GIG & SOA Sim / Stim Capabilities Integrate & Validate Net-Centric Simulations & SoS Emulate Red Cyber Warfare Capabilities Automate C2 Decision Process Measure & Analyze Net-Centric Test Environment Analyze Joint Mission Threads Near Real-Time Evaluate Net-Centric Data & Services Automated NR-KPP Analysis Measure Joint Mission Effectiveness (JMe) Analyze & Visualize Information Assurance & Operations Manage Net-Centric Test Battlespace Dynamically Manage Test Infrastructure Automated Planning and Scenario Development Cross Domain Solution Over Distributed Test Infrastructure NST Challenges“Technologies to. . . .”

NST Roadmap – Recreate the Net-Centric Test Battlespace T&E/S&T Investments • Emulate Tactical Edges NE2S – Network Effects Emulator System TRCE – TENA in a Resource Constrained Environment • Test Mission Thread w/Limited Participants • Technologies to V&V NST Env. • Integration/Responsiveness of VC with Theatre/Live Component • Representation of Cyber Threat to NSUT • Sim/Stim Irregular Warfare • Representation of Network Infrastructure and Env. NE2S • Integrate Internet into Family of Tested Nets TRCE • Integrate IO with Kinetic Sim’s Challenges Emulate Red Cyber Warfare Capabilities Flexible, Scalable LVC Environment For JMe Testing Integrate & Validate Netcentric Simulations & SoS Automate C2 Decision Process GIG & SOA Sim / Stim Capabilities T&E Capability Recreate the Net-Centric Test Battlespace

NST Roadmap – Measure and Analyze the Net-Centric Test Environment T&E/S&T Investments NECM – NST Evaluation Capability Module NEIV – Net-Centric Environment Instrumentation & Visualization NSAT – NR-KPP Solution Architecture Toolset NECM NEIV NSAT • Monitor/Analyze Critical Chaining Issues • Improving Near Real Time Analysis • Analyze impact of Cyber and IO Challenges Analyze Joint Mission Threads Near Real-Time Measure Joint Mission Effectiveness (JMe) Evaluate Netcentric Data & Services Analyze & Visualize Information Assurance & Operations Automated NR-KPP Analysis T&E Capability Measure and Analyze the Net-Centric Test Env.

NST Roadmap – Manage the Net-Centric Test Battlespace T&E/S&T Investments MENSA – Middleware Enhancements for Net-Centric Simulation Architecture ANSC – Analyzer for NST Confederations MLSCLS – Multi-Level Security Cross Layer Scheme PAM – Policy Based Adaptive Network Management RRM – Rapid Reconfiguration Module MENSA ANSC MLSCLS • Synch Synthetic Environments PAM • Manage Test Env. For Net-based systems RRM • Manage Tactically Dynamic Configured Networks • QoS to Critical Nodes Challenges Dynamically Manage Test Infrastructure Cross Domain Solution Over Distributed Test Infrastructure Automated Planning and Scenario Development T&E Capability Manage the Net-Centric Test Battlespace

T&E/S&T Project Critical Factors • T&E Need (Gaps) • Address a known T&E technological gap. • Understanding of the current T&E capabilities and techniques • Understand capabilities and techniques for the future of T&E • S&T Challenge • What needs to be done to effectively address the T&E Need • Efficiencies and test capacity improvements as a result of the technology developed or matured • Results of project will significantly advance technology and thereby significantly enhance current test capabilities • High Risk-High Payoff • The project improves testing rigor and capabilities relevant to Net-Centric technology improvements • Reduces the cost of government operations • Provides an improved capability to test operators

T&E/S&T Project Critical Factors • Wide Application • The technology can be applied at multiple sites, on multiple systems, in multiple organizations simultaneously • The technology effectively supports a distributed LVC Net-Centric test event • Transition Strategy • Identify potential partners, future consumers of the technology (government lab, agency, test activity, industry, etc). • Identify the intended use of the technology, and a plan to obtain buy-in to the technology from potential transition partners • Demonstrate a concise and sensible plan that will create a higher probability of a successful transition to the identified transition partners

Project Execution Plan (PEP) • PEP required contents: • Purpose and Work Scope • T&E Needs • S&T Challenges • Technology Readiness Levels • Budget • Project Schedule • Project Specifications • Phase Exit Criteria • Project Transition • List of Deliverables

Project Specifications

Netcentric Systems Test Technology Investments & Transitions (1 of 7)

Questions?

T-E Web Monthly Reports • Monthly reporting timeliness is critical • Timeliness of report submission is important to effectively communicate project status • Submission to Deputy EA required by the 5th of the month for prior month • Content: • Executive Summary of the project • Funding O&E • Activities for the month just completed • Significant accomplishments • Planned efforts for the next month • Problems/Issues • Brief explanation on use of funds

FAM Monthly Reports • Focus Area Monthly (FAM) reports • EA required to submit to PMO on the last Friday of each month • Highlight significant (positive and negative) events or issues in projects • Not a substitute for more timely notification to PMO of highly significant issues or events • Issues requiring immediate action • Events that could result in “bad press” • FAM reports are now auto-generated from T-E Web, so significant accomplishments and issues are very important to report each month Keep the Focus Area Monthly Reports simple—no lengthy description of normal, scheduled activities—just good news and problems

Back-Up Slides

Weekly Activity Reports • Due to EA every Wednesday by noon Pacific Time • Sample Content: • Netcentric Systems Test (NST) NST Evaluation Capability Module (NECM): The NECM project is developing an automated object-relationship graph algorithm . . . . • This technology will automate the arrangement of operational activities, control logic, sequence flows, and . . . • This algorithm enables improved user interaction to create a. . . • During the week of 11 January 2010, the NECM team implemented an initial Joint Personnel Recovery (JPR) joint mission thread (JMT). . . • The future efforts will include. . . • Status/Assessment: Green – Early engagement of transition partners to define appropriate use case to verify useful test technologies, and project is operating on schedule and on budget.

T&E/S&T Program Roles and Responsibilities • T&E/S&T Program Manager • Provides guidance to focus areas to ensure disciplined project selection and funding process • Provides program funds for administering and executing T&E/S&T Program • Reviews technical execution of focus areas and projects to ensure T&E/S&T Program objectives are being met • Reviews financial execution of focus areas and projects to ensure best value and fiscal responsibility • Provides TESTWeb and the T&E/S&T Acquisition Community Connection (ACC) Special Interest Area to support exchange of financial, technical, and programmatic information throughout the T&E/S&T Program • T&E/S&T Program Office Staff • Supports the T&E/S&T Program Manager • Reviews focus areas and projects to ensure technical objectives are being met • Reviews focus areas and projects to ensure financial objectives are being met • Provides recommendations to the T&E/S&T Program Manager on Program related matters • Interfaces with EAs, Deputies, WG members and PIs on behalf of T&E/S&T Program Manager as required

T&E/S&T Program Roles and Responsibilities • Focus Area Executing Agents • Primary point of contact for T&E/S&T Program Office within focus area • Maintain awareness of DoD T&E needs within focus area and technology developments related to focus area • Actively seek out high payoff T&E/S&T projects that address critical DoD T&E needs and mitigate identified T&E risks • Issue Requests for Information (RFIs) and Broad Agency Announcements (BAAs) through contractual channels • Responsible for technical and financial execution of projects approved by T&E/S&T Program Manager • Chair for focus area working group (WG) and responsible for maintaining Tri-Service WG T&E and S&T representation • Coordinate replacements with T&E/S&T Program Manager • Responsible for maintaining current project and focus area information in TESTWeb and T&E/S&T ACC Special Interest Area

T&E/S&T Program Roles and Responsibilities • Focus Area Working Group Members • Act as government leads on projects – where there is no intrinsic government lead • Provide coordination of T&E/S&T projects with Service related efforts • Identify and prioritize T&E needs • Support the EA on source selection efforts • Review ongoing focus area projects • Identify transition opportunities for projects • Project Principal Investigators • Execute projects approved by T&E/S&T Program Manager • Conduct research efforts • Provide technical progress reports to EA • Identify and actively pursue transition opportunities for T&E/S&T projects • Responsible for maintaining accurate financial information in TESTWeb

T&E/S&T Program Roles and Responsibilities • Government Lead • Attend major focus area reviews (when able) • Provide guidance to the project to ensure a disciplined engineering approach • Review technical execution of project to ensure NST Program objectives are being met; gaps are being satisfied • Provides quick review of monthly reports to ensure accuracy • Interfaces with EAs, Deputies, WG members and Principle Investigators as required • Maintain awareness of identified T&E needs within the NST focus area and technology developments related to focus area • Support the EA on source selection efforts • Support EA in Identifying transition opportunities for project

Publications • Professional publications are encouraged by the TRMC and PMO • Provides opportunities for transition of technology • Raises awareness of T&E/S&T Program to new communities • Public Release of Technical Information • EAs adhere to local procedures for release of technical information in EA’s focus area • Copies of all technical information to be released are to be sent to the T&E/S&T Principal Scientist prior to release • Use the ACC Special Interest Area https://acc.dau.mil/CommunityBrowser.aspx • Include acknowledgement of the T&E/S&T Program in all publications • Example: “The authors would like to thank the Test Resource Management Center (TRMC) Test and Evaluation/Science and Technology (T&E/S&T) Program for their support. This work is funded (or is partially funded) by the T&E/S&T Program through the (Put your organization’s name here) contract number xxxxxx.”

Release of Programmatic Information • The PMO must review and approve release of programmatic information prior to transmission outside the program • Applies to detailed project status, funding levels and execution, schedule, problem areas, and potential management issues • Also applies to Solicitations (BAAs, PRDAs, RFIs) • Not intended to restrict information exchange between EAs or core focus area working groups • This applies to written and verbal release of information • For publications, PMO requires a minimum of 10 days before EA’s next suspense on the publication • For verbal or email Request for Information, call or email PMO for approval before information is released • Option is to refer requestor to PMO for the programmatic information

Technology Readiness Levels (TRL)

TRL (Cont)

Definitions • Breadboard:A breadboard is an experimental model composed of representative system components used to prove the functional feasibility (in the lab) of a device, circuit, system, or principle without regard to the final configuration or packaging of the parts. Breadboard performance may not be representative of the final desired system performance, but provides an indication that desired performance can be achieved. • Brassboard:A brassboard is a refined experimental model that provides system level performance in a relevant environment. The brassboard should be designed to meet (at a minimum) the threshold performance measures for the final system. • Prototype:A prototype is a demonstration system that has been built to perform in an operational environment. A prototype device meets or exceeds all performance measures including size, weight, and environmental requirements. This device is suitable for complete evaluation of electrical and mechanical sub-system form, fit, and function design and characterization of the full system performance.

Accounting Definitions • Obligations • The amount of an order placed, contract awarded, service rendered, or other transaction that legally reserves a specified amount of an appropriation or fund for expenditure • Accruals • The costs incurred during a given period representing liabilities incurred for goods and services received, other assets acquired and performance accepted, prior to payment being made • Disbursements • The charges against available funds representing actual payment and evidenced by vouchers, claims, or other documents approved by competent authority • Expenditures • The total of disbursements plus accruals

Test & Evaluation/Science &Technology Net-Centric Systems Test (NST) Focus Area Overview