Introduction to Data Fusion and CMIF/CUBRC

1. John L. CrassidisProfessor, University at Buffalo, State University of New YorkAssociate Director, Center for Multisource Information FusionAerospace Control and Guidance Systems Committee Meeting

2. Introduction to Data Fusion and CMIF/CUBRC

3. Information Fusion Functional Model(Jt. Directors of Laboratories (JDL), 1993) Tracking Attributes ID/Events Relationships Aggregation Intent Lethality COA Opportunity • Point and Standoff Sensors • Data Sources • Intel Sources • Air Surveillance • Surface Sensors • Standoff Sensors • Space Surveillance Performance Context Consistency Level 0 — Sub-Object Data Association & Estimation: pixel/signal level data association and characterization Level 1 — Object Refinement: observation-to-track association, continuous state estimation (e.g. kinematics) and discrete state estimation (e.g. target type and ID) and prediction Level 2 — Situation Refinement: object clustering and relational analysis, to include force structure and cross force relations, communications, physical context, etc. Level 3 — Impact Assessment: [Threat Refinement]: threat intent estimation, [event prediction], consequence prediction, susceptibility and vulnerability assessment Level 4: Process Refinement: adaptive search and processing (an element of resource management) Detection Reports INFORMATION FUSION PROTOTYPE JEM JWARN3 GCCS Level 0Processing Sub-object DataAssociation & Estimation Level 1Processing Single-ObjectEstimation Level 2Processing SituationAssessment Level 3Processing Mission ImpactAssessment Methods: --Combinatorial Optimization --Linear/NL Estimation --Statistical --Knowledge-based --Control Theoretic Level 4Processing Process Assessment Data BaseManagement System SupportDatabase FusionDatabase

4. Center for Multisource Information Fusion (CMIF) CMIF Organizational Framework State University of New York CUBRC University Center Buffalo Info Fusion ChemBio Defense Hyper- sonics Intl Security Public Safety Medical Biotech ………. Engrg & Applied Sciences CMIF Industrial & Systems Engrg Shared Technical and Administrative Staffs Jointly Managed Major Research University Multidisciplinary Not-for-Profit Jim Llinas, Executive Director Moises Sudit, Managing Director Rakesh Nagi, ISE Chair John Crassidis, Associate Director Mike Moskal, Vice President of CUBRC

5. CMIF/CUBRC Business Model 6.1 6.2 6.3 6.4 6.5 TRL1 TRL2 TRL3 TRL4 TRL5 TRL6 TRL7 TRL8 TRL9 Application Driven Research and Development for Defense, Intelligence and Homeland Security Government Agencies (25+) and Industrial Partners (50+) Universities (~30+) • World Class Research Personnel and Facilities • Focus on Basic Research • Applied RDT&E Focus • Cleared Personnel and Facilities • Systems Engineering • SW/HW Development Development & Transition Engineering Applications Engineering, Fielding & Support USAMRIID Small Businesses (10+) • Specific Technologies Under Development

6. Digital Integrated Air Defense System (DIADS) • Capability to represent interactions of players at a high level of fidelity including • Real world capability to model tracker and correlation capabilities of current and future systems • Demonstrated interfaces to many other high fidelity models for aircraft platforms under test • Interfaces to many other specific threat system models • Description/Customer Needs: DIADS is an emulation fidelity model of the integrated threat air defense system. Important component in many test and training activities • Customer: Air Force • Importance: Key simulation in the evaluation of aircraft effectiveness • Comment: Used at training ranges and key airborne weapons platforms evaluations • Continuous development since 1996 • Derived from a Hardware-in-the-loop simulation • Demonstrated performance in Live, Virtual and Constructive Applications (LVC) • Used by both the test and training community • Capability to run as a mission level model but used more frequently in a multi-model environment

7. Center for Multisource Information Fusion (CMIF) • Mission: Information Fusion and related areas primarily but not exclusively for defense and homeland security applications • Basic and Applied Research in: • Multiple-sensor and instrumented systems • Synergistic Human-Multisensor systems • Real-time Decision-making using Hierarchical Fusion • Graph Theory and Optimization for Level 2/3 Fusion • Multi-modal information environments (speech+text+imagery+RFsensor+human input) • Applications: • Defense: Intelligence/Surveillance/Reconnaissance; Tactical Applications; Homeland Security • Non-Defense: Robotics; Conditioned-Based Maintenance; Medical; Transportation; Geology; Natural Disasters/Crisis Mgmt • History and Funding: • Started in 1996 with Air Force Research Lab Contract • Funding activity evolving; currently ~$10M/year • Scholarly: • Long-standing member of “JDL” fusion group and First President of Intl Society for Info Fusion • Extensive publishing by CMIF PI Team including books, Jl papers, conference papers and review boards • “Critical Issues” Workshops—5 years • CMIF is unique in American Universities as a research activity focused on IF technology for DHS/DoD • Consortium development to include other universities (SU, RIT and PSU) and industrial partners and development of a Graduate-level program in Data Fusion • Currently working on developing a consortium with TAMU and VPI as well

8. CMIF • Only Integrated Information Fusion (IF) Research Center in US Academia • In existence 15 years • Broad range of DoD/Agency sponsored research programs • ~15 Professor/Stakeholders • Systemic approach to IF capability development • Unclassified to Classified, 6.1 to Transition • Collaborations with • PSU • RIT, Syracuse • TAMU, VPI • Buffalo: • Ctr for Unified Biometrics • National Center for Geographic Information and Analysis • Center for Information Systems Assurance • Center for Document Analysis and Recognition • Wireless and Networking Systems Lab • Semantic Network Processing Systems Research Group

9. CMIF/CUBRC Projects

10. Multi-INT Tracker Correlator Intelligent Agent (MTCRIA) • Capability to accept data from different domains • Capability to accept potentially double counted data • Cross-correlate different domains to objects under surveillance • Build “Big Picture” with reduced workload for operators • Description/Customer Needs: Evolving set of algorithms to process data of unknown provenance in a Net-Centric SIGINT Focused Information System (NCSFIS) • Customer: Sierra Nevada Corporation • Importance: Process large volumes of data to assist operators in developing situation awareness • Comment: First integration performed. Additional capabilities in the “pipeline” • Recent effort for CUBRC – Sept 2007 • First integration involves ELINT and COMINT pre-processed reports on static objects • Pre-integration demonstration of measurement-level COMINT, MTI and IMINT • Interplay with Coarse-of-Action Mission-Planning to provide real-time optimization of surveillance asset usage

11. Relative Navigation of Air Vehicles • Problem to be solved • Fully derive relative air vehicle navigation equations • Includes both relative position and attitude • Used quaternion for relative attitude • Used relative LOS observations only • Texas A&M VISNAV sensor • Applications include UAV relative navigation without GPS and fully autonomous refueling • Energy centroid located more accurately than 1 part in 5,000 with proper design, calibration & signal processing, 1 part in 2,000 routinely achieved • 1 to 5 μs rise time  Can be sampled at very high frequency • With proper choice of optics, accuracy of energy centroid is a weak function of the depth of field

12. Formation Flying Attitude Determination • Problem to be solved • Consider only one set of LOS vectors between spacecraft • Rotation around LOS vector is not known • Under-deterministic case (standard attitude determination issue) • Can overcome problem by running filter with motion • Convergence and observability problems need to be overcome • Our goal is to exploit formation information to find a deterministic solution Derived a method that handles nonparallel beams, but also includes range errors. Less of a problem as distance increases Total of 6 “eqns” and 6 unknowns Deputy 2 Aircraft Deputy 1 Aircraft Total of 4 “eqns” and 6 unknowns Total of 2 “eqns” and 3 unknowns Chief Aircraft

13. Multiple Model Adaptive Estimation (MMAE) • Bank of parallel filters, multiple estimates • State estimate is a weighted sum of each filter’s estimate • Measurement residuals are “drivers” of adaptive process • Weights derived from Bayes rule • Weights are probabilities • Can work with nonlinear systems • EKF assumptions must be valid • Measurement residual must be Gaussian • CMIF Extension • Generalized MMAE (GMMAE) uses a window of residuals • Combines autocorrelation for isteps back with MMAE • When i = 0 standard MMAE is achieved • Goal is to improve convergence • Many applications • Current research involves L1/L2 integration • Currently extending to UKF GMMAE • Collaborations with Simon Julier Unknown System Tracking Example with i = 4 Real System MMAE Filter KF 1 KF 2 Process Noise Variance KF M Posterior PDF

14. MMAE Applications • Robust Target Tracking • Adaptive Signal Processing • Navigation applications • Adaptive Control • Communication Systems • Tactical Assessment Parameter Identification Filter Tuning MMAE Health Monitoring and Fault Detection • Sensor and actuator degradation faults • Structural and mechanical health monitoring • Reconfigurable (intelligent) systems • Higher level fusion applications

15. Decentralized Estimation FN – red (solid) LNN – all others (dashed) • Problem to be solved • Reduce single point failures in estimation system architectures • Fuse multiple filters (nodes) in optimal manner • Use optimal platform (sensor) motion to lower estimation errors • Fusion node (FN) uses only an estimate and covariance from each Local node (LN) • Our objective is to provide a robust estimation system architecture for dynamic problems • e.g. Target Tracking

16. Optimization Planning and Tactical Intelligent Management of Aerial Sensors (OPTIMAS) Project Description Discrete Optimization Models are designed to solve the “right” problems considering spatial as well as temporal constraints with the objective of maximizing the delivery of services by the entire fleet of UAVs: ● Maximize the servicing of certain predefined “targets” ● UAVs are constrained by the amount of time they can fly before refueling, the amount of service they can provide and a lower/upper amount of service the entire fleet should provide to a specific “target”. ● Consideration of threat, enemy attack, or natural hazards ● Novel Discrete Optimization techniques Management of Emerging Targets: ● Targets need to be classified into an appropriate priority ● UAVs scheduling and Sensor-Platform Assignment Significance Implementation of the OPTIMAS architecture into the overall Command and Control and Combat Systems (C2 and CS) Program will greatly enhance the decision-making effectiveness of maritime operators by increasing overall situation awareness and presenting optimal or high quality solutions to a suite of strategic and tactical decision making problems. The OPTIMAS suite of tools provides feedback to the operators and to the tools themselves both within and between the strategic and tactical level issues/problems. OPTIMAS facilitates the complex decisions to be made in an atmosphere of tight resources and dynamically changing environments taking advantage of the computational strength of computers in the human-computer interaction paradigm. Flight Dynamics – ‘Real’ and Approximated by Dubins Vehicles Dubins Vehicles: route tracking by a kinematic vehicle moving forward only with a lower bounded turning radius

17. Gaussian Sum Filter-A step towards exact nonlinear filter • Approximate the conditional pdfas a mixture of Gaussian components • The mean and covariance of each of the Gaussian component is propagated by using the extended Kalman filter or unscented Kalman filter • Two update schemes for the forecast weights • Continuous Dynamical Systems: minimize the Fokker-Planck-Kolmogorov Equation (FPKE) error • Discrete Dynamical Systems: minimize the integral square difference between the true pdf and its approximation • During the measurement update, Bayes rule is used to update the weights • Unique solution for weights is guaranteed • Future Work • Automatic selection of number of minimum number of Gaussian components required • Improve computational cost through parallelization • Many applications • Orbit determination, Attitude Estimation • Asteroid Collision Probability • GPS-less Localization • Plum Tracking • Currently extending to UKF Gaussian Sum Filter • Collaborations with Simon Julier