Topic 4: Sensor Processing

1. Topic 4: Sensor Processing David L. Hall

2. Topic Objectives • Introduce the input side of data fusion processing • Provide a brief survey of sensor types • Describe how a generic sensor works • Provide a basis for analysis of sensors for your selected application

3. Sensors: The Input Side of Data Fusion

4. Classes of Input • Traditional “hard” sensors • Observation of physical phenomena via physical sensors • “Soft” sensors • Humans as observer/reporters • Direct input (e.g., via cell phone) • Indirect input (web logs, etc.) • Emerging sensors • Emulation of biological sensors via new nano-scale technologies (e.g., electronic noses) The rapid evolution of sensor technology provides opportunities and challenges: Nano and micro-scale sensors include the capability for self-location, wireless communication in self-organizing networks, and web-based serves; sensors may be mobile and provide extensive on-sensor computations

5. Examples of New Sensors: Intel Mote • 32 bit processor • Blue tooth wireless comm. http://www.intel.com/research/exploratory/motes.htm

6. Smart Dust Vision www.sice.umkc.edu/~leeyu/Udic/Groups/SensorNetworksPresentationFINAL_0909.ppt

7. Examples of New Platforms for Sensor Deployment Military platforms New space-based environmental sensors Smart factories & buildings 10,000 surveillance cameras in London > 2.7 B Cell Phones

8. Examples of Military Sensor Platforms

9. Examples continued

10. Representative Observable-Sensor Pairings

11. The Frequency Spectrum of Sensors

12. Representative Sensor Characteristics

13. Adaptive Control Heuristic Data Time Function Inside a Generic Sensor ENERGY EMISSION SENSOR ELEMENTS Sensor Input SENSOR GUIDANCE/CONTROL • tasking • control data • tip-off • information • environmental • data Steering Control Energy Emission Signal Conditioning - Translation - Analog/Digital - Digital/Analog - Detection Signal Processing - Filtering - Transforms - Thresholding - Storage - Special Algorithms Information Processing/ Decision Making - Look-up Tables - Bit Map - Heuristics - Declaration Matrix Map Output Processing - Buffering - Data Conversion - Coordinate Transforms - Smoothing - Filtering image y(t) • Input Energy • intentional • unintentional • jamming

14. DATABASE • Target characteristics • Environment/sensor models • Signal models X(t) CANONICAL TRANSFORMATION natural Hilbert space representation of signals SIGNAL INFORMATION ANALYSIS characterize and analyze a signal SIGNAL PARAMETRIZATION and DETECTION determine a presence of signal and represent via attribute vector Time • Examples • Fourier • Wavelet • Gabor • Walsh • etc. • Examples • Time series • - State space • - ARMA • - HOS • Matched filter • Range-Doppler • Ambiguity function Time Series Processing Image • Examples • Feature extraction • - Peak detection • - Shape characterization • - Statistic summaries • Detection logic • - Hypothesis testing • - Thresholding • Tracking/geolocation • - Position estimation • - Kinematic estimation • Attribute estimation Single Sensor Target Detection and Parameter Estimation

15. Range Measurement Probability of Detection Vulnerability to Detection Vulnerability to CM SENSOR TYPE Detection Range Detection Time Target ID RELATIVE PERFORMANCE • Radar • Direct Beam • SAR • IR Imagery • Passive • Augmented • TV Imagery • Passive • Augmented • EW Sensors • RWR • ESM • Acoustic/ Seismic • Direct View Optics • Passive • Augmented • Optical Augmentation Sensors • E-O Tracking • IRSTR • LADAR Poor Fair Good * Steinberg, DFS-87. Sensor Selection Matrix (Steinberg)

16. Radar: Radio Detection & Ranging • Basic Measurements: • Radar cross-section • Frequency • Time • Derived Measurements: • Range • Azimuth • Elevation • Velocity • Target Size/Shape • Signature • State Determination • Position, velocity, identity • Limited target I/D from radar cross-section signature • Target size/shape may be obtained in limited cases SPY-1 Phased Array Radar 1. Skolnick, M.I., Ed., Radar Handbook, McGraw-Hill, NY, 1970. 2. Barton, D.L., Modern Radar System Analysis , Artech House, Norwood, MA, 1988. 3. Stein, A., "Bistatic Radar Applications in Passive Systems," Supplement to the January 1990 Journal of Electronic Defense , Horizon House Microwave, Norwood, Massachusetts.

17. LIDAR: Light Detection and Ranging • Basic Measurements: • Optical intensity • Range • Angular information (direction) • Derived Measurements: • Visual signature • State Determination • Location • LIDAR is the visual light equivalent of RADAR. Cruickshank, J.M., and R.C. Harney, Eds., Laser Radar Technology and Applications, SPIE Proceedings, Vol. 663, Quebec City, Canada, June 1986.

18. ELINT: Electronic Intelligence

19. Electronic Intelligence Sensors • Basic Measurements: • Amplitude • Frequency • Time • Derived Measurements: • Received signal-to-noise ratio • Polarization • Pulse shape • Pulse repetition interval • Radio frequency • State Determination • Identity • General and specific emitter identity • Analysis yields radar design characteristics Wiley, R.G., Electronic Intelligence: The Analysis of Radar Signals , Artech House, Norwood, MA, 1988

20. ESM: Electronic Support Measures • Basic Measurements: • RF Intensity/Amplitude • Frequency • Range/Time of Arrival • Angular information (direction) • Derived Measurements: • Signal signature • State Determination • Location • Type of emitter • . AN/ALR-606 Electronic Support Measures (ESM) Receiver (Northrop Grumman) EW Design Engineers Handbook 1989/1990, Supplement to the January 1990 Journal of Electronic Defense , Horizon House Microwave, Norwood, Massachusetts.

21. Human Skills: • Equipment • operator • Linguistics • Gisting • Analysis Tools: • Transforms • Graphics • Statistics • Database • Hardware: • Receivers • Turners • Demodulation • Filters Tactical Intelligence Technical Intelligence Communications Intelligence

22. IR: Infrared Warning • Basic Measurements: • IR intensity • Detection • Derived Measurements: • Direction • Intensity • Spectral Characteristics • State Determination • Identity • Location • Warning of heat sources approaching. Buser, R.G., and F.B. Warren, Eds., Sensors and Sensor Fusion, Proceedings of the SPIE , Vol. 782, Orlando, Florida, May 1987.

23. Sutton Coldfield Observer, SUTTON COLDFIELD, ENGLAND: “Even when the helicopter is flying between 800 and 1,000 feet”, say police intelligence, their heat seeking sensors can differentiate between a fleeing suspect and a heat producing device. The officers who rushed to the scene of the robbery at a builders site called for helicopter assistance, and the high tech sensor led them to the door of local, Barry Silvester. Detectives burst into the house to discover that the sensor had led them straight to a steaming hot compost heap in the back garden. Word to the wise

24. Synthetic Aperture Radar • Basic Measurements: • Coherent radar cross-section • Derived Measurements: • Target/platform shape • Target size • Aim point (direction) • State Determination • Identity • Enhanced radar resolution via coherent processing of • data collected by a stable moving antenna. http://www.nnsa.doe.gov/na-20/synthetic_aperture.shtml 1. Hovanessian, S.A.,Introduction to Synthetic Array and Imaging Radars , Artech House, Norwood, Massachusetts, 1980. 2. Fitch, J.P., Synthetic Aperture Radar, Springer-Verlag, NY, 1988. 3. Hovanessian, S.A., Introduction to Sensor Systems , Artech House, Norwood, Massachusetts, 1988.

25. Electro-Optical • Basic Measurements: • Picture elements (pixel) • Picture intensity and color • Derived Measurements: • Location • Size • Shape • State Determination • Position • Identity • - Can be used to obtain environmental data • Good for size/shape determination, but requires significant processing time. 1. Blouke, M.N., and D. Pophal, Eds., Optical Sensors and Electronic Photography, Proceedings of the SPIE , Vol. 1071, January 16-18 1989, Los Angeles, California.

26. TRACKED TRUCK CLUTTER TRACKED CLUTTER TRUCK CLUTTER TRACKED FLIR IMAGE: TRUE CLASSES FLIR CLASSIFICATIONS PIXEL-LEVEL FUSION TRACKED CLUTTER CLUTTER TRUCK TRACKED TRACKED TV IMAGE: TRUE CLASSES FEATURE-LEVEL FUSION TV CLASSIFICATIONS Representative Classification Results for FLIR and TV Imagery

27. NODE PROTOTYPE Breadboard Processor • Functions • NLOS acoustic/seismic detection, • bearing estimation, localization • Environmental data fusion for real- • time data confidence formulation • Low-cost research tool for proofing • concepts and sensors Battery Sensor Platform • Node Components • Acoustic array • Seismic/tiltmeter array • GPS, flux-gate compass/magnetometer • Temp/humidity, solar, soil sensors • PC/DSP breadboard processor • Wireless Ethernet networking • Input/Output • Acoustic and seismic waves, • environmental data, commands via • network is input • Digital files of detection, • confidences, position tracks, I/D, • predictions of performance in • current environment are outputs Example: Self-Locating/Self-Calibrating AcousticSensor

28. Examples of Emerging Sensors Use of Insect sensing organs (SPIE May 2003) NASA E-Nose (2004) Every soldier a sensor; 4/29/2004 Aviation Week Plants as observers and sensors

29. Both the natural physical environment and hostile attempts to degrade sensor effectiveness via countermeasures present complex problems in the application of data fusion techniques. Degraded Received Signal True Signal Propagation Atmospheric Noise Multi-path Jamming Attenuation Signal Propagation

30. Random signals may be added to the environment to decrease the signal-to-noise ratio (SNR). This is called JAMMING. JAMMING Normal Signal Noise DECEPTION Specific signals may be added to the environment to create false alarms at the receiver. This is called DECEPTION. Jamming and Deception

31. Camouflage & Deception in nature Examples of on-line photos of animal camouflage and mimicry http://www.oceanlight.com/lightbox.php?x=camoflage__fish_behavior__fish__animal http://rainforests.mongabay.com/0306.htm http://www.nytimes.com/2008/02/19/science/19camo.html?8br http://www.i-am-bored.com/bored_link.cfm?link_id=31398

32. KEY QUESTIONS: What can/should be sensed? What sensors are available? Physics of emissions, transmission, reflection, scattering, detection CM/CCM environment? How does clutter noise, multi-path, interference affect P(D), P(fa)? Sensor Performance? Mission constraints on sensing, data communication? Purpose: Select the best suite of sensors to detect, locate, characterize, and identify critical targets/entities Key Factors: Sensor selection/availability Accuracy vs timeliness vs computer resources Extent to which sensors are Smart Scheduling, utilization of sensors Contextual sensor performance - Environmental effects - Countermeasures Sensor control Parameters Stealth requirements Analysis Tools and Techniques: Static threat vs observability analysis Engagement timeline analysis Observation modeling - Observation prediction - Observation statistics Covariance error analysis Monte Carlo simulation Sensor Analysis Overview

33. Emerging network centric operations Ubiquitous connectivity Soldier and civilian hosted sensors Emerging “every soldier/civilian a sensor” concept New types of hybrid “hard/soft” sensors Sensors carried/worn by soldiers/civilians Sensors that monitor the condition of people (e.g., using the human body as a sensor for complex chem/bio phenomena and monitoring the body’s responses) Self-reporting via soldier as “continuous commentator” Use of animals and plants as sensors Emergent social phenomena – mining soldiers chat & blogs for emerging insights and unconscious insights Need for research in Development of a framework for fusing report-level data from human observers Mapping and modeling the data flow for fusion involving humans as “soft sensors” combined with conventional sensors Mixed input environments Traditional and new “hard” and “soft” sensing

34. General Issues • What is the general framework for human observing/reporting? • What is the role of the human as both provider and consumer of information? • What is the process for human observations (e.g., do humans provide information on a volunteer or ad hoc basis; can or should humans be tasked; is/should there be modulation/filtering/censuring of observations; • Should human observers be characterized (e.g., self-report of expertise; system “grading” of human observers, etc) – consider Accuweather example • What is the role of indirect reporting (e.g., blogging; news reports, chat, etc) • How should knowledge and reports be solicited? • How should knowledge be solicited (e.g., via structured entry forms; via prompted questions; etc. • What language/structure should be used (free-form text, specialized & restricted language; • How should uncertainty be represented (e.g., via fuzzy terms, confidence factors, subjective probabilities • How should 2nd order uncertainty be represented

35. Issues continued • What is the framework for human reporting uncertainty representation/fusion with traditional sensors? • How does “soft” data and web mined data flow into the traditional JDL model components? • Soft data provides contextual information (e.g., human judged relationships) & input directly into higher levels of JDL processing • Issues of legacy, out-of-sequence reports, significant differences in time scales require more sophisticated data management functions

36. Topic 4 Assignments • Preview the on-line topic 4 materials • Writing Assignment 4: Write a 2-page paper describing the challenges and issues of searching for the Loch Ness Monster (see on-line questions) and web resources. • Discussion 2: On-line discussion of the concept of camouflage in nature (and for human activities); what strategies are used to affect “observability”; can you find interesting examples on the web? • Team Assignment (T-3): For your selected application; provide a summary table of the anticipated information sources and sensors (e.g., provide a list of sensors and/or sources; for each sensor or source identify the type of data (what is observed or reported), the data rate (e.g., reports, signals, images per second, minute or day), any information about the data format, and information about the characteristics of the sensor/source (e.g., reliability, capability of observing in various environments, etc.)

37. Data Fusion Tip of the Week There’s no substitute for a good sensor that is appropriate to the situation or threat of interest: No amount of fusion of multiple poor sensors can substitute for an effective sensor!