Managing Data from Avian Radar Systems

1. Managing Data from Avian Radar Systems Edwin Herricks, PhD Siddhartha Majumdar

2. Introduction • 24/7 operation generates significant amounts of data • Ability to review archived avian radar data is essential • Data must be organized, distributed, and manipulated into useful data products • At each step of processing information is gained but data is lost

3. Radar Data Generation • Analog Data Generated by Radars on airfields • Rutter Card Digitizes Analog Data for the Radar Data Processor • Radar Data Processor extracts plots from energy data and links them to form tracks Data flow diagram of an Accipiter Radar Technologies, Inc. system

4. Radar Physics 1) Radar transmits electromagnetic energy pulses. 2) Energy reflects off all physical objects in it’s path. 3) Reflected energy is received by the radar. 4) Timing and intensity of reflections is recorded by the radar. The Radar Equation RCS affected by: • Material • Size • Orientation relative to the radar beam • Environment

5. Analog Data • Intensity of reflected energy is projected on axes of azimuth and range, or B-Scan Data • Coordinate transformation is necessary to convert to a polar plot, or Scan-converted Data

6. Digitization • Radars receive a constant stream of electromagnetic energy. • That stream must be quantified so that it is in a binary format that can be understood by the Radar Data Processor. • The rate at which analog values are measured and digitized is called the sampling rate. • Higher sampling rate gives more detail, and higher file sizes. • Commercial digitizers typically record at hundreds to thousands of samples per second. • Some loss of data inherent to the process.

7. Detection Extraction • Radar Data Processor takes digitized data and looks for potential targets. • Simplest form of extraction sets a constant minimum energy level required to constitute a detection. • More complex forms of extraction can vary that minimum energy level in space and time. • Gained – Potential targets of interest.Lost – Ability to reprocess with different “sensitivity.” • Any information contained below the threshold is lost.

8. Tracking • Radar Data Processor uses tracking algorithms to compare detections of successive scans in order to predict future scans. • Successful predictions lead to identifying tracks. • Additional parameters are recorded • Target history, heading, speed, etc • “False” tracks can be created from regularly occurring detections resulting from clutter. • No reprocessing can be done after this step!

9. Tradeoffs of Processing Data Detection Extraction Digitization Tracking File size Information available Flexibility to reprocess Each stage of processing presents an opportunity to save data, as well as some level of data loss.

10. Reprocessing Data • After Digitization, analog resolution is lost. • After Detection Extraction, ability to reprocess with different sensitivity is lost. • After Tracking, no reprocessing can be done.

11. Data Management • Data Generation • Remote access to radar • Transfer of data • Storage of data • Generation of data products • Dissemination of data products

12. Remote Access • Radars are typically installed relatively inaccessible areas. • Connectivity must be established to allow remote control and data transfer. • CEAT ASMP employs Virtual Network Computing (VNC) software to manipulate radar settings and coordinate data transfers.

13. VNC Access • Close to real-time control • Limited by connection speed • Remote Framebuffer Protocol • Pixel by pixel video stream and simple action commands • Cross platform compatibility • Low use of system resources

14. Data Transfer • Given the potential size and number of files created, it is best to automate these processes. • Typically only detection and track data is transferred. • Data available immediately after digitization can be prohibitively large. • Connectivity must allow enough throughput to keep up with generation of data. • Ideally connection is fast enough to transfer only during off-hours. • CEAT ASMP has found 1-2 megabytes per second is sufficient for most single-radar installations.

15. Data Storage • Amount of storage space necessary is unpredictable. • Amount of data proportional to amount of activity tracked by radar. • Ability to increase storage dynamically is important. • Data must be organized to facilitate future playback and manipulation.

16. CEAT ASMP Data Storage Server • Originally 2 Terabytes, now at 6 Terabytes capacity with 2.2 Terabytes full. • Located in University managed datacenter • Video surveillance, 24/7 monitoring • Redundant power and temperature control • Rolling 14 days of automatic incremental backup • Campus firewall • 12 Gigabyte/second fibrenetwork connection

17. Data Product Generation • Data products must be designed with the end-user in mind. • What information do they need? • How quickly can it be understood by a non-expert? • Where and how will it be viewed?

18. CEAT ASMP Data Products • Google Earth Enterprise Server • Live data available to client software via internet • Up to 1 hour stored for historical context • 24/7 Archived Data • 1 Hour historical Summaries • 8 Hour video playback

19. Google Earth Enterprise “Live” data display

20. Google Earth Enterprise Historical data layer

21. Historical Summaries • Tracks accumulate red trails for 1 hour segments • History cleared at the beginning of each segment

22. Video Playback • Variable playback delay • Data rate of playback dependant on computer processing speed

23. CEAT ASMP Data Product Generation • Detection and Track data generated locally at each radar • Data product generation automated wherever possible

24. Data Product Distribution • Security • How sensitive is the information specific to your tracked targets? • Radars track birds, planes, vehicles, etc, anything that moves. • Ease of Access • Can the end-user install client software? • Clear Organization of Data • By location, then radar, then date.

25. CEAT ASMP Data Product Distribution • Secure FTP Server • Requires client software to access • High security • Web Server • No client software required. Good for end-users with limited computer access. • Limited security • Google Earth Enterprise • Requires client software to access

26. Key Points • With each step of processing, information is gained but ability to reprocess is lost. • Data products must be tailored for the end-user. • Good organization of data and data products is vital to their future utility. • Network and computer limitations of end-users can limit data distribution options.

27. Questions?