Ilya Slutsker Data Base Development, Processing and Data Archive

1. Ilya SlutskerData Base Development, Processing and Data Archive • Data Collection • Archives • Processing • Data Access Delivery and Distribution

2. Data Collection • Historically, first data from Automatic Sun Photometers were collected manually. Starting in 1993 DCP (Data Collection Platforms) have become preferable way of getting data. • Today, about 92 % of data is collected using automatic DCP transmissions, and only about 8% is send using manual dump of the instrument memory to PC and submission of this dump to Aeronet server. • Main reasons for the manual downloads are 1) Unavailability of the direct satellite uplink – Far South and North, Landscape profiles 2) Local regulations. 3) Equipment malfunctioning. Most of those problem eventually can be remedied by using cellular or digital communication technology.

3. Until May 1993 We did not use DCP Satellites Uplinks In 1993 we started transmitting using GOES East from US, Canada and Brazil Soon we extended our transmission coverage area to GOES West – Alaska, Hawaii … Meteosat gave us access to European, African and Near East sites Finally by the end of the last century We covered Far East and Pacific using GMS

4. DCP Data Collection DCPs transmit data and satellites retransmit them to Data collection Stations. These Stations receive data, demodulate them, sort them and then send to users. Goes East and West dataare processed by NOAA DAPS at Wallops Island, Virginia. Meteosat are processed by EUMETSAT in Darmstadt, Germany GMS data are processed by Vitel built receiver, Located at the NOAA PTWS, Ewa Beach, Hawaii

5. Even manual data download changed very much during recent years. Now Site Managers can submit data on-line 24 hours a day, and the data Will be automatically archived and processed within 1 hour

6. After being submitted to Aeronet server Data are processed and placed to several archives EUMETSAT DAPS PTWS K7_upload.html Hourly Auto Upload Manual upload DCP Messages Archive K7 Archive DCP decoding K7 decoding Daily Reports Archive Processing Software Processed Data Archive Raw Data Archive

7. We maintain several archives that allow us properly process the data. • Filters Archive. It has spectral response function for filters installed in our instruments. Now the information about more that 3000 filters is stored there. • Sites Archive. All information of Aeronet sites where instruments have been operated. Currently 496 Sites. • Instrument Configurations Archive. Many instruments have their configurations changed over years. The archive keeps that information including types of instruments and filters used. • Calibration Archive. Sun , Sky and Flux calibrations for all instruments.

8. Filters Archive Filter Functions are sorted first by nominal channel (i.e. 1020, 440 …) Then by the year of shipment (1997, 1998 …) Then by the name of batch (1020_1, Lot2401 …) And finally by the number of the filter. We use the program “administrator” to manage this archive.

9. Sites Archive Sites are sorted first by types : Permanent, Seasonal, Temporary, by Region and Country/State, then by PI’s name. We can search sites using all the above parameters and also by the coordinates and name of the site. The archive is managed by “administrator”

10. Instruments Configurations Archive There have been more than 20 different instrument types over years. Most common are : Standard, Polarized and Advanced. Each type characterized by its own set of filters. The Archive is managed by “administrator”.

11. Calibration Archive This archive is managed by “demonstrat” program. Example – Calibration history.

12. Data Archive • This is where all the measurements data and Aerosol Property products are kept. • There are 2 archives : “Raw Data Archive” and “Processed Data Archive”. Processing Software Processed Data Archive Raw Data Archive Direct Solar Measurements Aerosol Optical Depth Sky Measurements (Almucantars, Principal Planes, BRDF) Quality Assured AOD Fluxes from Flux sensors Water Vapor Sky Radiance retrieval products ( Size distribution, Phase Function, Single Scattering Albedo, Refractive Index …) Temperature, Internal and External battery voltage, Dark current, resets, hardware statuses and errors

13. Processing Sequence and Software • Once the data are collected and archived in the Aeronet server, several processing tasks are performed. • Each of these tasks is managed by its own software. • Individual raw solar measurements are used and level 1 AOD and Water Vapor are computed. Runs every hour. • Daily blocks of raw solar measurements are used and level 1.5 cloud screened AOD and Water Vapor are computed. Runs every day. • Individual Almucantars and Principal Planes and Daily blocks of Level 1.5 AOD are used and Sky retrieval products are computed. Runs every day. • All these tasks run automatically • After they are performed, the data become available to our group. Normally all the data from the previous day are available by the noon of the current day. • The data can be reprocessed, due to 1) recalibration, 2) coordinates adjustments 3) quality assurance. Reprocessing programs are started by the authorized user and reprocess long records of data , sometimes months and years per run.

14. Data Access • All the data are accessible through several interface • On the System Level we have access to the binary files which can be backed up, cloned or, if necessary deleted. There are encoding and decoding programs that can retrieve the data into ascii text, that can be looked at, modified, and saved. • On the User Level users have access to “demonstrat” , which is data browsing tool. They can look at the data, download the data, and perform several research task. Authorized users can calibrate devices, reprocess data and raise the quality level of data. • On the Public Level everybody have access to the Web Site where they can look at the data and download them. • Our goal is to bring as much data as we can to the Public Level limiting Public Access by only actually changing the data. Only Authorized users are going to be able to modify the data. As for the raw data, they are locked and not modified. • Next slide will show some example of the “demonstrat” use.

15. Demonstrat Main Window Langley Calibration Window We can click on almost anything, And also drag the mouse across space and time. Right click on Site or Instrument number will limit selection to That site or instrument. AOT Time Dependence Window Almucantar Window

16. Conclusion • As the network of instruments and sites grows and now scattered across the world with no continents to spare, the reliability and performance of data storage and processing software have become of the essence. • Over the years we developed the system which collects, archives, processes and distributes Aeronet data from hundreds of sites and instruments and does it fully automatically in real time. • Although all the routine processing and storage is done automatically, we keep complete control over the system performance and can monitor and troubleshoot it at any moment. • The system development continues, as new data type and products become available.