Status of Cluster Active Archive (CAA) and Cluster Final Archive (CFA)

1. Status of Cluster Active Archive (CAA) and Cluster Final Archive (CFA) Harri Laakso, Philippe Escoubet, Arnaud Masson, and Matt Taylor European Space Agency (ESA) Science and Robotic Exploration Directorate Noordwijk, The Netherlands 2011 ILWS Science Workshop 28 Aug – 1 Sept 2011

2. What is CAA? http://caa.estec.esa.int/ • The CAA will contain all Cluster high-resolution data as well as all allied (supporting) data productsnecessary to the full interpretation of the full-resolution datasets • The entire high- and low-resolution datasets are fully calibrated(to the level achievable within the limited resources available) • The data are suitable for science use and publicationby the world-wide scientific community (however, see quality flags and caveat files and extended documentation for interpretation) • All data are provided in a standard format (CEF - Cluster Exchange Format) and with a complete set of metadata in machine readable form; all data can be downloaded either in CEF or CDF format • The CAA aims to provide user friendly services for searching and accessing these data

3. Updates in CAA User Interface • Web interface has stayed quite stable and no major updates recently • Command-line interface has been updated • Speed of requests has improved from minutes to seconds for regular requests • Small requests (<100MB) are executed immediately if the resources permit, otherwise it is scheduled (flag exists that user can make a scheduled request) • If request is executed immediately, the requested data file is returned to the user; if the request is scheduled the user gets a CAA_Info.log file as before • Streaming interface to the data is now available for beta-testing • Data are not written on CAA disk, but are directly streamed to the user • Usage: the data can be forwarded directly to an application or written in a file (and zip if requested) • wget “ hcp://caa.estec.esa.int/cgi--‐bin/stream_caa.cgi/?uname=[Uname]& pwd=[pwd] & dataset_id=C1_CP_EDI_SPIN&time_range= 2004-01‐03T00:00:00Z/2004-01-04T00:00:00Z&gzip=1” -O OUTFILE.cef.gz • Limitations: No error reports produced; will be applied to CEF files only

4. CAA Cross-Calibration Workshops • The Cross Calibration Workshops are a Critical Activity for the CAA and provides an active forum for discussion of CAA issues • Improves quality and standardisation of data • The minutes of the meetings with presentations can be found at • ftp://ftp.rssd.esa.int/pub/Cluster/MoM/CAA/CrossCalibration/

5. Examples of CAA Cross-Calibration Plots: Density CUSP MAGNETOSHEATH SOLAR WIND N H+ N i N e- N sounder

6. Cluster Boundary Crossings • Within the next 12 months, the CAA/BLIC project will determine the Cluster crossings of the bow shock, magnetopause, electron radiation belt and neutral sheet • Three first events are needed to add quality flags to some observations, e.g. to flag the instrument mode – region mismatch (BS and MP) or to flag the increased MCP noise (RB) • A Matlab code for semi-automated boundary recognition will be created for the work • Manual verification and correction of the boundary crossing time, i.e. the first and last crossings are verified for multiple boundary crossings • A quality flag will be included to indicate the type crossing (including e.g. a flag for quasi-parallel bow shock crossing or an unclear magnetopause crossing through the exterior cusp) and caveat remarks when necessary • The code will be made available for scientific use through CAA.

7. Example: Bow Shock Crossings Outward Inward B [nT] Total vCIS[cm-3] vCIS & nCIS & B nCIS [cm-3] nWHI [cm-3] nWHI & B nEFW [cm-3] nEFW & B EFW bitmask & B bitmask Cluster 1 March 9, 2001

8. Double Star Raw Data Reprocessing Workshops • The original raw data was found to have a large number of data gaps, noisy intervals, wrong time stamps etc. • it was decided that it is better to regenerate all raw data • New issues were found in regenerated data • When the problems have been resolved, the final science datasets will be generated and ingested on the CAA • The minutes of the meeting, with all presentations, can be found at • ftp://ftp.rssd.esa.int/pub/Cluster/MoM/CAA/DoubleStar/

9. European Cluster Assimilation Technology (ECLAT) • ECLAT provides context for the Cluster observations (EU-FP7) • How do Cluster observations relate to other observations of the magnetosphere, made from the ground or other space-borne observatories? • The Cluster Ground-based Working Group has provided contextual information for the duration of the Cluster mission to date on an ad hoc basis • This will be formalized, improved, and combined with a data-serving functionality within ECLAT

10. ECLAT Data Products for the CAA • A knowledge of the magnetospheric regions and boundaries Cluster is encountering • Region and boundary identification (OEAW) • An ability to know where Cluster is relative to other observatories in space or on the ground • Magnetic field tracing (SPSU) • The ionospheric conditions at the mapped footprint of the Cluster spacecraft • Miracle equivalent current mapping (FMI) • Knowledge of the large-scale magnetospheric state and behaviour • SuperDARN ionospheric flow patterns and global auroral imagery (ULEIC)

11. CFA vs CAA • The final Cluster data archive (CFA) will be located at ESAC (Madrid) • The CAA to CFA transition activities have started • CFA will use standard ESAC archive architecture (in order to simplify the maintenance in the future) • Services and user interfaces of CFA should be similar to the those of CAA • Role of CAA • the only interface to the data producers • CAA will validate and ingest the new files and new datasets on its database • Data will then be transferred to the CFA and be added in the CFA database • Up to 1 year after the end of operations, CAA will stay active • The rough transition schedule is • 2011: the CFA specification and a major part of implementation done • 2012: both CAA and CFA user interfaces will coexist, and CFA has further developments while the CAA GUIs are frozen • 2013 onwards: CFA is the sole interface for the access of Cluster data

12. Delivery Schedule of 2010-12 Data • All teams must deliver files more continuously than before (i.e. not once a year) • This helps the validation/ingestion tasks of the CAA • Files of some datasets are needed by other teams in their processing, and such files are needed even earlier • Delivery schedule for the last three years of spacecraft operations (assuming no further Cluster extension) • Measurements Delivery date • 2010 January-June 1 Dec 2010 • July-December 1 May 2011 • 2011 January-June 1 Dec 2011 • July-December 1 May 2012 • 2012 January-June 1 Dec 2012 • July-December 1 May 2013 • Final calibrated datasets (2000-2012) 1 Nov 2013

13. Near-Term Activities (Next 12 mo.) • Routine operations, data delivery and maintenance of the existing data access and value added services • System maintenance and upgrades • Production and ingestion of all dataset up to December 2011, and • Ingestion of WBD CDF data and translation to CEF for years 2001-TBD • Include the data mining tool to the CAA GUI • Calibration and cross-calibration of all measurements • 14th Cross-Calibration Workshop, October 2011 • 15th Cross-Calibration Workshop, April 2012 • Complete CAA cross-calibration analysis tools/plots • New data sources: • Production of boundary crossing datasets • Completion of Double Star reprocessing and production of datasets • Start of the production of ECLAT datasets • CAA-CFA transition activities

14. Cluster Active Archive statistics Now 1250 users

15. http://caa.estec.esa.int