BADC Workshop 2: BADC Services to Data Suppliers

1. BADC Workshop 2:BADC Services to Data Suppliers Royal Met. Soc. Conference – 14 September 2005 Ag Stephens et al.

2. Workshop Outline • Purpose of Workshop • To briefly present BADC provision to Data Suppliers • To gain feedback from the user community • Workshop plan • Presentation: BADC Services to Data Suppliers • Points for discussion… • How can we serve Data Suppliers better? • Which services need improving, and how? • What new services are required? • Evaluation form

3. Presentation Outline • Introduction and scope • The BADC and its data suppliers • The NERC Data Policy • Support to data suppliers • Data management planning • Archival, distribution & service infrastructure • Metadata • File names • Data format • Data submission • Campaign support • Getting help • Discussion

4. The BADC and its data suppliers The BADC • The NERC-designated Data Centre for atmospheric science is the BADC (under NCAS) • It currently holds over 30 TB of data including NWP forecasts, climate runs, instrumental and satellite products • It serves 7,000 users in the UK and overseas The BADC’s data suppliers • NERC-funded researchers (e.g. through Directed Mode Programmes or using a NERC facility such as the FAAM or UFAM instruments) — also the BADC data users! • Other research or data centres (e.g. Met Office, ECMWF, Eumetsat, ESA, NASA) • International research programmes (e.g. NDSC, EC-funded such as NitroEurope) http://badc.nerc.ac.uk/

5. The NERC Data Policy The NERC Data Policy stipulates: • NERC grant holders’ duties, e.g. • Get acquainted with the NERC Data Policy Handbook (*) • Offer the data generated by a NERC-funded project to the designated Data Centre • The Data Centres’ duties • Ensure appropriate data custody, validation, documentation, cataloguing and dissemination • Maintain and promote data stewardship standards • Set up data protocols (conditions of submission, access and use) • Assist UK researchers in locating and accessing data, including fetching data from external sources • Handle data-related queries (*) http://www.nerc.ac.uk/data/documents/datahandbook.pdf

6. Data Management Planning At the outset of a research programme/project/experiment: • Scoping study to determine: • scientific goals - external data needs • project duration - data sharing needs • staff and collaborators - investigators’ wishes • details on data to be produced & archived (nature, volume, flow,…) Data management plan (DMP) proposal and adoption (for large programmes): common dispositions and technical measures to meet the programme needs — in accordance with policies possibly already in force (e.g. international data policy, Freedom of Information Act, etc.) Data protocol (DMP executive summary) • - submission time-frame - conditions of access • retention time-period - conditions of use and publication

7. Metadata Metadata = data about the data • Metadata are essential to enable the: • user (a human or computer) to understand the data (physical nature, units, error estimates, scientific context, algorithms, instrument or model specifications, publication references, etc. — N.B. error flags, error bars may be integrated into the data body) • get connected information on research context (experiment, project, platform, contact, etc.) • read the data (format and layout) • find out about the existence of data, where the data are held and how to access them (discovery metadata). This information is required by data portals, browsers and search engines. Our NERC DataGrid project is developing metadata formats following international ISO standards to improve data discovery (see: http://ndg.nerc.ac.uk/????). Help on metadata - http://badc.nerc.ac.uk/help/metadata/

8. File names • The BADC archive is based on a browsable file system. • We encourage meaningful file names to allow: • identification of the file content without reading file • automated ingestion into the archive • automated use by handling software Help on file names - http://badc.nerc.ac.uk/help/file_naming.html

9. Data Formats The BADC encourages the use of NASA Ames (ASCII) and NetCDF (binary) data formats which: • have a history of successive improvements • allow and encourage inclusion of significant metadata (such as NetCDF’s Climate and Forecast [CF] Metadata Convention) • enable an ease of data exchange with collaborators • enable readability by existing software • Online BADC tools: • NASA Ames file format checker • NetCDF file CF compliance checker • Under development: • NetCDF/NASA Ames file format converter Help on formats - http://badc.nerc.ac.uk/help/formats

10. Data Submission • Requirements: • you must register as a BADC user • you must have been granted access to the relevant dataset • Data files are uploaded to the BADC incoming directory via: • the BADC web-based data file uploader • ftp to ftp.badc.rl.ac.uk NOTE: This process is data submission to the final archive. Uploading to the online workspace is not submitting the data to the BADC. • Data files are ingested (moved to the archive): • with some checking of metadata and data format

11. Near real time Met Office Products for the Convective Storms Initiation Project (CSIP) Campaign Support • Campaign Support includes: • The provision of meteorological data and products (such as synoptic charts, rain radar images) in near real time • - Sources: Met Office, ECMWF • The provision of near real time satellite data (MSG) • The provision of forecast trajectories calculated with forecast winds, either through the use of the BADC or Reading trajectory service • The provision of dedicated online workspaces or FTP space allowing fast exchange of preliminary data CSIP Collaborative Workspace allows secure upload and sharing of preliminary data for collaborators

12. Getting help/info • 2nd step -badc@rl.ac.uk • Points for discussion… • How can we serve Data Suppliers better? • Which services need improving, and how? • What new services are required? 1st step - http://badc.nerc.ac.uk/