The IUCr Diffraction Data Deposition Working Group Activities since IUCr Madrid

1. The IUCr Diffraction Data Deposition Working GroupActivitiessince IUCr Madrid John R. Helliwell, Tom Terwilliger and Brian McMahon john.helliwell@manchester.ac.uk terwilliger@lanl.gov bm@iucr.org

2. Members of the IUCr Diffraction Data Deposition Working Group WG Members Steve Androulakis (TARDIS representative) Sol Gruner (Diffuse scattering specialist and SR Facility Director)John R. Helliwell (Chair) (IUCr ICSTI Representative; Chairman,IUCrJournals Commission 1996-2005)LoesKroon-Batenburg (Data processing software)Brian McMahon (IUCr CODATA Representative) Tom Terwilliger (Representative of the Commission on Biological Macromolecules)John Westbrook (wwPDB representative and COMCIFS)Heinz-Josef Weyer (SR and Neutron Facility user)By invitation Chairs and delegates of IUCr CommissionsConsultants AlunAshton (Diamond Light Source (DLS); Data Archive leader there)Herbert Bernstein (Head of the imgCIF Dictionary Maintenance Group and member of COMCIFS)Frances Bernstein (Observer on data deposition policies)Gerard Bricogne (Active software and methods developer)Bernhard Rupp ( Macromolecular crystallographer)

3. Terms of reference • It is becoming increasingly important to deposit the raw data from scattering experiments; • A lot of valuable information gets lost when only structure factors are deposited. • A number of research centres, e.g. synchrotron and neutron facilities, are fully aware of the need and have established detector working groups addressing this issue.

4. Key terminology • Raw data • Processed data • Derived data

5. The data publication pyramid - the publishers' view Reilly, S., Schallier, W., Schrimpf, S., Smit, E. & Wilkinson, W. (2011). Report on Integration of Data and Publications. Available from http://www.stm-assoc.org/integration-of-data-and-publications/

6. Data flow in crystallography Raw experimental data Reduced/processed data Derived data Chemistry databases (CCDC) IUCr journals Experiment (synchrotron or laboratory) Structure solution and refinement (laboratory) Biological structure databases (PDB) Other journals retained by scientist archived at facility (~6 months) deposited Data reduction published/disseminated validated

7. Publication flow in IUCr journals structure factors .fcf CIF file .cif Chemistry databases (CCDC) IUCr journals Bibliographic databases (ISI, etc.) .fcf .cif Author Experiment (synchrotron or laboratory) Structure solution and refinement (laboratory) raw data (imgCIF) .xml, .rdfetc. .fcf .cif .pdf Publication Technical editing Validation Peer review .html preprint .pdf .html article of record .sgml Raw experimental data Reduced/processed data Derived data Published article Data reduction

8. Cost and benefits analyses versus obligations on researchers • Raw data has been out of reach as a viable option to archive but is now perceived within reach by a variety of research fields, learned societies (e.g. ICSU, CODATA, Royal Society etc.) • IUCr Journals has thus far ‘encouraged’ authors to retain raw data • Funding agencies believe their PIs are carefully retaining their ‘data’

9. Activities since Madrid • Organization and Communications • Core working group • Consultation group (dddwg@iucr.org list) • Public forum (http://forums.iucr.org) • Planning for workshop/briefings at Regional meetings

10. The IUCr DDD Forum • http://forums.iucr.org/ • Has provided a focus for documents and some discussion • It has led to around 500 views for some documents

11. Activities since Madrid • Organization and Communications • Analysis and Discussion • CCP4BB discussion: engagement and summary • ICSTI Insights articles • Data comparison paper (Kroon-Batenburg et al.) • Survey of facilities by IUCr Commission on Synchrotron Radiation

12. ICSTI Insights 1. The Living Publication has existed for many years for crystallographers John R. Helliwell and Brian McMahon 2. Continuous improvement of macromolecular crystal structures Thomas C. Terwilliger 3. Should the crystallographic community require the archiving of raw diffraction data from a crystal, a fibre or a solution? John R. Helliwell and Brian McMahon

13. Activities since Madrid • Organization and Communications • Analysis and Discussion • Pilot Experiments • University of Manchester repository • Diamond data sets DOI registration

14. Activities since Madrid • Organization and Communications • Analysis and Discussion • Pilot Experiments • Ongoing efforts • imgCIF/CBF development • PaNdata • MyTARDIS

15. imgCIF/CBF development • Vendor support for imgCIF/CBF support (primarily Pilatus 6m, but also all major vendors) • HDF5 backend for CBFlib (for seamless integration between imgCIF/CBF and NeXus/HDF5) • Metadata mapping between imgCIF and NeXus (H. J. Bernstein, Diamond) – will require minimal changes to imgCIF • New compression methods in anticipation of Dectris Eiger (2.5 gigapixels/second)

16. Aims of the DDD Workshop • Review progress after 1 year (launch was August 2011) • Review the choices for raw data archiving • Proceed towards a policy and plan for raw data archiving for approval by IUCr Executive Committee in IUCr Congress 2014 in Montreal

17. Options • Do nothing for ensuring raw data archiving; • Do what we can e.g. via centralised facilities raw data archiving along with Universities own data archives both as supplements to the processed data archiving at the CSD and PDB etc • Seek a blue skies solution where all raw data are compulsorily archived at centralised repositories.

18. Let's go to it!