1 / 22

Usability Issues Facing 21st Century Data Archives

Usability Issues Facing 21st Century Data Archives. Joey Mukherjee and David Winningham joey@swri.org. Future Scientists. Mission. Archive. Team. Write Papers. Current Archiving Goal. Raw Data. Processed Data. Data Iteration. Quality Data. Quality Data. Future Scientists.

danil
Download Presentation

Usability Issues Facing 21st Century Data Archives

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Usability Issues Facing 21st Century Data Archives Joey Mukherjee and David Winningham joey@swri.org

  2. Future Scientists Mission Archive Team Write Papers Current Archiving Goal Raw Data Processed Data Data Iteration Quality Data Quality Data

  3. Future Scientists Mission Home Institution Archive Permanent Archive Team Write Papers Current Archiving Reality Raw Data Processed Data Data Iteration Public Data Data Subsets Unchecked Data

  4. Future Scientists Archive Mission Team Write Papers New Goal Raw Data Processed Data Data Iteration Processed Data Processed Data

  5. Standardizing HOWTO • Make it easy • Make it useful • Make it extensible

  6. Make it Easy • Reading / writing files must be super easy (i.e. cheap!) • Either with tools or libraries • Tools can be command line or GUI

  7. Make it Useful • How do I look at it? • Plots/Analysis • What else can I do with it? • Read into IDL, Matlab, Excel, etc. • Must have immediate benefits

  8. Make it Extensible • Must be possible for others to add value added services • Must be able to hold varieties of data • Must agree to give up control on content

  9. Case Studies: HTML • Easy to create! • Once done, look at in browser • Embrace / Extend

  10. Case Studies: SPASE • Creation is slow and difficult • Once created, no real benefits yet • VxOs have embraced, no one extended yet

  11. Case Studies: IDFS • Until recently, difficult to create, complex • Once in, easy to look at, use, archive, etc. • Somewhat extensible

  12. Things right with IDFS • Efficient • Self documenting • Calibrations stored in text file • Science units derived instead of stored • Little to no reprocessing ever needed

  13. Other IDFS Benefits • Can store most types of space physics data from raw telemetry to highly processed science units • Reversible from science units to raw telemetry • Usable by data processor, scientist, and data archiver

  14. Things wrong with IDFS • Overly complex format and API • Not enough support in other tools - poor buy-in • Analysis routines merged with the file format - tried to do too much!

  15. Implementation Plan • Develop a simple file format that can contain any and all types of time series space physics data • Develop tools that allow someone to create and inspect files in this format • Merge in the best parts of IDFS, CDF, netCDF, HDF, FITS, etc... without breaking paradigm of simplicity

  16. Simple File Format • Format might already exist: • HDF5 • XML • JSON • Other data models?

  17. Making it useful • Get buy-in from visualization tools (SDDAS, DataShop, VisBard, IDL DLM, etc.) • Get buy-in from archives sites (PDS, PSA, NSSDC, etc.) • Seed money is essential

  18. Advantages • Providers • Users • Management

  19. Advantages: Providers • Instrument teams now have something to work toward • Can develop expertise

  20. Advantages: Users • Quick ways to create plots or access data • Expertise again!

  21. Advantages: Management • Homogenous archives are infinitely easier to manage and maintain • Value added services are a natural extension of quality archives

  22. Conclusion • Why now? Because SPASE is gaining traction, this is the next logical step. • This will save money for everyone in the long run. • Everyone benefits with value added services.

More Related