NEESgrid Data Technologies

1. NEESgrid Data Technologies Charles Severance January 8, 2004 NSF Site Visit

2. NEESgrid Data - Value Proposition • An RDF like store – Referential integrity long-term flexibility • Seamless data and meta data transport • Smooth integration of data with meta data • Extensible tools • Involved with sites through Experiment Based Deployment

3. NEESgrid Data – Current Elements • Local Repository • Central Repository • JAVA APIs – Run locally on the same system as a repository or over OGSA Web Services • NEES File Management Services • NEES Meta Data Services • NEES Data Mapping Services • Data Viewers • Streaming (numeric, X/Y graph) • Stored (X/Y graph, 2-D structure, video)

4. Current Elements Data Acquisition NEESpop Local Repository Grid and Web Services Data/MD Ingest Tools API Data Teamlets API Workstation NEESdata Data Servlets API Data tools Mapping Central Repository Data Teamlets API Data viewers

5. A Simple Experimental Scenario Developer System MD MD M M Data DAQ System Test Specimen API Labview Glue Researcher System Data Data

6. Repository Browser

7. Sample of the Video/Data Viewer Data Viewer

8. Mappings and the Data Viewer • NSDS (ISO 8601 Time channel) • Column data with time recorded as a column • Column – generate time • Column – generate time – trigger filter Channel units: g,g,in,kip Time ATL1 ATT1 2002-11-13T15:48:55.26499 -0.006409 0.004272 2002-11-13T15:48:55.36499 -0.005798 -0.003662 100.000 0.435 0.161 -1.016 -0.981 0.430 0.161 -1.016 -0.977 0.435 0.161 -1.016 -0.977 public class NEESDataMap { public static boolean repoMap(File mainFile, File mappingFile, String mapping) { // Code here } }

9. Data Ingestor

10. NEES Metadata Representation • NEES Markup Language (NEESML) • Provides an RDF-like structure capable of representing semantic information • XML is the syntax which is used • Logic is more “object oriented” • Can define objects • Can create objects • Can reference objects • Meta data is many different things…. • Goal if we EVER want to build reusable data tools, we have to represent the semantics inside the meta data rather than just the information

11. NEESML Table 1: Primitive types in NEESML

12. JAVA APIs • http://neesbox.ncsa.uiuc.edu/chef/doc/repo/api/

13. Remaining Work • Second Generation Repository API • Project Browser • Electronic Notebook • Data Turbine • Video as data • Schema/XML Ingestion • RDF Model/Data Ingestion • Curation Tools ***

14. Repository API • The NFMS and NMDS APIs are being combined into a single Repository API • Rich support for access control • Access control will use Community Authentication Service (CAS) from the Grid technology • Aligning with JSR-170 Java Content Repository • http://www.jcp.org/en/jsr/detail?id=170

15. Project Browser • Joint effort between the NEESgrid SI team and Oregon State technical developers • Based on a project browser prototype at Oregon State University • Provides a user friendly interface to Metadata elements - compliment to the project browser and electronic notebook • http://nees.orst.edu/prototype_4/

16. Electronic Notebook • Collaborative effort with the DOE SciDAC • Electronic notebook - metadata entry • Data mapping • Data provenance • Data display • Slide data/metadata jakarta.apache.org/slide/ • Ultimate integration will be via JSR-170 • www.scidac.org/SAM/ • collaboratory.emsl.pnl.gov/docs/collab/sam/samtechoverview.html

17. Data Turbine • Commercial, free data streaming toolkit • Developedby NASA

18. Data Turbine (cont) • Existing data viewers will be adapted to access and display data from data turbine • Data acquisition software will be adapted to place information in Data Turbine Channels • Metadata elements will be developed to represent data turbine live, stored, and derived channels • New efforts (video as data) will be developed from the ground up using Data Turbine • outlet.creare.com/rbnb/

19. Video as Data • Follow on to initial demonstration at ORST • Experiment based development: Minnesota • Design phase nearly complete • Joint effort, NEESGrid SI, ORST, Minnesota, UC Davis, Texas, and others as design solidifies

20. Video as data: User Views Still Image / Camera Control Still Image Viewer ^ < > ^ ~ < > < > Thumbnail + Audio + Data DT Main System Data Viewer < > + Camera Control Gateway

21. Schema/XML Ingestion • Several data efforts are expressing their data/models in Schema/XML (Cosmos, etc) • We are developing capabilities to parse XML and automatically extract relevant metadata for the repository • The entire XML file will be stored as data • This allows data developers to use tools like XMLSpy to develop their models.

22. RDF Integration • Some of the data and meta data task force members are using Protégé-2000 to develop their models and expressing them in RDF. • RDF and NEESML are very similar but not identical so it may be challenging to ingest any arbitrary RDF • We expect that we will be able to map a subset of RDF to NEESML for ingestion or adapt an RDF parser (Jena or Raptor) to ingest that subset directly into the repository

23. Curated Data Tools *** • Still evolving fine-grain requirements with community • Sites • Consortium DSAC has this as its focus • Some expected minimum requirements • Transfer between repositories • Workflow - implemented as ACLs (incoming, in-progress, published) • Will be extensions to Repository browser as well as a simple workflow tool

24. Conclusion • We are focusing on both core elements and the application of those elements • We are engaging the sites increasingly in the going forward development process • We have a lot of work - some of these efforts will continue post-transition with the sites taking an increasing role in the development