1 / 39

NEESgrid Data Overview

NEESgrid Data Overview. Charles Severance. Goals. Operates both as local and central (curated) repository using same toolset Object oriented - referential integrity - data and metadata are supported together Data migration and replication transparently including security, and metadata

angelina
Download Presentation

NEESgrid Data Overview

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. NEESgrid Data Overview Charles Severance

  2. Goals • Operates both as local and central (curated) repository using same toolset • Object oriented - referential integrity - data and metadata are supported together • Data migration and replication transparently including security, and metadata • Uses Grid Services throughout (OGSA/GT3) • Easily used access controls - Grid as “single sign on” • Data provenance • Data in support of research publication • Support for repeatable experiments • Data oriented research computation support • Support for workflow

  3. Data Meta Data Gathering Extracting Repository Mapping Vision: data on the Grid studies

  4. NEESgrid Data – Core Elements • Local Repository • Central Repository • JAVA APIs – Run locally on the same system as a repository or over OGSA Web Services • NEES File Management Services • NEES Meta Data Services • Data Viewers • Streaming (numeric, X/Y graph) • Stored (X/Y graph, 2-D structure, video)

  5. Core Elements Data Acquisition NEESpop Data/MD Ingest Tools API Local Repository Grid and Web Services Data Teamlets API Workstation NEESdata Data tools Central Repository Data Teamlets API Data viewers

  6. NEESgrid Data – Technologies • Grid • GRIDFTP is used for data transport • Grid Web Services are used to insure security and provide access control between systems over the Internet – also provide for credential passthrough • Grid credentials are used as part of login providing a single sign-on framework • CHEF • Provides a flexible mechanism for deploying GUI tools like the data viewers and data browsers.

  7. Researcher System A Simple Experimental Scenario Developer System DAQ System Test Specimen Labview Glue

  8. Simulation System Simulation System Simulation System Code Code Code A Simulation Scenario Developer System

  9. The MOST Scenario • Part of the run-up to NEESPop 2.0 • Used Beta of NEESpop and Beta of CHEF • Tested the data ingestion • Tested the metadata capabilities • Developed sample metadata • Tested mapping capabilities • System still available at https://cee-nees.cee.uiuc.edu/chef/

  10. Series of files Ftp Plug In Complete file (aggregated) NTCP NSDS Wires File I/O Site / Location NFMS/NMDS Computer Process UIUC/Newmark Colorado Test Specimen Test Specimen MatLab Host And Real-Time Target Control System LabView DAQ Shore-Western Sim Controller LabView DAQ UIUC Matlab NEESPop (1.1) NSDS NTCP Incoming FTP CO NEESPop (2.0) NTCP Ingest Repository NSDS CO UIUC NCSA NEESPop (1.1) Incoming FTP Ingest NCSA Meta NTCP UIUC NEESMost (Win XP) Matlab Computational Model NCSA Ingest MOST Data Flows

  11. Time ch01 ch02 0.00 0.01 3 6 4 8 0 2 3 5 3 3 5 3 4 0 3 4 0 6 8 The experiment was viewed using the standard NEES stored viewer with synchronized video and data and the ability to move back and forth DAQ NEESpop 2.0 Alpha Metadata

  12. DAQ 0 3 4 0 6 8 <experiment> <blah> <public-view> </experiment> University ofMinnesota 0 3 4 0 6 8 <experiment> <blah> <public-view> </experiment>

  13. NEES Markup Language (NEESML) • Provides an RDF-like structure capable of representing semantic information • XML is the syntax which is used • Logic is more “object oriented” • Can define objects • Can create objects • Can reference objects

  14. RDF/XML Versus NEESML • NEESml is topologically equivalent to RDF but more straightforward to use • A compromise between usability and functionality • Focused on solving the problems of ingesting types and data – rather than “cross-server ontology webs” • Used to build a reference set of ingestion tools • RDF is a moving target • Repository does not store either RDF or NEESML – It is an relational database tuned to store “three-tuples”

  15. Data Metadata The Slide There is a layer is where we develop tools which take advantage and begin to depend on of the “meaning” of the data – where we begin to depend on the meaning of a second. Where we make a viewer capable of viewing a certain type of object. This is where we build things which make use of knowledge. This layer will never be complete but it is a large focus of the coming months. Data Ingestors Search Data Mappers Concepts Data Viewers

  16. Looking for Data Models…Looking for Semantics …

  17. Strategies • Take Existing Data Models and Adopt • Build tools and data models at the same time • Find existing tools that produce types of data we find interesting • Build meta data extractors for important file formats • Build converters for things like Excel

  18. Why start with the ORST Model? • It provides good coverage of a well-understood scope • It has consensus across multiple sites • A prototype toolset exists around the model which is an important validation • http://nees.orst.edu/IT/data.model/ • We can adopt the core elements and extend as necessary

  19. http://nees.orst.edu/IT/data.model/docs/v1.3a.july16.pdf

  20. o:project specialCondions title startDate o:experiment[s] o:role[s] o:acknowledgement[s] o:facility title shortDescription longDescription <type id="o:project" title=”Project"> <specialConditions title=”Special Conditions"/> <title title=”Title"/> <startDate title=”Starting Date"/> <o:experiment allow="o:experiment” max="unbounded" /> <o:role allow="o:role” max="unbounded" /> <o:acknowledgements allow=“o:acknowledgement” max=“unbounded” /> </type> <type id="o:experiment" title=”Experiment"> <status title=”Status"/> <title title=”Title"/> <shortDescription title=”Description"/> <o:facility allow="o:facility” /> </type> <type id="o:experiment" title=”Experiment"> <title title=”Title"/> <shortDescription title=”Description"/> <longDescription title=”Description"/> </type> o:experiment status title shortDescription o:facility

  21. Go Forward – Core Elements • Implement Access Control • Implement Replication • Investigate RDF and its relationship to NEESML • Investigate provenance – would like to adopt from another project • Investigate mapping – Would like to adopt from another project

  22. Go Forward - Tools • Evaluate the ORST interface and use it to implement experiment-based interface to meta data repository • Investigate tools to represent structural data (like SAC data) • Extend and improve viewers – publish API so that sites can extend the viewers • Improve notebook • Single signon using CHEF/Grid credentials • Integration with Metadata • Smother integration with CHEF • Explore automated synchronized video and data capture and after-experiment replay of synchronized video and data (ORST UMinn) • Explore the capture of high quality still images as data (UMinn) • Investigate adopting a data-editing tool (XMLSpy)

  23. Go Forward – Data Models • Analyze the ORST model, determine core, convert to NEESML, pre-populate repositories with types, and develop usage documentation • Form core group between SI, ES, and CS to push data model issues forward – once groundwork is better defined – we can disperse into distributed teams • Use experiment based deployment to help us encounter new data needs over time

  24. How to prioritize model exploration and development • Focus on the following areas: • Areas where we have or are building tools • Areas where we already have incoming data in some format • Build the model through experiment based deployment - solve real problems in an open way and see if (with some adaptation) the solutions apply more broadly (i.e. Minnesota )

  25. Groovy look and feel Local Data Repository Repository Browser in CHEF Browse Create objects Upload / download data API documentation NEESML User Documentation Extensible data mapping in Java Data Viewer in CHEF Improved visually Configured by XML Can read data from repository or from urls Pre-populated with sample video and data formats Local Repositories pre-populated with SAC Data MOST Data ORST data model (subset) What is in Release 2.0?October 7

  26. NEESML Table 1: Primitive types in NEESML

  27. Repository Browser

  28. Ingestor

  29. API Documentation

  30. Configuring Events <event id="oregon" desc="Oregon Large Tank Test September 8, 2003" host=“/chef/org.nees.repo.data/retrieve-data?lfn=nacse_sample_01.txt&amp; static=yes&amp;mapping=nacse-" type="stored"> <channel id="00" desc="time" unit="Seconds" url="t" /> <channel id="01" desc="Offshore Wave Gauge" unit="" url="02" /> <channel id="02" desc="Wave gauge at the front face of the cylinder" unit="" url="03" /> <channel id="03" desc="Wave gauge at the back face of the cylinder " unit="" url="04" /> <channel id="04" desc="Pressure at the front face of the cylinder " unit="" url="05" /> <video id="01" desc="Video of cylinder" url=“/chef/retrieve-data/static/nacse_sample_01.avi" /> </event>    <type id="nees:storedEvent">     <data allow="io:file"/>     <url/>     <parseOptions/>      <storedTimeChannelsmin="1" max=“1" allow="nees:storedTimeChannel"/>     <storedDataChannels min=“0" max="unbounded" allow="nees:storedDataChannel"/>      <storedVideoChannels min=“0" max="unbounded" allow="nees:storedVideoChannel"/>      <storedStruct2DChannels min=“0" max="unbounded" allow="nees: stored2DStructureChannel "/>     <fileChannels max="unbounded" allow="nees:fileChannel"/>   </type> We may be able to get a patch out to switch this to NEESml and provide a simple entry tool.

  31. Mappings and the Data Viewer • NSDS (ISO 8601 Time channel) • Column data with time recorded as a column • Column – generate time • Column – generate time – trigger filter Channel units: g,g,in,kip Time ATL1 ATT1 2002-11-13T15:48:55.26499 -0.006409 0.004272 2002-11-13T15:48:55.36499 -0.005798 -0.003662 100.000 0.435 0.161 -1.016 -0.981 0.430 0.161 -1.016 -0.977 0.435 0.161 -1.016 -0.977 public class NEESDataMap { public static boolean repoMap(File mainFile, File mappingFile, String mapping) { // Code here } }

  32. Release 2.1 Data AspectsDecember 2003 • NEESpop • Notebook to metadata repository connection made • Closer integration of notebook into CHEF • First release of experiment tool (based on ORST) • Retool data viewers to be completely driven by Metadata objects rather than their own objects • More fine grained access control • Enhanced data models • Tools • Ingestion tools released • A limited set of pre-release video/image tools

  33. Further releases • Release 2.2 – March 04 • Driven by your needs as we encounter them • Perhaps some “nice to haves” from the SI team • Release 3.0 – June 04 • Very limited new functionality – maybe almost nothing new in the core components of the NEESpop

More Related