1 / 32

NEESGrid Data and MetaData Technology

NEESGrid Data and MetaData Technology. Kincho Law, Jun Peng, Jim Eng, Terry Weymouth, Paul Hubbard, Charles Severance. Goals. Data is online and persistent Data and Metadata are supported together Data migrates transparently including security, and metadata

thao
Download Presentation

NEESGrid Data and MetaData Technology

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. NEESGrid Data and MetaDataTechnology Kincho Law, Jun Peng, Jim Eng, Terry Weymouth, Paul Hubbard, Charles Severance

  2. Goals • Data is online and persistent • Data and Metadata are supported together • Data migrates transparently including security, and metadata • Data is completely secure with access controls but security does not get in the way • Data provenance - how was it gathered, how has it been manipulated? • Data in support of research publication • Support for repeatable experiments • Data oriented research computation support • Support for workflow

  3. Data Metadata The Slide There is a layer is where we develop tools which take advantage and begin to depend on of the “meaning” of the data – where we begin to depend on the meaning of a second. Where we make a viewer capable of viewing a certain type of object. This is where we build things which make use of knowledge. This layer will never be complete but it is a large focus of the coming months. Data Ingestors Search Data Mappers Concepts Data Viewers 09/2003

  4. NEESgrid Data – Core Elements • Local Repository • Central Repository • JAVA APIs – Run locally on the same system as a repository or over OGSA Web Services • NEES File Management Services • NEES Meta Data Services • Data Viewers • Streaming (numeric, X/Y graph) • Stored (X/Y graph, 2-D structure, video)

  5. Core Elements Data Acquisition NEESpop Data/MD Ingest Tools API Local Repository Grid and Web Services Data Teamlets API Workstation NEESdata Data tools Central Repository Data Teamlets API Data viewers

  6. Researcher System A Simple Experimental Scenario Developer System DAQ System Test Specimen Labview Glue

  7. Simulation System Simulation System Simulation System Code Code Code A Simulation Scenario Developer System

  8. Series of files Ftp Plug In Complete file (aggregated) NTCP NSDS Wires File I/O Site / Location NFMS/NMDS Computer Process UIUC/Newmark Colorado Test Specimen Test Specimen MatLab Host And Real-Time Target Control System LabView DAQ Shore-Western Sim Controller LabView DAQ UIUC Matlab NEESPop (1.1) NSDS NTCP Incoming FTP CO NEESPop (2.0) NTCP Ingest Repository NSDS CO UIUC NCSA NEESPop (1.1) Incoming FTP Ingest NCSA Meta NTCP UIUC NEESMost (Win XP) Matlab Computational Model NCSA Ingest MOST Scanario

  9. Overview of tools and technologies • Model development in RDF • Project Browser • Repository Browser • Electronic Notebook • Data Mappings • Data Viewer • Data Turbine • Data As Video • Still Cameras 11/2003

  10. RDF Integration • Some of the data and meta data task force members are using Protégé-2000 to develop their models and expressing them in RDF. • RDF and NEESML are very similar but not identical so it may be challenging to ingest any arbitrary RDF • We expect that we will be able to map a subset of RDF to NEESML for ingestion or adapt an RDF parser (Jena or Raptor) to ingest that subset directly into the repository 11/2003

  11. Models + Data Model Repo Data Load RDF <owl:ObjectProperty rdf:ID="hasPublications"> <rdfs:domain> <owl:Class> <owl:unionOf rdf:parseType="Collection"> <owl:Class rdf:about="#Project"/> <owl:Class rdf:about="#Task"/> </owl:unionOf> </owl:Class> </rdfs:domain> <rdfs:range rdf:resource="#Publications"/> </owl:ObjectProperty> Configure Models RDF/ OWL Configure

  12. Models + Data Model Repo Data Load RDF Configure Models Protégé - 2K RDF/ OWL Configure

  13. Repository Browser

  14. Electronic Notebook • Collaborative effort with the DOE SciDAC • Electronic notebook - metadata entry • Data mapping • Data provenance • Data display • Slide data/metadata jakarta.apache.org/slide/ • Ultimate integration will be via JSR-170 • www.scidac.org/SAM/ • collaboratory.emsl.pnl.gov/docs/collab/sam/samtechoverview.html

  15. DOE ELN / Example

  16. DOE ELN / NEESgrid Integration (to date) NEESgrid Repository MyProxy Chef Grid Security SAM / Slide Repository Technology Celebration

  17. DOE ELN / NEESgrid Integration (ultimate) MyProxy NEESgrid Repository Chef Grid Security

  18. Mappings and the Data Viewer • NSDS (ISO 8601 Time channel) • Column data with time recorded as a column • Column – generate time • Column – generate time – trigger filter Channel units: g,g,in,kip Time ATL1 ATT1 2002-11-13T15:48:55.26499 -0.006409 0.004272 2002-11-13T15:48:55.36499 -0.005798 -0.003662 100.000 0.435 0.161 -1.016 -0.981 0.430 0.161 -1.016 -0.977 0.435 0.161 -1.016 -0.977 public class NEESDataMap { public static boolean repoMap(File mainFile, File mappingFile, String mapping) { // Code here } }

  19. Data Turbine • Commercial, free data streaming toolkit

  20. Data Turbine (cont) • Existing data viewers will be adapted to access and display data from data turbine • Data acquisition software will be adapted to place information in Data Turbine Channels • Metadata elements will be developed to represent data turbine live, stored, and derived channels • New efforts (video as data) will be developed from the ground up using Data Turbine • outlet.creare.com/rbnb/ 11/2003

  21. Video as Data • Follow on to initial demonstration at ORST • Experiment based development: Minnesota • Design phase complete • Joint effort, NEESGrid SI, ORST, Minnesota, UC Davis, Texas, Buffalo, and others

  22. Data Turbine - Today rbnbjcap BT848 Axis DT Client Data Capture DAQ DT Client NTCP Plugin CTL NTCP Control Control Plugin DT Main System AXIS / DT Gateway NEES NSDS Driver

  23. Data Turbine Control Make Smoothie NTCP Control Control Plugin DT Main System Thumbs Technology Celebration

  24. DT Capturing Still Capture PTZ/ USB DT Client rbnbjcap BT848 DT Client Audio Encoder Audio DT Client Data Capture DAQ DT Client Each still capture produces two channels - Small 1-5fps stream + large single images when picture is taken Camera Control Control Plugin DT Main System NEES NSDS Driver Still Capture - Minnesota / Paul Hubbard Video capture - From Creare Audio capture - From Creare (TBD) Data Capture - From sites (upwards compatible) NEES NSDS Driver - Paul Hubbard Camera Control Plugin - Mich / Minn

  25. User Views / Still Camera Still Image / Camera Control ^ < > ^ ~ < > Thumbnail + Audio + Data Data Viewer < > + Control Plugin Thumbnail - uBuffalo / Umichigan Thumbnail viewer - Creare / Mich Camera Control Applet - Minn / Mich Quicktime Slicing tool - Mich (low) Stored Data Viewer - Mich JPEG Viewer - Creare Quickime Viewer - Apple DT Main System Thumbnail Process Quicktime Storage System Quicktime Slicing Tool

  26. MinnesotaMock - up If you area developer and interested in following / helping / participating in this activity, join the mailing list neesgrid-dv@neesgrid.org

  27. Tool List - To Do • Next release of repository • Integrate ELN into repository • DAQ Control Panel in CHEF • Set/Retrieve Metadata • Start / Stop • Ingest data from staging space • Data Turbine Control Panel in CHEF • Start / Stop / Configure Sources • Video | Audio | Data | Thumbnail • Control permanent storage of video

  28. To Do (cont) • NTCP Debugging and Monitoring in CHEF • Needed Data Turbine • Data as Video Client Tools in CHEF • New Monitor Tool • Still Image • Camera Control • Data Turbine Audio Capture • We may need to support XML Schema • QuickTime Capabilities • Archive, retrieve, slice, dice, convert, present • Probably will not be completed as part of SI effort

  29. Data Model Work • Data Curation Summit • Understand issues form go-forward plan • Meeting 3/18/2004 • Data and Metadata Task Force • Finish the tsunami and centrifuge models • DSAC Committee • Meeting 3/19/2004

  30. Summary • In September 2003, we met and “re-visioned” data • A bunch of requirements gathering and development has been done • The “high risk” elements are working now • There is more to do - We will run out of time • Evolutionary development approach - there will always be usable working code - we will stop when we run out of time

  31. The people… • Gokhan Peckan - Data Models • Kincho Law - Data Models / Software design • Jun Peng - Data Models / Software design • Jim Eng - Parse / Ingest / RDF / Project Browser • Jim Myers - Electronic Notebook • Terry Weymouth - Data Turbine • Paul Hubbard - DAQ and NTCP • Joe Futrelle - Data / Metadata Repository

More Related