1 / 32

NEES Grid Data Overview

NEES Grid Data Overview. Comments to Charles Severance (csev@umich.edu). Introduction. The data approach has evolved significantly in the past year Second version of the Data Repository (security, access control, performance improvements) Data Turbine as unified real-time storage

lafayette-isai
Download Presentation

NEES Grid Data Overview

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. NEES Grid Data Overview Comments to Charles Severance (csev@umich.edu)

  2. Introduction • The data approach has evolved significantly in the past year • Second version of the Data Repository (security, access control, performance improvements) • Data Turbine as unified real-time storage • NTCP has become increasingly capable • Model activity is bearing fruit (multiple) - Protégé / RDF / XML Schema • Data Curation summit has provided vision • We now have a notebook which captures metadata • As we see more detail in these areas, we find new areas that need exploration

  3. Boxology Data Models Notebook Central Repository NEES Grid Data Approach Local Repository Experiment Management Experiment Monitoring Data Acquisition Data Analysis

  4. Data Lifecycle Data Models Experiment Prep Experiment Management Data Monitoring Data Analysis Data Publishing Data Curation Data Discovery and Reuse

  5. Data/MetadataCaptureThroughout Data Models Experiment Prep Experiment Management Data Monitoring Data Analysis Data Publishing Data Curation Data Discovery and Reuse

  6. Data Models • Data models are developed in RDF • Local repository supports multiple simultaneous data models with cross-model linkages • Metadata browser (aka Project browser) becomes the Project Browser, Notebook Browser, Site Specification Database Browser • Metadata browser can federate multiple sources of Metadata

  7. Multiple Models Site Site Model Project Model Proj Person Facility Exp Equipment Trial Specimen Notebook Sensor Element Element Chapter Entry

  8. Overall Data Modeling Efforts NEES Site Site A Site B Site C Specifications Database Equipment People Equipment People ProjectDescription Trials Experiments Experiments Trials Domain Tsnumai Shake Table Centrifuge Geotech Specific Specimen Specimen Specimen Specimen models Common Units Sensors Elements Descriptions Data / Data Data Data Observations Ref. Source: Chuck Severance

  9. Models + Data Model Repo Data Load RDF <owl:ObjectProperty rdf:ID="hasPublications"> <rdfs:domain> <owl:Class> <owl:unionOf rdf:parseType="Collection"> <owl:Class rdf:about="#Project"/> <owl:Class rdf:about="#Task"/> </owl:unionOf> </owl:Class> </rdfs:domain> <rdfs:range rdf:resource="#Publications"/> </owl:ObjectProperty> Configure Models RDF/ OWL Configure

  10. Models + Data Model Repo Data Load RDF Configure Models Protégé - 2K RDF/ OWL Configure

  11. Experiment Preparation • Notebook • Allows the creation of material without needing a model • The model is pages, chapters, and “stuff” • It is all captured with data and metadata • A notebook can be attached to any object in the model structure (i.e. a project can have a notebook, a trial can have a notebook, etc…) • Resources • Discussions • Project Browser • Setup basic structured metadata for the experiment - Trials, descriptions, sensors, etc… This material is captured in accordance to and with the data model

  12. DOE ELN / Example

  13. Setting up and Experiment • Prior to running an experiment, the project browser will be used to create a trial, and experiment configuration, set up sensors, etc. • In some cases, setup information may be done on the DAQ itself and the configuration information may be pulled from the DAQ

  14. SiteSpecific ExperimentalSetup DAQ ProjectRelated C D ExperimentalElement DataElement NEESgrid Experiment Data Flow Project Browser Data Ingestion Experiment Control Data Model NEESGrid Data Repository Data Turbine Stored Viewer Streaming Viewer DAQ Disk

  15. Experiment Management • Simple reference implementations for • Experiment configuration (pull / push) • Experiment Start • Experiment Stop • Some combination of LabView and CHEF code

  16. Still Capture PTZ/ USB DT Client Video Frames BT848 DT Client Data Capture DAQ DT Client Capturing Video and Data Camera Control Gateway DT Main System Simulation Coordinator Site B Site A

  17. Data Monitoring Tools Still Image / Camera Control ^ < > ^ DT Main System ~ < > Camera Control Gateway Still image camera control Thumb- nail Creare viewers

  18. Working with Creare • We want to leverage Creare’s live capture and viewers • Integrated Live Video and Data Viewer • Audio capability in addition to Video • JMF DataSource Capability - Use JMStudio • SI will focus on the extraction, repository, data model, and stored viewer aspects

  19. Data Stored in Data Turbine Video Stills Data Time Step* 4 5 6 7 8 Wall Clock Time * Time Step is only present for Pseudo-dynamic

  20. A tool will be developed to extract data from Data Turbine and place it in the NEES repository in the appropriate format Video Channels Image Channels Data Channels The information will be stored in a format suitable for viewing using the stored viewer and appropriate metadata will be placed in the repository so that the information can be viewed This process is the primary new work in this plan Data Extraction / Ingestion

  21. Data Extraction For Analysis Time Step Channel xyz Start Time Step 1 End Time Step 9999 Data Extraction NEES Data Repository Pseudo-Dynamic Continuous Export Auto Export DT Main System

  22. Pseudo-Dynamic Extraction Video Stills Data Time Step* 4 5 6 7 8 Wall Clock Time

  23. Continuous Extraction Video Stills Data Wall Clock Time

  24. Stored Data Viewer Improvements • Interactive Mode allowing reconfiguration of views within the Applet (insta-view) • Linear combinations of data values • Ability to launch from the Project Browser • Looking at integration with notebook (i.e. launch from the notebook)

  25. Central Repository / Curation • Curation and the Central Repository are different than the local repository and the running / management of experiments • Data must be packaged, kept, indexed, and maintained for the long term

  26. Curation Flow • At some point, a project, experiment, etc is ready for curation. We must save all the information (models, notebooks, sensor data, etc) for transfer to the central repository Curation Bundle

  27. Data/MetadataCaptureThroughout Data Models Experiment Prep Experiment Management Data Monitoring Data Analysis Data Publishing Data Curation Data Discovery and Reuse

  28. Workflow in Central Repository • The workflow of the central repository will be defined over time - here are some sample concepts • Incoming materials collect in an inbox • The curator processed the materials - adds required metadata, checks incoming data models, distinguishes information, and makes the bundle ready for publication • Some data is published immediately, other data is held for a period of time (perhaps to allow for publication) • Published data can be searched and viewed used and downloaded • There are people in the curation loop • The software for this is non trivial and will evolve over time with requirements • Sometimes it will be necessary to alter/convert data to insure its value over time.

  29. Workflow in Central Repository Curation Bundle InBox Search Processed Curation Bundle Hold for Time Published Need Conversion

  30. Conclusion • This is a significant adjustment in priority • But not a significant shift in approach or architecture • All of the elements which have been discussed can still be delivered - the elements described herein are just higher priority.

More Related