1 / 29

Coding Provenance in Software and Matching Tools to Data

Coding Provenance in Software and Matching Tools to Data. OPeNDAP Provenance Project And ESIP ToolMatch Project. Patrick West, Tetherless World Constellation Rensselaer Polytechnic Institute. What is Provenance.

caroun
Download Presentation

Coding Provenance in Software and Matching Tools to Data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Coding Provenance in Softwareand Matching Tools to Data OPeNDAP Provenance Project And ESIP ToolMatch Project Patrick West, Tetherless World Constellation Rensselaer Polytechnic Institute

  2. What is Provenance • Provenance is information about entities, activities, and people involved in producing a piece of data or thing. • In Data Science we’re interested in keeping track of, or being able to trace back, how a data product was generated and from what. • E.G. As part of the Ecosystem Status Report there’s an interesting plot in one of the chapters which I’m interested in learning more about.

  3. Generating a Plot

  4. How did I get there?

  5. I know how it was generated • Because I’m the one who added the plot to the document • I know how the plot was generated • I wrote parts of the software in OPeNDAP Hyrax that’s doing the data access, manipulation, and transformation • So I know: . A plot is generated by accessing a set of data using OPeNDAP Hyrax; which generates a DAP DataDDS object by reading in a set of NetCDF files, constraining and projecting the data, running a server side function or two, doing an aggregation; and then using that data product to generate the plot.

  6. Generating a Plot IPython Notebook OPeNDAP Hyrax cell Reads in Data OPeNDAP Request URL cell cell BUT I WANT TO KNOW MORE Badda Bing Badda Boom Spits out data cell Uses data Generates plot

  7. Some informationI WANT to know • How was that plot generated? • What software was used to generate the plot and any intermediary data? • What data files were read in to generate the plot, what was done to the data, and by what? • Where did those data files come from? What parameters are in there? What sensors measured those parameters? Tell me information about the measuring of the data.

  8. Generating a Plot IPython Notebook OPeNDAP Hyrax Where did the data files come from? cell Reads in Data OPeNDAP Request URL cell cell Spits out data cell Uses data Generates plot

  9. Linked Data • I also am interested in the developers of the software and who publishes the software, the licensing of the software, and how I could use it. • I’m interested in what IPython Notebooks are, what they can do, and whether I could use them for other projects. • And I want to be able to let the “owner” of the data files know that I’ve used the results of an access in a publication, presentation, article, or whatever.

  10. What the project focuses on OPeNDAP Hyrax OPeNDAP Hyrax OLFS BES Request URL NetCDF dap Server Side Functions aggregate transform

  11. W3C Prov

  12. Prov-O :dds_of_reading a prov:Entity; dcterms:format opendap:DataDDS; prov:wasGeneratedBy [ a prov:Activity; prov:used <http://test.opendap.org/dap/data/h5/monday.h5> [ a vsto:Dataset, prov:Entity, toolmatch:DataCollection; toolmatch:hasAccessURL <http://test.opendap.org/dap/data/h5/monday.h5>; ]; prov:used <http://test.opendap.org/dap/data/h5/tuesday.h5> [ a vsto:Dataset, prov:Entity, toolmatch:DataCollection; toolmatch:hasAccessURL <http://test.opendap.org/dap/data/h5/monday.h5>; ]; prov:wasAssociatedWith <opendapi:software/hdf5_handler/2.1.1>; ]; .

  13. Prov-O :aggregated_dds a prov:Entity; dcterms:format opendap:DataDDS; prov:wasGeneratedBy [ a prov:Activity; prov:used :constrained_dds; prov:wasAssociatedWith <opendapi:software/ncml_module/1.2.2>; ]; . :result a foaf:Document; nfo:fileName "thursday.h5"; dcterms:format netcdf; prov:wasGeneratedBy [ a prov:Activity; prov:used :aggregated_dds; prov:wasAssociatedWith <opendapi:software/fileout_netcdf/1.2.1>; ]; . :constrained_dds a prov:Entity; dcterms:format opendap:DataDDS; prov:wasGeneratedBy [ a prov:Activity; prov:used :dds_of_reading; prov:wasAssociatedWith <opendapi:software/BES/3.12.0>; ]; .

  14. DOAP – Description of a Project

  15. DOAP – Description of a Project <http://opendap.tw.rpi.edu/instances/software/BES> a doap:Project, prov:Entity; doap:name "OPeNDAP Back-End Server (BES)"; doap:developer <http://tw.rpi.edu/instances/PatrickWest>; doap:developer <http://tw.rpi.edu/instances/DanHalloway>; doap:developer <http://tw.rpi.edu/instances/James_Gallagher>; doap:developer <http://tw.rpi.edu/instances/NathanPotter>; doap:homepage <http://opendap.org/download/hyrax?q=BES_software>; doap:vendor <http://tw.rpi.edu/instances/OPeNDAP>; doap:repository <http://opendap.tw.rpi.edu/instances/Repository>; doap:bug-database <http://scm.opendap.org/trac/>; doap:release <http://opendap.tw.rpi.edu/instances/software/BES/3.12.0>; doap:description "BES is a high-performance back-end server software framework that allows data providers more flexibility in providing end users views of their data."; doap:license <http://opendap.tw.rpi.edu/instances/License>; . <http://opendap.tw.rpi.edu/instances/software/BES/3.12.0> a doap:Version, prov:Entity; prov:specializationOf <http://opendap.tw.rpi.edu/instances/software/BES>; doap:name "BES-3.12.0"; doap:revision "3.12.0"; doap:download-page <http://opendap.org/download/hyrax/1.9>; doap:repository <http://scm.opendap.org/svn/tags/bes/3.12.0>; doap:license <http://opendap.tw.rpi.edu/instances/License>; doap:created 2013-08-27; .

  16. DOAP – Description of a Project <http://opendap.tw.rpi.edu/instances/Repository> a doap:SVNRepository; doap:location <http://scm.opendap.org/svn/> doap:browse <http://scm.opendap.org/svn/> . <http://opendap.tw.rpi.edu/instances/License> dc:description "This software is distributed under the GNU Lesser General Public License <http://www.gnu.org/licenses/gpl.html>"; doap:name "GNU LESSER GENERAL PUBLIC LICENSE"; rdfs:seeAlso <http://www.gnu.org/licenses/gpl.html>; . <http://opendap.tw.rpi.edu/id/opendap/D9IH6677D3I6HDIHD36IHDI7DH> # The hash above is: HASH(config file, BES version that read it) a prov:Agent; prov:wasDerivedFrom <http://opendap.tw.rpi.edu/instances/software/hdf5_handler/2.1.1>, <http://opendap.tw.rpi.edu/instances/software/BES/3.12.0>, <http://opendap.tw.rpi.edu/instances/software/ncml_module/1.2.2/>, <http://opendap.tw.rpi.edu/instances/software/fileout_netcdf/1.2.1>; . prov:wasDerivedFrom :config_file_hash; # b/c BES set it up: prov:wasAttributedTo <http://scm.opendap.org/svn/tags/bes/3.9.2>; .

  17. What We’re Trying • The BES loads shared modules at startup that handle specific tasks • Our first attempt was to use something called a Reporter that reports on the completion of a request, but it’s too after the fact. • Second thought is that the modules themselves add provenance information on the fly, which to me is ideal, but is unrealistic. • The probably implementation is that the BES, the software framework that communicates with the modules, is where the provenance is tracked.

  18. What’s next • Get more use cases about what types of information we want to collect • Write the story about what we’re trying to do • Come up with software use cases for the implementation • Continue discussing provenance with the core OPeNDAP group • Continue to work with the original Prov group (Tim, Jim, and Stephan) in discussions

  19. Questions

  20. ToolMatch Usecase • "I need data for Carbon dioxide (CO2) concentrations, a climate change indicator, for the summer of 2012, that can be accessed via OPeNDAP Hyrax and plotted as a timeseries.” • "I need data with measurements of atmospheric aerosol optical depth sliced along latitude and longitude, returned as netcdf data, and accessible in MatLab."

  21. Using SADL

  22. Inferencing * Equivalent Class DataCollection <Aqua_AIRS_Level2_Plus_AMSU> and(isAccessedBy value OPeNDAP) or (hasDataStorageFormat value NetCDF) and (usesGridType value AuxiliaryLatLonGrid) or (usesGridType value RegularLatLonGrid) and usesConvention value ClimateForecast_CF * Subclass Of mappedBy value IDV and mappedBy value McIDAS-V and mappedBy value Panoply Inferred

  23. Inferencing * Equivalent Class DataCollection and (isAccessedByvalue OPeNDAP) or(hasDataFormatvalue NetCDF) and usesConventionvalueCF1Convention and usesConventionvalue RegularLatLonGrid * Subclass Of mappedByvalue Ferret and mappedByvalue GrADS Inferred

  24. Inferencing * Equivalent Class DataCollection and (isAccessedByvalue GrADSDataServer) or(isAccessedByvalue Hyrax) or (isAccessedByvalue ThreddsDataServer) or (isAccessedByvalue erddap) * Subclass Of isAccessedByvalue OPeNDAP Inferred

  25. Resulting Query The resulting query to find the set of tools available to visualize a data collection becomes very simple DESCRIBE ?tool WHERE { <data_collection> toolmatch:visualizedBy ?tool . ?tool rdf:type toolmatch:Tool . }

  26. The Result Description Tools

  27. Where are weand what’s next • We’ve got part of the ontology done • We’ve got stuff in the triple store • We need to complete the dataset ontology piece • We need to verify the ontology and rules • We need crowd sourcing for more tools and information about tools • Patrick needs to understand rules better

  28. Questions

  29. References OPeNDAP Provenance Project • Prov Overview - http://www.w3.org/TR/prov-overview/ • OPeNDAP Prov - https://github.com/tetherless-world/opendap/ • OPeNDAP LODSPeaKr - http://opendap.tw.rpi.edu/index.html • OPeNDAP Endpoint - http://opendap.tw.rpi.edu/virtuoso/sparql • OPeNDAP – http://opendap.org ToolMatch Project • ToolMatch - http://wiki.esipfed.org/index.php/ToolMatch • ToolMatch Virtual Server - http://toolmatch.tw.rpi.edu/ • ToolMatch Schema - http://toolmatch.tw.rpi.edu/docs/index • ToolMatch Endpoint - http://toolmatch.tw.rpi.edu/sparql

More Related