Metadata, Provenance, and Search in e-Science

1. Metadata, Provenance, and Search in e-Science Beth Plale Director, Center for Data and Search Informatics School of Informatics Indiana University

2. Credits:PhD students Yogesh Simmhan, Nithya Vijayakumar, and Scott Jensen. Dennis Gannon, IU, key collaborator on discovery cyberinfrastructure

3. Nature of Computational Science Discovery • Extract data from heterogeneous databases, • Execute task sequences (“workflows”) on your behalf, • Mine data from sensors and instruments and responding, • Try out new algorithms, • Explore data through visualization, and • Go back and repeat steps again: with new data, answering new questions, or with new algorithms. • How is this discovery process supported today? • Through cyberinfrastructure. • CyberInfrastructure that supports • On demand knowledge discovery • Automated experiment management (data and workflow) • Data protection, and automated data product provenance tracking.

4. CyberInfrastructure: framework for discovery • Plug and play data sources and analysis tools. Complex what-if scenarios. Through • User portal • Personal metadata catalog of data exploration results • Data product index/catalog • Data provenance service • Workflow engine and composition tools • Tied together with Internet-scale event bus. • Results publishable to digital library.

5. Cyberinfrastructure for computing: DSI DataCenter Supports analysis, use, visualization and search research. Supports multiple datasets.

6. Distributed services provide functionali capability

7. Vision for Data Handling • Capturing metadata about data sets as generated is key • Syntatic: file size, date of creation and • Semantic or domain specific: spatial region, logical time • Context of file is key search parameter • Provenance, or history of data product, needed to assess quality • Volume of data used in computational science too large: manage on behalf of user • Indexes help efficiency

8. The Realization in Software Workflow graph Application services Compute Engine User’s Browser Workflow Engine App factory Event Notification Bus Portal server MyLEAD Agent service Data Management Service Data Catalog service Provenance Collection service MyLEAD User Metadata catalog Data Storage

9. Infrastructure is portal based - that is, all services are available through a web server

10. Gateway Services Proxy Certificate Server (Vault) Application Deployment Workflow engine Resource Registry Community & User Metadata Catalog Events & Messaging Resource Broker Core Grid Services Security Services Information Services Self Management Resource Management Execution Management Data Services Resource Virtualization (OGSA) Compute Resources Data Resources Instruments & Sensors e-Science Gateway Architecture User’s Grid Desktop Grid Portal Server [1]Service Oriented Architectures for Science Gateways on Grid Systems, Gannon, D., et al.; ICSOC, 2005

11. LEAD-CI Cyberinfrastructure • Workflows run on the LEADgrid and on Teragrid. • Portal and persistent back-end web services run on LEADgrid. • Data storage resources for storing user-generated data products are provided by Indiana University.

12. Typical weather forecast runs as workflow Visualization Pre-Processing Assimilation Forecast Terrain data files ETA, RUC, GFS data IDV viz arpstrn Ext2arps-ibc Ext2arps-lbc Surface data files WRF Radar data (level II) arpssfc 88d2arps arps2wrf wrf2arps ADAS assimilation Radar data (level III) arpsplot Surface, upper air mesonet & wind profiler data nids2arps Satellite data ~400 Data Products Consumed & Produced –transformed– during Workflow Lifecycle mci2arps

13. To set up workflow experiment, we select a workflow (not shown) then set model parameters here

14. Supported community data collections

15. Data Integration Local view: crosswalk point of presence supports crawling, publishes difference list as LEAD Metadata Schema (LMS) documents CASA radar Collection, Months (ftp) Globally integrated view: Data Catalog Service Oklahoma Boolean search query Latest 3 days Unidata IDD Distribution (XML web server) • Crawler crawls catalogs; • Builds index of results; • Web service API; • Boolean search query with spatial/temporal support Indiana List of results as LEAD Metadata Schema documents Web service API Level II and III radar, latest 3 days (XML web server) Colorado ETA, NCEP, NAM, METAR, etc. (XML web server) Index XMLDB native XML database and Lucene for index Colorado crosswalks

16. LEAD Personal Workspace • CyberInfrastructure extends user’s desktop to incorporate vast data analysis space. • As users go about doing scientific experiments, the CI manages back-end storage and compute resources. • Portal provides ways to explore this data and search and discover it. • Metadata about experiments is largely automatically generated, and highly searchable. • Describes data object (the file) in application-rich terms, and provides URI to data service that can resolve an abstract unique identifier to real, on-line data “file”.

17. Searching for experiments using model configuration parameters: 2 attributes selected

18. Searching for experiments based on model parameters: 4 returned experiments; one displayed

19. How forecast model configuration parameters stored in personal catalog Forecast model configuration file handed off to plugin that shreds XML document into queriable attributes associated with experiment

20. Data.In.1 Data.Out.1 Application A Data.In.2 Config.A What & Why of Provenance • Derivation history of a data product • What (when, where) application created the data • Its parameters & configuration • Other input data used by application • Workflow is composed from building blocks like these. So provenance for data used in workflow gives workflow trace • Data Provenance::Data.Out.1 • Process: Application_A • Timestamp: 2006-06-23T12:45:23 • Host: tyr20.cs.indiana.edu … • Input: Data.In.1, Data.In.2 • Config: Config.A

21. The What & Why of Provenance • Trace Workflow Execution • What services were used during workflow execution? • Validate if all steps of execution successful? • Audit Trail • What resources were used during workflow execution? • Data Quality & Reuse • What applications were used to derived data products? • Which workflows use a certain data product? • Attribution • Who performed the experiment? • Who owns the workflow & data products? • Discovery • Locate data generated by a workflow • Locate workflows containing App-X that succeeded

22. Message Bus WS-EventingService API Query for Workflow, Process, & Data Provenance Karma Provenance Service Provenance Browser Client Provenance Listener Provenance Query API Activity DB Subscribe & Listen to Activity Notifications WS-Messenger Notification Broker Workflow–Started & –Finished Activities Publish Provenance Activities as Notifications Application–Started & –Finished, Data–Produced & –Consumed Activities Workflow Engine Workflow Instance 10 Data Products Consumed & Produced by each Service Orchestration Service 1 Service 2 Service 9 Service 10 … 10C 10P 10P 10C 10P/10C 10P/10C Collection Framework A Framework for Collecting Provenance in Data-Centric Scientific Workflows, Simmhan, Y., et al., ICWS Conference, 2006

23. Generating Karma Provenance Activities • Instrument applications to publish provenance • Simple Java Library available to • Create provenance activities • Publish activities as messages • Jython “wrapper” scripts use library to publish provenance & invoke application • Generic Factory toolkit easily converts applications to web service • Built-in provenance instrumentation

24. Sample Sequence of Activities appStarted(App1) info(‘App1 starting’) fileReceiveStarted(File1) -- do gridftp get to stage input file File1 -- fileReceiveFinished(File1) fileConsumed(File1) computationStarted(Code1) -- call Fortran code Code1 to process input files -- computationFinished(Code1) fileProduced(File2) fileSendStarted(File2) -- do gridftp put to save output file File2 -- fileSendFinished(File2) publishURL(File2) appFinishedSuccess(App1, File2) | appFinishedFailed(App1, ERR) flush()

25. Performance perturbation

26. Standalone tool for provenance collection and experience reuse: future direction

27. Forecast start time can also be set to occur on severe weather conditions (not shown here)

28. Weather triggered workflows • Goal is cyberinfrastructure that allows scientists and students to run weather models dynamically and adaptively in response to weather events. • Accomplished by coupling events processing and triggered forecast workflows • Vijayakumar et al (2006) presented framework for this purpose • Events-processing system does temporal and spatial filtering. • Storm detection algorithm (SDA) detects storm events in remaining streams • SDA returns detected storm events • Events processing system generates trigger to workflow engine

29. Continuous stream mining • In stream mining of weather, events of interest are anomalies • Event processing queries can be deployed to sites in the LEAD grid (rectangles) • Data streams delivered to each site through Unidata Internet Data Dissemination system • CEP enables real-time response to the weather query computation node data generation source

30. Example CEP query • Scientists can set up a 6-hour weather forecast over a region of say a 700 sq. mile bounding box, and submit a workflow that will run sometime in the future • CEP query detects severe storm conditions developing in the region • The forecast workflow is started at a future point in time as determined by the CEP query

31. Stream Provenance Tracking • Data stream provenance - derivation history of data product where data product is derived time-bounded stream • Stream provenance can establish correlations between significant events (e.g., storm occurrences) • Anticipate resource needs by examining provenance data and discover trends in weather forecast model output • Determine when next wave of users will arrive, and where their resources might need to be allocated

32. Workflow graph Application services Compute Engine User’s Browser Workflow Engine App factory Event Notification Bus NEXRAD Streams Portal server Mining queries MyLEAD Agent service Data Management Service Calder Stream Mining Service Data Catalog service MyLEAD User Metadata catalog Data Storage Doppler Radars Stream processing as part of cyberinfrastructure • SQL-based queries responding to input streams event-by-event within stream and concurrent across streams • Each query generates time-bounded output stream

33. User Query Planner Service Rowset service aggregating derived streams Obtain continuous query Compile SQL to TCL query Create Ring Buffer Distribute query Setup buffer to aggregate results Deploy queries Query start / stop / distribution plan chance Results if any Provenance Service Updates on Stream rates, approximations etc Queries Process updates and store in DB Computational mesh executing query execution engines DB Provenance Service in Calder Process flow / invocation Calder internal messaging WS-Messenger notifications

34. Provenance Update Handling Scalability • Update processing time - time taken from instant user sends a notification to instant provenance service completes corresponding update • Experiment • Bombard provenance service at different update rates by simulating many clients sending provenance updates simultaneously • Measure incoming rate at provenance service and overall time taken for handling each update. • Overhead includes time to create message, send and receive through WS-Messenger, process message and store it in DB

35. Problem: • Severe weather can bring many storms over a local region of interest • It is infeasible and unnecessary to run weather model in response to each of them • Solution: • Group storm events into spatial clusters • Trigger model runs in response to clusters of storms

36. Spatial Clustering: DBSCAN algorithm* • DBSCAN is a density-based clustering algorithm and it can do spatial clustering location parameters are treated as features. • DBSCAN algorithm has two parameters • ε: radius within which a point is considered to be a neighbor of another point • minPt: minimum number of neighboring points that a point has to have to be considered as a core point. • The two parameters determine the clustering result * Mining work done by Xiang Li, University of Alabama Huntsville

37. Data • WSR88D radar data on 3/27/2007 • Total of 134 radar sites covering CONUS • The time period examined is between 1:00 pm to 6:00pm EST. • The 5 hrs time period is divided into 20 time interval with each interval of 15 min. Storm events within the same time interval is clustered Storm events detected at 1:00 pm – 1:15 pm * Mining work done by Xiang Li, University of Alabama Huntsville

38. DBSCAN result K-means result Algorithm comparison: DBSCAN and K-means Time period: 1:00 pm – 1:15 pm Number of clusters: 3 Conclusion: DBSCAN algorithm performs better than k-means algorithm

39. Future Work • Publication of provenance to digital library • Generalized support for metadata systems • Enhanced support for mining triggers • Personal weather predictor • LEAD framework packaged into single 8-16 core multicore machine • Expands educational opportunities: suitable for small schools • Engage communities beyond meteorologists

40. Thank you for the interest. Thanks to my many domain science and CS collaborators, to my students, and to the funding agents.Please feel free to contact me at plale@indiana.edu