1 / 9

Experience with Fusion Workflows

Norbert Podhorszki, Bertram Lud ä scher Department of Computer Science University of California, Davis. kepler-project.org. UC DAVIS Department of Computer Science. Experience with Fusion Workflows. New Challenges.

baxter
Download Presentation

Experience with Fusion Workflows

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Norbert Podhorszki, Bertram Ludäscher Department of Computer Science University of California, Davis kepler-project.org UCDAVIS Department of Computer Science Experience with Fusion Workflows

  2. New Challenges • The CPES project brought new challenges for Kepler and workflow automation people • Remote computations, services and tools • Long running simulations, large amounts of data • One-time-passwords • Workflow = “Glue” • Scientists only need to connect individual components together • Automate tedious processes (logins, copies of data, control, start-stop) • Do it reliably • Show what is going on

  3. Workflows • Real-time Monitoring of Simulation: • Transfer current data set to a secondary resource • Execute short analysis/visualization routines • Display result • Archival and post-processing • Transfer, pack and archive data sets on the fly

  4. Kepler actors for CPES • Job submission to various resource managers • Permanent SSH connection to perform tasks on a remote machine • Generalized actors (workflows themselves) for specified tasks: • Watch a remote directory for simulation timesteps • Execute an external command on a remote machine • Tar and archive data in large junks to HPSS • Transfer a remote image file and display on screen • Control a running SCIRun server remotely • Above actors do logging/checkpointing • the final workflow can be stopped / restarted

  5. Plasma physics simulation on 2048 processors on Seaborg@NERSC (LBL) Gyrokinetic Toroidal Code (GTC) to study energy transport in fusion devices (plasma microturbulence) Generating 800GB of data (3000 files, 6000 timesteps, 267MB/timestep), 30+ hour simulation run Under workflow control: Monitor (watch) simulation progress (via remote scripts) Transfer from NERSC to ORNL concurrently with the simulation run Convert each file to HDF5 file Archive files to 4GB chunks into HPSS Transfer Convert Archive Archival Workflow Monitor

  6. Monitoring Workflow

  7. Future Plans • Currently we have specialized actors that should be generalized for other disciplines and systems • “watching for” simulation output • safe and robust transfer, recovery from failure • archiving to different MSS, with different security policies, robust to failures and maintenance periods • Next workflow is cyclic, not just streaming • couple two simulations on two resources, transfer data and control between them • use local job manager for code execution • What about provenance management? • main reason to use scientific workflow system e.g. in bioinformatics workflows – needed for debugging runs, interpreting results, etc.

  8. Author: Tim McPhillips, UC Davis There is more, e.g., how to get from messy to neat & reusable designs?

  9. The Answer (YMMV) • Collection-Oriented Modeling & Design (COMAD) • embrace an assembly line metaphor • data = taggednestedcollections • e.g. represented as flattened, pipelined token streams:

More Related