1 / 8

Pegasus in FutureGrid and Experiment Management

Pegasus in FutureGrid and Experiment Management. Presented by Jens-S. Vöckler. Current JIRA tasks.

kenna
Download Presentation

Pegasus in FutureGrid and Experiment Management

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Pegasus in FutureGrid and Experiment Management Presented by Jens-S. Vöckler http://futuregrid.org

  2. Current JIRA tasks -+- FG 412 (master task) | +- FG 407 Pegasus available for use on each resource | | +- FG 838 Get Pegasus into FG vanilla VM images | | | `- FG 839 minimal input installation shell scripts | | +- FG 413 Pegasus available on test-bed | | `- FG 414 (milestone) Pegasus avail for use completed | +- FG 415 Pegasus documentation and maintenance | | +- FG 436 Use-case documentation | | +- FG 671 workflow architecture document | | +- FG 672 experiment management | | `- FG 753 dev-doc describes activities for FG-SW plan | +- FG 409 Immediate resource provisioning workflow | +- FG 410 Time-sensitive resource provisioning workflow | +- FG 418 Workflow repository requirement | +- FG 419 Pegasus tutorial | +- FG 420 End-2-end experiment management workflows | `- FG 421 Workflow repository… (few non-related tasks) http://futuregrid.org

  3. Proposed Restructuring • Two basic, parallel-running, root-tasks: • Pegasus on FutureGrid • Helps to draw domain science users • Provides familiar context • Envision FG-specific tutorial, support, etc. • Pegasus as part of Experiment Management • Workflow capabilities considered essential for EM • Includes notion of repeatable experiment • Related is the experiment repository http://futuregrid.org

  4. [1] Pegasus on FutureGrid • Pegasus installation on FutureGrid (in progress) • Pegasus installation on FG helpful minimal input shell scripts (in progress) • Pegasus onto Nimbus (done) • Pegasus onto Eucalyptus (update necessary) • Pegasus onto OpenStack(in progress) • Pegasus onto bare-metal (to be done) • Pegasus on FG tutorial - Nimbus based (started) • Create Pegasus “submit host” FG VM (to be done) • Create Pegasus “worker node” FG VM (mostly done, see above) • Use either cloudinit.d or Wrangler to "connect" above ( 1.2.1 + N x 1.2.2 ) • Write Pegasus tutorial based on this setup (to be done) • Pegasus on FG documentation (to be done) • Document worker-node behavior inside VMs • Document worker-node behavior on bare-metal • Document behavior as submit-host • Pegasus immediate provisioning workflow (prototype done) • Pegasus time-sensitive provisioning workflow (to be done) http://futuregrid.org

  5. [2] Pegasus in Experiment Management • Pegasus as part of FG Experiment Management (EM) • Better define "what is an experiment” (still discussing) • Design the "missing pieces" of Experiment Management • Identify EM pieces that are covered by Pegasus pieces. • Research what parts of FG can be taken into EM • FG rain client for regular and dynamic provisioning • FG image management to provision and configure vetted VMs • FG Moab for advanced scheduling HPC operations • Design Experiment Catalog a.k.a. workflow repository. • Probably more that will surface later • Consider inter-operability between PWMS and EH • Record EH activity into something that replays in PWMS. • Consider leveraging EH scripts inside PWMS workflows. http://futuregrid.org

  6. [2] Pegasus in Experiment Management (Cont.) • Create end-to-end experiment workflow • Define use-case based on manual periodogram work. • Document use-case in FG SW documentation. • Divide use-case into goal and steps. • Populate Experiment Catalog with Pegasus experiments • Experiment Management Documentation • EM Tutorial • Similar to Pegasus tutorial http://futuregrid.org

  7. PWMS Milestones (proposed) Milestones depend on timely arrival of the remainder of year 3 and year 4 funds. http://futuregrid.org

  8. EM Milestones (proposed) Year 4 milestones will be revisited next year. http://futuregrid.org

More Related