1 / 12

Terence Critchlow, Xiaowen Xin, Bertram Ludaescher, Ilkay Altintas Mladen Vouk, Zengang Cheng,

A Quick Update on the TSI and PIW workflows SDM All Hands March 2-3, 2005 https://www-casc.llnl.gov/sdm/. Terence Critchlow, Xiaowen Xin, Bertram Ludaescher, Ilkay Altintas Mladen Vouk, Zengang Cheng, Lots of Others…. UCRL-PRES-209990. Example workflows available for download

hovan
Download Presentation

Terence Critchlow, Xiaowen Xin, Bertram Ludaescher, Ilkay Altintas Mladen Vouk, Zengang Cheng,

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Quick Update on the TSI and PIW workflowsSDM All Hands March 2-3, 2005https://www-casc.llnl.gov/sdm/ Terence Critchlow, Xiaowen Xin, Bertram Ludaescher, Ilkay Altintas Mladen Vouk, Zengang Cheng, Lots of Others…. UCRL-PRES-209990

  2. Example workflows available for download Documentation exists and is improving List of (somewhat dated) publications https://www-casc.llnl.gov/sdm/ Web page is up and running

  3. Update notebook Update notebook Update web page Update web page Extract Update notebook Get Variables Create neutrino vars Create Chem vars Update web page Remap coordinates Derive other vars Write diagnostic file Generate plots Update notebook Tool-1 Tool-2 Update web page Tool-3 Tool-4 TSI-1 Workflow Error notification Web interface for Simulation configuration submission Identify config files Move files to correct locations Submit batch request at NERSC Are files consistent Delay Generate directory listing Are there new files? Identify complete files Job still running? Delay Check on job status Empty list Count files Transfer completed correctly Delete file Transfer files to HPSS Transfer completed correctly Transfer files to SB Generate movie Generate thumbnails

  4. Update notebook Update notebook Update web page Update web page Extract Update notebook Get Variables Create neutrino vars Create Chem vars Update web page Remap coordinates Derive other vars Write diagnostic file Generate plots Update notebook Tool-1 Tool-2 Update web page Tool-3 Tool-4 TSI-1 Workflow Remote job submission - done Error notification Web interface for Simulation configuration submission Identify config files Move files to correct locations Submit batch request at NERSC Are files consistent Delay Generate directory listing Are there new files? Identify complete files Job still running? Delay Check on job status Empty list Count files Transfer completed correctly Delete file Transfer files to HPSS Transfer completed correctly Transfer files to SB Generate movie Generate thumbnails

  5. Update notebook Update notebook Update web page Update web page Extract Update notebook Get Variables Create neutrino vars Create Chem vars Update web page Remap coordinates Derive other vars Write diagnostic file Generate plots Update notebook Tool-1 Tool-2 Update web page Tool-3 Tool-4 TSI-1 Workflow Real-time file transfer - done Error notification Web interface for Simulation configuration submission Identify config files Move files to correct locations Submit batch request at NERSC Are files consistent Delay Generate directory listing Are there new files? Identify complete files Job still running? Delay Check on job status Empty list Count files Transfer completed correctly Delete file Transfer files to HPSS Transfer completed correctly Transfer files to SB Generate movie Generate thumbnails

  6. Update notebook Update notebook Update web page Update web page Extract Update notebook Get Variables Create neutrino vars Create Chem vars Update web page Remap coordinates Derive other vars Write diagnostic file Generate plots Update notebook Tool-1 Tool-2 Update web page Tool-3 Tool-4 TSI-1 Workflow Data analysis – not yet started Error notification Web interface for Simulation configuration submission Identify config files Move files to correct locations Submit batch request at NERSC Are files consistent Delay Generate directory listing Are there new files? Identify complete files Job still running? Delay Check on job status Empty list Count files Transfer completed correctly Delete file Transfer files to HPSS Transfer completed correctly Transfer files to SB Generate movie Generate thumbnails

  7. TSI-1 Workflow • Developed modular, configurable, actors that can be used elsewhere • File filtering • Use regular expressions to filter file list • File transfer • User specified command transfers files between two remote locations • SSH • Executes remote command, catches errors • Retry • Repeat an action until successful or until user defined limit is reached

  8. Top level view of TSI workflow

  9. Submit actor Debug actor lets you test without actually submitting job Submit job using llsubmit Return the job number

  10. Added logging Modularized workflow a bit more Added debugging and testing code New visualization actor PIW Workflow

  11. Next Steps • Working through deadlock / race condition problems in Ptolemy PN director • Discovered because of test suite / debug actor • Need to be able to “merge” multiple streams • Data analysis section of TSI-1 workflow • More documentation

  12. Next talks • Steve Parker • SCIRun and SPA integration status • Mladen Vouk • On Large Data Flow Scientific Workflows: An Astropysics Case Study • Bertram Ludaescher • Kepler / SPA Extensions for Scientific Workflows – Now and Upcoming

More Related