1 / 17

Pages

Pages. A File System based eScience Workbench Roger Menday Jülich Supercomputing Centre, Germany. Pages is …. … a Problem Solving Environment … inspired by UNICORE experience … acknowledging that it is difficult to beat the flexibility of doing things from the command line

coty
Download Presentation

Pages

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Pages A File System based eScience Workbench Roger Menday Jülich Supercomputing Centre, Germany

  2. Pages is … … a Problem Solving Environment … inspired by UNICORE experience … acknowledging that it is difficult to beat the flexibility of doing things from the command line … reflecting common usage patterns … following ‘convention over configuration’ … a unified description, execution and results … doing some challenging workflow tasks quite naturally and doing some easy things ‘quirkily’ … not perfect for all cases

  3. ‘Tinkering’ • An important part of the eScience process • Not always possible to specify everything up front • Iterative process • Want to react to changes • Start before knowing where it will end up … • Purely user driven workflow is useful • So is partially assisted workflow

  4. Define Submit Execute Retrieve This parsing stage is the one which leads to static-ness. => Must eliminate it ! parse

  5. Define Submit Execute Retrieve Eliminated ! (but what does this unified workflow look like ?) parse

  6. Define Submit Execute Retrieve Now able to directly interact with the workflow during all stages of the workflow a bit like a spreadsheet Enhance / Redefine

  7. Strategy • The working environment of the eScientist is the file-system • Through following conventions and using simple command-line tools, a workflow environment is provided by <pages> reading/writing the file-system in harmony with the user reading/writing the file-system

  8. API u.follow(‘.containsjobs.containsjob’) { |i| i.delete } Deletes all jobs across my Grid u.follow(‘.containsstorage[name=‘home’]’) { |i| i.grind(3, ‘*.mp3’) { |i| i.backup( …dest… ) } } Find all mp3 files in all home storages (up to a depth of 3 in the hierachy), then backup

  9. Side Note Such an API works best by having a unified model of the Grid - a graph of interlinked resources. And a uniform way of addressing these resources and interacting with them.

  10. Command line … mirroring the API gak follow .containsjobs.containsjob | gak delete gak follow .containsstorage[name=‘home’] | gak grind –d 3 ‘*.mp3’ | gak backup –f storage

  11. Bench, Book, Chapter, Page • books on a bench … a book for each research interest (?) • a book has a number of chapters • chapters consist of pages In addition, it would be nice to share one or more of my books with my colleagues …

  12. Start from the command line (or using API) • Initialisation of a ‘long running task’ recorded • … in the file-system, at your PWD • Each task is a page

  13. A collection of pages in a chapter • Directory is a chapter • Express dependencies between chapters • Directed Acyclic Graph • Dependencies recorded in a file for each chapter • A triggered chapter will trigger its contained pages (the book analogy breaks a little here)

  14. The workbench will be monitored • For each page scripts are invoked as lifecycle events are reached (BEFORE, AFTER) • Similarly for each chapter, scripts are invoked (STARTED, READY, DONE)

  15. Dependencies Explicit Implicit Implied processing order for the individual pages with a single chapter • … expression of dependencies • i.e. between chapters there is a explicit workflow ‘please read chapters 2 & 3 before reading chapters 6’

  16. Assistance at the command line pages graph • draws graph of current workbook pages contents • lists all chapters pages link ‘n’ • link chapters pages clone • clones current book pages clone • clones current book pages book • list all books / creates new one pages www • publishes workbench as website pages facebook • publishes workbench as facebook website

  17. Finally • Package workflow as zip/tar files • Behaviour via file-system/command line/API • Insert new behaviour through lifecycle scripts • Hold, pause, request input • Share results (selectively) to web • As much (or as little) automation as you need

More Related