1 / 21

Support for P arameter S tudy a pplications in the P-GRADE Portal

Support for P arameter S tudy a pplications in the P-GRADE Portal. Cevat Şener Dept. Of Computer Engineering, METU. General structure of a P arameter S tudy (PS) applications. Algorithm 2. Algorithm 2. Algorithm 2. Algorithm 2. Algorithm 2. Algorithm 2. Algorithm 2. Algorithm 2.

chill
Download Presentation

Support for P arameter S tudy a pplications in the P-GRADE Portal

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Support for Parameter Study applications in the P-GRADE Portal Cevat Şener Dept. Of Computer Engineering, METU

  2. General structure of a Parameter Study(PS) applications Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2 SPMD parallelism: Single Program Multiple Data

  3. AdvancedPS applications Algorithm 1 Cut inputintosmaller pieces Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2

  4. AdvancedPS applications Algorithm 1 Cut inputintosmaller pieces Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 2 Algorithm 3 Aggregate result

  5. PS applications inP-GRADE Portal 2.5 Complete workflow Files in the same LFC catalog (e.g. /grid/gilda/sipos/myinputs) Results produced in the same catalog

  6. Initial input data Generator component(s) Generate orcut input intosmaller pieces Collector component(s) Aggregate result Advanced PS applications inP-GRADE Portal 2.5 Complete workflow Files in the same LFC catalog (e.g. /grid/gilda/sipos/myinputs) Results produced in the same catalog

  7. Parallel execution inside a workflow node (SIMD/MIMD/MISD) • Parallel execution among workflow nodes(SIMD/MIMD/MISD) Multiple jobs run parallel Each job can be a parallel program Third level of parallelism Multiple instances of the same workflow process different data files • Parameter study execution of the workflow(SIMD)

  8. Turning a WF into a PS By turning at least of the open input ports into a “PS Input port” the WF is turned into a Parameter Study

  9. Turning a WF into a PS /grid/gilda/sipos/InputImages Image.0 Image.1 /grid/gilda/sipos/XCoordinates XCoordinate.0 XCoordinate.1 /grid/gilda/sipos/YCoordinates YCoordinate.0 YCoordinate.1 2 x 2 x 2 = 8 execution of the whole workflow /grid/gilda/sipos/Output ImagePart.0 ImagePart.1 . . .

  10. PS workflows 4 x 3 normal workflow execution 1 PS workflow execution PS port: 4 instances of the input file PS port: 3 instances of the input file = This provides the 3rd level of parallelism resulting a very large demand for Grid resources

  11. PS workflow execution

  12. PS workflow detailed vieweWorkflows Total = Init + Submitted + Rescue + Error + Finished

  13. Auto generator Pre defined program logic (static binary) Generates text files User can control text file content by templates and patterns Custom generator User provides generator program logic Useful to generate binary content (e.g. image files, audio files, …) Generators • Generate input files for parameter study workflows • Saves these files on SEs, register them with LFNs into the LFC catalog

  14. Collector • Collects output files and perform collective operation on them. E.g. • Standard deviation • Average • Statistics • Evaluation and find the “best” result • … • User provides the program logic • Portal provides data transfer • Refer in your code to input files as local files • No need to use any Grid API in your code

  15. Workflow management window showing Generator and Collector Collector phase is inactive until each eWorkflow has terminated

  16. Workflow management window showing Generator and Collector The Output(s) of the collector(s) can be downloaded.

  17. The generic PS workflow application First phase: Generators executed Second phase: eWorkflows executed Third phase: collectors executed

  18. Introduction to the practicals

  19. 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 15 Hands-on exerciseMatrix multiplication Multiplication job 1 2 3 4 5 6 7 8 12 2 1 3 1 1 1 3 3 3 Matrix2

  20. Hands-on exerciseMatrix multiplication 1 2 3 4 5 6 7 8 X Auto generator 9 <= X <=15,step 3 Auto generator Multiplication job 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 12 1 2 3 4 5 6 7 8 15 2 1 3 1 1 1 3 3 3 Input files registered in the LFC catalog Matrix2 Output files registered in your the LFC catalog

  21. Hands-on exerciseTurn the matrix operations WF into a PS AB[*,0]Tx AB[*,1] A B A * B A * B [ *, 0 ] A * B [ *, 1 ] A * B [ *, 0 ]T ( A * B [ *, 0 ]T ) * ( A * B [ *, 1 ] ) B

More Related