1 / 32

Grid Application Support by the P-GRADE Portal

Grid Application Support by the P-GRADE Portal. Gergely Sipos sipos @ sztaki.hu. MTA SZTAKI Computer and Automation Research Institute Hungarian Academy of Sciences www.lpds.sztaki.hu/pgportal pgportal @ lpds.sztaki.hu. Outline. Motivation of our work P-GRADE Portal in a nutshell

kamea
Download Presentation

Grid Application Support by the P-GRADE Portal

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Grid Application Support by the P-GRADE Portal Gergely Sipossipos@sztaki.hu MTA SZTAKI Computer and Automation Research InstituteHungarian Academy of Scienceswww.lpds.sztaki.hu/pgportal pgportal@lpds.sztaki.hu CEOS WGISS Meeting, May 8-12, 2006, Budapest

  2. Outline • Motivation of our work • P-GRADE Portal in a nutshell • Grid application development • Grid application execution and monitoring • Support for collaborative work • Conclusions CEOS WGISS Meeting, May 8-12, 2006, Budapest

  3. Layered view of Grid systems E-Scientists Applications Graphical grid tools:P-GRADE Portal Application toolkits, portals Command line grid tools Higher-level grid services: brokering, data replica, … Grid middleware:e.g. Globus, EGEE, … Basic Grid services:security, job submission, info, … Resources to be shared CEOS WGISS Meeting, May 8-12, 2006, Budapest

  4. Motivation to theP-GRADE Portal • Fast evolution of Grid middleware technologies and tools: • GT2, OGSA, GT3 (OGSI), GT4 (WSRF), LCG-2, gLite, … • Many production Grids are now freely available for e-Scientists • EGEE (LCG-2  gLite), UK NGS (GT2), US Open Science Grid (GT2  GT4), NorduGrid (ARC), … • The same set of services are available everywhere, but implemented in different ways • Computation services, data services, security services, (brokers) • Let’s provide a technology-neutral, intuitive graphical interface for the most common Grid middleware services! CEOS WGISS Meeting, May 8-12, 2006, Budapest

  5. P-GRADE Portal in a nutshell • General purpose, workflow-oriented computational Grid portal. Supports the development and execution of workflow-based Grid applications. • Based on standard portlet framework (Gridsphere) • Easy to expand with new portlets (e.g. application-specific portlets) • Easy to tailor to community needs • Grid services supported by the portal: The P-GRADE Portal hides middleware technologies and solves Grid interoperability problem at the workflow level CEOS WGISS Meeting, May 8-12, 2006, Budapest

  6. What is a P-GRADE Portal workflow? • a directed acyclic graph where • Nodes represent jobs (batch programs to be executed on a computing resource) • Ports represent input/output files the jobs expect/produce • Arcs represent file transfer operations • semantics of the workflow: • A job can be executed if all of its input files are available CEOS WGISS Meeting, May 8-12, 2006, Budapest

  7. Parallel execution inside a workflow node • Parallel execution among workflow nodes Multiple jobs run parallel Each job can be a parallel program Two levels of parallelism • The workflow concept of the P-GRADE Portal enables the efficient parallelization of complex problems • Semantics of the workflow enables two levels of parallelism: CEOS WGISS Meeting, May 8-12, 2006, Budapest

  8. Ultra-short range weather forecast (Hungarian Meteorology Service) Forecasting dangerous weather situations (storms, fog, etc.),crucial task in the protection of life and property 25 x Processed information: surface level measurements, high-altitude measurements, radar, satellite, lightning, results of previous computed models 10 x 5 x 25 x • Requirements: • Execution time < 10 min • High resolution (1km) CEOS WGISS Meeting, May 8-12, 2006, Budapest

  9. Executable Executable Portal Client in Geneva EGEE (CE) Service Invocation WestFocus GT4 Grid (UK) Executable Leeds Poznan Executable UoW Budapest Brunel Multi-Grid portal: Demonstration @ EGEE User Forum Portal Server in London NGS GT2 (UK) P-GRADE Portalserver Manchester Job Repository (GEMLCA) CEOS WGISS Meeting, May 8-12, 2006, Budapest

  10. SAVE WORKFLOW START EDITOR The typical user scenarioPart 1 - development phase Certificate servers Gridservices Portal server OPEN & EDIT or DEVELOP WORKFLOW CEOS WGISS Meeting, May 8-12, 2006, Budapest

  11. TRANSFER FILES, SUBMIT JOBS DOWNLOAD PROXY CERTIFICATES MONITOR JOBS VISUALIZE JOBS and WORKFLOW PROGRESS SUBMIT WORKFLOW DOWNLOAD RESULTS DOWNLOAD RESULTS The typical user scenarioPart 2 - execution phase Certificate servers Gridservices Portal server CEOS WGISS Meeting, May 8-12, 2006, Budapest

  12. Opening the workflow editor The editor is a Java Webstart applicationdownload and installation is only a click! CEOS WGISS Meeting, May 8-12, 2006, Budapest

  13. Workflow Editor Define DAG of batch jobs: • Drag & drop components:jobs and ports • Connect ports by channels(no cycles, no loops) • Define job and port properties CEOS WGISS Meeting, May 8-12, 2006, Budapest

  14. Job property window • Properties of a job: • Binary executable • (Type of executable) • (Number of processors) • Command line parameters • The resource to be used for the execution: • Grid • Broker / resource CEOS WGISS Meeting, May 8-12, 2006, Budapest

  15. Support for manual resource selection:information system browser The information system portlet queries EGEE and Globus information systems CEOS WGISS Meeting, May 8-12, 2006, Budapest

  16. Support for broker-based resource selection • Select a broker Grid/VO for the job • (Specify extra ranks & requirements for the job in Job Description Language) • The broker will find the best resource for your job! CEOS WGISS Meeting, May 8-12, 2006, Budapest

  17. Defining input/output data File type: input:required by the job output:produced by the job File location:local:my desktop remote: grid storage site File name: unique name of the file File storage type: Permanent:final result of the WF Volatile:only used for inter-job data transfer CEOS WGISS Meeting, May 8-12, 2006, Budapest

  18. The typical user scenarioExecution phase – step 1: Certificate servers Gridservices DOWNLOAD PROXY CERTIFICATES Portal server CEOS WGISS Meeting, May 8-12, 2006, Budapest

  19. Prerequisite of workflow execution: Download user certificates • Certificate server details: • Hostname • Port number • User name • Password • Certificate parameters: • Lifetime CEOS WGISS Meeting, May 8-12, 2006, Budapest

  20. Result of certificate download Multiple certificates for multiple Grids/VOs can be available on the portal server at the same time! HUNGRID resources SEE-GRID resources CEOS WGISS Meeting, May 8-12, 2006, Budapest

  21. Workflow Execution(observation by the workflow portlet) CEOS WGISS Meeting, May 8-12, 2006, Budapest White/Red/Green color means the job is initial/running/finished state

  22. Workflow Execution(observation by the workflow portlet) CEOS WGISS Meeting, May 8-12, 2006, Budapest White/Red/Green color means the job is initial/running/finished state

  23. Workflow Execution(observation by the workflow portlet) CEOS WGISS Meeting, May 8-12, 2006, Budapest White/Red/Green color means the job is initial/running/finished state

  24. Workflow Execution(observation by the workflow portlet) CEOS WGISS Meeting, May 8-12, 2006, Budapest White/Red/Green color means the job is initial/running/finished state

  25. Workflow Execution(observation by the workflow portlet) CEOS WGISS Meeting, May 8-12, 2006, Budapest White/Red/Green color means the job is initialised/running/finished

  26. The portal monitors and visualizes parallel jobs On-Line application monitoring • The portal monitors and visualizes workflows: • Input data transfer • Job execution • Output data transfer CEOS WGISS Meeting, May 8-12, 2006, Budapest

  27. Downloading the results… • Download small files in a single ZIP file • Keep large files in the Grid for future analysis! CEOS WGISS Meeting, May 8-12, 2006, Budapest

  28. Collaborative work by the P-GRADE Portal • Export and import workflows and execution results from/into the portal • Share legacy applications as services with other portal users • Share successfully executed jobs as services with other portal users • Integrate other users’ legacy applications and jobs into your workflow CEOS WGISS Meeting, May 8-12, 2006, Budapest

  29. References • P-GRADE Portal service is available for • SEE-GRID infrastructure • HUNGRID VO of EGEE • Central European VO of EGEE • GILDA: Training VO of EGEE • UK National Grid Service • Under preparation for • US Open Science Grid, Economy-Grid, Swiss BioGrid, BioInfoGrid, Turkish Grid • P-GRADE portal can be installed for any public/private EGEE or Globus Grid/VO Please contact us! CEOS WGISS Meeting, May 8-12, 2006, Budapest

  30. Main conclusion:Easy-to-use, technology-neutral Grid portal for e-Scientists • The P-GRADE Portal hides differences ofGrids • Globus X – LCG2 – gLite Grid interoperability • Transparent switching between Grid technologies • Graphical tools for application development, execution and monitoring • Seq & parallel components can be integrated into large Grid applications • Your code does not have to contain grid specific calls • Support for collaborative research • Share workflows • Share legacy applications and jobs as application services • Built by standard portlet API • customizable to specific application areas and user groups(such as earth scientists…) CEOS WGISS Meeting, May 8-12, 2006, Budapest

  31. Further information… • Visit the www.lpds.sztaki.hu/pgportal site(manuals, slide shows, installation procedure, etc.) • Visit or request a training event!(see list of events on P-GRADE Portal homepage) • Lectures, demos, hands-on tutorials, application development support • Get an account for a portal service: • VOCE portal - SZTAKI • SEEGRID portal – SZTAKI • HUNGrid portal – SZTAKI • NGS portal – University of Westminster • Install a portal for your community: If you are the administrator of a Grid/VO SZTAKI is pleased to help you install a portal for your community! CEOS WGISS Meeting, May 8-12, 2006, Budapest

  32. Latest news! Thank you!Questions? • P-GRADE Portal tutorial in Paris (free event!) 22-23, June – lectures, hands-on, application support See details on P-GRADE Portal homepage! • Joint Regional CE EGEE and SEE-GRIDSummer School on Grid Application Support 3-8 July, Budapest, SZTAKI www.egee.hu/grid06 CEOS WGISS Meeting, May 8-12, 2006, Budapest

More Related