1 / 25

Grid - korisnicki pristup i razvoj aplikacija

Grid - korisnicki pristup i razvoj aplikacija. Meteo VO Training, Belgrade 24. June 2008. Branko Marovic RCUB - UoB. SEE-GRID-2 Applications Adaptation & Support. Application lifecycle defined in “Gridification Guidelines” document.

chill
Download Presentation

Grid - korisnicki pristup i razvoj aplikacija

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Grid - korisnicki pristup i razvoj aplikacija Meteo VO Training, Belgrade 24. June 2008 Branko Marovic RCUB - UoB The SEE-GRID-SCI initiative is co-funded by the European Commission under the FP7 Research Infrastructures contract no. 211338

  2. SEE-GRID-2 Applications Adaptation & Support • Application lifecycle defined in “Gridification Guidelines” document. • SG Gridification Guide (wiki pages) structured and contributed by developers. • Policy: Applications initially supported at the country level. • Application Support Group (ASG) created for gridification support. • Application Review Committee(ARC) created to assess existing and new candidate applications. • Application selection criteria defined by ARC. • MoU Between Applications and SEEGRID-2. Application Gridification 2/26

  3. SEE-GRID-2 Strategy for New Applications • There is “Gridification Guideline” document to provide initial help to developers. (application life-cycle, policy and methodology to run and support an application on Grid) • There is “SG Gridification Guide” on SEE-GRID-2 Wiki to provide comprehensive help to developers (porting an existing application to the Grid) • ARC (Application Review Committee) recommendations push applications towards collaborations with other applications and projects. • Applications are required to provide update on their established user community and collaborations within their periodic Application Progress Report. Application Gridification 3/26

  4. Application Gridification 4/26

  5. SEE-GRID2 Applications Application Gridification 5/26

  6. Support the deployed applications • Application support group (ASG)created to provide fast and continuous gridification support to developers. • Experienced developers & admins • ASG team members assigned for each application. • National level application support • SEE-GRID - global level application support • SG Gridification Wiki online. • SEE-GRID helpdesk available and used for problems. • Useful Helpdesk Tickets for Developers. • SEE-GRID Helpdesk “Applications Support” knowledge base. • GOODs responsible to dump useful tickets of the week. • Work in close collaboration with WP5 (training) and WP3 (software requirements, maintenance of performance) Application Gridification 6/26

  7. Application Lifecycle Application Gridification 7/26

  8. SEE-GRID-2Gridification Guides • An initial guide for the developer including the application life-cycle, VO and site policies, gridification methodology: • Gridification Guideline • http://www.see-grid.eu/content/modules/downloads/SEEGRID2-WP4-TR-008-Gridification_Guideline-n-2006-11-22.doc • Detailed wiki pages with illustrations for gridification issues: • SG Gridification Guide • http://wiki.egee-see.org/index.php/SG_Gridification_Guide Application Gridification 8/26

  9. SEE-GRIDGridification Guide • Relevant topics for application developers identified trough online questionnaire system • Some investigation areas identified as well – candidates for future GG topics • Gridification guide will provide information on these topics • Wiki as collaboration medium Application Gridification 9/26

  10. SEE-GRIDGridification Guide 1 Job Management 1.1 Running Jobs (T1) 1.2 Software Installation Management (T3) 2 File Management and Access 2.1 Data Management (T2) 2.2 Data Access (T4) 2.3 Logical Files (T6) 2.4 Advanced Replica Management (T2, T6 related investigation area) 3 Inter-Job Communication 3.1 Workflows (T7) 3.2 MPI (T9) 3.3 Interactive & Pilot Jobs (T1 related investigation area) 3.4 Java networking 4 Information Services 4.1 Application-Level Event Logging & Performance (T5) 4.1.1 Applications and Systems monitoring tools 4.1.2 Benchmarking 4.2 Metadata Catalogues 5 Development Environments and Portals 5.1 Web Authentication and Authorization (T8) 5.2 P-GRADE Portal (T1-T9 related investigation area) 5.3 P-GRADE (T1-T9 related investigation area) Application Gridification 10/26

  11. Application Gridification 11/26

  12. VO Resource Allocation Policies • Seegrid VO is used for all applications: • Utilization of groups and roles:/seegrid/COUNTRY_CODE/SHORT_APP_NAME/Role=production/seegrid/COUNTRY_CODE/SHORT_APP_NAME/Role=software • For gridification testing phase, • no role or group is used, all sites support all applications. • For deployed applications, • sites are encouraged to support as many applications as possible. • MoU template between SEE-GRID-2 Project and the Application created. Application Gridification 12/26

  13. SEE-GRID-SCI Application Gridification 13/26

  14. Meteorology VO • Members • National Observatory of Athens (coordinator) • South Environment and Weather Agency of Serbia • Hydrometeorological Institute of Montenegro • RBI of Zagreb • University of Zagreb • Federal Hydro-Meteorological Institute of Bosnia and Herzegovina • Republic Hydrometeorogical Institute, Banjaluka • Two sets of applications for initial deployment: • Regional scale Multi-model, Multi-analysis ensemble forecasting system • Study of the interaction of airflow with complex terrain Application Gridification 14/26

  15. Developer Resources • Grid environment is constantly evolving, but • Useful features persist • New are constantly being added • Bugs are being fixed • Gained knowledge remains relevant, must be updated • Applications can be easily migrated to new/updated APIs • gLite User Guide • https://edms.cern.ch/file/722398/1.2/gLite-3-UserGuide.pdf • WMS user guides • https://edms.cern.ch/document/572489/1 • SEE-GRID Gridification Guide • http://wiki.egee-see.org/index.php/SG_Gridification_Guide • SEEGRID Wiki • http://wiki.see-grid.eu/ • gLite documentation • http://glite.web.cern.ch/glite/documentation/ Application Gridification 15/26

  16. Grids: a foundation for e-Research computers software Grid sensor nets instruments Shared data archives Diagram derived from Ian Foster’s slide colleagues Enabling a whole-system approach Collaborative research / engineering / public service … Application Gridification 16/26

  17. Načini upotrebe grida • Komandna linija, JDL, input/output sandbox i/ili ručna manipulacija podacima • Primena skriptova u generisanu poslova, obradi i manipulaciji podacima • Korišćenje postojećih Grid portala • Primena grid API-ja i servisa iz koda koji se izvršava na gridu • Pravljenje aplikativno specifičnih pristupnih tačaka-servisa Application Gridification 17/26

  18. Complexities of grid applications • Simple jobs – submitted to WMS to run in batch mode • Job invokes grid services • To read & write files on SE • Monitoring • For outbound connectivity (interactive jobs) • To manage metadata • … • Complex jobs • An environment controls multiple jobs on users’ behalf • High-level services • Portals with workflow • Software written for the VO (or by the user) • … Application Gridification 18/26

  19. Invocation of applications • From the UI • Command Line Interfaces / Scripts • APIs • Higher level tools • From desktop Windows applications • Use Grids without awareness of them! • But gLite not (yet) fully supporting Windows (more info: http://jessica.trigrid.it/grid2win/) • From portals • For recurring tasks: “core grid services” as well as application layer • Accessible from any browser • Tailored to applications • Different portal solutions, and wide range of capabilities. Second part of this course: P-GRADE Portal Application Gridification 19/26

  20. Characteristics of VOs • What is being shared? • resources of storage and/or compute cycles • software and/or data • Distinct groups of developers and of users? • Some VOs have distinct groups of developers and users… • Biomedical applications used by clinicians,…. • …. Some don’t • Physics application developers who share data but write own analyses • Effect: need to • hide complexity from the 1st type of VOs • expose functionality to 2nd type of Vos • many security issues Application Gridification 20/26

  21. Challenges Research software is often Created for one user: the developer Familiarity makes it useable Short-term goals: Used until papers are written and then discarded Grid applications are often used by a VO Without support from developer In new contexts and workflows • Grid application developers are • In a research environment • Yet their s/w must have: • Stability • Documentation • Usability • Extendability i.e. Production quality Need expertise in: • software engineering • application domain • grid computing Application Gridification 21/26

  22. Consequences • Team work! • Engaged in world-wide initiatives – reuse, don’t make your own! Cross disciplines for solutions. • From research to production software: ~5 times the effort. • “80% of the time for last 10% of the functionality & reliability” • Standardisation is key • For re-use, for dynamic configuration of services,.. • Both for middleware and domain specific • Need to follow a deliberate development process • Waterfall? Rapid prototyping? • Requirements engineering, design, implementation, validation, deployment • Engaged with the user community Application Gridification 22/26

  23. Basic tasks while Porting applications to the Grid • Developing a non-grid application (or inheriting and updating an ancient one); • Go/no-Go decision about gridification • Is it suitable for the Grid environment? • “Cost/profit” analysis / feasibility study Typical Questions Groups: • Current structure of the application • Dependencies of the application • Available resources (manpower, knowledge, etc.) • Requirements for the gridified application • Expected impact of gridification • Requirements for the grid infrastructure More info: Application Description Template http://www.lpds.sztaki.hu/gasuc/?m=4 Application Gridification 23/26

  24. Basic taskswhile Porting applications to the Grid (contd.) • Grid environment access • Requesting Certificates / VO membership • Accessing Grid environment • Appropriate VO UI machine account for command line • Portal GUI account; • Executing, Testing and Debugging the application; • Testing the non-grid application (debugging in Grid environment is a hard task), creating use cases for single (non-grid) runs; • Constructing the job suite – JDL (Job Description Language) files, executables, auxiliary scripts and input/output data files; • Submitting the job to the Grid as small-scale pilot application; Application Gridification 24/26

  25. Basic tasks while Portingapplications to the Grid (contd.) • Executing, Testing and Debugging the pilot application; • IF something goes wrong THEN GOTO4; • IF everything seems to work THEN increase the scale of the application (increase problem size, amount of used resources); • Optimizing the grid application; Application Gridification 25/26

More Related