1 / 48

Grid Tutorial

Grid Tutorial. Cameron Kiddle Grid Research Centre University of Calgary. Acknowledgments. Rob Simmonds Mark Fox Phil Rizk Abhishek Gaurav Andrey Mirtchovski Nayden Markatchev. Outline. Grid Computing Overview Basic Grid Services

yosef
Download Presentation

Grid Tutorial

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Grid Tutorial Cameron Kiddle Grid Research Centre University of Calgary

  2. Acknowledgments • Rob Simmonds • Mark Fox • Phil Rizk • Abhishek Gaurav • Andrey Mirtchovski • Nayden Markatchev Grid Tutorial

  3. Outline • Grid Computing Overview • Basic Grid Services • security, data management, resource management, information services • Advanced Grid Services • meta schedulers, workflow managers, replica location service • Future of Grid Computing Grid Tutorial

  4. Grid Computing • Aimed at providing users with a single system view of a set of distributed resources that span administrative domains • Aids in solving large scale problems • Aids in collaborative research efforts Grid Tutorial

  5. Current Grid Projects • CERN LCG (LHC Computing Grid) • GriPhyN (Grid Physics Network) • NEES (Network for Earthquake Engineering Simulation) • BioSimGrid • IVOA (International Virtual Observatory Alliance) Grid Tutorial

  6. WestGrid and Grid • Is WestGrid a computational grid? • Provides grid enabled resources • Security services • Data transfer tools • Job submission services • WestGrid resources can be part of computational grids Grid Tutorial

  7. Global Grid Forum (GGF) • Leads the global standardization effort for grid computing • Open Grid Services Architecture (OGSA) Grid Tutorial

  8. Open Grid Services Architecture • Infrastructure Services • Resource Management Services • Data Services • Context Services • Information Services • Self-Management Services • Security Services • Execution Management Services Grid Tutorial

  9. Grid Middleware • Globus Toolkit • Globus Alliance • Open source • Legion • University of Virginia • Now commercial product developed by Avaki • Focuses on data grid / enterprise integration Grid Tutorial

  10. Globus Toolkit • Collection of grid services to aid in development of grid applications and environments • GT-2 – pre standards (defacto standard) • GT-3 – Open Grid Services Infrastructure (OGSI) • GT-4 – Web Services Resource Framework (WSRF) – beta version Grid Tutorial

  11. Basic Grid Services • Security Services • GSI (Grid Security Infrastructure) • Data Management • GridFTP • Resource Management • GRAM (Grid Resource Allocation and Management) • Information Services • MDS (Monitoring and Discovery Service) Grid Tutorial

  12. Grid Security Infrastructure (GSI) • Based on • Public key cryptography • X.509 certificates • Secure Socket Layer (SSL) • Provides • Mutual authentication • Single sign-on Grid Tutorial

  13. Certificates • Used to identify and authenticate users and services • Includes four primary pieces of information • Subject name • Public key • Identity of Certificate Authority (CA) • Digital signature of the Certificate Authority • Provides global name space • Encoded in X.509 certificate format (IETF) Grid Tutorial

  14. Mutual Authentication • Process of two parties proving to each other who they say they are • Both parties must trust CAs that signed each other’s certificates • Each party sends certificate to other • Make sure certificate is valid • Make sure other party really is the party identified in the certificate Grid Tutorial

  15. Single Sign-On • Sign-on only once to computational grid • Use a proxy to avoid re-entering password • Consists of new certificate and private key • Certificate signed by owner, not CA • Less secure – limited lifetime • Both proxy certificate and owner’s certificate sent for mutual authentication • grid-proxy-init Grid Tutorial

  16. GSI Demo Grid Tutorial

  17. MyProxy • Developed by NCSA (National Center for Supercomputing Applications) • Credential repository • Allows proxy credential to be received from any machine • Can allow trusted servers to renew proxy credentials • myproxy-init, myproxy-get-delegation Grid Tutorial

  18. MyProxy Demo Grid Tutorial

  19. File Transfers • gsiscp • scp version based on GSI • Easy to use, but provides poor performance for large file transfers • GridFTP • Based on FTP (File Transfer Protocol) • GSI security on control and data channels • Modification of TCP buffer sizes • Parallel transfers • Third party transfers Grid Tutorial

  20. File Transfer Performance Grid Tutorial

  21. File Transfer Demo Grid Tutorial

  22. Grid Resource Allocation and Management (GRAM) • Provides a single standard interface for remote job submission and resource management • Requires users and application developers to learn only one method to gain access to a large variety of local management systems • globus-job-run, globus-job-submit Grid Tutorial

  23. Job Submission Demo Grid Tutorial

  24. Monitoring and Discovery Service (MDS) • Provides information about availability and status of resources • Uses LDAP (Lightweight Directory Access Protocol) to access this information • Grid Resource Information Service (GRIS) • Provides resource information • Grid Index Information Service (GIIS) • Provides caching services and search capabilities Grid Tutorial

  25. Advanced Grid Services • Meta-schedulers • Workflows • Replica Location Service (RLS) • Abstract Workflows Grid Tutorial

  26. Meta-Schedulers • Condor-G (Condor) • University of Wisconsin-Madison • Moab Grid Scheduler (Silver) • Cluster Resources Inc. Grid Tutorial

  27. Condor-G • Can submit and manage jobs at multiple sites • Uses Globus mechanisms to communicate with resources at different sites • Can advertise sites that jobs can be submitted to through Class Ads Grid Tutorial

  28. Class Ad • Used to represent characteristics and constraints of resources and jobs • Consists of a list of attributes (name/value pairs) • Used in matchmaking of jobs to resources Grid Tutorial

  29. Condor-G Demo Grid Tutorial

  30. Workflows C1 E1 A D B C2 E2 C3 Grid Tutorial

  31. DAGMan • Directed Acyclic Graph Manager • Node = program • Edge = dependency • Monitors Condor log files to enforce ordering of dependencies Grid Tutorial

  32. DAGMan Input File • Programs in the DAG • Pre/post processing scripts • Dependencies • Number of times to retry • Macro definitions Grid Tutorial

  33. Sample DAGMan Input File A JOB A A.condor JOB B B.condor JOB C C.condor JOB D D.condor PARENT A CHILD B C PARENT B C CHILD D B C D Grid Tutorial

  34. Rescue DAGs • Aids in resubmission of uncompleted portions of workflows • Produced if DAGMan can not make any further progress in a DAG due to node failures • Same as original DAG but nodes that successfully completed before are marked as DONE Grid Tutorial

  35. Life3D Workflow Animation Life3D Simulation Rendering Grid Tutorial

  36. Life3D Workflow - II blackhole (SFU) 2. Life3D Simulation 1. lattice (UofC) Data Storage Grid Research Center WestGrid 3. grc15 7. octarine 4. Rendering 5. 6. Animation Grid Tutorial

  37. Life3D Workflow - III grc15 octarine blackhole blackhole lattice Stage Data Stage Data Animation Life3d Simulation Rendering Stage Data Stage Data Grid Tutorial

  38. DAGMan Demo Grid Tutorial

  39. Life3D Animation Grid Tutorial

  40. Replica Location Service (RLS) • Co-developed by Globus and DataGrid projects • Provides a mechanism for registering and locating replicas • Maintains mappings between logical names and target names • Logical names – unique identifiers for data content that may have one or more physical replicas • Target name – physical location of a replica or other logical names representing the data Grid Tutorial

  41. Chimera Virtual Data System • Part of the GriPhyN project • Distributed with the Virtual Data Toolkit • Keeps track of how data products are derived Abstract Workflow Concrete Workflow Partial Workflow Chimera Pegasus (VDL) (DAG) (DAX) Virtual Data Language DAG XML RLS MDS TC (Transformation Catalog) Grid Tutorial

  42. Chimera • Input - Virtual Data Language (VDL) • Transformation – an executable program • Derivation – an execution of the transformation along with logical input and output file names • Output – Abstract Workflow (DAX) • Workflow activities independent of specific grid resources Grid Tutorial

  43. Pegasus • Input – Abstract Workflow • Output – Concrete Workflow (DAG) • Workflow activities assigned to specific Gird resources • Uses RLS to determine data file locations • Uses Transformation Catalog to map logical transformations to physical locations • Uses MDS to determine available resources Grid Tutorial

  44. Chimera Example transitfile add20 add10 infile (blackhole) outfile (blackhole) (nexus) (lattice) Grid Tutorial

  45. Chimera Demo Grid Tutorial

  46. Security Services (GSI) grid-proxy-init myproxy-init myproxy-get-delegation Data Management (GridFTP) globus-url-copy Resource Management (GRAM) globus-job-run globus-job-submit Information Services MDS Meta-scheduler Condor-G Concrete Workflows DAGMan Replica Location Service (RLS) Abstract Workflows Chimera/Pegasus Summary Basic Grid Services Advanced Grid Services Grid Tutorial

  47. Future of Grid Computing • Integration of Web Services and Grid Services • Web Services Resource Framework (GT-4) • Standards are more established • Involvement of more developers • Better tools • Grid-enabled applications • Common library interface • “run_job_on_grid” • Wider spread use Grid Tutorial

  48. Contact Information Cameron Kiddle kiddlec@cpsc.ucalgary.ca http://grid.ucalgary.ca/ http://pages.cpsc.ucalgary.ca/~kiddlec/ Grid Tutorial

More Related