1 / 35

Open Grid Computing Environments: Advanced Gateway Support Activities

Open Grid Computing Environments: Advanced Gateway Support Activities. RT Project Review October 7 th , 2010. Software Strategy. We develop and package downloadable software Focus on gadget container and tools for running science applications on grids and clouds.

gerik
Download Presentation

Open Grid Computing Environments: Advanced Gateway Support Activities

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Open Grid Computing Environments: Advanced Gateway Support Activities RT Project Review October 7th, 2010

  2. Software Strategy • We develop and package downloadable software • Focus on gadget container and tools for running science applications on grids and clouds. • Provide a tool set that can be used in whole or in part. • If you just want GFac, then you can use it without buying an entire framework. • Outsource security, information services, data and metadata, etc to other providers. • MyProxy, TG IIS, Globus, Condor, XMC Cat, iRods, etc.

  3. NSF SDCI Award: 2010-2013 • Indiana: Marlon Pierce (PI), Suresh Marru (Co-PI), Raminder Singh, Rob Quick, Gregor von Laszewski, Gerald Guo • Purdue: Carol Song (Co-PI), David Braun, Lan Zhao • NCSA/UIUC: Sudhakar Pamidighantam (Co-PI), Shaowen Wang, Yan Liu • UTHSCSA: Emre Brookes (Co-PI), Borries Demeler

  4. Project Highlights • Full Circle Development • Directly fund both software developers and gateway consumers. • Directly supported (non-IU) gateways: • UltraScan (UTHSCSA), GridChem (NCSA), SimpleGrid/GISolve (UIUC), Purdue CCSM and Environmental Data Gateways • Among the most used TG gateways. • Sustainability strategy: Apache Incubator for workflow suite of tools • XBaya, GFac, and supporting services.

  5. UltraScan High Level Overview User Web Server MySQL DB US LIMS GridControl TeraGrid UTHSCSA Jacinto Terascale storage TIGRE/Globus Network High Performance Computing Clusters

  6. UltraScan Middleware • Immediate Goals: Use GFAC as a replacement job submission service. • GRAM 2, 4, 5 independence • Significant effort into GRAM5 testing on Ranger. • Longer term goals • Integrate with TG information services to provide better job scheduling. • OGCE Resource Prediction Service • Support UNICORE job management. Current Architecture

  7. Gram5 Testing Steps • Developed Testing harness to run different UltraScan cases. • Started with small number of jobs and increased the concurrency later • Watched job behavior of the job on resource and monitored the gram log • There were lot of issue which we found from the logs and working with Globus team to fix them • Recorded all the job run data to create a google gadget to create graph for different runs on different resources.

  8. TG Resources and Patterns Patterns:

  9. Learning from this exercise on Ranger • There were problems with Sun Grid Engine job manager (sge.pm) not loading system environment and user environment settings. • You can't load modules using Gram parameters. • On Ranger all the libraries are modules and user may want to load some special module version for his job. • Example is if you have different version of your code using different modules.  • On Ranger user default shell is C shell and Gram runs using bash shell. This is not a problem but users need to be informed where to add user environment. • You can look at detailed results on following http://teragridforum.org/mediawiki/index.php?title=GRAM_5_Testing

  10. GridChem Science Gateway • A chemistry/material Science Gateway for running computational chemistry codes, workflows, and parameter sweeps. • Integrates molecular science applications and tools for community use. • 400+ users heavily using TeraGrid. One of the consistent top5 TeraGrid Gateway users. • Supports all popular Chemistry applications including Gaussian, GAMESS, NWChem, QMCPack, Amber and MolPro, CHARMM • ParamChem is a follow-on project to develop workflows for chemical parameter studies and provide the infrastructure to execute them.

  11. Cyberenvironments for ParameterizationComputational Reference Data Generation

  12. GridChem , ParamChem Integration • Immediate Goals: integrate XBaya workflow composer with GridChem user interface and middleware. • Target SC10 • Longer term goals: provide part of GridChem next generation middleware; support large scale ParamChem workflows.

  13. SimpleGrid (UIUC) SimpleGrid is a Gateway teaching tool that is also the core of GISolve

  14. SimpleGrid Gadgets Yan and Gerald’s work from our Oct 6th hacking session.

  15. SimpleGrid, GISolve • Short term goal: develop SimpleGrid Gadgets deployable into gadget container. • Must meet security requirements • Support PHP development • Support interactivity requirements • Integrate YUI JavaScript libraries with Gadget JavaScript. • Longer term goals: investigate workflow, job management tools. Apply to GISolve

  16. Purdue CCSM and Data Portals • Short terms goals: Develop CCSM and data management gadgets and necessary backing middleware. • Support GWT development • Interactivity and security requirements. • Significant requirements overlap with SimpleGrid • Longer term goals: Build gateways out of gadgets hosted by multiple containers; examine workflow and other tools.

  17. Conclusion • We focus initially on one component per gateway. • SimpleGrid, CCSM, Data Portal: gadgets • Other gadget based gateways at UC • GridChem: Xbaya • UltraScan: GFac • Goal is to establish an Apache-style meritocracy for contributed code. • Making distributed teams work: hacking retreats, SourceForge, Jira, Google apps

  18. Acknowledgments • The following student interns have contributed to the project over the last year • Ye Fan • Patanachai Tangchaisin • Suresh Kumar Deivasigamani • Roland Mai • Sashikiran Challa

  19. Backup Slides

  20. OGCE Gateway Tool Adaption & Reuse LEAD LEAD Experiment Builder, XRegistry Interface GFac, XBaya, XRegistry, FTR Eventing System GridChem Xbaya, GC Middleware GridChem Ultrascan OGCE Re-engineer, Generalize, Build, Test and Release GFac, Eventing System Resource Discovery Service BioVLab OVP/ RST/ MIG TeraGrid User Portal XBaya, GFac ODI GPIR, File Browser Workflow Suite, Gadget Container OGCE Team Bio Drug Screen Swarm->GFac Gadget Container, GTLab, Javascript Cog, XRegistry Interface, Experiment Builder, Axis2 Gfac, Axis2 Eventing System, Resource Prediction Service, Swarm EST Pipeline Swarm->GFac Future Grid GFac, Xbaya, … 21 21

  21. Putting It All Together

  22. Software Strategy • Focus on gadget container and tools for running science applications on grids and clouds. • Provide a tool set that can be used in whole or in part. • If you just want GFac, then you can use it without buying an entire framework. • Outsource security, information services, data and metadata, etc to other providers. • MyProxy, TG IIS, Globus, Condor, XMC Cat, iRods, etc.

  23. Advanced Support Scenarios • GridChem/ParamChem workflow support • UltraScan Job Submission (GFAC) • EST Pipeline • Bioinformatics pipeline for managing mass job submission.

  24. More Information • This is downloadable, packaged software. • Apache Maven build system provides everything you need to to build the gadget container, gadgets, workflow composer, and backing services. • Get code by anonymous SVN checkout. • Email: mpierce@cs.indiana.edu, smarru@cs.indiana.edu, ogce-discuss@googlegroups.com • OGCE Web Site: www.collab-ogce.org • Blog/News Feed: http://collab-ogce.blogspot.com/

  25. Acknowledgements and People • Funding by TeraGrid GIG, RP and by OCI SDCI • IU: Marlon Pierce, Suresh Marru, Raminder Singh, Archit Kulshrestha, Zhenhua Guo • TACC: Maytal Dahan, Rion Dooley • SDSC: Nancy Wilkins-Diehr, Jeff Sale • SDSU: Mary Thomas

  26. GridChem Advanced Support • IU Personal – Suresh Marru, Marlon Pierce, Raminder Singh, Ye Fan • Gateways Area Director – Nancy Wilkins-Diehr • GridChem Team – Sudhakar Pamidigantam, Rion Dooley, Vikram Gazula & Rest of GridChem team.

  27. Empirical ForceFields Parameterization Need Process Lack of Accurate Force Fields Produce Erroneous Property Estimation Vanommeslaeghe et al. J. Comp.Chem 2010, 31, 671-690

  28. Cyberenvironments for ParameterizationComputational Reference Data Generation

  29. Workflow Manager • Parameter definitions • Optimization Monitor • Model/Reference Data Definition • Optimization • Incomplete? • Optimization Job Completed? • Merit Function Specification • Paramater testing Model • Expert • Interface • Optimization Methods Choice • Successful Testing • Paramater Sensitivity Analysis • Consistency Checker • Update Parameter Database with new set • Optmization Job Launcher • Notification of End of Workflow Molecular Force Field CyberenvironmentsParameter Initialization and optimization Workflow

  30. OGCE Gateway Tool Adaption & Reuse LEAD LEAD Experiment Builder, XRegistry Interface GFac, XBaya, XRegistry, FTR Eventing System GridChem XBaya GridChem Ultrascan OGCE Re-engineer, Generalize, Build, Test and Release Gfac, Eventing System Resource Discovery Service BioVLab OVP/ RST/ MIG TeraGrid User Portal XBaya, GFac ODI GPIR, File Browser Workflow Suite OGCE Team Bio Drug Screen Workflow Suite Gadget Container, GTLab, Javascript Cog, XRegistry Interface, Experiment Builder, Axis2 Gfac, Axis2 Eventing System, Resource Prediction Service EST Pipeline Workflow Suite Future Grid ??? 31 31

  31. Cyberinfrastructure Layers User Interfaces Gateway Abstraction Interfaces Web Enabled Desktop Applications Web/Gadget Container Web/Gadget Interfaces Information Services Application Abstractions Monitoring User Management Fault Tolerance Gateway Software Provenance & Metadata Management Registry Workflow System Security Auditing & Reporting Resource Middleware Cloud Interfaces SSH & Resource Managers Grid Middleware Compute Resources Local Resources Computational Grids Computational Clouds Color Coding OGCE Gateway Components Complimentary Gateway Components Dependent resource provider components

  32. LEAD/OGCE Layered Workflow Architecture Workflow Interfaces (Design & Definition) XBaya GUI (Composition, Deploying, Steering & Monitoring) Flex/Web Composition Gadget Interface for Input Binding BPEL 2.0 Python Workflow Specification Scufl BPEL 1.0 Java Code Pegasus DAG Workflow Execution & Control Engines Apache ODE Dynamic Enactor Condor DAGMan Jython Interpreter GBPEL Taverna

  33. GFac Current & Future Features Apache Axis2 Globus Registry Interface Scheduling Interface Input Handlers Monitoring Interface Campus Resources Output Handlers Fault Tolerance Data Management Abstraction Amazon Eucalyptus Job Management Abstraction Auditing Checkpoint Support Unicore Condor Color Coding Existing Features Planned/Requested Features

  34. Gateway Computing Environments (GCE10)

More Related