380 likes | 497 Views
GridLab Applications. Gabrielle Allen, Zhou Lei, Archit Kulshrestha Center for Computation & Technology, LSU. Modeling Black Holes. Einstein’s equations of General Relativity: G ab = 0 One of the most complex set of equations in physics Mixed hyperbolic-elliptic initial value problem
E N D
GridLab Applications Gabrielle Allen, Zhou Lei, Archit Kulshrestha Center for Computation & Technology, LSU
Modeling Black Holes • Einstein’s equations of General Relativity: Gab = 0 • One of the most complex set of equations in physics • Mixed hyperbolic-elliptic initial value problem • 3 Dimensional, multiple scales • 12 or more equations, with thousands of terms • Require diverse distributed collaborations & many large scale simulations: • Parallelization, optimization • Parallel IO, data management, viz • Checkpointing, steering, interaction Cactuscode.org t=100 t=0
Application Scenarios • Intelligent Automatic Parameter Surveys • Dynamic Migration: faster/cheaper/bigger machine • Multiple Universe: create job clone to investigate steered parameter • Automatic Component Discovery and Loading: Needs of job change • Automatic Convergence Testing • Look Ahead: Spawn off and run coarser resolution to predict likely future • Spawn Independent/Asynchronous Tasks • send to cheaper machine, main simulation carries on • Routine Profiling • best machine/queue, choose resolution parameters based on queue • Dynamic Load Balancing and Optimization: inhomogeneous loads, multiple grids, network and cache hierarchies.
Application Scenarios • Automatically record details about simulations and results. • Store locations of output data files, along with searchable metadata. • Ontologies for describing application area: use for meta data, connection to journal publications etc. • All done in a community accessible way
SCOOP: SURA Coastal Ocean Observing Program (twiki.sura.org) NOAA/ONR Data coming in from sensors all over Gulf: Realtime Operational Grid Feeds in to models on LSU SuperMike and other Grid sites Indication of strengthening storm changing direction: Must Act! Multiple simulations, expertise, and data connected together: FL, LA, notification, steering sensors Early warning, preparation: Insurance industry, petroleum industry, local economy all save millions, lives saved Hurricanes on the Grid LSU, GoMOOS, Texas A&M, UaH, UNC/MCNC, U. Florida, U. Miami, VIMS
Ubiquitous Computing and Monitoring System (UCoMS) • $2.4M DOE/BOR project • ULL, LSU, Southern, Landmark • Petroleum Engineering, CS, ULL • Deploy sensor networks across Gulf • Data collected to provide input to simulations, tasks farmed out • Results collected (transmitted back) • http://www.ucoms.org
Application Tools for Grids • Application developers need the right kinds of tools and environments to use Grids • APIs, toolkits, debuggers, … • Focused on the application requirements (MPI, Fortran, legacy, ….) • Hide the complexities of the Grid • Work in a familiar way (avoid new paradigms) • Functioning and reliable heterogeneous testbed environments for developing and testing • Support new and dynamic application scenarios, as generic as possible.
GridLab Project • EU Funded (5th Framework): Jan 2002 • Partners: PSNC, AEI, ZIB, MASARYK, VU, SZTAKI, ISUFI, Cardiff, NTUA, Chicago, ISI, Wisconsin, Sun, Compaq • GridSphere Portal • Applications • Numerical Relativity (Cactus) • Gravitational Waves (Triana) • Mobile Users • Europe-wide Test Bed (Linux, SGI, Hitachi, …) • Grid Application Toolkit (GAT) • Range of Grid Services • http://www.gridlab.org
Grid Application Toolkit (GAT) • Application oriented access to Grid through a standard API • GATFile_Move • GATResource_Find • GATFile_Logical • Independent of Grid infrastructure and available services. • C, C++, Java, Python, [Perl/Fortran] • All platforms: Unix, Windows • API driver for GGF SAGA-RG
GAT Motivation • Use of Grids by applications is lagging behind the development of Grid technologies, big jump from prototypes and demonstrations to real production use of Grids needed by applications. • Problems: • Missing or immature grid services. • Changing environment, new software versions. • Different and evolving interfaces to the “grid”. • “Functioning” Grid is needed to develop applications. • Interfaces are not simple for scientific application developers.
Example: Remote File Copying Application “Copy my file from there to there ..” SOAP WSDL Corba OGSA Other Monitoring Profiling Information Logging Security Notification Resource Management Application Manager Migration Data Management GLOBUS Other Grid Infrastructure?
Copy a File: Globus GASS int RemoteFile::GetFile (char const* source, if (source_url.scheme_type == GLOBUS_URL_SCHEME_GSIFTP || char const* target) { source_url.scheme_type == GLOBUS_URL_SCHEME_FTP ) { globus_url_t source_url; globus_ftp_client_operationattr_init (&source_ftp_attr); globus_io_handle_t dest_io_handle; globus_gass_copy_attr_set_ftp (&source_gass_copy_attr, globus_ftp_client_operationattr_t source_ftp_attr; &source_ftp_attr); globus_result_t result; } globus_gass_transfer_requestattr_t source_gass_attr; else { globus_gass_copy_attr_t source_gass_copy_attr; globus_gass_transfer_requestattr_init (&source_gass_attr, globus_gass_copy_handle_t gass_copy_handle; source_url.scheme); globus_gass_copy_handleattr_t gass_copy_handleattr; globus_gass_copy_attr_set_gass(&source_gass_copy_attr, globus_ftp_client_handleattr_t ftp_handleattr; &source_gass_attr); globus_io_attr_t io_attr; } int output_file = -1; output_file = globus_libc_open ((char*) target, if ( globus_url_parse (source_URL, &source_url) != GLOBUS_SUCCESS ) { O_WRONLY | O_TRUNC | O_CREAT, printf ("can not parse source_URL \"%s\"\n", source_URL); S_IRUSR | S_IWUSR | S_IRGRP | return (-1); S_IWGRP); } if ( output_file == -1 ) { printf ("could not open the file \"%s\"\n", target); if ( source_url.scheme_type != GLOBUS_URL_SCHEME_GSIFTP && return (-1); source_url.scheme_type != GLOBUS_URL_SCHEME_FTP && } source_url.scheme_type != GLOBUS_URL_SCHEME_HTTP && /* convert stdout to be a globus_io_handle */ source_url.scheme_type != GLOBUS_URL_SCHEME_HTTPS ) { if ( globus_io_file_posix_convert (output_file, 0, printf ("can not copy from %s - wrong prot\n", source_URL); &dest_io_handle) return (-1); != GLOBUS_SUCCESS) { } printf ("Error converting the file handle\n"); globus_gass_copy_handleattr_init (&gass_copy_handleattr); return (-1); globus_gass_copy_attr_init (&source_gass_copy_attr); } globus_ftp_client_handleattr_init (&ftp_handleattr); result = globus_gass_copy_register_url_to_handle ( globus_io_fileattr_init (&io_attr); &gass_copy_handle, (char*)source_URL, &source_gass_copy_attr, &dest_io_handle, globus_gass_copy_attr_set_io (&source_gass_copy_attr, &io_attr); my_callback, NULL); &io_attr); if ( result != GLOBUS_SUCCESS ) { globus_gass_copy_handleattr_set_ftp_attr printf ("error: %s\n", globus_object_printable_to_string (&gass_copy_handleattr, (globus_error_get (result))); &ftp_handleattr); return (-1); globus_gass_copy_handle_init (&gass_copy_handle, } &gass_copy_handleattr); globus_url_destroy (&source_url); return (0); }
API layer between applications and the grid infrastructure: Higher level than existing grid APIs, hide complexity, abstract grid functionality through application oriented APIs Insulate against Rapid evolution of grid infrastructure State of Grid deployment Choose between different grid infrastructures Application developers use and develop for the grid independent of the state of deployment of the grid infrastructure Service developers can make their software available to many different applications. GAT Solution
Copy a File: GAT/C++ #include <GAT++.hpp> GAT::Result RemoteFile::GetFile (GAT::Context context, std::string source_url, std::string target_url) { try { GAT::File file (context, source_url); file.Copy (target_url); } catch (GAT::Exception const &e) { std::cerr << "Some error: " << e.what() << std::endl; return e.Result(); } return GAT_SUCCESS; }
Example: Remote File Copy Application GAT:File.Copy gridftp cp scp gsiscp RFT wget sftp newest Project Specific GridLab GridJIT
Grid Application Toolkit • Standard API and Toolkit for developing portable Grid applications independently of the underlying Grid infrastructure and available services • Implements the GAT-API • Used by applications • GAT Adaptors • Connect to capabilities/services • Implements well defined CPI (mirrors GAT-API) • Interchangeable adaptors can be loaded/switched at runtime • GAT Engine • Provides the function bindings for the GAT-API
GAT API Scope • Files • Resources • Events • Information exchange • Utility classes (error handling, security, preferences)
GAT Adaptors • Default Adaptors: • Basic functionality which will work on a single isolated machine (e.g. cp, fork/exec) • GridLab Adaptors: • Connect to services developed through the GridLab project (GRMS, Mercury, Delphoi, iGrid) • Globus Adaptors: • Core Globus functionality: GRAM, MDS, GT-RLS, GridFTP • Others: DRMAA, Condor, SGE, gsissh, …
GGF SAGA-RG • Simple API for Grid Applications • Global Grid Forum Research Group • Motivated by GAT-API and other efforts to develop a real common standard • DRMAA, Java-COG, GAT-API, … • Meeting at LSU Dec04 will provide an initial draft • A standard API, implemented and supported by vendors, will greatly advance the deployment and adaoption of Grid applications
Simple Examples Many short examples and “one liners” included in GAT distribution: • Read from a remote file • Read from a logical file • Move a file • Spawn a subtask to a remote machine • Information about remote/logical file (e.g. file size, permissions)
Example: Spawn a Subtask GAT::Table sdt; sdt.add ("location", "/bin/date"); GAT::Table hdt; hdt.add ("machine.type", "i686"); GAT::SoftwareDescriptionsd (sdt); GAT::HardwareResourceDescriptionhrd (hdt); GAT::JobDescription jd (context, sd, hrd); GAT::ResourceBroker rb (context, prefs); GAT::Job j = rb.submit (jd);
Task Farming on the Grid Task farming infrastructure implemented in Cactus TFM GAT used for starting remote TFMs TFM TFM Designed for the Grid TFM TFM Tasks can be Anything (MPI, Single proc)
Black Holes on the Grid • Task farm small Cactus black hole simulations across GGTC Grid • Parameter survey: black hole corotation parameter • Results steer a large production black hole simulation
Migration of Legacy Apps • Using e.g. wrapper scripts the GAT can be used to help Grid-enable “legacy” applications. Use a GAT wrapper to start Legacy App Send a checkpoint request to the GAT Wrapper Legacy Application writes checkpoint file GAT Wrapper Exits – Kills Legacy App GAT Wrapper registers checkpoint file to logical file Legacy Application is started within GAT Wrapper GRMS starts the GAT Wrapper on target host GRMS Replicates logical File on target Host
Notification and Information Replica Catalog “The Grid” GridSphere Portal SMS Server Mail Server
GridSphere Portal • Portlet based: JSR 168 • Collaboration focal point • Interact, share data • Start, monitor remote jobs • Move/browse files • Track and monitor announced jobs • Application specific portlets • Cactus, UCoMS, SCOOP
Portlets • JSR-168 lays the foundation for a new open-standard for Web portal development frameworks. • Multiple portlets can be composed in a portal page. • Portlets define how to construct and deliver Web content as modular components. • Portlets can be “maximized” or “minimized”. • Portlets support various modes (View, Edit, Help, Configure) • Users can subscribe to portlets.
GridSphere Feature List • Portlets compatible with IBM's WebSphere 4.2. • Support for easy integration of "third-party portlets" • Higher-level model for building complex portlets. • Customized layouts. • Built-in support for Role Based Access Control (RBAC) separating users into guests, users, admins and super users. • Sophisticated portlet service model. • Documentation (DocBook) HTML/PDF guides and tutorials. • GridSphere core portlets offer base functionality including login, logout, user and access control management. • Grid Portlets add general Grid Capabilities.
Grid Portlets • Resource Registry Portlet (Grid/Administration Tab) • Administrators can customize for particular VO • Resource Browser Portlet (Grid/Resources Tab) • Resource information from MDS/iGrid etc • Credential Retrieval Portlet (Grid/Credentials Tab) • MyProxy • Job Submission Portlet (Grid/Jobs Tab) • GRAM, GRMS • File Browser Portlet (Grid/Files Tab) • GridFTP • …and many more coming… • Implements generic Grid Portlet API (similar to GAT)
Links • GridLab: • http://www.gridlab.org • Grid Application Toolkit • http://www.gridlab.org/GAT • GridSphere: • http://www.gridsphere.org
Mardi Gras Conference Frontiers of Grid Applications and Technologies Louisiana State University Baton Rouge February 3rd to 5th, 2005 http://www.mardigrasconference.org