1 / 21

Atlas Grid Status - part 1

Atlas Grid Status - part 1. Jennifer Schopf ANL U.S. ATLAS Physics and Computing Advisory Panel Review Argonne National Laboratory Oct 30, 2001. Globus/Atlas Interactions. Condor (G). GRAPPA/ Gardner. GridView and other Monitoring De, Schopf, Yu. GRAM. GSI. MDS/ GIIS/GRIS. Magda/

ivan-santana
Download Presentation

Atlas Grid Status - part 1

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Atlas Grid Status - part 1 Jennifer Schopf ANL U.S. ATLAS Physics and Computing Advisory Panel Review Argonne National Laboratory Oct 30, 2001

  2. Globus/Atlas Interactions Condor (G) GRAPPA/ Gardner GridView and other Monitoring De, Schopf, Yu GRAM GSI MDS/ GIIS/GRIS Magda/ Wenaus GridFTP Grid Data Access/Malon Replica Cat Replica Mgr PacMan/ Youssef Packaging Jennifer Schopf, ANL

  3. Globus ToolkitTM • Core protocols and services • Grid Security Infrastructure • Grid Resource Access & Management • MDS information service & monitoring • GridFTP data access & transfer • Data Grid technologies • Replica catalog, replica management service • Reliable file transfer • Defacto-standard for Grid projects: • GriPhyN, PPDG, NEES, EU DataGrid, ESG, Fusion Collaboratory, DISCOM, NASA IPG, NSF TeraGrid, DOE Science Grid, EU DataGrid, UK Grid Center, U.S. GRIDS Center, Access Grid, GridPort, MPICH-G2, Condor-G, GrADS, and others Jennifer Schopf, ANL

  4. Globus Status • New release • In alpha4 now, beta before SC ‘01 (Nov 15), 4Q01 release • New packaging: enables modular binary and source distributions • GRAM 1.5 (job submission): enhanced robustness • MDS-2.1 (information service): security, better performance, etc. • GridFTP- secure large file transfer • Replica Mgt- Data management, catalogs for replicas • Continuing work • Community Authorization Services • Reliable file transfer • Java, other “Commodity Grid” toolkits Jennifer Schopf, ANL

  5. ATLAS GriPhyN/iVDGL • GriPhyN funded Fall 2000, $11.9M/5 years • iVDGL funded Fall 2001, 5 years, $13.65M/5 year • Both involve ATLAS, CMS, Ligo and SDSS • ATLAS support: 2.5 FTE IU, 1.5 FTE BU, $331K/5yr IU HW • R. Gardner is the ATLAS lead, J. Schopf is the CS liaison • GriPhyN proposal emphasizes virtual data requirements, collaboration between experiments • iVDGL proposal emphasizes testbed and infrastructure issues across experiment Jennifer Schopf, ANL

  6. GriPhyN • Principal ATLAS GriPhyN/iVDGL deliverables: • 2001: Testbed with GriPhyN VDT 1.0, packaged with PacMan • 2002: Serving DC1 data • 2003: Dataset re-creation/Data signature • Additional efforts: • Monitoring: Dantong Yu, BNL, and J. Schopf,ANL, joint with PPDG • GRAPPA: Rob Gardner and Randy Bramley, IU Jennifer Schopf, ANL

  7. GriPhyN Testbed Issues • GriPhyN is defining VDT 1.o • SW install for GriPhyN/PPDG, compatible with EDG as well • Globus 2.0 beta when it’s released (10/30) • GSI, GridFTP, MDS, repl, cat stuff, etc (Gram 1.5) • GDMP 2.0 (supports flat files) • Condor 6.3.1 (also Condor-G, Dagman) • Extra tools for ATLAS • objectivity 6.1 • Magda • Still need to resolve CA issue between EDG and US test sites Jennifer Schopf, ANL

  8. GriPhyN ATLAS Goal 1Serving DC1 Data (July-Dec 2002) • Limited reconstruction analysis job using grid job submission interface • Serving the data results form DC1 • As part of DC1, data must be tagged with meta data for ease of access • Minimal keywords would be sufficient • Magda already implements portions of this • Job submission with minimal smarts • Extend GRAPPA work • Move compute resources to data sites Jennifer Schopf, ANL

  9. GriPhyN ATLAS Goal 2Dataset Re-creation (Jan-Sept 2003) • Goal: be able to re-create a data file • Need to evaluate what parameters need to be kept track of • Need to evaluate data needed for full data signature • Need to develop a metric for evaluating success - what is good enough? Jennifer Schopf, ANL

  10. GRAPPA:Grid Access Portal for Physics Applications • Provide a point of access to ATLAS grid resources • IU (Physics, CS), Northwestern (ECE), ANL (CS), BU (Physics), • Provide a simple interface for physicists to submit and monitor jobs on the Grid • Web-based as well as script-based • Ability to “replay” • Compatible with both ATLSim and Athena architecture • Adaptable and/or “extensible” to new developments in Grid software, Athena, etc Jennifer Schopf, ANL

  11. GRAPPA Components • User Interface • Job submission • Monitoring • Bookkeeping • Resource selection Jennifer Schopf, ANL

  12. Current Status • Simple prototype that allows users to submit an Athena job from a web interface to the Condor pool on the Atlas IU cluster (via Globus) • Next steps • Adding more Athena functionality to the interface (e.g., user's defined libraries) • Experiment with other job launch mechanisms • Condor-G and DAGMAN description language • Web Services Flow Language as a more general workflow description • Explore interfaces for multiple metadata and replica catalog systems. • http://lexus.physics.indiana.edu/~griphyn/grappa/ Jennifer Schopf, ANL

  13. Grid-Enabled Data Access in Athena • David Malon, ANL • Integrate Grid Data Access techniques (Globus replica catalog and/or GDMP) into the Athena event selection module • When a file is needed, a check is done to see if the file is local • if not, use Grid Data Access tools to make it local Jennifer Schopf, ANL

  14. Grid-Enabled Data Access in Athena • Athena and Globus • Search Globus replica catalog and select • Transfer file using the protocol associated with the location object in the catalog (gsiftp, https/globus-url-copy from a remote gass_server, ...) • Athena and GDMP • Work to date uses GDMP 1.2.2, which had not yet incorporated the Globus replica catalog • Uses the GDMP import/export catalogs • Supports certain subscription-based approaches • Automatically updates Objectivity/DB internal catalog when Objectivity database files are transferred Jennifer Schopf, ANL

  15. Status • Paper presented at CHEP • Prototype up and running on ANL systems • GDMP-based prototype work done between CERN and Milan • Next Steps: • GDMP will use Globus data replica work • Use GDMP to copy files • Requirements and design work for interfaces between Athena data producers and metadata catalogs and replica management services (e.g., Magda) • Metadata work – extending the Athena Event Selector properties to allow for data signature/virtual data Jennifer Schopf, ANL

  16. Monitoring • Joint working group set up between PPDG and GriPhyN to investigate monitoring issues • Led by J. Schopf and D. Yu • Monitoring currently defined very broadly: • Is this router configured correctly? • Has the application finished using that file yet? • What information do I need to determine where to run my application? Jennifer Schopf, ANL

  17. GridView (Kaushik De) • Tool to show the status of the 8 test bed machines on the web • Uses Globus GRAM to query sites every 30 mins • Hostname, Uptime, Idletime, # users, and Load average • Next steps include • Integration with Globus information service (MDS) • Visualization as part of cross PPDG/GriPhyN monitoring work Jennifer Schopf, ANL

  18. Jennifer Schopf, ANL

  19. Application Level Monitoring (Taylor) • Initial testbed to incorporate GRID monitoring capabilities into Athena • Collaboration with Valerie Taylor, David Quarrie, and others • Very long ramp-up due to difficulty of “outsiders” running an Athena application • Working on developing an Auditor for Athena • http://www.ece.nwu.edu/EXTERNAL/vtaylorlab/prophesy.html Jennifer Schopf, ANL

  20. Current Status • Mailing list has been set up • pg-monitoring@mcs.anl.gov • Defining usage cases • Sensors • Predictors • Archiving • Will gather requirements and look at extending Globus MDS as a common framework to meet these requirements Jennifer Schopf, ANL

  21. Summary • Globus • GriPhyN/iVDGL • Grappa • Grid-Enabled Data Access • Monitoring and Visualization Jennifer Schopf, ANL

More Related