Atlas Grid Status - part 1

1. Atlas Grid Status - part 1 Jennifer Schopf ANL U.S. ATLAS Physics and Computing Advisory Panel Review Argonne National Laboratory Oct 30, 2001

2. Globus/Atlas Interactions Condor (G) GRAPPA/ Gardner GridView and other Monitoring De, Schopf, Yu GRAM GSI MDS/ GIIS/GRIS Magda/ Wenaus GridFTP Grid Data Access/Malon Replica Cat Replica Mgr PacMan/ Youssef Packaging Jennifer Schopf, ANL

3. Globus ToolkitTM • Core protocols and services • Grid Security Infrastructure • Grid Resource Access & Management • MDS information service & monitoring • GridFTP data access & transfer • Data Grid technologies • Replica catalog, replica management service • Reliable file transfer • Defacto-standard for Grid projects: • GriPhyN, PPDG, NEES, EU DataGrid, ESG, Fusion Collaboratory, DISCOM, NASA IPG, NSF TeraGrid, DOE Science Grid, EU DataGrid, UK Grid Center, U.S. GRIDS Center, Access Grid, GridPort, MPICH-G2, Condor-G, GrADS, and others Jennifer Schopf, ANL

4. Globus Status • New release • In alpha4 now, beta before SC ‘01 (Nov 15), 4Q01 release • New packaging: enables modular binary and source distributions • GRAM 1.5 (job submission): enhanced robustness • MDS-2.1 (information service): security, better performance, etc. • GridFTP- secure large file transfer • Replica Mgt- Data management, catalogs for replicas • Continuing work • Community Authorization Services • Reliable file transfer • Java, other “Commodity Grid” toolkits Jennifer Schopf, ANL

5. ATLAS GriPhyN/iVDGL • GriPhyN funded Fall 2000, $11.9M/5 years • iVDGL funded Fall 2001, 5 years, $13.65M/5 year • Both involve ATLAS, CMS, Ligo and SDSS • ATLAS support: 2.5 FTE IU, 1.5 FTE BU, $331K/5yr IU HW • R. Gardner is the ATLAS lead, J. Schopf is the CS liaison • GriPhyN proposal emphasizes virtual data requirements, collaboration between experiments • iVDGL proposal emphasizes testbed and infrastructure issues across experiment Jennifer Schopf, ANL

6. GriPhyN • Principal ATLAS GriPhyN/iVDGL deliverables: • 2001: Testbed with GriPhyN VDT 1.0, packaged with PacMan • 2002: Serving DC1 data • 2003: Dataset re-creation/Data signature • Additional efforts: • Monitoring: Dantong Yu, BNL, and J. Schopf,ANL, joint with PPDG • GRAPPA: Rob Gardner and Randy Bramley, IU Jennifer Schopf, ANL

7. GriPhyN Testbed Issues • GriPhyN is defining VDT 1.o • SW install for GriPhyN/PPDG, compatible with EDG as well • Globus 2.0 beta when it’s released (10/30) • GSI, GridFTP, MDS, repl, cat stuff, etc (Gram 1.5) • GDMP 2.0 (supports flat files) • Condor 6.3.1 (also Condor-G, Dagman) • Extra tools for ATLAS • objectivity 6.1 • Magda • Still need to resolve CA issue between EDG and US test sites Jennifer Schopf, ANL

8. GriPhyN ATLAS Goal 1Serving DC1 Data (July-Dec 2002) • Limited reconstruction analysis job using grid job submission interface • Serving the data results form DC1 • As part of DC1, data must be tagged with meta data for ease of access • Minimal keywords would be sufficient • Magda already implements portions of this • Job submission with minimal smarts • Extend GRAPPA work • Move compute resources to data sites Jennifer Schopf, ANL

9. GriPhyN ATLAS Goal 2Dataset Re-creation (Jan-Sept 2003) • Goal: be able to re-create a data file • Need to evaluate what parameters need to be kept track of • Need to evaluate data needed for full data signature • Need to develop a metric for evaluating success - what is good enough? Jennifer Schopf, ANL

10. GRAPPA:Grid Access Portal for Physics Applications • Provide a point of access to ATLAS grid resources • IU (Physics, CS), Northwestern (ECE), ANL (CS), BU (Physics), • Provide a simple interface for physicists to submit and monitor jobs on the Grid • Web-based as well as script-based • Ability to “replay” • Compatible with both ATLSim and Athena architecture • Adaptable and/or “extensible” to new developments in Grid software, Athena, etc Jennifer Schopf, ANL

11. GRAPPA Components • User Interface • Job submission • Monitoring • Bookkeeping • Resource selection Jennifer Schopf, ANL

12. Current Status • Simple prototype that allows users to submit an Athena job from a web interface to the Condor pool on the Atlas IU cluster (via Globus) • Next steps • Adding more Athena functionality to the interface (e.g., user's defined libraries) • Experiment with other job launch mechanisms • Condor-G and DAGMAN description language • Web Services Flow Language as a more general workflow description • Explore interfaces for multiple metadata and replica catalog systems. • http://lexus.physics.indiana.edu/~griphyn/grappa/ Jennifer Schopf, ANL

13. Grid-Enabled Data Access in Athena • David Malon, ANL • Integrate Grid Data Access techniques (Globus replica catalog and/or GDMP) into the Athena event selection module • When a file is needed, a check is done to see if the file is local • if not, use Grid Data Access tools to make it local Jennifer Schopf, ANL

14. Grid-Enabled Data Access in Athena • Athena and Globus • Search Globus replica catalog and select • Transfer file using the protocol associated with the location object in the catalog (gsiftp, https/globus-url-copy from a remote gass_server, ...) • Athena and GDMP • Work to date uses GDMP 1.2.2, which had not yet incorporated the Globus replica catalog • Uses the GDMP import/export catalogs • Supports certain subscription-based approaches • Automatically updates Objectivity/DB internal catalog when Objectivity database files are transferred Jennifer Schopf, ANL

15. Status • Paper presented at CHEP • Prototype up and running on ANL systems • GDMP-based prototype work done between CERN and Milan • Next Steps: • GDMP will use Globus data replica work • Use GDMP to copy files • Requirements and design work for interfaces between Athena data producers and metadata catalogs and replica management services (e.g., Magda) • Metadata work – extending the Athena Event Selector properties to allow for data signature/virtual data Jennifer Schopf, ANL

16. Monitoring • Joint working group set up between PPDG and GriPhyN to investigate monitoring issues • Led by J. Schopf and D. Yu • Monitoring currently defined very broadly: • Is this router configured correctly? • Has the application finished using that file yet? • What information do I need to determine where to run my application? Jennifer Schopf, ANL

17. GridView (Kaushik De) • Tool to show the status of the 8 test bed machines on the web • Uses Globus GRAM to query sites every 30 mins • Hostname, Uptime, Idletime, # users, and Load average • Next steps include • Integration with Globus information service (MDS) • Visualization as part of cross PPDG/GriPhyN monitoring work Jennifer Schopf, ANL

18. Jennifer Schopf, ANL

19. Application Level Monitoring (Taylor) • Initial testbed to incorporate GRID monitoring capabilities into Athena • Collaboration with Valerie Taylor, David Quarrie, and others • Very long ramp-up due to difficulty of “outsiders” running an Athena application • Working on developing an Auditor for Athena • http://www.ece.nwu.edu/EXTERNAL/vtaylorlab/prophesy.html Jennifer Schopf, ANL

20. Current Status • Mailing list has been set up • pg-monitoring@mcs.anl.gov • Defining usage cases • Sensors • Predictors • Archiving • Will gather requirements and look at extending Globus MDS as a common framework to meet these requirements Jennifer Schopf, ANL

21. Summary • Globus • GriPhyN/iVDGL • Grappa • Grid-Enabled Data Access • Monitoring and Visualization Jennifer Schopf, ANL