Summary of Category 3 HENP Computing Systems and Infrastructure

1. Summary of Category 3HENP Computing Systems and Infrastructure Ian Fisk and Michael Ernst CHEP 2003 March 28, 2003

2. Introduction • We tried to break the week into themes • We Discussed Fabrics and Architectures on Monday • Heard general talks about building and securing large multi-purpose facilities • As well as updates from a number of HEPN computing efforts • We discussed emerging hardware and software technology on Tuesday • Review of the most recent pasta report and update of commodity disk storage work • Software for flexible clusters: MOSIX. Advanced storage and data serving: CASTOR, ENSTOR, dCache, Data Farm and ROOT-IO • We discussed Grid and other services on Thursday • Grid Interfaces and Storage Management over the grid • Monitoring services • It was a full week with a lot to discuss. Special thanks to all those who presented. • There is no way to cover very much of what was presented in a thirty minute talk. CAS 2002-10-24 Ian M. Fisk, UCSD 2

3. General Observations • Grid functionality is coming quickly • Basic underlying concepts of distributed, parasitic, and multi-purpose computing are already being deployed in running experiments • Early implementation of interfaces for grid services to fabrics • I would expect by the time the LHC experiments have real data that the tools and techniques will have been well broken-in by experiments running today • Shift to commodity equipment accelerated since the last CHEP • I would argue that the shift is nearly complete • At least two large computing centers admitted to having nothing in their work rooms but Linux systems and a few Suns to debug software • This has resulted in the development of tools to help handle this complicated component environment • With notable exceptions high energy computing does not work well together • The individual experiments often have subtly different requirements, which results in completely independent development efforts CAS 2002-10-24 Ian M. Fisk, UCSD 3

4. Distributed Computing • Example from CDF: Central Analysis Facility is very well used • Future (very near future) plan is • to deploy satellite analysis farms • to increase the computing • resources. CAS 2002-10-24 Ian M. Fisk, UCSD 4

5. Distributed Computing • Peter Elmer presented how the Babar experiment has been able to take advantage of distributed computing resources for primary event reconstruction • By splitting their prompt calibration and event reconstruction, they now take advantage of 5 reconstruction farms at SLAC and 4 in Padova CAS 2002-10-24 Ian M. Fisk, UCSD 5

6. Parasitic Computing • Bill Lee presented the CLuED0 work of the D0 experiment • CLuED0 is a cluster of D0 desktop machines which along with some custom management software provides D0 with 50% of their analysis CPU cycles parasitically. • Heterogeneous system with distributed support • The US LHC experiments submitted a proposal on Monday which, among many other topics, discussed the use of economic theories to optimize resource allocations. • Techniques already used in D0 CAS 2002-10-24 Ian M. Fisk, UCSD 6

7. Multipurpose Computing • Fundamental to a grid connected facility is the ability to support multiple experiments at a minimum and ideally multiple disciplines • The people responsible for computing systems have been thinking about how to make this possible, because so many regional computing centers have to support multiple experiments and user communities. • John Gordon gave an interesting talk on whether it was possible to build a multipurpose center • John identified 6 categories of problems and discussed possible solutions • Software levels • ‘experts’ • Local rules • Security • Firewalls • The accelerator centres CAS 2002-10-24 Ian M. Fisk, UCSD 7

8. Early Interfacing of Grid Services to Fabrics • Alex Sim gave a talk on the Storage Resource Manager: SRM Functionality • Manage space • Negotiate and assign space to users, Manage “lifetime” of spaces • Manage files on behalf of a user • Pin files in storage till they are released, Manage “lifetime” of files • Manage action when pins expire (depends on file types) • Manage file sharing • Policies on what should reside on a storage resource at any one time • Policies on what to evict when space is needed • Get files from remote locations when necessary • Purpose: to simplify client’s task • Manage multi-file requests • A brokering function: queue file requests, pre-stage when possible • Provide grid access to/from mass storage systems • HPSS (LBNL, ORNL, BNL), Enstore (Fermi), JasMINE (Jlab), Castor (CERN), MSS (NCAR), … CAS 2002-10-24 Ian M. Fisk, UCSD 8

9. Early Implementation • The functionality of SRM is impressive, leads to interesting analysis scenarios • Equally interesting is the number of places that are prepared to interface their storage to the WAN using SRM • Robust file replication between BNL and LBNL CAS 2002-10-24 Ian M. Fisk, UCSD 9

10. Shift to commodity equipment CAS 2002-10-24 Ian M. Fisk, UCSD 10

11. Benefits and Complications • The benefit is very substantial computing resources at a reasonable hardware cost. • The complication is the scale and complexity of the commodity computing cluster • A reasonably big computing cluster today might be 1000 systems • With all the possible hardware problems associated with 1000 systems bought from the lowest bidder • Considerable amount of deployment, integration, and development effort to create tools that allow a shelf or rack of linux boxes to behave like a computing resource. • Configuration Tools • Monitoring Tools • Tools for systems control • Scheduling Tools • Security Techniques CAS 2002-10-24 Ian M. Fisk, UCSD 11

12. Configuration Tools • We heard an interesting talk from Thorsten Kleinwork on install and running systems at CERN • Systems are installed with kickstart and RPMs • CERN and several other centers are deploying the configuration tools from EDG WP4 • Pan & CDB (Configuration Data Base) for describing hosts: • Pan is a very flexible language for describing host configuration information: • Expressed in templates (ASCII) • Allows includes (inheritance) • Pan is compiled into XML, inside CDB • XML is downloaded and the information provided by CCConfig, which is the high level API • Complicated even to track what it is you have. • We had an interesting presentation from Jens Kreutzkamp from DESY about how they track their IT assets. CAS 2002-10-24 Ian M. Fisk, UCSD 12

13. System Status Page Monitoring Tools • Systems are complicated consisting of many components this has lead to the development of lots of monitoring tools • Very functional, complete and scalable though complicated to extend tools like NGOP, which Tanya Levshina presented CAS 2002-10-24 Ian M. Fisk, UCSD 13

14. Monitoring Tools (cont.) • On the opposite end where examples of extremely lightweight monitoring packages for Babar presented by Matthias Wittgen. • Monitors CPU and network usage as well packets sent to disk and number of processes • Writes it to a central server where it is kept on a flat file. CAS 2002-10-24 Ian M. Fisk, UCSD 14

15. Tools for system control • Andras Horvath presented a technique for secure system control and reset access for a reasonable cost • This solutions doesn’t scale to 6000 boxes • System Andras is implementing consists of serial connections for console access and relays attached to the reset switch on the motherboard for resets CAS 2002-10-24 Ian M. Fisk, UCSD 15

16. Security Techniques • Number of systems in these large commodity clusters makes for interesting security work • Doubly so when worrying about making grid interfaces • The work to secure the BNL facility was presented • Work prioritizing their assets and forming responses for security breaches CAS 2002-10-24 Ian M. Fisk, UCSD 16

17. Field doesn’t cooperate well • This is not necessarily a problem, nor is it a criticism, simply an observation • One doesn’t see a lot of common detector building projects, maybe it isn’t surprising that there aren’t a lot common computing development efforts • I noticed during the week that there is a lot of duplication of effort, even between experiments that are geographically close • We have forums for exchange like HEPIX and the Large Cluster Workshop meetings • Even with these, we don’t seem to do much development in common • There are notable exceptions • Alan Silverman presented the work to write a guide to building and operating a large cluster • Their noble if somewhat ambitious goal is to “Produce the definitive guide to building and running a cluster - how to choose, acquire, test and operate the hardware; software installation and upgrade tools; performance mgmt, logging, accounting, alarms, security, etc, etc” CAS 2002-10-24 Ian M. Fisk, UCSD 17

18. Grid Projects • The grid projects are another area in which the field is working effectively together • A number of sites indicated the desire to use common tools developed by EDG Work Package 4 • Good buy in from fabric managers about the use of SRM • Software deployment through the VDT CAS 2002-10-24 Ian M. Fisk, UCSD 18

19. Conclusions • It was a long and interesting week • Apologies for not being able to summarize everything • We had very interesting discussions and presentations yesterday about how to interface the fabrics and the grid services • I also didn’t get a change to cover some of the hardware and software R&D results • I encourage people to look at the web page. Almost all the talks were posted. CAS 2002-10-24 Ian M. Fisk, UCSD 19