High Energy Physics & Computing Grid

1. High Energy Physics & Computing Grid Univ. of Texas @ Arlington Dr. Yu

2. Outline • High Energy Physics • The problem • A solution • An example of implemented solution • Accomplishments • Future plans • Summary

3. Matter Molecule Atom Nucleus Baryon Quark (Hadron) u Electron (Lepton) High Energy Physics Structure of Matter 10-14m 10-9m 10-10m 10-15m <10-19m 10-2m Condensed matter/Nano-Science/Chemistry protons, neutrons, mesons, etc. p,W,L... top, bottom, charm, strange, up, down Atomic Physics Nuclear Physics <10-18m

4. Mysteries in High Energy Physics? • The “STANDARD MODEL” has been extremely successful (Precision 10-6) • BUT… many mysteries • Why somany quarks/leptons?? • Whyfour forces?? Unification? • Why is there large particle- antipaticle asymmetry? • Does Higgs particle exist? • Where doesmasscome from?? • Are there other theories??

5. High Energy Physics • Definition: A field of Physics pursues for fundamental constituents of matter and basic principles of interactions between them  How is universe created, and how does it work? • Use large particle accelerators • Use large particle detectors

6. Discovered in 1995 Discovered in 2000 The Standard Model • Assumes the following fundamental structure:

7. Chicago  CDF p DØ Tevatron p Fermilab Tevatron and LHC at CERN • World’s Highest Energy proton-proton collider in 2 years • Ecm=14 TeV (=44x10-7J/p 1000M Joules on 10-4m2) • Equivalent to the kinetic energy of a 20t truck at a speed 6150 mi/hr • Present world’s Highest Energy proton-anti-proton collider • Ecm=1.96 TeV (=6.3x10-7J/p 13M Joules on 10-4m2) • Equivalent to the kinetic energy of a 20t truck at a speed 80 mi/hr

8. High Energy Physics • Definition: A field of Physics pursues for fundamental constituents of matter and basic principles of interactions between them  How is universe created, and how does it work? • Use large particle accelerators • Use large particle detectors • Large, distributed collaborations • ~600/experiment for currently operating experiments • ~2000/experiment for future experiments • WWW grew out of HEP to expedite communication between collaborators

9. Calorimeter (dense) Muon Tracks Charged Particle Tracks Energy Scintillating Fiber Silicon Tracking Interaction Point B EM hadronic Magnet Wire Chambers Particle Detection electron photon jet muon We know x,y starting momenta is zero, but along the z axis it is not, so many of our measurements are in the xy plane, or transverse neutrino -- or any non-interacting particle missing transverse momentum

10. 30’ 30’ 50’ DØ Detector: Run II • Weighs 5000 tons • Can inspect 3,000,000 collisions/second • Will record 50 collisions/second • Records ~12.5M Bytes/second • Will record 2 Peta bytes in the current run.

11. DØ Central Calorimeter 1990

12. p `p Data Reconstruction How are computers used in HEP? Digital Data

13. How does an Event Look in the DØ Detector? Highest ET dijet event at DØ CH “calorimeter jet” hadrons FH  EM “particle jet” Time “parton jet”

14. The Problem • Current Experiments at Tevatron • Has been taking data for the past 3 years and will continue throughout much of the decade The immediacy!!! • Current data size close to 1PB and will be over 4 PB by the end (~100km stack of 100GB disk drives) • 10 – 20 times (~100PB) increase at the future experiments

15. ~50M events/mo

16. The Problem • Current Experiments at Tevatron • Has been taking data for the past 3 years and will continue throughout much of the decade The immediacy!!! • Current data size close to 1PB and will be over 4 PB by the end (~100km stack of 100GB disk drives) • 10 – 20 times (~100PB) increase at the future experiments • Detectors are complicated  Need many people to construct and make them work • Collaboration is large and scattered all over the world

17. Typical HEP Collaboration at Present ~700 Collaborators ~80 Institutions 18 Countries

18. Large Hadron Collider (LHC) CERN, Geneva: 2007 Start • pp s =14 TeV L=1034 cm-2 s-1 • 27 km Tunnel in Switzerland & France CMS TOTEM 5000+ Physicists 250+ Institutes 60+ Countries First Beams: Summer 2007 Physics Runs: from Fall 2007 ALICE : HI LHCb: B-physics Atlas H. Newman

19. The Problem • Current Experiments at Tevatron • Has been taking data for the past 3 years and will continue throughout much of the decade The immediacy!!! • Current data size close to 1PB and will be over 4 PB by the end (~100km stack of 100GB disk drives) • 10 – 20 times (~100PB) increase at the future experiments • Detectors are complicated  Need many people to construct and make them work • Collaboration is large and scattered all over the world • Development and improvements at remote institutions • Optimized resource management, job scheduling, and monitoring tools • Efficient and transparent data delivery and sharing • Use the opportunity of having large data set in furthering grid computing technology • Improve computational capability for education • Improve quality of life

20. What is a Computing Grid? • Grid: Geographically distributed computing resources configured for coordinated use • Physical resources & networks provide raw capability • “Middleware” software ties it together

21. How do HEP physicists communicate?

22. Old Deployment Models Started with Fermilab-centric SAM infrastructure in place, … …transition to hierarchically distributed Model 

23. Central Analysis Center (CAC) Normal Interaction Communication Path Occasional Interaction Communication Path …. RAC RAC ... … IAC IAC IAC IAC …. …. DAS DAS DAS DAS DØ Remote Analysis Model (DØRAM) Fermilab Regional Analysis Centers Institutional Analysis Centers Desktop Analysis Stations

24. KSU OU/LU KU Aachen Bonn Wuppertal UAZ Mainz Ole Miss UTA GridKa (Karlsruhe) LTU Rice Munich Mexico/Brazil DØRAM Implementation UTA has the first and the only US RAC DØ Southern Analysis Region (DØSAR) formed around UTA

25. What can accomplished in an analysis region? • Construct end-to-end service environment in a smaller, manageable scale • Train and accumulate local expertise and share them • Form a smaller group to work coherently and closely • Draw additional resources from variety of funding sources • Promote interdisciplinary collaboration • Increase intellectual resources for the experiment • Enable remote participants to be more actively contribute to the collaboration • Form a grid and use it for DØ • Simulated data (Monte Carlo) production • Actual data reconstruction • Actual and simulated data analyses • Promote and improve IAC’s group stature

26. DØSAR Consortium • Second Generation IAC’s • Cinvestav, Mexico • Universidade Estadual Paulista, Brazil • University of Kansas • Kansas State University • First Generation IAC’s • University of Texas at Arlington • Louisiana Tech University • Langston University • University of Oklahoma • Tata Institute (India) Each 1st generation institution is paired with a 2nd generation institution to help expedite implementation of D0SAR capabilities • Third Generation IAC’s • Ole Miss, MS • Rice University, TX • University of Arizona, Tucson, AZ • USTC China • Korea University, Korea Both 1st and 2nd generation institutions can then help the 3rd generation institutions implement D0SAR capabilities

27. DØSAR Accomplishments • The only established US analysis region within DØ • Constructed and activated a Regional Analysis Center • Formed and activated five new MC production farms • Data access capability implemented in 70% of the sites • Employed and developed and implemented many useful monitoring tools • Ganglia and MonaLISA • McFarmGraph, McPerM, McQue, and McFarmDB

28. UTA, The New Way • 84 P4 Xeon 2.4GHz CPU = 202 GHz • 7.5TB of Disk space • 100 P4 Xeon 2.6GHz CPU = 260 GHz • 64TB of Disk space • Total CPU: 462 GHz • Total disk: 73TB • Total Memory: 168Gbyte • Network bandwidth: 68Gb/sec

29. Various Monitoring Applications Ganglia: Operating since Apr. 2003 McGraph: Operating since Sept. 2003 McQue: Operating since June 2004 McPerM: Operating since Sept. 2003

30. ot MonaLISA Grid Resource Monitoring

31. DØSAR Accomplishments • The only established US analysis region within DØ • Formed and activated five new MC production farms • SAM stations are installed in eight sites (three more to go..) • Constructed and activated a Regional Analysis Center • Employed and developed and implemented many useful monitoring tools • Started contributing beyond MC production • Accumulated large expertise in many areas • Successfully brought in additional computing and human resources

32. DØSAR Computing & Human Resources

33. DØSAR Accomplishments • The only established US analysis region within DØ • Formed and activated five new MC production farms • SAM stations are installed in eight sites (three more to go..) • Constructed and activated a Regional Analysis Center • Employed and developed and implemented many useful monitoring tools • Started contributing beyond MC production • Accumulated large expertise • Successfully brought in additional computing and human resources • Formed the DØSARGrid and producing simulated events on it

34. GUI to Access Grid

35. Ded. Clst. Ded. Clst. Desktop. Clst. Desktop. Clst. Client Site Sub. Sites SAM How does the DØSARGrid work? DØ Grid JDL Exe. Sites Reg. Grids

36. DØSAR Accomplishments • The only established US analysis region within DØ • Formed and activated five new MC production farms • SAM stations are installed in eight sites (three more to go..) • Constructed and activated a Regional Analysis Center • Employed and developed and implemented many useful monitoring tools • Started contributing beyond MC production • Accumulated large expertise • Successfully brought in additional computing and human resources • Developed McFarm interface to SAMGrid • Formed the DØSARGrid and start producing MC events on the grid • Promote inter-disciplinary collaborations

37. What next? • We will participate in large scale DØ data processing in a few months • 5 TB of data has been pre-staged in preparation • Must perform data analysis in the region using the regional resources • Large number of data sets has been transferred to UTA • HEP Students are working on their theses analyses using these data sets • Physics Undergraduate students are working on their class projects using this data • Preparing the transition into future experiment and exploit it in DØ • Improve local infrastructure, such as network bandwidths

38. DØ and ATLAS Production DPCC online Network Bandwidth Usage at UTA

39. What next, cnt’d? • Transform into a legitimate, active Virtual Organization within the global grid picture  Should be gradually done within the next year • Participate in existing US (Open Science Grid) and European (Enabling Grid for E-science in Europe) • Fully utilize the involvement with future experiments • Turn DØSAR into DOSAR (Data Oriented Super Analysis Region) • Continue promoting interdisciplinary collaboration • Actively participate and lead grid computing efforts in the respective states • Employ the grid computing technology not just for research but also for education

40. Seattle Clev Chicago New York Pitts Denver Sunnyvale KC Wash DC Raleigh Tulsa LA Albuq. Phoenix San Diego Atlanta Dallas Jacksonville El Paso - Las Cruces Pensacola Baton Rouge Houston San Ant. NLR – National LambdaRail 10GB/sec connections

41. Grid in other disciplines? • Nuclear physics • Bioinformatics • Genetics • Meteorology • Medical science and medicine • Homeland security

42. Conclusions • To understand the fundamentals of nature, High Energy Physics • Uses accelerators to look into extremely small distances • Uses large detectors to explore nature • Uses large number of computers to process data • Large amount of data gets accumulated  need computing grid to perform expeditious data analysis • Computing grid needed for other disciplines with large data sets • HEP is an exciting endeavor in understanding nature • Physics analyses at one’s own desktop using computing grid is close to be a reality • UTA plays a leading role in HEP research and shaping the future of computing Grid • Computing grid will soon revolutionize everyday lives soon…