1 / 34

Grids and HEP Computing coordination ICFA seminar Guy Wormser IHEPCCC chair LAL Orsay

Grids and HEP Computing coordination ICFA seminar Guy Wormser IHEPCCC chair LAL Orsay. Grids and HEP. Around 1998, a new concept « GRID computing » landed in HEP physics computing:

zorion
Download Presentation

Grids and HEP Computing coordination ICFA seminar Guy Wormser IHEPCCC chair LAL Orsay

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Grids and HEP Computing coordination ICFA seminar Guy Wormser IHEPCCC chair LAL Orsay

  2. Grids and HEP • Around 1998, a new concept « GRID computing » landed in HEP physics computing: • “A computational grid is a hardware and software infrastructure that provides dependable, consistent, pervasive, and inexpensive access to high computational capabilities.” (The Grid, I. Foster, C. Kesselman, 1998) • It had immediately a large worldwide impact in HEP • HEP is a very distributed field, naturally structured into virtual organisations • Huge computing needs for LHC and other projects • Embarassingly parallel style of computing well matched to simple Grid architecture • The network bandwith “explosion” Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  3. HEP commitment to Grids • 2000-2003 : Exploratory phase. Several R&D projects in the world to develop the middleware, build mid size test beds, port some applications • US : PPDG, Griphyn • Europe : DATAGRID • Several initiatives in Asia • 2002: Decision to build a grid infrastructure as THE TOOL for LHC computing. Creation of the LCG project (LHC Grid Computing). Point of no return! • 2004 Large scale deployment of the world largest distributed computing infrastructure(s): • LCG, EGEE, OSG (US), NAREGI(Japan), etc…. • >200 sites, 15 000 processors, 5 PB of storage Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  4. Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  5. Grids: a great oportunity for interdisciplinary contacts • Grids have become a great vehicle for promoting interdisciplinary contacts between HEP, many other application fields and computing scientists • Bioinformatics • Medecine • Chemistry • Fusion science • Earth sciences • Astrophysics/astronomy • Neuroinformatics • Climate • Finance • ….. • HEP can be proud to have been a key player in these endeavours Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  6. 2004 Service Usage • VOs and users on the production service • Active HEP experiments: • 4 LHC, D0, CDF, Zeus, Babar • Active other VO: • Biomed, ESR (Earth Sciences), Compchem, Magic (Astronomy), EGEODE (Geo-Physics) • 6 disciplines • Registered users in these VO: 500 • In addition to these there are many VO that are local to a region, supported by their ROCs, but not yet visible across EGEE • Scale of work performed: • LHC Data challenges 2004: • >1 M SI2K years of cpu time (~1000 cpu years) • 400 TB of data generated, moved and stored • 1 VO achieved ~4000 simultaneous jobs (~4 times CERN grid capacity) Number of jobs processed/month Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  7. Data Challenges – LHCb This is more than 3 times what was possible at CERN alone June-September 2004 • 186 M events in ~250k jobs . 61 Tbyte data • CPU: ~370 kSI2k years (43 LCG and 20 DIRAC sites) • Increased usage of LCG with time – 75% by September; 50% LCG overall • Up to 3500 concurrent running jobs in LCG 3-5 106/day LCG use restart LCG use pause Plus LCG 1.8 106/day DIRAC alone Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  8. D0 MC efficiency on LCG2 since Xmas (but small statistics) Efficiency 98 % System running monitored very closely by run-manager in close contact with sites Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  9. APPLICATIONS ported on EGEE Earth Observation by Satellite Hydrology Solid Earth Physics Meteorology Climate Geosciences Chemistry of the Mars Upper Atmosphere Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  10. SEISMOLOGY[1] Fast Determination of mechanisms of important earthquakes (IPGP: E. Clévédé, G. Patau) Challenge Provide results 24h -48h after its occurrence 5 Seisms already ported: Peru, Guadeloupe, Indonesia (Dec.), Japon, Indonesia (Feb.) Application to run on alert • Collect data of 30 seismic stations from GEOSCOPE worldwide network • Select stations and data • Definition of a spatial 3D grid +time • Run for example 50-100jobs Peru earthquake, 23/6/2001, Mw=8.3 Data used: 15 Geoscope Stations Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  11. SEISMOLOGY [2] Indonesian EQ 25 March 2005 Thanks to the grid, the best of the 30,000 solutions could be found within 30 hours after the earthquake took place. This best solution was later confirmed by comparing it with results from other analysis methods. Many CPUs (>100) from time to time without reservation Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  12. HYDROLOGY [1] Flood Prediction: Experimental data and Models. L. Hluchy, Viet Tran, M. Ciglan …. (II-SAS, Bratislava Slovaquie) Cascade of simulations Data sources CHALLENGE Complex application: Assimilation of data in models and cascade of simulations Meteorological simulation Hydrological simulation Hydraulic simulation Portal Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  13. Korba coastal aquifer Management of water resources in Mediterranean area (SWIMED) G. Lecca (CRS4 Italy), P. Renard (Unine, CH), J. Kerrou (INAT, Tunisia), R. Ababou (IMFT, Fr) Tunisia 45 km Cape Bon Peninsula 70km south-east of Tunis Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  14. GEOSCIENCES • Generic seismic platform software, based on Geocluster commercial software developed by CGG • Includes 400 geophysical modules, implemented on EGEE • Used by both academics and private companies. • Free of charge for Academics, with charge for R&D Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  15. EARTH SCIENCE ORGANISATION • Coordination Earth Science: Monique Petitdidier (IPSL) • 14 Research Laboratoiries and Institutes; One organisation: ESA, 3 Private companies, CGG, DutchSpace et Iguassu Software Systems • Bulgaria, France, Italy, Germany, Greece, The Netherlands, Russia, Slovakia, Switzerland, Tchequie • Around 40 persons registered in the VOs Welcome to new partners Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  16. Status of Biomedical VO RLS, VO LDAP Server: CC-IN2P3 4 RBs: CNAF, IFAE, LAPP, UPV 15 resource centres ( ) • 17 CEs (>750 CPUs) • 16 SEs 4 RBs 1 RLS 1 LDAP Server PADOVA BARI Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  17. Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  18. Malaria Data challenge (summer 2005) Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  19. HEP computing coordination: IHEPCCC • HEPCCC committee created in Europe in ~1980 to coordinate HEP computing. Mainly centered around CERN at the time. Membership: Directors of the main European computing centers • Important role as a discussion forum and coordinating body throughout the years. Creation of HTASC, a permanent technical advisory body. HEPCCC father of the DATAGRID initiative. • In 2001, it was realized that it was mandatory to include membership from all over the world to the increased worldwide nature of HEP computing • IHEPCCC was created in 2003 as a ICFA subcommittee, « sister » to SCIC, organizer of this workshop Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  20. IHEPCCC mandate Preamble • …an efficient information exchange on computing issues between the major HEP centers in the world. Missions • …forum between the main persons in charge of HEP computing, by gathering and distributing information about all relevant issues in HEP computing, and especially those with a global nature. • Typical examples : new technology trends, computing centers strategic policies, security issues, recommendation of standard practices, presentation of R&D results, comparison of various equipments performances. • The other missions include: • Issuing statements and recommendations concerning computing in the HEP community. • Serving as an interface to other scientific domains on matters of computing. • Working in close connection with the ICFA SCIC, the physics regional organizations, and the HICB coordinating the grid projects in HEP. • Reporting to ICFA (ie grabbing the attention of all the HEP major lab directors) • http://www.fnal.gov/directorate/icfa/Int'l_HEPCCC.html Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  21. IHEPCCC membership Asia • Setsuya Kawabata(KEK-Japan) , Hesheng Chen (Beijing-China), Dongchul Son (Daegu- South Korea), S. Banerjee(Tata-India) , Geoff Taylor (Melbourne-Australia) North America • Paul Avery (Gainesville-US), Bruce Gibbard (BNL-US) , Richard Mount (SLAC-US) , Randall Sobie (Victoria-Canada), Vicky White(FNAL-US), Frank Wuerthwein (San Diego-US) Europe • Manuel Delfino (Barcelona-Spain) , Matthias Kasemann (DESY-Germany) , Mirco Mazzucato (Padova-Italy) , Wolfgang von Rueden (CERN) , Michal Turala (Crakow-Poland), Peter Watkins (Birmingham, UK), Guy Wormser (Orsay-France), Chair Rest of the world • Viacheslav Ilyin (Moscow-Russia), Abdeslam Houmadda (Casablanca-Morocco), Alberto Santoro (Rio, Brazil) Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  22. Consensus on objectives A/ Guarantee the best possible conditions for the advent on a Grid-based HEP world wide computing A.1/ Make sure that all conditions are met to foster the development of viable HEP grid(s) A.2/ Make sure that they are interoperable and secure A.3/ Make sure that all regions of the HEP world have an easy access to them B/ Foster collaboration and training in HEP computing B.1/ Make sure that the best possible conditions are met to foster the common development of software tools and standards B.2/ Make sure that the best possible conditiond are met for best practice exchange and training Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  23. IHEPCCC meetings • Inaugural meeting May 2004 in Barcelona • http://agenda.cern.ch/fullAgenda.php?ida=a041864 • CERN Oct 2004, just after CHEP04 • http://agenda.cern.ch/fullAgenda.php?ida=a044205 • South Korea, May 28 2005 • Seattle Nov 12-18 2005. Colocated with SC2005 • Feb 2006 Mombai, colocated with CHEP2006 • Nov 2006 Europe • Spring 2007 US Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  24. Technical Advisory Body • IHEPCCC chose to use HEPiX as the main provider of ad hoc task forces experts • Contact taken with HEPiX chairpersons • First cases to study : • HEP VO issues • Linux Operating systems. Report now available, thanks to A. Silverman et al. http://agenda.cern.ch/askArchive.php?base=agenda&categ=a053314&id=a053314s0t3/document • Future potential topics: • Better coordination of HEP sofware development • Videoconference and other collaborating tools • Data storage management • All suggestions welcome Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  25. The last IHEPCCC meeting in Korea May 27 • Communications The presentation from many countries or regions were done during the digital divide workshop organisnized by SCIC (D. Son and H. Newman) . They can be found at : http://chep.knu.ac.kr/HEPDG2005/index.html • Highlights of the digital divide workshop. Summary by H. Newman: http://chep.knu.ac.kr/HEPDG2005/upload/fri_s5/ICFADDKoreaConclude052705s3.ppt • HEPiX-IHEPCCC interaction • HEP VO • Training group status report: • Software development in HEP • Grid projects • New initiatives • large procurements for LHC. • Licensing issues Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  26. Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  27. Digital divide issues • HEP is playing an important role in promoting science of many developping countries • The Digital divide workshop in Korea in May was a very good illustration of this phenomemon. The virtusous circle: science/grids/high speed network is proving very efficient in many countries see • http://chep.knu.ac.kr/HEPDG2005/index.html • CHEP06 in India • Sharing the knowledge around the Mediterranea conf in Casablanca Sept 05 • Creation of a training task force within IHEPCCC (V. Ilyin chair) • Census of all existing training programs in HEP • Preparation of World Summit on IST : Tunis, Durban,… • SCIC is preparing a report. IHEPCCC will help • Second common workshop between SCIC and IHEPCCC in fall 2006 in Europe Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  28. Towards a HEP VO • GOAL: take advantage on the new technologies to offer minimum computing services to any bona fide HEP member in any HEP lab. • Solve the travelling physicist problem • Create new services for the community • Initial proposal to Base it on Certificate technology • Requires to maintain an accurate list of HEP VO members. Has to rely on the institutions. Requires a list of institutions • Technical implementation to be discussed with experts • Slow progress along this line due to large security concerns and parallel progress of tehcnologies (wireless and laptop are now fully widespread, although it is not the full solution to the problem) • Reorientation towards guidelines on list on minimal services and how on provide them. Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  29. Software issues • The degree of duplication of the various software efforts was really striking • For instance, one Mass storage software per big lab • Many different grid flavors • From the most tiny technical piece to the large frameworks • Could be partly be reduced by more information • How to promote the co development of HEP software (there are some success stories like GEANT4) • Present trend to go away fom commercial software but some licensing issues remain cf MATLAB • Work on licensing issues at HEP level • IHEPCCC will prepare a matrix using CHEP04 presentations and calssifications to document which lab is doing what in all aspects of HEP software Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  30. Global GridCommunity Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  31. The Grid bazaar • Probably useless and counter-productive to push for one unified GRID • AT least insist on interoperability and minimal set of grids • But INSIST very strongly that the LHC computing is ONE PROJECT IHEPCCC recommendation in preparation The LHC computing project should be considered as a whole. Different flavours of middleware are presently used in various regions of the world and , although it could be argued that is not ideal, this situation is very likely not going to change in the coming years. Therefore, IHEPCC strongly recommends • i) to pursue and develop all efforts aiming at the maximum interoperability. In doing so, it is important to keep in mind that the LHC grid usage for physics analysis purposes will traverse the present geographical boundaries of these middleware implementations. • ii) to review the full LHC computing project as a single integrated project • iii) to adopt names for the various middlewares and the LHC computing grid project that allow to better capture the collective nature fo the enterprise Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  32. Procurement and licensing issues • a) Most of us will enter a phase of large procurements for LHC. Although it is clearly a sensitive domain, exchanging information between Hep centers either before (to negociate and/or together) or after the procurement (to provide data that can help others to get low cost) can lead to huge savings at the global scale. Can we make a recommendation to ICFA, ie to put together a group of procurement experts from the HEP centers. • b) The question of scope if licences for commercial software used in HEP came up. • -how to be as inclusive as possible when a big lab negociates with a vendor to nclude as many other labs as possible (beware that the labs that ill not in the deal may have to pay a much higher price) • -how can we deploy licensed software on a grid if needed (a working group on that topic exists in EGEE). Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  33. IHEPCCC summary • IHEPCCC recently created and off for a good start as a discussion forum and information exchange at the worldwide scale in HEP • HEPiX has been picked up as the natural companion for technical advices, on a voluntary basis • IHEPCCC provides a good way to attract attention of world lab directors on computing issues. • Very fruitful session in Korea with many new initiatives • Comprehensive digital divide report showing the impact of HEP in the virtuous circle • Linux OS status report • Training working group • HEP VO • Software development • Recommendation on LCG • Procurement and licensing • Report on IHEPCCC activities made during the LCG deployment board on June 22 Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

  34. Conclusion • Grids : hard work but a great success story so far • Huge infrastructure in place, other sciences embarked, very high political visibility, HEP role clearly recognised • IHEPCCC created as an exchange forum between HEP computing centers • Presently LHC computing plays a key role in this field • IHEPCCC brings input on other aspects : non LHC HEP activities (accelerator based or not), general issues • Useful channel to lab directors thru ICFA meetings • Very good collaboration with SCIC, the computing network related ICFA subcommittee Guy Wormser, ICFA seminar, Daegu, Korea, September 30, 2005

More Related