1 / 57

My Journey through Scientific Computing

My Journey through Scientific Computing. August 28 2013, FNAL René Brun/CERN*. Disclaimer. In this talk I present the views of somebody involved in some aspects of scientific computing as seen from a major lab in HEP.

dyanne
Download Presentation

My Journey through Scientific Computing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. My Journey throughScientific Computing August 28 2013, FNAL René Brun/CERN*

  2. Disclaimer • In this talk I present the views of somebody involved in some aspects of scientific computing as seen from a major lab in HEP. • Having been involved in the design and implementation of many systems, my views are necessarily biased. R.Brun

  3. CV with experiments • Jun 1973: Thesis Nuclear Physics • Jul 1973 : ISR: R602 with C.Rubbia (reconstruction) • Oct 1974: SPS: NA4 with C.Rubbia (simulation/recons) • Feb 1980: LEP: OPAL at LEP (simulation and framework) • 1991-1993 :LHC: ATLAS/CMS simulation • Sep 1994: SPS: NA49 at SPS (exile) • Sep 1996: LHC: ALICE (simulation and framework) R.Brun

  4. CV with general software • 1974 : HBOOK • 1975: GEANT1, Lintra, Mudifi • 1977: GEANT2, HPLOT, ZBOOK • 1980: GEANT3 • 1983: ZEBRA • 1984: PAW • 1995: ROOT • 2012: GEANT4+5 vector prototype concepts R.Brun

  5. Machines From Mainframes ===== Clusters Walls of cores GRIDs & Clouds R.Brun

  6. Machine Units (bits) 16 32 36 48 56 60 64 pdp 11 univac many cdc many nord50 besm6 A strong push to develop portable machine independent I/O systems With even more combinations of exponent/mantissa size or byte ordering R.Brun

  7. User machine interface R.Brun

  8. Systems in 1980 End user Analysis software 10 KLOC Experiment Software 100 KLOC Libraries HBOOK, Naglib, cernlib 500 KLOC RAM 1 MB OS & fortran 1000 KLOC Tapes CDC, IBM Vax780 R.Brun

  9. Systemstoday End user Analysis software Networks 10 Gbit/s 0.1 MLOC Disks 1o PB Experiment Software 4MLOC Frameworks like ROOT, Geant4 5MLOC CLOUDS RAM 16 GB OS & compilers 20 MLOC GRIDS Hardware Hardware Hardware Hardware Clusters of multi-core machines 10000x8 R.Brun

  10. Tools & Libs Geant4+5 geant4 geant3 geant1 geant2 bos minuit hbook root paw zbook zebra hydra R.Brun

  11. General Software in 1973 • Software for bubble chambers: Thresh, Grind, Hydra • Histogram tool: SUMX from Berkeley • Simulation with EGS3 (SLAC), MCNP(Oak Ridge) • Small Fortran IV programs (1000 LOC, 50 kbytes) • Punched cards, line printers, pen plotters (GD3) • Small archive libraries (cernlib), lib.a R.Brun

  12. Software in 1974 • First “Large Electronic Experiments” • Data Handling Division == Track Chambers • Well organized software in TC with HYDRA, Thresh, Grind, anarchy elsewhere • HBOOK: from 3 routines to 100, from 3 users to many • First software group in DD R.Brun

  13. GEANT1 in 1975 • Very basic framework to drive a simulation program, reading data cards with FFREAD, step actions with GUSTEP, GUNEXT, apply mag-field (GUFLD). • Output (Hist/Digits) was user defined • Histograms with HBOOK • About 2,000 LOC R.Brun

  14. ZBOOK in 1975 • Extraction of the HBOOK memory manager in an independent package. • Creation of banks and data structures anywhere in common blocks • Machine independent I/O, sequential and random • About 5,000 LOC R.Brun

  15. GEANT2 in 1976 • Extension of GEANT1 with more physics (e-showers based on a subset of EGS, mult-scattering, decays, energy loss • Kinematics, hits/digits data structures in ZBOOK • Used by several SPS experiments (NA3, NA4, NA10, Omega) • About 10,000 LOC R.Brun

  16. Problems with GEANT2 • Very successful small framework. • However, the detector description was user written and defined via “if” statements at tracking time. • This was becoming a hard task for large and always evolving detectors (case with NA4 and C.Rubbia) • Many attempts to describe a detector geometry via data cards (a bit like XML), but the main problem was the poor and inefficient detector description in memory. R.Brun

  17. GEANT3 in 1980 • A data structure (ZBOOK tree) describing complex geometries introduced , then gradually the geometry routines computing distances, etc • This was a huge step forward implemented first in OPAL, then L3 and ALEPH. • Full electromagnetic showers (first based on EGS, then own developments) R.Brun

  18. (HYDRA,ZBOOK)GEM ->ZEBRA • HYDRA and ZBOOK continuous developments, both having nice and complementary features. • In 1981 the GEM project launched, developed with no contacts with experiments fail to deliver a working system in 1982. • In 1983 the director for computing decided to stop GEM and HYDRA and support the ZBOOK line, mainly because of the success of GEANT3 based on ZBOOK. • I decided to collaborate with J.Zoll (HYDRA) to develop ZEBRA, combining ZBOOK and HYDRA. • This decision made OPAL and L3 happy, but ALEPH decided to use BOS from DESY. R.Brun

  19. GEANT3 with ZEBRA • ZEBRA was very rapidly implemented in 1983. • We introduced ZEBRA in GEANT3 in 1984. • From 1984 to 1993 we introduced plenty of new features in GEANT3: extensions of the geometry, hadronic models with Tatina, Gheisha and Fluka, Graphics tools. • In 1998, GEANT3 interface with ROOT via the VMC (Virtual Monte Carlo) • GEANT3 has been used and still in use by many experiments. R.Brun

  20. Graphics/UI in the 80s • CORE system (Andy Van Dam) in the US • GKS in Europe • Xwindow wins with Xterms and workstations • We design HIGZ (interface to graphics and ZEBRA) with many interfaces (CORE, GKS, X, PHIGS,etc) • KUIP for User Interface (VT100, work-stations, xterms) R.Brun

  21. PAW • First minimal version in 1984 • Attempt to merge with GEP (DESY) in 1985, but take the idea of ntuples for storage and analysis. GEP was written in PL1. • Package growing until 1994 with more and more functions. Column-wise ntuplesin 1990. • Users liked it, mainly once the system was frozen in 1994. R.Brun

  22. Vectorization attempts • During the years 1985->1990 a big effort was invested in vectorizing GEANT3 (work in collaboration with Florida State University) on CRAY/YMP, CYBER205,ETA10. • The minor gains obtained did not justify the big manpower investment. GEANT3 transport was still essentially sequential and we had a big overhead with vectors creation, gather/scatter. • However this experience and failure was very important for usand many messages useful for the design of GEANT5 many years later. R.Brun

  23. So far so good • The years 1980->1989 were pretty quiet (ie no fights). Fortran 77 was the main language and LEP our main target. The SSC simulations (GEM, SDC) were along the same lines. • The ADAMO system had been proposed as an implementation of the Entity Relationship Model, but its use remained confidential (ALEPH/ZEUS), same for the Jazelle system from SLD. • In 1989 Tim Berners Lee joined my group in DD to implement a system allowing access to a central documentation on the IBM via RPCs (Remote Procedure Calls), but he developed something else. This coincided with a lot of controversy about future languages, data structures management, data bases, user interfaces, documentation systems, etc. R.Brun

  24. 1991: Erice Workshop • This workshop was supposed to see an agreement on the directions and languages for the next generation experiments, instead a very confusing picture emerged. • The MOOSE project created at CERN to investigate languages such as Eiffel, C++, ObjectiveC, F90. But this project failed to produce anything concrete. • At the same time my group was very busy with the LEP analysis with PAW and the SSC and LHC simulations with GEANT3 (ATLAS and CMS). R.Brun

  25. Erice workshop 1991 Where to go with DS tools & languages ? Powerfultools but programmingwiththemwasodd R.Brun

  26. 1992: CHEP Annecy • Web, web, web, web………… • Attempts to replace/upgrade ZEBRA to support/use F90 modules and structures, but modules parsing and analysis was thought to be too difficult. • With ZEBRA the bank description was within the bank itself (just a few bits). A bank was typically a few integers followed by a dynamic array of floats/doubles. • We did not realize at the time that parsing user data structures was going to be a big challenge!! R.Brun

  27. Parallelism in the 80s & early 90s • Many attempts (all failing) with parallel architectures • Transputers and OCCAM • MPP (CM2, CM5, ELXI,..) with OpenMP-like software • Too many GLOBAL variables/structures with Fortran common blocks. • RISC architectures or emulators perceived as a cheaper solution in the early 90s. • Then MPPs died with the advent of the Pentium Pro (1994) and farms of PCs or workstations. R.Brun

  28. Consequences • In 1993/1994 performance was not anymore the main problem. • Our field invaded by computer scientists. • Program design, object-oriented programming , move to more sexy languages was becoming a priority. • The “goal” was thought less important than the “how” • This situation deteriorates even more with the death of the SSC. R.Brun

  29. 1993: Warning Danger • 3 “clans” in my group • 1/3 pro F90 • 1/3 pro C++ • 1/3 pro commercial products (any language) for graphics, User Interfaces, I/O and data bases • My proposal to continue with PAW, develop ZOO(ZEBRA Object-Oriented) and GEANT3 geometry in C++ is not accepted. • EvolutionvsRevolution R.Brun

  30. 1994: What next? • SSC down, LHC up. SSC refugees joining LHC development groups. • DRDC projects: Objectivity, GEANT4 • Time to think, think, think and learn new things (OO,UML,C++,Eiffel, O2, ObjectStore, Objectivity,..) • Discard several proposals and choose exile to NA49 • Fall94: first version of histogram package in C++, including some I/O attempts. Now in a better position to estimate development time, C++ pros&cons, Objectivity cons. • Christmas94: YES, let’s go with ROOT R.Brun

  31. 1995: roads for ROOT • The official line was with GEANT4 and Objectivity, not much room left for success with an alternative product when you are alone. • The best tactic had to be a mixture of sociology , technicalities and very hard work. • Strong support from PAW and GEANT3 users • Strong support from HP (workstations + manpower) • In November we were ready for a first ROOT show • Java is announced (problem?) R.Brun

  32. 1996: Technicalities • Histogram classes (3 versions) + Minuit • Hand written Streamers • From PAW/KUIP style to CINT • Collection classes (STL , templates, hesitations..) • ATLFAST++ (fast simulation of ATLAS based on ROOT) • Letter from the director of computing against the use of ROOT by experiments (except NA49). Problem for ALICE. • LHC++ official alternative to ROOT R.Brun

  33. 1997: Technicalities++ • ROOTCINT generated Streamers with parsing of C++ header files, including user classes. • Many improvements and new packages • Automatic classes documentation system (THTML), first manual and tutorials. • gAlice converted to AliRoot • Interest from RHIC (Phobos and Star) • CHEP97 in Berlin with focus on GEANT4, Objectivity,LHC++,JAS R.Brun

  34. 1998: work & smile • RUN II projects at FNAL • Data Analysis and Visualization • Data Formats and storage • ROOT competing with HistoScope, JAS, LHC++ • CHEP98 (September) Intercontinental Chicago • ROOT selected by FNAL, followed by RHIC • Vital decision for ROOT, thanks FermiLab!!! • But official support at CERN only in 2002 R.Brun

  35. ROOT Trees vs Objectivity • Compared to the best OODBMS candidate in 1995 (Objectivity) ROOT supports a persistent class thatmaybe a subset of the transient class. • ROOT supports compression (typicalfactors 3 to 6), file portability and access in heterogeneous networks • ROOT supports branchsplittingthatincreasesdrastically the performance whenreading. • It isbased on classical system files and does not require the nightmare of a central data base. • ROOT TTreeCacheallows efficient access to LAN and WAN files. R.Brun

  36. Input/Output: Major Steps User written streamers filling TBuffer member-wise streaming for STL collections<T*> streamers generated by rootcint TreeCache automatic streamers from dictionary with StreamerInfos in self-describing files parallel merge member-wise streaming for TClonesArray R.Brun

  37. ROOT evolution • No time to discuss the creation/evolution of the 110 ROOT shared libs/packages. • ROOT has gradually evolved from a data storage, analysis and visualization system to a more general software environment replacing totally what was known before as CERNLIB. • This has been possible thanks to MANY contributors from experiments, labs or people working on other fields. • In the following few slides, I show the current big systems assigned to at least one developer. R.Brun

  38. Code repository and Management • From CMZ to CVS  SVN GIT • Make  cmake • Build and test infrastructure • Distribution • Forum, mails, Savannah->JIRA • Documentation, User Guide R.Brun

  39. CINT -> CLING • CINT, originally developed by MasaGoto(HP/Taligent/Japan) in 1991 has been gradually upgraded across the years. • Work in progress to replace CINT by CLING based on the CLANG C++ compiler from Apple. • With CLING many C++ limitations of CINT will be eliminated and full support for C++11 provided at the command line, script via a JIT compiler. R.Brun

  40. 2-D Graphics • Extremely important area that requires a lot of effort to support a large number of styles, options. • Support for graphics on screen with different back-ends for Linux, Macs/Ipad , Windows, Android, etc • Support for many output formats (.ps, .eps, .pdf,.tex,.gif,.png,.jpg, .C, .root • Support for interaction, picking, reflexion, etc R.Brun

  41. 3D Graphics • Again many back-ends: X, OpenGL, GDK/Windows, Cocoa/Mac, • Event viewing • Ouput drivers • User Interfaces R.Brun

  42. Geometry package • Originally from GEANT3, then considerable developments. • Many interfaces: to/from GEANT3 and GEANT4 • New developments (parallelism, thread safety and vectorization) in the context of the GEANT4+5 project R.Brun

  43. User Interface • UI library for X, GL, GDK, Cocoa, Ipad, • Pre-defined menu, pop-up, pull-down • Interface builder • C++ automatic code generator • Qt interface R.Brun

  44. Maths&Stats • Mathematical functions • Matrix packages • Random number generators • MINUIT, Minuit2,.. • RooStats, RooFit • TMVA,… R.Brun

  45. PyRoot/Python • Full time to work to support an easy Python interface • Old interface based on CINT • New interface based on CLING R.Brun

  46. PROOF • Use multi-process techniques to speed-up interactive analysis. • Evolved and again evolving to be used via automatic interfaces to the GRID systems • PROOFLight interesting for multi-core laptops. R.Brun

  47. Systems in 2030 ? End user Analysis software Networks 100 Gbit/s 1 MLOC Networks 100 Gbit/s Networks 10 Tbit/s Experiment Software 50MLOC Disks 1o00 PB Frameworks like ROOT, Geant5 20MLOC CLOUDS on demand RAM 10 TB OS & compilers 100 MLOC GRIDS Hardware Hardware Multi-level parallel machines 10000x1000x1000 Hardware Hardware R.Brun

  48. Parallelism: key points Minimize the sequential/synchronization parts (Amdhallaw): Verydifficult Run the same code (processes) on all cores to optimize the memory use (code and read-only data sharing) Job-levelisbetterthanevent-levelparallelism for offline systems. Use the good-oldprinciple of data localityto minimize the cache misses. Exploit the vectorcapabilitiesbut becarefulwith the new/delete/gather/scatterproblem Reorganizeyour code to reducetails R.Brun

  49. Data Structures & parallelism C++ pointers specific to a process event event vertices Copying the structure implies a relocation of all pointers I/O is a nightmare tracks Update of the structure from a different thread implies a lock/mutex R.Brun

  50. Data Structures & Locality sparse data structures defeat the system memory caches For example: group the cross-sections for all processes per materialinstead of all materials per process Group objectelements/collections suchthat the storage matches the traversalprocesses R.Brun

More Related