ROOT Project Status Major developments Directions

1. ROOT Project StatusMajor developmentsDirections DESY 5 December 2005 René Brun CERN Overview, Major Developments, Directions

2. ROOT: a 10 years old project • Started in January 1995 in NA49 • First public presentation in November 95 • Cooperation with Masa Goto/CINT in April 96 • First implementation of ROOT with gAlice in December 97 (TGeant3)->VMC • FNAL chooses ROOT for I/O and data analysis • Announced at CHEP98 in Sept 98 in Chicago • RHIC experiments follow immediately • Hoffmann computing Review in 2000/2001 • LCG project in 2002 (Blueprint RTAG) • ROOT: LCG project in 2005 : Overview & Major Developments

3. Applications Area Organizationin LCG Phase I decisions strategy Architects forum Applications manager ROOT User - provider Applications area meeting POOL project SEAL project PI project Simulation project SPI project consultation Overview & Major Developments

4. Applications Area Organizationin LCG Phase II Same organization as in Phase I. ROOT and SEAL projects merge Applications area meeting POOL project ROOT +SEAL project Simulation project SPI project consultation ROOT project structured in work-packages Overview & Major Developments

5. New ROOT team structure The work-packages SEAL Lorenzo Moneta DICT Philippe Canal BASE Fons Rademakers MATH Lorenzo Moneta I/O & Trees Philippe Canal 2-D/3D graphics Olivier Couet GUI Ilka Antcheva PROOF Fons Rademakers GEOM/VMC Andrei Gheata Overview & Major Developments

6. ROOT version 5 • pro release 4.04/02 3 May (new Users Guide) • 1st dev release 5.02 28 June • 2nd dev release 5.04 20 September • 3rd dev release 5.06 2 Nov • Pro release 5.12 15 December (New Guide) See detailed Release Notes http://root.cern.ch/root/Version50400.news.html Overview & Major Developments

7. ROOT 2005 workshop • September 28-30 at CERN • 115 registered participants • 41 talks • 9 posters • See talks at ROOT web page Overview & Major Developments

8. ROOT: large user community ROOT is LGPL Overview & Major Developments

9. Base Work Package New infrastructure features Miscellaneous Overview, Major Developments, Directions

10. TArchiveFile and TZIPFile • TArchiveFile is an abstract class that describes an archive file containing multiple sub-files, like a ZIP or TAR archive. • The TZIPFile class describes a ZIP archive file containing multiple ROOT sub-files. Notice that the ROOT files should not be compressed when being added to the ZIP file, since ROOT files are normally already compressed. To create the file multi.zip do: • The ROOT files in an archive can be simply accessed like this: • A TBrowser and TChain interface will follow shortly. zip –n root multi file1.root file2.root TFile *f = TFile::Open("multi.zip#file2.root") or TFile *f = TFile::Open("root://mymachine/multi.zip#2") Overview & Major Developments

11. Replica of a DB subset T0 T1 local TZipFile remote TZipFile http, xrootd, castor, dcache.. Overview & Major Developments

12. SLAC’s New File Server - xrootd • The file server xrootd (eXtended ROOT daemon) has been developed by Andy Hanushevsky of SLAC. • The server exploits a multithreaded architecture to provide high-performance file based access, focusing on scalability and fault tolerance. The server is being extensively used by BaBar, Star and Alice. • The xrootd file server will in the near future replace the current daemon rootd. • xrootd and its plugin facility is heavily used by PROOF Overview & Major Developments

13. The New xrootd Client - TXNetFile • The new client class TXNetFile implements the xrootd protocol and is provided to open a file via the xrootd daemon. • TXNetFile can detect when it talks to on old rootd daemon and return a TNetFile. • To open a file via xrootd, just use the standard static method TFile::Open() as for opening via rootd. Overview & Major Developments

14. Class TGrid (abstract interface) //--- General GRID const char *GridUrl() const const char *GetGrid() const const char *GetHost() const const char *GetUser() const const char *GetPw() const const char *GetOptions() const Int_t GetPort() const //--- Catalogue Interface virtual TGridResult *Command(const char *command, Bool_t interactive = kFALSE, UInt_t stream = kFALSE) virtual TGridResult *Query(const char *path, const char *pattern, const char *conditions, const char *options) virtual TGridResult *LocateSites() virtual TGridResult *ls(const char*ldn ="", Option_t*options ="") virtual Bool_t cd(const char*ldn ="",Bool_t verbose =kFALSE) virtual Bool_t mkdir(const char*ldn ="", Option_t*options ="") virtual Bool_t rmdir(const char*ldn ="", Option_t*options ="") virtual Bool_t register(const char *lfn , const char *turl , Long_t size, const char *se, const char *guid) virtual Bool_t rm(const char*lfn , Option_t*option ="") //--- Job Submission Interface virtual TGridJob *Submit(const char *jdl) virtual TGridJDL *GetJDLGenerator() //--- Load desired plugin and setup conection to GRID static TGrid *Connect(const char *grid, const char *uid, const char *pw, const char *options) Overview & Major Developments

15. Access to File Catalogues eg Alien FC Same style interface could be implemented for Other GRID File Catalogues Overview & Major Developments

16. TGrid example with Alien // Connect TGrid alien = TGrid::Connect(“alien://”); // Query TGridResult *res =alien.Query (“/alice/cern.ch/user/p/peters/analysis/miniesd/”, ”*.root“); // List of files TList *listf = res->GetFileInfoList(); // Create chain TChain chain(“Events", “session"); Chain.AddFileInfoList(listf); // Start PROOF TProof proof(“remote”); // Process your query Chain.Process(“selector.C”); Overview & Major Developments

17. Auto Loading of Plugins • Support for auto-loading libraries when an unknown class is being referenced. • The auto-loading mechanism reads the files $ROOTSYS/etc/system.rootmap, ~/.rootmap and ./.rootmap (via TEnv) to try to map the unknown class to a library. • If the library is found it, and the libraries on which it depends, are loaded. • The rootmap files are created with the rlibmap tool when executing "make map". • Example: in an interactive session, one can do directly without having to do TLorentzVector v; gSystem->Load(“libPhysics”); Overview & Major Developments

18. TMacro • This class allows for storing a C++ macro in a ROOT file. • In addition to being stored in a ROOT file a TMacro can be executed, edited, etc. TMacro m("Peaks.C"); //macro m with name "Peaks" is created //from file Peaks.C m.Exec(); //macro executed with default arguments m.Exec("4"); //macro executed with argument m.SaveSource("newPeaks.C"); TFile f("mymacros.root","recreate"); m.Write(); //macro saved to file with name "Peaks" Overview & Major Developments

19. Using PCRE for Reg Exp’s • new class TPRegexp which uses the Perl Compatible Regular Expressions library. • Well know, rich, regular expression syntax. • It is interfaced to TString and other class and methods now using TRegexp. • TRegexp will of course stay for backward compatibility. Overview & Major Developments

20. Dict Work Package New version of Reflex New version of rootcint rootcint  CINT rootcint -> Reflex ->Cintex ->CINT rootcint ->gccxml -> Reflex -> CINT Adapt PyRoot to Reflex Adapt CINT to Reflex Overview, Major Developments, Directions

21. Dictionaries : root only Root meta C++ CINT DS ROOT X.h CINT API rootcint CINT XDictcint.cxx Overview & Major Developments

22. Dictionaries : situation today REFLEX API lcgdict XDictlcg.cxx REFLEX DS X.xml Root meta C++ cintex gccxml CINT DS ROOT CINT API X.h rootcint CINT XDictcint.cxx Overview & Major Developments

23. Dictionaries : situation in the future Python CINT Root meta C++ Reflex/Cint DS ROOT CINT/Reflex API rootcint -cint XDictcint.cxx rootcint -reflex X.h rootcint -gccxml Overview & Major Developments

24. IO work-package Consolidation, Consolidation, Consolidation Support for STL collections More cases in auto schema evolution Better support for references Bitmap index TreeSQL Overview, Major Developments, Directions

25. ROOT I/O: STL Collections • ROOT now supports I/O of all STL containers • std::vector<T> std::list<T> std::set<T> std::deque<T>std::map<K,T> std::multimap<K,T> • And implicitly (through std::deque) std::queue<T> std::stack<T> • STL collections are saved in split mode • Objects are split (but: NOT if pointers) • Quick pre-selections on trees • Interactivity: Trees can be browsed • Save space (see $ROOTSYS/test/bench):std::vector<THit>:compression 5.38std::vector<THit*>:compression 3.37 Overview & Major Developments

26. Float, double and space…(1) • Math operations very often require doubleprecision, but on saving single precision is sufficient… • New data type: Double32_tIn memory: doubleOn disk: float or integer Overview & Major Developments

27. Float, double and space… (2) • Usage (see tutorials/double32.C): Double32_t m_data; // [min,max<,nbits>] • No nbits,min,max: saved as float • min, max: saved as int 32 bits precision explicit values or expressions of values known to Cint (e.g. “pi”) • nbits present: saved as int with nbit precision higher precision than float for same persistent space Overview & Major Developments

28. Float, double and space… (3) Increase precision Save space Overview & Major Developments

29. File types & Access in 5.06 user Local File X.xml TTreeSQL TFile TKey/TTree TStreamerInfo TSQLServer TSQLRow TSQLResult http rootd/xrootd Oracle Local File X.root MySQL Dcache Castor PgSQL RFIO Chirp SapDb Overview & Major Developments

30. Bitmap Indices • Bitmap indices are efficient data structures for accelerating multi-dimensional queries: • E.g. pT > 195 AND nTracks < 4 AND muonTight1cm > 12.4 • Supported by most commercial database management systems and data warehouses • Optimized for read-only data • However, because an efficient index may be as big as the data, we think that it is only appropriate for things like event meta data catalogues Overview & Major Developments

31. Query Performance - TTreeFormula vs. Bitmap Indices Bitmap indices 10X faster than TTreeFormula Overview & Major Developments

32. Data analysis with bitmap indices Event catalogue Bitmap index Direct use by PROOF slaves to select events query Event list Overview & Major Developments

33. Math work-package MathCore MathMore Minuit2 Smatrix Linear & Robust Fitter Splot Overview, Major Developments, Directions

34. MATH work-package : News • MathCore library with basic Math functionality • Basic Special and statistical functions • Physics and geometry vectors • MathMore library • C++ interface to function and algorithm from GSL • Extra math functions, Adaptive integration, derivation, root finders • Minuit2 • New OO implementation of Minuit • Interface to ROOT TVirtualFitter • Linear and Robust Fitter • sPlot Overview & Major Developments

35. New Math Libraries organization Overview & Major Developments

36. MATH work-package : Plan • Complete MathCore with Random numbers • Adapt ROOT classes to MathCore • TF1,2,3, Fitting • Virtual Fitter extensions • corresponding changes in ROOT fitting and roofit • Fully integrate and extend new Minuit • Fitting GUI • Box plots, qqplots • Many new tools required for LHC Physics analysis (PHYSTAT05 Oxford) Overview & Major Developments

37. Linear Fitter (0) • To fit functions linear in parameters • Polynomials, hyperplanes, linear combinations of arbitrary functions • TLinearFitter can be used directly or through TH1, TGraph, TGraph2D::Fit interfaces • When used directly, can fit multidimensional functions Overview & Major Developments

38. Linear Fitter (1) • Special formula syntax: • Linear parts separated by “++” signs: • “1 ++ sin(x) ++ sin(2*x) ++ cos(3*x)” • “[0] + [1]*sin(x) + [2]*sin(2*x) + [3]*cos(3*x)” • Simple to use in multidimensional case • “x0 ++ x1 ++ exp(x2) ++ log(x3) ++ x4” • Polynomials (pol0, pol1…) and hyperplanes (hyp1, hyp2, …) are the fastest to compute • By default, polynomials in TH1, TGraph::Fit functions now go through Linear Fitter • Data to be used for fitting is not copied into the fitter Overview & Major Developments

39. Linear Fitter (2) • Advantages in separating linear and non-linear fitting: • Doesn’t require setting initial parameter values • The gain in speed Overview & Major Developments

40. Robust fitting (0) • Least Trimmed Squares regression – extension of the TLinearFitter class • Motivation: least-squares fitting is very sensitive to bad observations • Robust fitter is used to fit datasets with outliers • The algorithm tries to fit h points (out of N) that have the smallest sum of squared residuals Overview & Major Developments

41. Robust fitting (1) • Highbreakdown point - smallest proportion of outliers that can cause the estimator to produce values arbitrarily far from the true parameters Graph.Fit(“pol3”, “rob=0.75”, -2, 2); 2nd parameter – fraction h of the good points Overview & Major Developments

42. Multivariate covariance • Minimum Covariance Determinant Estimator – a highly robust estimator of multivariate location and scatter • Motivation: arithmetic mean and regular covariance estimator are very sensitive to bad observations • Class TRobustEstimator • The algorithm tries to find a subset of h observations (out of N) with the minimal covariance matrix determinant Overview & Major Developments

43. Multivariate covariance • Left – covariance ellipses of a 1000-point dataset with 250 outliers • Right – distances of points from the robust mean, calculated using robust covariance matrix • High breakdown point • Indices of outlying points can be returned Overview & Major Developments

44. sPlot – A statistical tool to unfold data distributions Left – Projection plot cut on the likelihood ratio Excess of events – Signal? Background? Right – sPlot – no cut getting rid of background by statistical methods Signal!!! Overview & Major Developments

45. News in TH1 and TF1 • TH1: • Chi2 test • Mean & RMS error, skewness and kurtosis • TF1: • Derivatives (1st, 2nd and 3rd) • Improved minimization – a combination of grid search and Brent’s method (golden section search and parabolic interpolation) Overview & Major Developments

46. Graphics work-package zillions of micro/mini features http://couet.home.cern.ch/couet/POW_files/frame.htm reimplement (TGaxis) GL with new GUI GL for dynamic tracks GL in Pad Overview, Major Developments, Directions

47. Overview & Major Developments

48. TImageDump • Many extensions to the libAfterImage library to support line, marker and (filled) polygon drawing. Accessible via TASImage. • The new class TImageDump uses TASImage and derives from TVirtualPS to allow the saving of canvases in gif, jpg, png, tiff, etc., image formats in batch mode: • Or to display any gif, jpg, png, tiff in a canvas, do: $ root –b root [0] .x hsimple.C root [1] c1->Print("c1.gif"); TCanvas *c1; TImageDump *imgdump = new TImageDump("test.png"); c1->Paint(); imgdump->Close(); Overview & Major Developments

49. GL in Pad Overview & Major Developments

50. Overview & Major Developments