Distributed Data Analysis

1. Distributed Data Analysis Intel discussion meeting CERN 5 Oct 2003 René Brun CERN ROOT for Data Analysis

2. Intel: Distributed Data Analysis

3. ROOT Trends Parallelism on the GRID Batch/Interactive Access to Catalogs Resource Brokers Process migration Progress Monitors Proxies/caches Virtual data sets PAW or ROOT like PAW Efficient Access to large and structured event collections Interaction with user & experiment classes Histogram Ntuple viewers Data Presenters Intel: Distributed Data Analysis

4. Memory <--> TreeEach Node is a branch in the Tree Memory T.GetEntry(6) 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 T.Fill() 18 T Intel: Distributed Data Analysis tr

5. 8 leaves of branch Electrons A double-click to histogram the leaf 8 Branches of T Intel: Distributed Data Analysis

6. The Tree Viewer & Analyzer A very powerful class supporting complex cuts, event lists, 1-d,2-d, 3-d views parallelism Intel: Distributed Data Analysis

7. 0 1 2 3 4 5 0 6 0 1 7 1 2 2 8 3 3 9 4 4 10 5 5 6 6 11 7 7 12 8 8 13 9 9 10 14 10 11 15 11 12 16 12 13 13 17 14 14 15 18 16 15 17 16 18 17 18 Tree Friends Entry # 8 Public read User Write Public read Intel: Distributed Data Analysis tr

8. Tree Friends Collaboration-wide public read Processing time independent of the number of friends unlike table joins in RDBMS Analysis group protected user private x Root > TFile f1(“tree1.root”); Root > tree.AddFriend(“tree2”,“tree2.root”) Root > tree.AddFriend(“tree3”,“tree3.root”); Root > tree.Draw(“x:a”,”k<c”); Root > tree.Draw(“x:tree2.x”,”sqrt(p)<b”); Intel: Distributed Data Analysis

9. Data Volume & Organisation 100MB 1GB 10GB 100GB 1TB 10TB 100TB 1PB 1 1 5 50 500 5000 50000 TTree TChain A TFile typically contains 1 TTree A TChain is a collection of TTrees or/and TChains A TChain is typically the result of a query to the file catalogue Intel: Distributed Data Analysis

10. Data Volume & Processing TimeUsing technology available in 2003 100MB 1GB 10GB 100GB 1TB 10TB 100TB 1PB ROOT 1 Processor P IV 2.4GHz 2003 : Time for one query using 10 per cent of data 1” 10” 1’ 10’ 1h 10h 1day 1month batch Interactive PROOF 10 Processors 1” 1” 10” 1’ 10’ 1h 10h 1day 10days PROOF 100Processors 1” 1” 1” 10” 1’ 10’ 1h 10h 1day PROOF/ALIEN 1000Processors 1’ 10’ 1h 10h Intel: Distributed Data Analysis

11. Data Volume & Processing TimeUsing technology available in 2010 100MB 1GB 10GB 100GB 1TB 10TB 100TB 1PB ROOT 1 Processor XXXXX 2010 : Time for one query using 10 per cent of data 1” 1” 10” 1’ 10’ 1h 10h 1day 10days batch Interactive PROOF 10 Processors 1” 1” 1” 10” 1’ 10’ 1h 10h 1day PROOF 100Processors 1” 1” 1” 1” 10” 1’ 10’ 1h 10h PROOF/ALIEN 1000Processors 1’ 10’ 1h Intel: Distributed Data Analysis

12. Interactive Local Analysis • On a public cluster, or the user’s laptop. • Tools like PAW or successor are used for visualization and ntuples/trees analysis. Intel: Distributed Data Analysis

13. GRID: Interactive AnalysisCase 1 • Data transfer to user’s laptop • Optional Run/File catalog • Optional GRID software Optional run/File Catalog Analysis scripts are interpreted or compiled on the local machine Trees Remote file server eg rootd Trees Intel: Distributed Data Analysis

14. GRID: Interactive AnalysisCase 2 • Remote data processing • Optional Run/File catalog • Optional GRID software Optional run/File Catalog Analysis scripts are interpreted or compiled on the remote machine Trees Remote data analyzer eg proofd Commands, scripts Trees histograms Intel: Distributed Data Analysis

15. GRID: Interactive AnalysisCase 3 • Remote data processing • Run/File catalog • Full GRID software Run/File Catalog Analysis scripts are interpreted or compiled on the remote master(s) Trees slave Trees Trees Trees slave Remote data analyzer eg proofd slave Commands, scripts slave Trees Histograms,trees Trees slave Trees Trees slave Intel: Distributed Data Analysis