1 / 22

Filesytems and file access

Filesytems and file access. Wahid Bhimji University of Edinburgh, Sam Skipsey , Chris Walker …. This session:. 2 Themes: What are the demands made on storage by LHC data analysis – how might they change What solutions ( filesystems or other) help us to meet such demands.

jerom
Download Presentation

Filesytems and file access

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Filesytems and file access Wahid Bhimji University of Edinburgh, Sam Skipsey, Chris Walker …. Wahid Bhimji – Files access

  2. This session: 2 Themes: • What are the demands made on storage by LHC data analysis – how might they change • What solutions (filesystems or other) help us to meet such demands Wahid Bhimji – Files access

  3. Demands - Data access • LHC data analysis at Tier2s involves large files read from servers over the network. • Significant load and bottlenecks on storage. Tuning is affected by : • File access pattern • File size (e.g. whether can be cached in RAM) • Access method (direct local access, copy to WN etc.) • And more… • Problems can be “fixed” by the VO or site. Wahid Bhimji – Files access

  4. Examples – rfio access A reminder of STEP last year (Sam/ Glasgow) • “Rfio” access • Large data files. • “random” file access • Large rfio buffers –> Disk server Memory exhausted, network saturated Solution Employed: • Smaller RFIO buffers and copy to WN But ….. Wahid Bhimji – Files access

  5. copy to WN – moves the problem .. • More recent tests using FILESTAGER • Disk seek count hits max for drive Solution: • SSDs (see Sams talk)? • More WN disks? Or …: Smaller files/ Sequential access ?.... Wahid Bhimji – Files access

  6. Heart of the matter - ROOT I/O • Fundamental to most LHC analysis… • Need not be the bad guy. Root Data structure – “Tree” has multiple “Entries” – (physics “events”) and “Branches” (Objects – e.g. a Track). Wahid Bhimji – Files access

  7. Problems • Order that elements of Tree written into file • Ordering by branches –compresses well - good for extracting a single branch • BUT: in experiments data, there are lots of branches, branches are uneven sized, buffers are fixed (and small) so • Branches for the same event are scattered in the file Wahid Bhimji – Files access

  8. Aside on testing • “ROOT Test” - Read through the Tree with Tree->GetEntry, observe pattern with TTreePerfStats • Athena / CMSSW - Real analysis jobs • Hammercloud tests - Multiple continuous athena jobs Wahid Bhimji – Files access

  9. “Random” access – ROOT test on Atlas AOD file • RFIO access (128k buffer) • Scatter in read • Poor cpu Efficiency (~ 15 % ) Wahid Bhimji – Files access

  10. Tuning / Filesystem can make a difference • GPFS: 3 x faster than rfio access (in this test for this file) • Buffering Wahid Bhimji – Files access

  11. CMS – similar picture Brian Bockelman https://twiki.cern.ch/twiki/bin/view/Sandbox/CmsIOWork Wahid Bhimji – Files access

  12. Reordering helps • Reordered ATLAS AOD Tree using CloneTree(-1,"SortBasketsByEntry"); • Factor 5 quicker in reading through Entries (or rfio) Wahid Bhimji – Files access

  13. “new” ATLAS files • Recentlyboth CMS and ATLAS have been optimising no. of branches and writing baskets in event order. • Files are clearly reordered - CPU eff = ~90% (for this ROOT test) Wahid Bhimji – Files access

  14. HammercloudLocal Access CPU eff / event rate Indicative results – many things can effect these – further testing needed http://gangarobot.cern.ch/hc/1243/test/ http://gangarobot.cern.ch/hc/1246/test/ Wahid Bhimji – Files access

  15. Copy to WN with reordered files • HDD < 120 Seeks/s Much Happier! Wahid Bhimji – Files access

  16. TTree Caching • Group into a buffer all blocks from used branches, sort in ascending order and merge so read sequentially. • Learn from the first few events what branches are of interest. • Claims to reduce “typically by a factor 10000 the number of transactions with the disk and in particular the network with servers like xrootdor dCache.” (vector reads) • Configuration still under investigation • Size of cache / training • Behavior with rfio – inc. possible segfaults. • Not yet used “by default” in ATLAS – config likely to be site specific Wahid Bhimji – Files access

  17. Shows further improvements e.g. Dcache- GuenterDuckeck Wahid Bhimji – Files access

  18. Conclusions • I/O pattern of software can cost money for site • Much of this is closely coupled to storage choices and configuration • Close feedback from sites to VOscan help access method (and site tuning) to become (and stay) optimal • Hotch-Potch of evidence – could do with systematic study Wahid Bhimji – Files access

  19. Memory <--> TreeEach Node is a branch in the Tree Memory T.GetEntry(6) 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 T.Fill() 18 T Rene Brun - Sinaia Wahid Bhimji – Files access tr

  20. ROOT I/O -- Split/ClusterTree version Tree entries Streamer Branches Tree in memory File Wahid Bhimji – Files access

  21. Andrew Lahiff – RAL - CMS Reconstruction (4000 events) Skimming (4000 events) Wahid Bhimji – Files access

  22. Wahid Bhimji – Files access

More Related