1 / 48

STACS: Storage Access Coordination of Tertiary Storage for High Energy Physics Applications

STACS: Storage Access Coordination of Tertiary Storage for High Energy Physics Applications Arie Shoshani, Alex Sim, John Wu, Luis Bernardo*, Henrik Nordberg*, Doron Rotem* Scientific Data Management Group Computing Science Directorate Lawrence Berkeley National Laboratory.

gada
Download Presentation

STACS: Storage Access Coordination of Tertiary Storage for High Energy Physics Applications

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. STACS: Storage Access Coordination of Tertiary Storage for High Energy Physics Applications Arie Shoshani, Alex Sim, John Wu, Luis Bernardo*, Henrik Nordberg*, Doron Rotem* Scientific Data Management Group Computing Science Directorate Lawrence Berkeley National Laboratory * no longer at LBNL

  2. Outline • Short High Energy Physics overview (of data handling problem) • Description of the Storage Coordination System • File tracking • The Query Estimator (QE) • Details of the bit-sliced index • The Query Monitor • coordination of “file bundles” • The Cache Manager • tertiary storage queuing and tape coordination • transfer time for query estimation

  3. Optimizing Storage Management for High Energy Physics Applications Data Volumes for planned HENP experiments STAR: Solenoidal Tracker At RHIC RHIC: Relativistic Heavy Ion Collider

  4. Particle Detection Systems Phenix at RHIC STAR detector at RHIC

  5. Result of Particle Collision (event)

  6. Typical Scientific Exploration Process • Generate large amounts of raw data • large simulations • collect from experiments • Post-processing of data • analyze data (find particles produced, tracks) • generate summary data • e.g. momentum, no. of pions, transverse energy • Number of properties is large (50-100) • Analyze data • use summary data as guide • extract subsets from the large dataset • Need to access events based on partialproperties specification (range queries) • e.g. ((0.1 < AVpT < 0.2) ^ (10 < Np < 20)) v (N > 6000) • apply analysis code

  7. Size of Data and Access Patterns • STAR experiment • 108 events over 3 years • 1-10 MB per event: reconstructed data • events organized into 0.1 - 1 GB files • 1015 total size • 106 files, ~30,000 tapes (30 GB tapes) • Access patterns • Subsets of events are selected by region in high-dimensional property space for analysis • 10,000 - 50,000 out of total of 108 • Data is randomly scattered all over the tapes • Goal: Optimize access from tape systems

  8. EXAMPLE OF EVENT PROPERTY VALUES I event 1 I N(1) 9965 I N(2) 1192 I N(3) 1704 I Npip(1) 2443 I Npip(2) 551 I Npip(3) 426 I Npim(1) 2480 I Npim(2) 541 I Npim(3) 382 I Nkp(1) 229 I Nkp(2) 30 I Nkp(3) 50 I Nkm(1) 209 I Nkm(2) 23 I Nkm(3) 32 I Np(1) 255 I Np(2) 34 I Np(3) 24 I Npbar(1) 94 I Npbar(2) 12 I Npbar(3) 24 I NSEC(1) 15607 I NSEC(2) 1342 I NSECpip(1) 638 I NSECpip(2) 191 I NSECpim(1) 728 I NSECpim(2) 206 I NSECkp(1) 3 I NSECkp(2) 0 I NSECkm(1) 0 I NSECkm(2) 0 I NSECp(1) 524 I NSECp(2) 244 I NSECpbar(1) 41 I NSECpbar(2) 8 R AVpT(1) 0.325951 R AVpT(2) 0.402098 R AVpTpip(1) 0.300771 R AVpTpip(2) 0.379093 R AVpTpim(1) 0.298997 R AVpTpim(2) 0.375859 R AVpTkp(1) 0.421875 R AVpTkp(2) 0.564385 R AVpTkm(1) 0.435554 R AVpTkm(2) 0.663398 R AVpTp(1) 0.651253 R AVpTp(2) 0.777526 R AVpTpbar(1) 0.399824 R AVpTpbar(2) 0.690237 I NHIGHpT(1) 205 I NHIGHpT(2) 7 I NHIGHpT(3) 1 I NHIGHpT(4) 0 I NHIGHpT(5) 0 54 Properties, as many as 108 events

  9. Opportunities for optimization • Prevent / eliminate unwanted queries=> query estimation (fast estimation index) • Read only events qualified for a query from a file (avoid reading irrelevant events)=> exact index over all properties • Share files brought into cache by multiple queries=> look ahead for files needed and cache management • Read files from same tape when possible=> coordinating file access from tape

  10. The Storage Access Coordination System (STACS) Query estimation / execution requests Query Estimator (QE) Bit- Sliced index User’s Application open, read, close Caching Policy Module Query Monitor (QM) Disk Cache file purging file caching Cache Manager (CM) file caching request File Catalog (FC)

  11. A typical SQL-like Query SELECT * FROM star_dataset WHERE 500<total_tracks<1000 & energy<3 -- The index will generate a set of files {F6 : E4,E17,E44, F13 : E6,E8,E32, …, F1036 : E503,E3112} that the query needs -- The files can be returned to the application in any order

  12. File Tracking (1)

  13. File Tracking (2)

  14. File Tracking query1 start query2 start query3 start All 3 queries

  15. Typical Processing Flow User Code Query Estimator Index Event Iterators Query Object Query Monitor Policy Module File Catalog Cache Manager STACS 1 new Query Quick Estimate 2 Execute Full Estimate 3 execute 18 done whichFileToCache 12retrieve 5 4request 6 14release FileID ToCache 13 7 stage 11staged 15purge 17purged 16purge Local Disk 8 file info 10 File Caching 9 File Caching Request

  16. The Storage Access Coordination System (STACS) Query estimation / execution requests Query Estimator (QE) Bit- Sliced index User’s Application open, read, close Caching Policy Module Query Monitor (QM) Disk Cache file purging file caching Cache Manager (CM) file caching request File Catalog (FC)

  17. Bit-Sliced Index:used by Query Estimator • Index size • property space: 108 events x 100 properties x 4 bytes = 40 GB • index requirements: • range queries (10 < Np < 20) ^ (0.1 < AVpT < .2) • number of properties involved is small: 3-5 • Problem • how to organize property space index

  18. QUAD-tree KD-tree indexing over all properties • Multi-dimensional index methods • partitioning MD space (KD-trees, n-QUAD-trees, ...) • for high dimensionality - either fanout or tree depthtoo large • e.g. symmetric n-QUAD-trees require 2100 fanout • non-symmetric solutions are order dependent

  19. Partitioning property space • One possible solution • partition property space into subsets • e.g. 7 dimensions at a time • Performance • good for non-partial range queries (full hypercube) • bad if only few of the dimensions in eachpartition are involved in query • S. Berchtold, C. Bohm, H. Kriegel, The Pyramid-Technique: Towards Breaking the Curse of Dimensionality, SIGMOD 1998 • best for non-skewed (random) data • best for full hypercube queries • for partial range (e.g. 3 out 100) close to sequential scan

  20. property 2 property 1 property n 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 1 1 1 0 1 1 1 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 1 1 1 0 1 1 1 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 1 1 1 0 1 1 1 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 . . . Bit-Sliced Index Solution: Take advantage that index need to be is append only • partition each property into bins • (e.g. for 0<Np<300, have 20 equal size bins) • for each bin generate a bit vector • compress each bit vector (run length encoding)

  21. Run Length Compression Uncompressed: 0000000000001111000000000 ......0000001000000001111111100000000 .... 000000 Compressed: 12, 4, 1000,1,8,1000 Store very short sequences as-is Advantage: Can perform: AND, OR, COUNT operations on compressed data

  22. 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 1 1 1 0 1 1 0 0 0 0 0 1 1 1 0 1 1 1 0 1 1 1 0 1 1 0 0 0 0 0 0 0 0 0 0 0 1 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Bit-Sliced Index Advantages Advantages: • space for index very small - can fit in memory • Need only touch properties involved in queries (vertical partitioning) • Need only touch bins involved min-max Query Estimation in memory only !!

  23. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Inner Bins vs. Edge Bins Edge bin Edge bin Range(x) Range(y)

  24. 0 1 0 0 0 0 0 1 0 0 0 1 0 0 0 0 1 0 0 0 0 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 . 0 0 0 0 0 . 1 0 0 0 0 1 0 0 0 0 . 1 0 0 0 0 0 0 0 1 0 1 0 0 0 0 1 0 0 0 0 Vertical Bit-Sliced Event partitions index list (on disk) (in memory) edge edge bin bin events in bin1 events in bin2 file list, and events properties that qualify range in each conditions Events in edge bin ~ 20-40 GB ~ 50-100 MB

  25. Experimental Results on Index • Simulated dataset (hijing) • 10 million events • 70 properties • property space • BSI: 2.5 GB • Oracle: 3.5 GB • index size • BSI: 280 MB (4 MB/property) • Oracle: 7 GB (100 MB/property) • index creation time • BSI: 3 hours (~2.5 min / property) • Oracle: 47 hours (~40 min / property)

  26. Experimental Results on Index • Run a “count” query (preliminary) • BSI: • 1 property: 14 - 70 sec (depending on size of range) • 2 properties: 90 sec (both about half the range) • => linear with # of bins touched • Oracle: • 1 property: comparable (counts only) • 2 properties: > 2 hours ! • use one index, loop on table • => Need to tune Oracle • run “analyze” on indexes, choose policy • bitmap index - did not help • => After tuning: 12 Min

  27. The Storage Access Coordination System (STACS) Query estimation / execution requests Query Estimator (QE) Bit- Sliced index User’s Application open, read, close Caching Policy Module Query Monitor (QM) Disk Cache file purging file caching Cache Manager (CM) file caching request File Catalog (FC)

  28. File Bundles:Multiple Event Components Files of Component A Files of Component B e1 e2 e3 Component A of event e1 Component B of event e1 e4 File 2 File 1 e5 e6 e7 e8 e9 File 4 File 3 File Bundles: (F1,F2: e1,e2,e3,e5), (F3,F2: e4,e7), (F3,F4: e6,e8,e9)

  29. A typical SQL-like Queryfor Multiple Components SELECT Vertices, Raw FROM star_dataset WHERE 500<total_tracks<1000 & energy<3 -- The index will generate a set of bundles {[F7, F16: E4,E17,E44], [F13, F16: E6,E8,E32], …} that the query needs -- The bundles can be returned to the application in any order -- Bundle: the set of files that need to be in cache at the same time

  30. File Weight Policy for ManagingFile Bundles • File weight (bundle) = 1 if it appears in a bundle, = 0 otherwise • Initial file weight = SUM (all bundles for each query) over all queries • Example: • query 1: file FK appears is 5 bundles • query 2: file FK appears is 3 bundles Then, IFW (FK) = 8

  31. File Weight Policy for ManagingFile Bundles (cont’d) • Dynamic file weight: the file weight for a file in a bundle that was processed is decremented by 1 • Dynamic Bundle Weight

  32. How file weights are usedfor caching and purging • Bundle caching policy • For each query, in turn, cache the bundlewith the most files in cache • In case of a tie, select the bundle with the highest weight • Ensures that a bundle that include files needed by other bundles/queries have priority • File purging policy • No file purging occurs till space is needed • Purge file not in use with smallest weight • Ensures that files needed in other bundles stay in cache

  33. Other policies • Pre-fetching policy • queries can request pre-fetching of bundlessubject to a limit • Currently, limit set to two bundles • multiple pre-fetching useful for parallel processing • Query service policy • queries serviced in Round Robin fashion • queries that have all their bundles cachedand are still processing are skipped

  34. . . . . . . . . . . . . . . . Managing the queues Files Being Processed Query Bundle File Queue Set Set Files in Cache

  35. File Tracking of Bundles Bundle (3 files) formed, then passed to query Bundle shared by two queries Bundle was found in cache Query 1 starts here Query 2 starts here

  36. Summary • The key to managing bundle caching and purging policies is weight assignment • caching - based on bundle weight • purging - based on file weight • Other file weight policies are possible • e.g. based on bundle size • e.g. based on tape sharing • Proving which policy is best - a hard problem • can test in real system - expensive, need stand alone • simulation - too many parameters in query profile can vary: processing time, inter-arrival time, number of drive, size of cache, etc. • model with a system of queues - hard to model policies • we are working on last two methods

  37. The Storage Access Coordination System (STACS) Query estimation / execution requests Query Estimator (QE) Bit- Sliced index User’s Application open, read, close Caching Policy Module Query Monitor (QM) Disk Cache file purging file caching Cache Manager (CM) file caching request File Catalog (FC)

  38. Queuing File Transfers • Number of PFTPs to HPSS are limited • limit set by a parameter - NoPFTP • parameter can be changed dynamically • CM is multi-threaded • issues and monitors multiple PFTPs in parallel • All requests beyond PFTP limit are queued • File Catalog used to provide for each file • HPSS path/file_name • Disk cache path/file_name • File size • tape ID

  39. File Queue Management • Goal • minimize tape mounts • still respect the order of requests • do not postpone unpopular tapes forever • File clustering parameter - FCP • If the file at top of queue is in Tapei and FCP > 1 (e.g. 5) then up to 4 files from Tapei will be selected to be transferred next • then, go back to file at top of queue • Parameter can be set dynamically 4 F(Ti) 3 F(Ti) 2 F(Ti) 5 1 F(Ti)

  40. File Queue Management • Goal • minimize tape mounts • still respect the order of requests • do not postpone unpopular tapes forever • File clustering parameter - FCP • If the file at top of queue is in Tapei and FCP > 1 (e.g. 4) then up to 4 files from Tapei will be selected to be transferred next • then, go back to file at top of queue • Parameter can be set dynamically 4 F4(Ti) 3 F3(Ti) 2 F2(Ti) 5 1 F1(Ti) Order of file service

  41. File Caching Order fordifferent File Clustering Parameters File Clustering Parameter = 1 File Clustering Parameter = 10

  42. Transfer Rate (Tr) Estimates • Need Tr to estimate total time of a query • Tr is average over recent file transfers from thetime PFTP request is made to the time transfer completes. This includes: • mount time, seek time, read to HPSS Raid,transfer to local cache over network • For dynamic network speed estimate • check total bytes for all file being transferredover small intervals (e.g. 15 sec) • calculate moving average over n intervals(e.g. 10 intervals) • Using this, actual time in HPSS can be estimated

  43. Dynamic Display of Various Measurements

  44. Query Estimate • Given: transfer rate Tr. • Given a query for which: • X files are in cache • Y files are in the queue • Z files are not scheduled yet • Let s(file_set) be the total byte size of all files in file_set • If Z = 0, then • QuEst = s(Y’)/Tr • If Z = 0, then • QuEst = (s(T)+ q.s(Z))/Trwhere q is the number of active queries F4(Y) T F3(Y) F2(Y) F1(Y)

  45. Reason for q.s(Z) 20 Queries of length ~20 minutes launched 20 minutes apart 20 Queries of length ~20 minutes launched 5 minutes apart Estimate bad - request accumulate in queue Estimate pretty close

  46. Error Handling • 5 generic errors • file not found • return error to caller • limit PFTP reached • can’t login • re-queue request, try later (1-2 min) • HPSS error (I/O, device busy) • remove part of file from cache, re-queue • try n times (e.g. 3), then return error“transfer_failed” • HPSS down • re-queue request, try repeatedly till successful • respond to File_status request with “HPSS_down”

  47. Summary • HPSS Hierarchical Resource Manager (HRM) • insulates applications from transient HPSS and network errors • limits concurrent PFTPs to HPSS • manages queue to minimize tape mounts • provides file/query time estimates • handles errors in a generic way • Same API can be used for any MSS, suchas Unitree, Enstore, etc.

  48. Web pointers • http://gizmo.lbl.gov/stacs • http://gizmo.lbl.gov/~arie/download.papers.html -- to download papers • http://gizmo.lbl.gov/stacs/stacs.slides/index.htm-- a STACS presentation

More Related