1 / 11

Peta-Cache: Electronics Discussion I Presentation Ryan Herbst, Mike Huffer, Leonid Saphoznikov

Peta-Cache: Electronics Discussion I Presentation Ryan Herbst, Mike Huffer, Leonid Saphoznikov Gunther Haller haller@slac.stanford.edu (650) 926-4257. Content. Collection of information so base-line is common Couple of slides how it seems to be done presently Part 1: Option 3: (Tuesday)

verena
Download Presentation

Peta-Cache: Electronics Discussion I Presentation Ryan Herbst, Mike Huffer, Leonid Saphoznikov

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Peta-Cache: Electronics Discussion I Presentation Ryan Herbst, Mike Huffer, Leonid Saphoznikov Gunther Haller haller@slac.stanford.edu (650) 926-4257

  2. Content • Collection of information so base-line is common • Couple of slides how it seems to be done presently • Part 1: Option 3: (Tuesday) • Alternative architecture with skim builder and event server • Part 2: Option 1 and 2 (Thursday) • Using Flash with minimum changes to present architecture • Using Flash in more changes to present architecture

  3. Collection of Info (to be checked) • Total number of BBR events: ~ 400 Million • Original dataflow events: ¼ to ½ Mbytes -> reconstruction results in 16-kbytes raw events • From raw events (16 kbytes) generate • Mini’s • Micros (~2.5 kbytes) (MC’s ~ 5kbytes) • Tags (512 bytes) • Raw (reconstructed) events: 400 Million x 16 kbytes = 6 Terrabytes • Fraction of them are needed most often • Estimate: • 200 skims total • Large skims use 10-20% of total # of events • Smallest use ~<1% of total • Question of how many of those are needed how frequently • Presently about 6-10 ms disk-access time • What is event analysis time • Very preliminary benchmark for analysis with little processing including decompression • 35 ms if data in memory • 70 ms if data on disk • What is the number as a function of users

  4. More info • Now: each type of *new* skim involves linking new code into the babar (to zeroth order, single image!) application, which entails a significant amount of time for creating a new build, testing and worrying about the effect of its robustness on the rest of operation etc. • What does Redirector do? (check): effectively maps an event *number* requested by the user to a file some where in the storage system (the combination of all the disks over all the tape system). Basically, if the user doesn't have an event in its xdroot server, it finds out which one has the file on the disk it serves and then asks the original server to make *copy* of it onto its *own* disks. If no server has the file then the user's xrootd server goes to the tape (HPSS) system and fetches it • Xrootd server: ~ $6k • Disk: ~$2/Gbyte (including cooling, etc)

  5. Raw Events - Tags – Skims Raw-Data (12-kbytes) Micros (2.5 KBytes) • Tag data and sometimes micro data is used for skims • Run over tag data with defined algorithm and produce a skim • Pointer skim: collection of tags with pointers to raw-data • Deep skim: contains also raw-data (e.g. for transfer to remote institution) • About 200 different skims. Time to get a new skim can be about 9 months due process (formalism to make sure skims are of interest to many) • One goal is to be able to produce easily and fast new skim Skim Tag data (512 Bytes) Tag data (512 Bytes) Tag data (512 Bytes) Collection of tags (with pointers to raw-data or can contain raw-data Alls tags, or all micros, etc Contains interesting characteristics and pointer to raw data

  6. Present Architecture (needs check) 4.8 TB Mass Storage (RAID-5) 3 TB usable 4.8 TB Mass Storage (RAID-5) 3 TB usable • Skims: go thru all tags (sometimes micro’s, but never raw-events) on disk to make collection of tags (which is a skim) • Client broadcasts to xrootd servers the first event to be fetched in a specific skim • If no xrootd servers responds (i.e. time-out) then go to tape and put event into xrootd server • So populate the disks with events which have been requested • If other xrootd has it, it copies it over • Still read ~2-6 Tbytes/day from tape? • Does xrootd server really know what it has in its disks? • is redirector involved for every event to find where event is from event number client is supplying? • What if disk is full? How recover new space? • Seems inefficient use of space since many copies 40 sets, to be extended to 80 sets Fibre-Channel xrootd server xrootd server Tape 1-G Ethernet Switch Redirector Client (1, 2, or 4 core) Client (1, 2, or 4 core) Up to 1,500 cores in ~ 800 units?

  7. Present Architecture (needs check) con’t 4.8 TB Mass Storage (RAID-5) 3 TB usable 4.8 TB Mass Storage (RAID-5) 3 TB usable • At the end, all tags and events which client desires on a single xrootd server • Why are all of those copied into one place, because the client or other clients reuses them many times later? • What is xrootd server really doing? • Xrootd server fetches event from its disk and delivers to client • So select clients for a server which uses same skims? • Client decompresses event for analysis 40 sets, to be extended to 80 sets FiberChannel xrootd server xrootd server Tape 1-G Ethernet Redirector Client (1, 2, or 4 core) Client (1, 2, or 4 core) Up to 1,500 cores in ~ 800 units?

  8. Goals (really need requirements) • Reduce time/overhead to make new skim • Goal that each user can make own “skim” quickly • Reduce time to get data to client • Presently analysis time and time to get next event is not concurrent but sequential. • To get next event it queries all xrootd servers • If one has it, it is copied into its (for that analysis) xrootd server • If not, its xrootd server gets it from tape • Issue is latency to get data from disk • Minimize duplicate storage

  9. Option with Skim Builder and Event Server (1) Tape Disk Storage Disk Storage • Skim builder and event server have access to all disks • Client gives skim builder code to build skim • Results are given to client • Client broadcast/multicast to event servers the request of events in the list (name of list and name of client) • Event server is selected which fits the criteria best • Maybe already has the list, may not doing anything, maybe whoever has most in the list is selected, that is flexible (or optionally a specific event server can be locked to a client (directly IO) • If no event in cache then after initial latency going to disk, get all the data • Event server may multicast its events to be delivered to all which requested any events in the list • Clients don’t have to get events in list in order • Any client can get events at 100 Mbytes/s independent of number of clients • Event server is cache box (e.g. 16 Gbytes of mem) • Pipe between event servers and storage is slower but averaged due to cache • Client could consume ~< 50 events/sec. Number of clients, e.g. 4,000. each events is 16kbytes. So average thruput 3.6Gbyte/sec To file system Skim builder (s) Event Server Event Server Ethernet/PCI-E/etc Optionally direct IO Client (1, 2, or 4 core) Client (1, 2, or 4 core) Client (1, 2, or 4 core) Up to 1,500 cores in ~ 800 units?

  10. Event Processing Center file system fabric disks pizza box as skim builder switch HPSS switch(s) switch (s) sea of cores fabric out protocol conversion pizza box in protocol conversion Event processing node

  11. Pizza box block diagram (In) PCI Express x16 PLX8532 x4 PPC 405 PLX8508 RLDRAM II IGbyte x4 x16 PLX8508 XILNIX XC4VFX40 (Out) PCI Express PLX8532

More Related