Streaming From A New & Bewildered User’s Point of View

1. Adam Lyon FNAL Tools Meeting 5/13/02 Streaming From A New & Bewildered User’s Point of View

2. What I’m used to • D0 Run 1 (Jet+MET Squark & Gluino search) • Virtual Trigger/Filter streams (Missing ET) in D0Dad • Keep list of processed runs • Submit list to big VAX database. Cross fingers. Get luminosity. • Remove events where VAX database reported no data. • If luminosity constants changed, rerun big VAX database job. Painful (“no data runs” could change) • CLEO (b  sg , dileption asymmetry) • One big stream was made (hadronic events) plus other specials • Skim the entire hadronic dataset (10s of millions of events II+II.V) • Luminosity kept in Begin Of Run information. Analysis job just adds up luminosity it sees. No corrections necessary. ALL BOR records saved to personal skims. No runs failed to process.

3. Streaming documents • Look under Detector/Data Streaming • Greg Landsberg “Documentation Page” last updated August 2001 • Vicky White: “Conceptual Design …” • Heidi Schellman: “Plain English explanation…” • Greg Landsberg: “Strawman proposal” / Talk / Use case • Jon Hays’: “Streaming and Level 3”

4. Streaming Basics • Avoid processing all of the data for your analysis • An event goes into one and only one stream (Exclusive streaming) • Files produced by online must start and end at a luminosity boundary • Events are categorized by L3 physics objects (“primal streams”) that triggered. Make a bitmap. Lots of combinations. • Physical streams are some combination of the primal streams. Manageable number. • Datalogger writes physical streams to separate tape families (though small streams may live in one family)

5. Simple one-to-one: 2nphysical streams Always n decisions No ambiguity Priority: Flexible (too flexible?) Can sometimes decide in < n decisions Can produce few streams, though must be careful that streams are not big You have to think! What if Muon & Electron? Primal categories  Physical Streams[see Hays’ document]

6. Tools • If users had to figure things out by hand, chance of getting it right would be small • Tool: Determine physical stream(s) from required triggers/filters • SAM produces list of files given streams

7. Luminosity implications • Bad luminosity blocks:User’s should not analyze events within those blocks from any stream • Will the framework skip these events? • What if luminosity blocks are declared bad after user processing? • Failed Reco processing:A file in a stream kills reco (SAM rejects the output file). Events from the affected luminosity blocks in other streams should not be analyzed. • Are the affected luminosity blocks declared bad? • What if a file from a small stream fails reco (large luminosity)? • How do we do this “cross stream communication?”

8. … Luminosity implications • Failed user processing • User has many jobs making many files “skims” that are merged at the end • Failed jobs are noted. Program is fixed/rerun or failed luminosity determined and subtracted (is that easy to determine?)

9. Conclusions • Streaming possibly complex: Tools needed to help users • Luminosity calculation potentially very complicated. Tools needed to give accurate result. • Speed?E.g. Vicky’s paper states that luminosity calculation should start from EVERY RAW file that fed desired stream(s). Are there database issues here?