dfos: tutorial

1. dfos: tutorial www.eso.org/~qc/dfos/ • Lesson 1: • dfos - vision and architecture • Directory structure • Installation • Tips and tricks • Lesson 2: • Association • Lesson 3: • Operations

2. Note added in proof: active buttons dfosExplorer: • use active buttons (‘refresh’, red button, config file editor) dfoMonitor: • compare web version (passive) and local (active) version • http://www.eso.org/~qc/monitor/DFO/

3. Association: principles • daily workflow: • processing • packing (increm.) • Both need: association (organization, classification, association = OCA) Where: $DFO_RAW_DIR/$DATE What? CALIB, FLAT, 1x1 binning, 4port readout, V_BESS filter With whom? Other raw FLATs, a master_bias

4. Association: principles Principles: • follow rules and properties (rather than: file names, nightlog entries, …) • createAB: reads rules from config files, knows about properties from property files: raw_list, mcalib_list, vcalib_list) • work on a data set • a DFO night (normally) • use time-match rule to associate calib products • CLOSEST: smallest |t| (default) • NEXT: smallest positive t (CALIBs later than SCIENCE) • ATTACHED: same OBS_ID + smallest |t| SCIENCE CALIB C A N

5. Association: principles • Use match rules: • common propertyobject1 has same property as object2:SCIENCE to find a MASTER_FLAT with the same filter • fixed-propertyobject1 has some property, match it with object2 having always a specified property:SCIENCE to find a MASTER_DARK with DIT = 2.0 sec

6. Association: definitions • Basic data sets • data set which does not need another data set (cannot split further) • synonyms: setting, setup, configuration • example: all VIMOS files with filter = U_BESS, 2x2 bin, fast readout, same quadrant U_BESS, 2x2, fast, Q1 U_BESS, 2x2, fast, Q2 V_BESS, 2x2, fast, Q1 U_BESS, 1x1, fast, Q1

7. Association: definitions • match keys • set of parameters to define a basic data set (INS, DET; OBS) • raw-raw • raw-product • product-product • packing • example: ins.grat.wlen + ins.grat.order + ins.grat.name + ins.slit.name • concept known already from the mcalib names!

8. Calibration map • … is: • a 2D map of mutual interdependencies of raw and product files (CALIB and SCIENCE) • a user-friendly but not yet machine-readable representation of association rules • translation into config files needed • sketch --> calibration map --> config files

9. Association Block • Text file with QUALIFIER CONTENT • Generalized Reduction Block (RB) or Set of Frames (SOF) • holds all association info (for processing and packing) • for tradition, it contains the RB and SOF explicitely • name: <root name of first ARCFILE>[_tpl].ab • stored: in $DFO_AB_DIR, later moved to $DFO_LOG_DIR/<DATE>

10. Cascaded and virtual calibrations • Columns in calib map: connected by nodes • horizontal connections: required calib products • if not existing at runtime of createAB: virtual, otherwise real • virtual mcalibs = predicted names of mcalibs, required to exist at AB execution time (but not earlier) • key concept to facilitate batch processing of complete nights and batches of nights in case of complex calibration maps • of course: if an AB crashes, all cascaded ABs will also crash! • Hence this works fine in a stable environment BIAS FLAT gimasterbias gimasterflat master Bias

11. dfos tools for association AB creation/editing: • createAB: AB creation • processAB: execute AB processing • updateAB: update AB content after product certification and distribution AB reading: • createJob: create AB execution file • updateDP: read ABs and create package increments • getStatusAB: provide AB status infrastructure: • listMcalib (mcalib list); extractAB (gets all associated files); helpAB

12. Association config files • config files: structured into three parts • config.assoc (‘the wholy gral’) • specific rule files • fitsreport_assoclists.cfg: fitsreport configuration config.assoc rawtype_1 (BIAS) ... rawtype_2 (FLAT) ... rawtype_3 (STD) ... BIAS.rules FLAT.rules STD.rules fitsreport_assoclists.cfg

13. config.assoc • config.assoc has: classification • definitions of grouping criteria: DPR, TPL keys • DO_CLASS: expected by recipes, not under your control! • RAW_TYPE: short form of DO_CLASS, under your control (must be uniquely ‘greppable’: e.g. IFLAT and DFLAT is fine, but not IFLAT and FLAT!) • PACK_DIR: string defining the name of the final package directory (OB scheme) • pipeline info: • recipe names • cascade sequence • event • packing info: • additional file types to pack (e.g. gif, ps, paf) • plus tool configuration (N_MCAL_LIST etc.)

14. Configuration: rules files • Rules files: • one per RAW_TYPE (e.g. BIAS.rules, FLAT.rules etc.) • has all information for a column of the association map • associated files: • [raw for processing; need not be defined] • raw_assoc (for packing) • mcalib (processing) • mcal_assoc (packing) • match keys for these file types • RAW_MATCH, RASSOC_MATCH, CALIB_MATCH, MASSOC_MATCH • optional recipe parameters • cascade info: pro_catg to be used for other data sets • trigger, waitfor events: used for scheduling DRS (REI)

15. Configuration: fitsreport…cfg • fitsreport_assoclists.cfg • fitsreport is used by createAB and listMcalib to create parameter lists • raw files: raw_list (filled by createAB, from scratch) • virtual mcalibs: vcalib_list (new entry for every predicted product under CASC in rule files; incremental fill/deplete) • real mcalibs: mcalib_list (created by listMcalib, from scratch) • used in technical (unformatted) mode, includes file names and path • has structure from fitsreport, plus some additional info

16. Demo • Demo: • run listMcalib and create mcalib_list • delete vcalib_list and observe it growing • run createAB and view raw_list • use dummy config.assoc with BIAS only; use 2005-01-10; check ABs • add FFLATs • add WAVE • check fresh ABs: SOF part, RB part • check processed, updated ABs: status