Framework What has been happening?

FrameworkWhat has been happening? Jim Kowalkowski

Framework Quick Update • Still inherit from Package, implement one or more interface(s) • Executables still currently built using libraries and standalone object files. • Flow and Interface parameters have sensible defaults • Interfaces (Hooks) are automatically made available • Dataflow dispatching available • Release version string available

Main program object framework.o Framework related libraries libframework.a libio_packages.a libevpack.a libd0om.a ... Analyze package has a full list (it might have more than required) Package libraries libemreco.a libcpsreco.a one required for each package you use Package registration objects RegEMReco.o RegCPSReco.o one required for each package you use Parts Needed

Contents of Framework RCP PackageName = “Controller” // optional but useful string Packages = “read_event prereq conejet_5 conejet_7” // Define what these package instances mean RCP read_event = <example ReadEvent> RCP prereq = <example SomeCalorimeterStuff> RCP conejet_5 = <example ConeJet5>RCP conejet_7 = <example Conejet7> (That is all!)

Complex Framework RCP If you must change ordering or limit available interfaces bool OverrideInterfaces = true bool OverrideFlow = true string Interfaces = "decide process finish" string Flow = "decide process" string Packages = “read_event prereq conejet_5 conejet_7” RCP read_event = <example ReadEvent> RCP prereq = <example SomeCalorimeterStuff> RCP conejet_5 = <example ConeJet5>RCP conejet_7 = <example Conejet7>

Interfaces (Hooks) • Default order of execution is defined in • framework/interfaces/Order.hpp • A child controller has different ordering than the parent framework controller • Flow interfaces defined in • framework/interfaces/Flow.hpp • Standalone interfaces defined in • framework/interfaces/StandAlone.hpp

The Flow Interfaces (Default Order) • Generate - events are read in here (io_packages) • Decide - things like runInit discovered and handled here • Builder - mess with the event, like dropping chunks • Merge - combine some events (io_packages) • Filter - stop further processing of event if need be • Process - do reconstruction • Analyze - produce ntuples entries and fill histograms • Tag - mark individual chunks and the event for output • Finish - fill ntuple (flush) • Dump - debugging output for the event • Output - write the event

Child Flow (Default Order) • Filter • Process • Analyze • Tag • Dump

The Standalone Interfaces • fileOpen - file name that’s been opened • fileClose - file name that’s been closed • runInit - a run is starting • runEnd - a run is ending • luminBegin - a luminosity section start • luminEnd - a luminosity section end • jobSummary - end of the job processing

Simple Package Example .hpp: Class Processor : public fwk::Package, fwk::Process { public: Processor(fwk::Context* cxt):Package(cxt),Process(cxt) { … } ~Processor(); fwk::Result processEvent(edm::Event& e); // should write statusReport, reinitialize, and flush here too }; .cc: FWK_REGISTRY_IMPL(Processor,"$Name: $") Reg.cc: FWK_REGISTRY_DECL(Processor) This is all that is required!!!

Package Initialization • Do not use the Context* passed into the constructor, treat it as a void* and pass it to the base classes • Get at parameters via method Package::packageRCP( ) • Get access to the framework RCP file using method Package::frameworkRCP(). Useful if you want globally available flags or other things.

Using the Framework • Implement the processEvent() interface for reconstruction purposes • From Package, implement • statusReport( ) • Report summary of state (statistics) to cout • reinitialize(RCP r) • this should call setParameters(r), probably should clear histograms, etc. • flush( ) • Flush any buffers your package contains (histograms, log streams, etc.)

Please… • Stop putting Flow/Interfaces parameters into your framework RCP files unless you are really using them. They are not used. • Stop creating packages with unnecessary framework methods that have been obsolete for years. • Put good version strings into the registry macros (most everyone does this).

Version Information • From package code: • include framework/utilities/Functions.hpp • call getProductionVersion(), it returns an std::string • From command line: • my.exe -version • my.exe -show • Try “-mem” option, it will report, between package, each time memory usage increases.

ReadEvent emuCut tauReco multiLeptonCut susyAnalysis Using Framework Groups • Assume the following packages exist: • Single lepton cut package (emuCut) • Tau reconstruction package (tauReco) • Tri-Lepton cut package (multiLeptonCut) • SUSY analysis package (susyAnalysis) • Assume the packages use the proper hooks • We do not want to make any code changes • Want to tune Tau reco parameters • We want to run the following sequence:

Using Groups Standard Flow not good enough, we would need: Generate -> Filter -> Process -> Filter -> Analyze Could use framework group instead, revised sequence would be: Main: ReadEvent -> emuCut -> tauReco -> Controller2 Controller2: multiLeptonCut -> susyAnalysis Now default Flow ok and everything works just fine.

Group RCP Example Main.rcp: string PackageName = “Controller” string Packages = “reader emu_cut tau_reco susy” RCP reader = < example doread > RCP emu_cut = < example emu_cut > RCP tau_reco = < example tau_reco > RCP susy = < example dosusy > doread.rcp: string PackageName = “ReadEvent” emu_cut.rcp: string PackageName = “emuCut” tau_reco.rcp: string PackageName = “tauReco” multi_cut.rcp: string PackageName = “multiLeptonCut susy.rcp: string PackageName = “susyAnalysis” dosusy.rcp: string PackageName = “Controller” string Packages = “multi_cut susy” RCP multi_cut = < example multi_cut > RCP susy = < example susy >

Creating a “Hook” // File: MyHooks.hpp // helper macros with example use #include “framework/interfaces/Maker.hpp // hook with no arguments, named “myhook”, class MyHook, method dumbHook FWKI_NOARG(”myhook",MyHook,dumbHook) // hook with one arg, named “muochunk”, class MuoStuff, method muoChunk, // object type MuoHitChunk, managed by framework smart pointer FWKI_ARG(”muochunk",MuoStuff,muoChunk,MuoHitChunk,fwk::Ref<MuoHitChunk>) // hook with one const argument and access to the framework work queue FWKI_CONSTARG_WQ(”find",Find,findObj,edm::Event,d0_Ref<edm::Event>)

Using the New Hooks #include “MyHooks.hpp” class MyPkg : public MyHook, public MuoStuff, public Find { public: fwk::Result dumbHook(); fwk::Result muoChunk(MuoHitChunk& e); // can add objects to the framework work queue with findObj fwk::Result findObj(const edm::Event& e, fwk::WorkQueue& q); fwk::Result findObj(fwk::WorkQueue& q); ... }; fwk::Result MyPkg::findObj(const Event& e, fwk::WorkQueue& q) { Thandle<MuoRawData> raw = e.find_me_this_thing; fwk::Ref<MuoHitChunk> m = createChunk(raw); fwk::pushQueueBack(q, muo_msg_id, m); return Result::success; }

New Hook Notes • It is not difficult to create new hooks • The documentation is poor • There are plenty of examples in the framework package • Come see me if you are interested

Use of ErrorLog at D0 // using the protected member of the Package class… fwk::Result MyPkg::processEvent(edm::Event& e) { … if(error occurred) error_log(ELwarning,”FWK") << “help me” << '\n' << endmsg; // anywhere in your code… void func(edm::Event& e) { ErrorLog elog(“MyFuncPackage”); ... if(error occurred) elog(ELwarning,”DEATH”) << “help me” << ‘\n’ << endmsg; }

ErrorLog Usage • Both will produce the same framework context information • run/event numbers • currently executing package instance/method • The package name will be the name assigned in the RCP file for the first one • The package name will be “MyFuncPackage” for the second, each time it is used

Package Coding Rules • No caching event data between events • No global variables • No removing pointer from handle • use the handle as a pointer • Do not include headers from another package’s private directory • Do reconstruction in the processEvent( ) interface • Do not cast away const-ness

Error Handling • Catch exceptions thrown by your own algorithms • Framework catches all exceptions that exit your package; logs the message and program dies • Using return codes • Currently any package returning failure aborts further processing of that event (from that group) • Proposal for changes given at earlier talk

Best Practice • Do not put using namespace in any header • Use forward declarations wherever possible • Prefer ++it to it++ • Do not expose details unnecessarily • e.g.: Do hide bit encodings from users • Use meaningful member names, rather than slot numbers in an array • Watch out for return by value, could be performance killer. • Use “fillme” method.

“fillme” Method Not very good: std::vector<int> X::getValues() { std::vector<int> ret; … ret.push_back(x); ... return ret; } Better: size_t X::getValues( std::vector<int>& fillme ) { … fillme.push_back(x); … return num_of_things_in_fillme; }

Discussion Item • Framework package to configure it? • Produce statistics? • Separate messages into different files? • Destination that sends email? Does anything need to be done about control of the ErrorLogger?

Dataflow Dispatcher • Processing based on input objects becoming available • DFPackages are Packages that can declare input/output object names • Get invoked with a collection of all the inputs they are waiting for. • Dispatcher creates network of packages based on inputs and outputs • Dispatcher verifies that packages will get input objects • Supports nesting of DFPackages • Can pass arbitrary objects to DFPackages, not just Events

More About Dataflow • Trigger simulator is using it • Can do many, many things that need to be documented • Standard hooks work (runInit/etc.) • Works with standard Packages (io_packages) • See me if you are interested

Trivial Dataflow Example Class L2Worker: public fwk::DFPackage { void L2Worker::inputs(list<string>& ilist) { ilist.push_back(“event”); ilist.push_back(“l2inp”); } void L2Worker::outputs(list<string>& olist) { olist.push_back(“l2global”); olist.push_back(“l2out”); } fwk::Result DFProc::ready(DataStore& dstore) { … vector<L2Inp*> inps; edm::Event* e = getEvent(dstore); queryPointers(dstore,inps,”l2inp”); produceItem(dstore,new L2Global,”l2global”); produceItem(dstore,new L2out,”l2out”); } };

Dataflow Configuration Example ReadEvent event * l1em L2Presenter L2Source l1caltt l1em l1caltt l1muo l1muo L2EM L2CalTT L2Muo l2global l2global l2l3data l2l3data l2global l2em l2l3em l2caltt l2l3caltt L2Global l2l3data l2muo l2l3muo l2l3global l2l3data * L2Collator

Dataflow DataStore queryPointers(DataStore& d,vector<X>& out, Action::Id id=0) queryAliases(DataStore& d,vector<X>& out, Action::Id id=0) query(DataStore& d,vector<X>& out, Action::Id id=0) produceSignal(DataStore& d, Action::Id) produceManaged(DataStore& d, ITEM managed_obj, Action::Id id, Action::Id alias=0) produceManaged(DataStore& d, ITEM managed_obj, Action::Id id, ITER first_alias, ITER last_alias) produceItem(DataStore& d, ITEM* obj, Action::Id id, Action::Id alias=0) produceItem(DataStore& d, ITEM* obj, Action::Id id, ITER first_alias, ITER last_alias) produceRaw(DataStore& d, ITEM* obj, Action::Id id, Action::Id alias=0) produceRaw(DataStore& d, ITEM* obj, Action::Id id, ITER first_alias, ITER last_alias)

DataStore Filling • produceRaw • Takes a pointer to any object. No memory management. Object ownership is not passed to framework • produceItem • Takes a pointer to any object. Ownership is passed to framework (and lifetime control) • produceManaged • Takes a smart pointer to any object. Ownership is through users smart pointer

Framework What has been happening?