1 / 15

JAM Job & Application Monitoring

JAM Job & Application Monitoring. G.Donvito G.Maggi M.Maggi. Overview . Introduction Goals The Alpha Release 0.0.2 The Data Flow C++ API and CL for Clients SCRAM Integration JOB Integration Perspectives. Introduction. Applications deal with large volume of data

lore
Download Presentation

JAM Job & Application Monitoring

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. JAMJob & ApplicationMonitoring G.Donvito G.Maggi M.Maggi

  2. Overview • Introduction • Goals • The Alpha Release 0.0.2 • The Data Flow • C++ API and CL for Clients • SCRAM Integration • JOB Integration • Perspectives

  3. Introduction Applications deal with large volume of data Application splitting Monitoring, logging and bookkeeping, quality assurance and interactive control for the very-end user

  4. Introduction • Easy data storage service installation • Simple/No Maintenance • Data transfer via basic C++ API • Command Lines for scripting • Smart and hidden data organization A friendly package

  5. Goals • Describe the application with tags • Organize the split applications in sets • Retrieve the values of the tags online • Store the infos in a psuedo-filesystem • Define rules on tags to classify the application as good/maybe/bad • Peek any remote file online • Very thin Forwarder for “hidden” WNs

  6. The alpha release 0.0.2 • Client-Server based on gSoap-2.7.0 • Storage with MySQL >= 4.1.0 (XML-based storage in quick progress) • Most of functionalities implemented (no output peeking yet) • The API is complete • Some protections need to be added This is not the production release

  7. The Data Flow • The application connect and send(tag,value) with non blocking command • The Receiver, forked by the application, update, compact and forward the tags at scheduled (configurable) intervals and on disconnect • The Forwarder, on gate keepers, collect the requests of all WNs, retry on failure, and peek file on WNs on user demands • The Server store and organize the tags • Time stamps are always stored with any tag to log any action.

  8. C++ API the simple send #include "JAMWriter/JAMWriter.h" #include "JAMWriter/JAMHandle.h" jam::JAMHandle* myhandle = jam::JAMWriter::getHandle(); myhandle->connect(url,"maggim","jamdb","SimpleTest"); myhandle->send<int>("MyIntTag",312) flaot myfl=3.1415927 myhandle->send<float>("MyFloatTag",myfl) std::string buffer="This is just a test"; myhandle->send<std::string>("MyStringTag",buffer); myhandle->disconnect(true); delete myhandle;

  9. C++ API the bulk send SCHEMA definition jam::ParameterSchema* scimpl = jam::ParameterHandle::schema(); scimpl->insert("var1",typeid(int).name()); scimpl->insert("var2",typeid(float).name()); scimpl->insert("var3",typeid(std::string).name()); myhandle->defineSchema(*scimpl); POLICY definition jam::ParameterPolicy* poimpl = jam::ParameterHandle::policy(); poimpl->buildPolicy("('var2' > 0 or 'var3'=='ciccio')"); poimpl->buildPolicy("'var1'==8"); myhandle->definePolicy(*poimpl); BULKSEND myhandle->bulksend(ParameterList);

  10. Command Lines from WN Neglecting the connection options, that can be set in a conf file sendJAM -t tag –v value -h Can be used to send shell values, and are interpreted as string -h option keeps the Receiver alive Special tags can be sent from UI to trigger special actions

  11. Command Lines from UI • JAMAdministrator • JAMRegisterUser • JAMChangePasswd • JAMListSetId • JAMListJobId • JAMGetSchema • JAMDumpSet • JAMStatus Hopefully self-explanatory Some will be replaced with filesystem-like command

  12. SCRAM Integration Scram allows the integration very easily • Add an ExetrnalRef to BuildFile • Add in .SCRAM/$SCRAM_ARCH/admin jam 0.0.2 0.0.2 • Add a jam_0.0.2.dat file with the relevant paths (root_n.n.n.dat is a good example…)

  13. JOB integration From UI the submission can be registers via special tags jamSend –t SpecialTag –v jobid A start/job of job can be sent within the job jamSend –t StartofJob –v whatever –h In the application (disconnect(true)) jamSend –t EndOfJob –v whatever

  14. ToInclude List for the beta release • TimeOut handling: short (retry) long (log file ) • Secure accounting • POSIX filesys command: jamls, jamcp, jamrm, jammv, jammkdir to replace existing commands • Implementation of the output peeking • Complete the Forwarder logic • A safer Server in high performance regime

  15. Perspectives • Gain experience in CMS environment • Gain experience with very-end users… • Improve the reliability of the server in the kHz regime • Fast cycles for minor releases: each 2- weeks • First beta release in May ‘05

More Related