1 / 11

Analysis train - status

Analysis train - status. M. Gheata ALICE offline week, March 2010. Current train wagons. Several new tasks in all PWG in the train. Total number of tasks that can be currently run in a single session: 67 PWG2: 30, PWG3: 17, PWG4: 20+ (depending on how many jet finders)

dard
Download Presentation

Analysis train - status

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Analysis train - status M. Gheata ALICE offline week, March 2010

  2. Current train wagons • Several new tasks in all PWG in the train. Total number of tasks that can be currently run in a single session: 67 • PWG2: 30, PWG3: 17, PWG4: 20+ (depending on how many jet finders) • The number of tasks seem to be doubling every 6 months. We maintain one train macro that can run several different trains. There are in addition trains grouping PWG specific tasks which are run regularly by PWG responsible’s

  3. Output size • Per 10000 events (50 ESD files)

  4. Train monitoring page Several types of trains now grouped by input and output data type. Merging of final results delayed – to be made automatic

  5. alitrain – now a production user • Processing on demand simulation data • Produce AOD + several delta AOD for new productions • Delta AOD’s in several passes • Processing reconstruction passes as these become available • Running the pilot train – see presentation of Andreas • Semi-automatic system to run grid analysis • The idea is to define input data, define train composition, push the button, collect results • In real life not as ideal as that (merging and publishing results requires extra intervention)

  6. Getting results

  7. alitrain – current procedure • AN branch made every Monday • Based on user requests, we select train type • ESD+MC->AOD + deltas, ESD+MC->analysis • AOD->deltas+analysis, AOD+deltas->analysis • Data ESD->pilot train • Test and fix the branch -> tag + Savannah reports. Produce analysis input files via alien plugin, submit centrally via LPM (production manager) • Start local merging when analysis is done (manual) • Run Terminate, copy merged results and images back in alien, publish folder with results in the train monitoring page • Some parts need to be made automatic

  8. Learned lessons • The success of the procedure depends on some critical points (assuming well behaving user code) • Availability of data -> replication on more SE’s • Job duration -> Balancing the work to maximum few hours make success rate increase • Availability of resources -> bad configuration of sites and SE’s can be blockers • We cannot run all tasks in one train anymore • Job duration ~15 min/file -> max. 20-30 files per subjob • Analysis on AOD input is several times faster, MC processing is slow • Producing AOD’s and increasing the number of AOD-friendly tasks may become very important

  9. Automatic train profiling • New tools are available for profiling analysis in a train • We run them now automatically in the central train in the testing phase • A summary plot is showing eventual problems • Suspicious wagons are profiled individually using the new ProfileTask() method (see presentation of Andrei) • Actions are taken

  10. VM allocated by each task per event New tools VM allocated by all tasks in UserCreateOutputObjects and Terminate VM total

  11. Summary • The analysis train has now a big number of user wagons • Different analysis types already run in different trains • Scheduling according user requests • Work is done for making the train running procedure more automatic • Generating a merging JDL, publishing results • New profiling tools will allow finding problems faster

More Related