1 / 20

JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow

JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow. Contents. What is JIM? What is SAMGrid? How does JIM relate to SAMGrid? How does JIM work? Using JIM What will JIM do once complete? Deployment Plan. What is JIM ?.

sebastian-lane
Download Presentation

JIM and SAMGrid for CDF Mòrag Burgon-Lyon University of Glasgow

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. JIM and SAMGrid for CDFMòrag Burgon-LyonUniversity of Glasgow JIM for CDF

  2. Contents • What is JIM? • What is SAMGrid? • How does JIM relate to SAMGrid? • How does JIM work? • Using JIM • What will JIM do once complete? • Deployment Plan JIM for CDF

  3. What is JIM? Job Management Infrastructure and Information Management • Job Management Infrastructure is the framework allowing job submissions for execution on a cluster that has enough resources to complete the job. • Information Management is knowing what resources are available for use. JIM for CDF

  4. What is SAMGrid? • SAMGrid is a grid infrastructure whose goal is to allow globally distributed computing for current experiments, CDF and DØ. • Minos is strongly considering SAM • Communicating with LHC experiments CMS (Fermilab) and Atlas (Brookhaven) JIM for CDF

  5. How does JIM relate to SAMGrid? Complements Sequencial Access via Metadata to provide complete grid services Job Management Information and Monitoring Data Handling JIM for CDF

  6. How does JIM relate to SAMGrid? JIM for CDF

  7. How does JIM relate to SAMGrid? • JIM allows a user to submit jobs to SAMGrid and to access the output files on completion. • JIM chooses which resources will be used to execute a job. This decision is based on the nearest location of the required data files. • SAM executes the submitted job on the chosen cluster providing the I/O data management for the files. JIM for CDF

  8. How does JIM relate to SAMGrid • Condor MMS was expanded for SAMGrid • Broker can query SAM station to see how much data is already present. • Globus toolkit is used for job transfer and monitoring JIM for CDF

  9. How does JIM work? Submission Client Submission Client Match Making Service Match Making Service User Interface User Interface Broker Queuing System Queuing System Information Collector Information Collector JOB Data Handling System Data Handling System Data Handling System Data Handling System Execution Site #1 Execution Site #n Computing Element Computing Element Computing Element Storage Element Storage Element Storage Element Storage Element Storage Element Grid Sensors Grid Sensors Grid Sensors Grid Sensors Computing Element JIM for CDF

  10. How does JIM work? • Distinguishes grid level (global) scheduling (selection of a cluster to run) from local scheduling (distribution of the job within the cluster). • Distinguishes structured jobs (where the details are known to Grid middleware) from unstructured jobs (where the whole job is mapped onto a single cluster). JIM for CDF

  11. Using JIM – Submitting a job User creates a jdl file such as the example shown: sam_dataset = jpmm08-1file executable = retrieve.sh input_sandbox = /home_scotgrid/m/mlyon/test/testjob cpu-per-event = 1s job_manager = sam job_type = sam_analysis sam_universe = prd sam_experiment = cdf log = testjob.log output = testjob.out error = testjob.err arguments = Download Output Yet? group = test instances = 1 JIM for CDF

  12. Using JIM – Submitting a job • The job is submitted by typing: samg submit testjobfile.jdf • The progress of the job can be viewed by selecting the submission site from the list: http://samgrid.fnal.gov:8080 • Select the job from the list. Details of the job state are displayed • Once the job has been completed the output may be downloaded JIM for CDF

  13. Using JIM – Viewing Map JIM for CDF

  14. Using JIM – Viewing submission sites JIM for CDF

  15. Using JIM – Viewing submitted jobs JIM for CDF

  16. Using JIM – Downloading output JIM for CDF

  17. What will JIM do once complete? • Current functionality allows job submission and retrieval • Next steps: • Making the web-download secure • User support and defect fixing through testing phase • Roll out of SAMGrid to all CDF sites with available resources • Add more brokering criteria • Directors review of Run II computing has recommended the expansion of SAM to be a lab-wide product JIM for CDF

  18. Deployment Plan Two plans running in parallel: • Testing JIM with non-trivial submissions: • Large Monte Carlo Simulation • Reconstruction • Analysis • JIM software installations: • Glasgow ScotGrid IBM and CDFG Dell cluster • Oxford Dell cluster • Liverpool Dell cluster • UCL Dell cluster • Wisconsin In progress Complete Planned for October JIM for CDF

  19. Deployment Plan • Glasgow University has a complete installation of SAMGrid on both ScotGrid and the CDF cluster. This installation is being tested with Monte Carlo simulation. • Oxford University will be installed in the first week of October. • Other UK institutions and Wisconsin to follow JIM for CDF

  20. Credits • Thanks to Gabriele Garzoglio and the JIM team at Fermilab for providing material for this presentation. JIM for CDF

More Related