1 / 45

Tier-2 in India for Alice Susanta Kumar Pal, VECC, Kolkata

International Workshop on Large Scale of Computing February 8-10, 2006 Variable Energy Cyclotron Centre 1/AF, Bidhan Nagar Kolkata – 700 064, India. Tier-2 in India for Alice Susanta Kumar Pal, VECC, Kolkata. What is GRID computing?

chelsea-yates
Download Presentation

Tier-2 in India for Alice Susanta Kumar Pal, VECC, Kolkata

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. International Workshop on Large Scale of Computing February 8-10, 2006 Variable Energy Cyclotron Centre 1/AF, Bidhan Nagar Kolkata – 700 064, India Tier-2 in India for Alice Susanta Kumar Pal, VECC, Kolkata IWLSC, Feb 8-10, 2006, Kolkata, India

  2. What is GRID computing? • Computational Grid is a collection of distributed, heterogeneous resources which can be used as an ensemble to execute large-scale applications. The emphasis is on : • Distributed supercomputing • High throughput & data intensive applications. • Large scale storage • High speed network connectivity • Why ALICE is interested in GRID? • 1 year of Pb-Pb running: 1 PBytes of data • 1 year of p-p running : 1 PBytes of data • Simulations : 2 PBytes • Total Data storage: 4 Pbtyes/year • ALICE computing requirements: • Simulations, Data reconstruction & analysis will use about 10,000 PC-years. • GRID is the solution for ALICE • Connect high performance computers from all collaborating countries with a high speed secured network. implementing one virtual environment that is easy for the “end user’’. IWLSC, Feb 8-10, 2006, Kolkata, India

  3. CERN – Tier 0 2.5 Gbps IN2P3 622 Mbps RAL FNAL Tier 1 155 mbps 155 mbps 622 Mbps Uni n Lab a Tier2 Uni b Lab c   Department  Desktop MONARC report: http://home.cern.ch/~barone/monarc/RCArchitecture.html The MONARC Multi-Tier Model (1999) IWLSC, Feb 8-10, 2006, Kolkata, India

  4. The opportunity of Grid technology CMS ATLAS Lab m Uni x regional group CERN Tier 1 Uni a UK USA Lab a France Tier 1 Tier3 physics department Uni n CERN Tier2 LHCb ………. Italy CERN Tier 0 Desktop Lab b Germany ………. Lab c  Uni y Uni b physics group   Tier 0 Centre at CERN LHC Computing Model2001 - evolving The opportunity of Grid technology The LHC Computing Centre IWLSC, Feb 8-10, 2006, Kolkata, India

  5. What is a Tier-2? The following is taken from theLCG RTAG 6on Regional Centre Category and Service Definition. A possible categorization scheme for regional centres could be based on service qualities as follows: Tier1 * CPU cycles (Grid enabled computing elements), advance reservation * Disk storage, resident and temporary (Grid enabled storage elements), advance reservation * Mass storage (Grid enabled storage elements), advance reservation * State-of-the-art network bandwidth, quality of service * Commitment to provide access to primary/master copy of data over lifetime of LHC * Commitment to provide long-term access to specific analysis data * Commitment to resource upgrades as required * 24/7 services and resource support * National support role * Training and user support * Interactive support for particular applications IWLSC, Feb 8-10, 2006, Kolkata, India

  6. Tier2 * CPU cycles (Grid enabled computing elements) * Disk storage, maybe temporary only (Grid enabled storage elements) * May have mass storage * Sufficient network bandwidth for inter-operability * A weaker commitment to provide access to data over LHC lifetime * A weaker commitment to provide long-term access to specific analysis data * A weaker commitment to resource upgrades * Focused user support * 24/7 service but with no guaranteed short-time "crash" response Tier3 * CPU cycles (Grid enabled computing elements) * Local storage (not necessarily Grid enabled) * Focused commitment to data access or resource upgrade * Only local user support * Focused services for agreed and day-by-day analysis activities * Local interactive support Tier4 * Enable Grid access * Provide experiment specific tools IWLSC, Feb 8-10, 2006, Kolkata, India

  7. RAW data delivered by DAQ undergo • Calibration and Reconstruction which • produce for each event 3 kinds of objects: • ESD object • AOD object • Tag object • Done in Tier-0 site. • Further reconstruction and calibration of RAW • data will be done at Tier 1 and Tier 2. • The generation, reconstruction, storage and • distribution of Monte-Carlo simulated data • will be the main task of Tier 1 and Tier 2. • DPD (Derived Physics Data) objects will be • Processed in Tier 3 and Tier 4. IWLSC, Feb 8-10, 2006, Kolkata, India

  8. Why Tier 2 ? • Tier-2 is the lowest level to be accessible by the • whole collaboration • 2. Each sub-detector of ALICE has to be associated with minimum Tier-2 because of large volume of calibration data • PMD and Muon Arm are the important sub-detectors of ALICE • We are solely responsible for PMD – from conception to commissioning IWLSC, Feb 8-10, 2006, Kolkata, India

  9. ALICE Setup LHC Utilization -- ALICE Cost 120 MCHF HMPID TOF TRD TPC Size: 16 x 26 meters Weight: 10,000 tons PMD ITS Muon Arm PHOS Indian contribution to ALICE : PMD, Muon Arm IWLSC, Feb 8-10, 2006, Kolkata, India

  10. ALICE Layout : another view IWLSC, Feb 8-10, 2006, Kolkata, India

  11. ALICE PMD (=2.3 – 3.5) Two Halves In Vertical When not in use Honeycomb counters for ALICE PMD are Modified edition of STAR PMD Two planes of honeycomb proportional counters 3 X0 thick lead converter Arranged in two halves in vertical plane Installed at z=360 cm from I.P. PMD in ALICE : Fully Indian Contribution IWLSC, Feb 8-10, 2006, Kolkata, India

  12. ALICE PMD Layout (Data taking position) Converter + Support Plate Super Module 8 Supermodules in 2 planes Total channels Preshower+Veto = 221,184 Unit Module 48 Nos. Total Cooling by air circulation Unit module components Honeycomb (4608 cells) Top, bottom PCBs IWLSC, Feb 8-10, 2006, Kolkata, India

  13. Components of a unit module 4 X 16 cells for one MCM Board TOP PCB (details) 32-pin connector Copper honeycomb Edge frame Bottom PCB IWLSC, Feb 8-10, 2006, Kolkata, India

  14. PMD unit module assembled Unit cell parameters 421 mm Cell cross-section 23 mm2 Cell depth 5 mm Centre-to-centre dist. 5 mm Cell wall thickness 0.4 mm Anode wire 20 µm dia (Au-plated tungsten) Anode-cathode distance 750 µm (on PCB) Unit Module, 4608 cells 421 mm X 260 mm Fabrication of unit modules at Kolkata, Jammu and Bhubaneswar Final assembly at Kolkata IWLSC, Feb 8-10, 2006, Kolkata, India

  15. PMD Parameters IWLSC, Feb 8-10, 2006, Kolkata, India

  16. muon tracking quadrants assembled in Kolkata IWLSC, Feb 8-10, 2006, Kolkata, India

  17. 0 0 0 0 ADC>0 0 0 Cell-to-cell gain calibration for PMD Need : To have uniform response throughout the detector In data look for single isolated cell ADC spectrum How to do : Test beam studies tell us – hadron response from our detector STAR PMD Isolated cell ADC spectra Pulse height spectra landau in a cell : Mean can be used for calibration Single cell is affected IWLSC, Feb 8-10, 2006, Kolkata, India

  18. Cell-to-cell gain calibration for PMD • ALICE PMD has 221184 cells to be calibrated • For this we need isolated cell ADC spectra for each cell • Minimum number of entries need for a good Landau distribution ~ 1000 • We need at least 1 Million events for calibration • So data volume ~200Kx0.25x4BytesX1Million ~ 200GB Although calibration is done once for a period of Running, it may be advisable to check the calibration constants time-to-time. IWLSC, Feb 8-10, 2006, Kolkata, India

  19. MANAS Calibration for Tracking Detectors of MuonSpectrometer With and Without MANAS Calibration – preproduction batch : gain spread 5% Expected Gain Spread in MANAS production : ~ 2.3 % Channel gain calibration may not be essential any more Production Batch IWLSC, Feb 8-10, 2006, Kolkata, India

  20. Expected Data rate for MuonArm • Physics data • Total number of readout pads in FMS: 1.1 X 106 • Trigger rate – 1 kHz • Average occupency ~ 2.5% • 1 month of Pb-Pb data == 300 TB Pedestal data: Every run ~ 5 MB (σ values of each channel needs to be known for analysis. One pedestal run/hour • 40 GB/month Electronic Calibration data The frequency will depend on the observed gain spread (if it is <2.5% then uncalibrated resolution will be satisfactory) Every run ~10 MB (for 2 point calibration) GMS data Comparable or less than pedestal data. Estimation from SINP group IWLSC, Feb 8-10, 2006, Kolkata, India

  21. Required Computing Resources for Tier 2 By 2006: # CPU(Intel P3) Disk Space (TB) Tier 0: 28800 727 Tier 1 + 2 7200 238 Total 6 Tier 1 centers and for each Tier 1 there will be several (~ 5-6) Tier 2 Centers. Tier 2 centres should have the capacity roughly 30% of Tier1 + 2 capacity. Bandwidth: Tier 2 centers rely entirely on associated Tier 1 centers for Reconstruction data storage. For efficient and fast analysis a bandwidth of 1.5 Gb/s is a reasonable value – although smaller values are viable. IWLSC, Feb 8-10, 2006, Kolkata, India

  22. Our Experience with Alice-Grid & PDC’04 • The following H/W and S/W infrastructures were used : • 8 Node Cluster consisting of dual Xeon CPU & 400GB Disk Space. • PBS Batch System with one Management server and eight Clients under OSCAR cluster management environment • ALICE Environment ( AliEn ) was installed • Data Grid has been registered at cern.ch • AliROOT, GEANT and other production related packages are tested successfully in both ways • Linked with CERN via 2Mbps available Internet link and Participated in PDC’04 IWLSC, Feb 8-10, 2006, Kolkata, India

  23. RDBMS Database Proxy ADBI DBD DBI User Application File & Metadata Catalogue API (C/C++/perl/java) LDAP Authenticator Resource Broker FS User Interface External Libraries Computing Element Modules Core CLI Config Mgr V.O. Packages & Commands SOAP/XML Storage Element GUI Monitor/Gatekeeper Web Portal Package Mgr (…) Logger Low level High level Our Experience with Alice-Grid & PDC’04 AliEn Architecture in General Core Middleware Core Components & Services Interfaces IWLSC, Feb 8-10, 2006, Kolkata, India

  24. Workload Management Reliable Job Submission Matchmaking Logging & Bookkeeping Data Management Replica Management Metadata Management Resource Management Gatekeeper (batch Interface) Storage (disk, tape, etc.) Database (SQL, …) Network Information System Service Discovery Service State Information Security Authentication Authorization Our Experience with Alice-Grid & PDC’04 Grid Services IWLSC, Feb 8-10, 2006, Kolkata, India

  25. Our Experience with Alice-Grid & PDC’04 IWLSC, Feb 8-10, 2006, Kolkata, India

  26. Preparing for next PDC • Changed the cluster software from OSCAR to QUATTOR • OSCAR • Performance degrades with more Nos. of Nodes ( > 32 Nodes ) • QUATTOR • Perfomence does not degrade with increase of nodes • Better performance with more Nos. of Nodes • Dedicated Band-Width for Tier2@Kolkata -> 4 Mbps • Separate Domain Name for Tier2; ‘tier2-kol.res.in’ • Addition of More Nos. of CPUs (48-Xeon) and Storage(7TB) • Installation of gLite is in progress IWLSC, Feb 8-10, 2006, Kolkata, India

  27. GAS WM DM TQ FTQ ACE PM FC JW (JA) CE SE L JC L RC CR (LSF,..) SRM Preparing for next PDC gLite Architecture GAS Grid Access Service WM Workload Mgmt DM Data Mgmt RB Resource Broker TQ Task Queue FPS File Placement Service FTQ File Transfer Queue PM Package Manager ACE AliEn CE (pull) FC File Catalogue JW Job Wrapper JA Job Agent LRC Local Replica Catalogue LJC Local Job Catalogue SE Storage Element CE Computing Element SRM Storage Resource Mgr CR Computing Resource (LSF, PBS,…) gLite Middleware Services API IWLSC, Feb 8-10, 2006, Kolkata, India

  28. Present Status VECC- Cluster : High Availability Quattor CERN Management Node ROUTER Switch 4Mbps Management Node (Stand-by) Gigabit Network Internet Cloud FireWall Tier-2@Kolkata SINP- Cluster : High Availability Quattor IWLSC, Feb 8-10, 2006, Kolkata, India

  29. Present Status IWLSC, Feb 8-10, 2006, Kolkata, India

  30. Present Status IWLSC, Feb 8-10, 2006, Kolkata, India

  31. Present Status grid.veccal.ernet.in graphs last hour sorted descending IWLSC, Feb 8-10, 2006, Kolkata, India

  32. Tier-II Centre for ALICE (Update on VECC and SINP Activities) • New Domain name “tier2-kol.res.in’ has been registered and work is going on • CONDOR Batch System is running with one server and eight Clients under QUATTOR cluster management environment • AliROOT, GEANT and other production related packages are tested successfully in both ways • ALICE Environment ( AliEn ) at present NOT running • Data Grid has been registered at cern.ch • Linked with CERN via 2Mbps available Internet link. • 4MBPS band-width is already installed and commissioned IWLSC, Feb 8-10, 2006, Kolkata, India

  33. Main Room Six Auto Switchable AC units • Main Room • Six Auto Switchable AC units • NW to VECC main Main Room Six Auto Switchable AC units 20ft x 19ft 30ft x 20ft VECC SINP UPS room 10ft x 19ft UPS room 20ft x 15ft UPS room Tier-II Centre for ALICE (Update on VECC and SINP Activities) • Infrastructure Status: • Fully Equipped GRID Room is Ready • (Sufficient to accommodate projected Tier-II H/W) IWLSC, Feb 8-10, 2006, Kolkata, India

  34. VECC Grid Computing Facility Room IWLSC, Feb 8-10, 2006, Kolkata, India

  35. Indian ALICE Collaborators • VECC, SINP - Kolkata • IOP, Bhubaneswar • Rajasthan Univ., Jaipur • Panjab Univ., Chandigarh • Jammu Univ., Jammu • AMU, Aligarh Proposed Indian ALICE-GRID Project CERN Chandigarh Kolkata Modem Router Modem Alice-Grid Router Jammu Jaipur Internet Back-Bone Router Modem Router Modem Gateway Router +Modem Aligarh Bhubaneswar Router Modem Router Modem IWLSC, Feb 8-10, 2006, Kolkata, India

  36. Contribution towards LCG-GRID PROJECTS • LCG-AliEn-SEInterface using GridFTP • 1st-July’03 to 18th Dec’03 • Status: • The project was successfully completed before schedule • Milestones of the Project were as follows : • Test Bed Installation for Grid Environment The configuration consists of one central server and two Sites. • Certification Authority Server has been installed A simple certification authority has been installed to generate certificates for authentication purpose. • Installation of GridFTP Library under AliEn The GridFTP daemon in.ftpd has been used as server and globus-url-copyhas been used as client • Development of AliEn-SE Interface via GridFTP These newly developed modules along with necessary GridFTP libraries and changes made in existing AliEn Code have been committed to CVS Server at CERN. • QualityAssuranceandTestEnvironment • ForAliEn-ARDA*Prototype • 23rd Dec’03 to 31st March’04 • The Project was successfully completed. • Milestones of the Project were: • Exploration and Design of Test Scripts using perl. • Implementation of Test Scripts for each Individual perl sub-module of AliEn. Individual perl sub-modules of AliEn code were tested for proper functionalities. It Generates a detailed report of the individual tests and maintains a log. • Validation of Test-Scripts and Procedures. • Testing Modules with perl Harnessing Environment. The Complete Suit was tested at CERN under perl Harnessing Environment for testing AliEn online and generating online consolidated report of the test. • Inline Documentation to the extent possible. ARDA Architectural Road-Map towards Distributed Analysis IWLSC, Feb 8-10, 2006, Kolkata, India

  37. To Summarize • Infrastructure is ready to accommodate Tier-2 H/W • Upgradation of present AliEn cluster with gLite is in progress • Middleware are installed in the Current Hardware with latest Scientific Linux OS platform • With limited resources, VECC took part in ALICE Production-Data-Challenge(PDC-2004) • Getting ready for Next PDC with upgraded infrastructure • Upgradation of CE and SE is in the process as per requirement • One FTE and 2 Engineers with 30% of time are engaged • Two more FTEs are approved Thank you IWLSC, Feb 8-10, 2006, Kolkata, India

  38. PMD V0 IWLSC, Feb 8-10, 2006, Kolkata, India

  39. PMD Split Position IWLSC, Feb 8-10, 2006, Kolkata, India

  40. Super Module and Unit module arrangement of ALICE PMD IWLSC, Feb 8-10, 2006, Kolkata, India

  41. Super Module Type B UM : - 12 - FEE Boards in a row - 6 Nos. of rows IWLSC, Feb 8-10, 2006, Kolkata, India

  42. Super Module Type A UM: 24-FEE Boards in a Row 3- Rows IWLSC, Feb 8-10, 2006, Kolkata, India

  43. Contribution towards LCG Project • LCG-AliEn Storage Element Interface (addendum-5) • 1st-July’03 to 18th Dec’03 • Value: 150 KCHF • Test Suite for perl harnessing with AliEn (addendum-6) • 23rd Dec’03 to 31st March’04 • Value: 150 KCHF • VECC as a part of the Alice Data-Challenge Team run offline production on the existing infrastructure. • Currently main emphasis is on participation in ALICE Physics Data Challenge 2005-06. • Future Projects: • Development of Test Environment for ADRA (Architectural • Roadmap for Distributed Analysis) code • Testing ARDA code under Perl Test harness guidelines • Part in EGEE ( Enabling Grid for E-Science Euro) prototype development IWLSC, Feb 8-10, 2006, Kolkata, India

  44. Our Experience with Alice-Grid & PDC’04 Security: Overview • User side • Getting a Certificate • Becoming a member of the VO • Server side • Authentication /CA • Authorization / VO IWLSC, Feb 8-10, 2006, Kolkata, India

  45. Our Experience with Alice-Grid & PDC’04 Security: AuthenticationOverview Once in every year Once in every year cert-signing CA grid-cert-request grid-cert-request cert-signing ca/url update SERVICE USER Once in life-time of the VO (only DN) host-request cert-request host-cert certificate convert CA-certificate cert.pkcs12 Every night/week crl Registration grid-proxy-init gridmap proxy-cert VO-LDAP mkgridmap Once in every 12/24 Hours host/proxy certificate exchanged Every night/week IWLSC, Feb 8-10, 2006, Kolkata, India

More Related