170 likes | 186 Views
Explore the data processing infrastructures at LIP-Lisbon, including testbeds, production infrastructure, and managing queues, supporting EGEE, Int EU Grid, and EELA projects. Learn about resources, virtual machine testbeds, and shared usage policies among group members. Discover insights into grid computing and water facts. Contact goncalo@lip.pt for more information.
E N D
Data processing infrastructures @ LIP-Lisbon Luso, 12th January 2008 Gonçalo Borges Grid Computing Team LIP-Lisbon
Testbeds @ LIP-Lisbon • Local test & development infrastructure • Virtual machine testbed • Aimed to LIP/Grid admins experiencies • Pre-Production infrastructure • Test EGEE middleware releases • Aimed for testing VO applications & grid middleware deployment and performance • Production infrastructure • EGEE, Int.EU.Grid, LIP FARM • Where grid users and LIP users run their applications goncalo@lip.pt LIP Grid Computing
VANESSA: Virtual machine testbed • After MONICA and AMANDA... • then it came VANESSA! • VANESSA: A physical machine supporting a virtual testbed • 4 VMs implemented via Xen technology and “LVMs partitions” • The images of the VMs can be recorded at any moment in time • OS/Software breaking is allowed • Used to develop new grid software • SGE integration with EGEE middleware • Used to test grid middleware and applications • Under different OS and environments goncalo@lip.pt LIP Grid Computing
Pre-Production Testbed • Small EGEE Grid Cluster • 1 Computing Element; 1 Storage Element • DPM with ~100 GB of storage space • 6 CPUs on the Worker Nodes • Intel Pentium IV 2.0 GHz, Intel Celeron 2.4 GHz • Used to test Grid middleware • After middleware certification and before deploying it in EGEE production infrastructure • Tries to simulate a real production enviroment • Same services as production, real users submitting jobs, ... • Middleware updated or re-installed almost every week • Also really used for important tasks • DILIGENT Press Release goncalo@lip.pt LIP Grid Computing
LIP Production Testbed / LIP Farm • LIP is envolved in 3 different grid projects • EGEE, Int.EU.Grid and EELA. • Support to more than 17 VOs • EGEE: atlas, auger, biomed, cms, compass, dteam, ops, swetest • Int.EU.Grid: ibrain, icompchem, imain, iplanck, itest, itut,iusct • EELA: edteam, eela • LIP groups giving resources to the LIP FARM • calo, cms, pet, compass, calo • Total of Resources = 88 cores • EGEE and Int.EU.Grid middleware installed everywhere • All machines may work as Grid Worker Nodes • Just a matter of configuration at the batch system level goncalo@lip.pt LIP Grid Computing
Int.EU.Grid CE (SGE sub) EGEE/EELA CE (SGE sub) lnlip01.lip.pt (SGE sub) WNs (SGE exec hosts) Resources architecture sge01.lip.pt (SGE Qmaster) goncalo@lip.pt LIP Grid Computing
Groups & resources • Grid and local queues • Most machines: Dual core AMD Opteron 2.2 GHz goncalo@lip.pt LIP Grid Computing
Managing queues • By default all jobs run in i386 queues • Unless a x86_64 queue is requested at submission time • slowlip_on_i386 • Very short time queue • Just to test if your application is running • Have to explicitely ask for it at submission time • fastlip_on_i386 • ~ 1/2 of the LIP IT machines • The other 1/2 is reserved to the grid • ~ 1/2 of each group machines • The other 1/2 is reserved to the group • fastlip_on_x86_64 • ~ 1/2 of each x86_64 group machines goncalo@lip.pt LIP Grid Computing
calolip (seq #0) compasslip (seq #0) fastlip_on_i386 (seq #1) Scheduling policies for LIP users • LIP users start by filling their group machines • When filled, start to fill other group shared machines • The next job to be run in a group machine is always from a group member, if there are group members pending jobs... • Sharing policies between groups members goncalo@lip.pt LIP Grid Computing
Usage (Number of submitted jobs) Max = 33093 jobs; Total = 158125 jobs goncalo@lip.pt LIP Grid Computing
LIP usage (Number of submitted jobs) Max (Grid) = 3841 jobs goncalo@lip.pt LIP Grid Computing
Max (Grid) = 72043 hours; Total = 173601 hours Usage (Cpu Time) goncalo@lip.pt LIP Grid Computing
Max (Grid) = 11402 hours LIP usage (Cpu Time) goncalo@lip.pt LIP Grid Computing
Usage (Cpu Time/Job) goncalo@lip.pt LIP Grid Computing
Usage (Memory/Job) goncalo@lip.pt LIP Grid Computing
Usage (“Efficiency”) goncalo@lip.pt LIP Grid Computing
“O organismo humano expele, diariamente, entre 2 a 3 litros de àgua, seja pela transpiração, pela respiração ou pela urina.” • “97% são águas salgadas; 2% são águas doces; 0,99% são águas doces subterrâneas e na forma de vapor. • Restam 0,01% de água doce disponível... grande parte já poluída!” • “Uma pessoa com peso e saúde normal pode viver até 30 dias sem comida. Essa mesma pessoa poderá morrer em 72 horas caso não possa beber água.” “Talk” que mais àgua meteu! goncalo@lip.pt LIP Grid Computing