1 / 29

Infrastructure and Basic Grid Services

Infrastructure and Basic Grid Services. Working Group Kick-Off Meeting SWITCH, 2008 January 15. Agenda. 09:45 - 10:15: Charter of the WG, relation to EB and other WGs

jaguar
Download Presentation

Infrastructure and Basic Grid Services

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Infrastructure and Basic Grid Services Working Group Kick-Off Meeting SWITCH, 2008 January 15

  2. Agenda • 09:45 - 10:15: Charter of the WG, relation to EB and other WGs • 10:15 - 11: 45 : From every institution a short summary what they currently have installed and what they are planning to do in 2008. • 11:45 - 12:30: Exchange of middleware experiences: Every participant, who has first hand experiences with a certain grid middleware gives a short presentation with the strengths and weaknesses of this particular middleware from his point of view. • 12:30 - 13:30 Lunch • 13:30 - 14:30: Discussion, which middlewares should be supported and how • 14:30 - 16:00: Next steps: Identification of sub-projects and assignment to groups of 3-4 persons

  3. Charter – Focus/Purpose • One of the mission statements of SwiNG is the establishment of a sustained Swiss Grid Infrastructure. There are two basic components necessary to do so: • a set of shared hardware resources (computing, data and instruments) and • a set of Grid services linking the resources and making them accessible for use. • The resources available to the Swiss Grid Infrastructure are provided by the individual member institutions and their sub-units. Concerning the first point, the manner by which the resources are shared has to be established and adapted to the constraints of each individual partner. In addition, mechanisms have to be set up by which sufficient incentive is created to share and keep sharing the resources, as well as to attract new shared resources. • Concerning the second point, the basic Grid middleware services that are independent of the applications running on the Grid have to be set up and operated by the SwiNG member institutions. The services have also to be defined, categorized, monitored and maintained, and the exact mechanisms how to do so defined.

  4. Scope • This activity builds on previous Swiss effort [R1-2]. In the scope of SwiNG we solicit the active participation of all members that have any kind of shareable resource. The initial work of the WG will define the detailed mechanisms of the resource sharing. This includes drafting the necessary service level agreements for the participating partners, the detailed mechanisms of accounting, access, monitoring, etc. Processes have to be defined and set up by which middleware is selected, deployed and operated. • Strategic decisions concerning the infrastructure and basic Grid services are taken on the level of the Association, i.e. the Executive Board and the Assembly. The WG is responsible to prepare the necessary material for such decisions. • The ISBG-WG may re-scope itself at a later time into smaller subsets later, once the bootstrapping of the infrastructure is done and the processes are understood. • The scientific requirements will be defined by other WGs but collected by us.

  5. Goals • There are many examples of existing Grids and other countries’ national infrastructures that can be taken as basis for this WG. We are in the comfortable position to have a lot of experience to draft from ([R3-R7] and more), picking the best concepts for the Swiss academic landscape, which has its own peculiarities. • The WG is expected to define the processes and procedures to set up and operate the initial Swiss Grid Infrastructure, and actually putting them to practice. A very important aspect of the WG is to define the tasks that need dedicated funding to take place.

  6. Inventory • From every member group a short summary what they currently have installed and what they are planning to do in 2008. • What is there in terms of resources (compute, data, instruments) • Current projects • Interests • Goals for 2008 • Experience, lessons learned from previous projects

  7. SWITCH • What is there in terms of resources (compute, data, instruments) • Grid Team is a subset of the Middleware team; 5 people (not all full-time) • gLite-based installation running central services • WMS, File Catalogs, VOMS, Info systems • Current projects • EGEE-II activities – shibboleth interoperability in JRA1 • Middleware security focus • Goals for 2008 and beyond • EGEE-III starting this spring • Core focus of SWITCH in general is on the national scope, hence very active in SwiNG, importance of national scope will grow • Provide central services • PKI • AAI / SLCS • Non-browser based applications for Shibboleth • NO computing resources • AAA Grid Projects – define projects, attract funding

  8. PSI • What is there in terms of resources (compute, data, instruments) • Using gLite, previously also AliEn • Current projects • LCG Grid, Science Driven • Data analysis is extremely important (several PB/year to analyze) • System for data sharing • Securing data access • Goals for 2008 • Detector is giving data this year, ramping up • Entering production this year, supply a stable production Grid • Building an LCG Tier3 at PSI, 120CPUs/70TB • Interests • Provide access to local resources • Experience, lessons learned from previous projects • Small dev team makes for flexible testing • Selection of supported requirements is important • There may be only few questions concerning science requirements (cpu, data, instrument access) • gLite can cover data access aspects, but support is very expensive – don’t know of an alternative at this scale

  9. EPFL – DIT • What is there in terms of resources (compute, data, instruments) • Desktop Grid with 300 CPUs multi-platform (45% linux, 50% windows, rest MacOSX) • Condor-based, only available at nights and week-ends • Current projects • Condor Desktop grid • Interests • Providing resources to the users inside and outside • Users are physics and chemistry today • Goals for 2008 • Add more compute nodes to the Condor desktop grid (there are 3000 unused nodes yet) • Improve NorduGrid interface • Experience, lessons learned from previous projects

  10. Uni BE - IT • What is there in terms of resources (compute, data, instruments) • Ubelix cluster (500cores) • Several science users • NorduGrid is installed • gLite has been tried but is not available • Current projects • Parallel filesystem: Lustre • Interests • Providing standardized access • Goals for 2008 • Enlarge cluster with more hardware • Access to cluster should be standardized Swiss-wide • Experience, lessons learned from previous projects

  11. FGCZ • What is there in terms of resources (compute, data, instruments) • There are a lot of instruments (mass spectrometry, etc) • Cluster that is used is at the IT • Several TB of data • Current projects • Data centric • B-Fabric infrastructure • Based on open source projects to do reliable data mgmt • Store data and annotate data (metadata) • Share data – security, access • Avaki (sybase) • However this is commercial now • Interests • Data management, data sharing issues, not only CPU cycles • Build basic services allowing to provide secure access to distributed data • Promote B-Fabric • Goals for 2008 • Extension of B-Fabric wrt workflows, new instruments, etc • Working with Unilever R&D and CSI-D (they use B-Fabric) • Promote B-Fabric, establish a community, extend opensource subsystems (e.g Cocoon). Allows generating new interfaces. • Experience, lessons learned from previous projects • Avaki • Grid Transactions (GGF)

  12. UZH-OCI Comp chem and Grid computing • What is there in terms of resources (compute, data, instruments) (all linux) • Dedicated cluster for Grid projects 2 frontend nodes 8 compute nodes (dual core) [TEST] • Special purpose servers • Production-style cluster > 100 nodes for comp.chem, not shared [PRODUCTION] • Dedicated infrastructure for SwissRe project • Globus4, Nordugrid, Unicore, and a lot more.. • Websphare, DataSynapse • Current projects • SwissRe collaboration on financial loss prediction (insurance) • SwiNG • PRAGMA • SUK project dealing with NorduGrid and AAI • SEPAC • Chemomentum (grid workflow based infrastructure) FP6 EU project • And a few more related projects like GEMSTONE, all related to comp.chemistry, workflows, web services, etc • Interests • Provide infrastructure to adapt application code to Grid infrastructures • Provide a real production-style Grid infrastructure to user groups (usable and stable), bridge the gap between application and infrastructure • Goals for 2008 • Double nodes for dedicated grid cluster • Continue SwiNG involvement and all other projects • Attract/secure more 3rd party funding • Experience, lessons learned from previous projects • Funding drives everything • Data will become more important (complexity for us, not size)

  13. UZH-IT • What is there in terms of resources (compute, data, instruments) • Hardware: 384dualCPU matterhorn cluster, attached experimentally to swiss bio grid • United Devices grid (currently unused) • Current projects • Small test-cluster (32x8 nodes) two different fast interconnects, possibly can be used for MPI-grid testing + 10TB disk • Interests • Serve the users • Make grid services accessible for non-experts (biologists, economists, ..) • Applications for UD • MPI on the grid • Goals for 2008 • Build up 2000CPU(or core) cluster for medicine faculty • Make a few blade centers available and accessible by grid users • Consolidation of data storage infrastructure, hierarchical storage managers • Experience, lessons learned from previous projects • UD is administrator friendly but not coder friendly • 64bit opteron – seems still be problematic for Nordugrid and shibboleth

  14. UniGe – HEP • What is there in terms of resources (compute, data, instruments) • 61 computers, 184cores . Fileservers, 3 service nodes. 75TB disk. • SLC4 plus solaris. Using Torque and Maui, with Nordugrid • Current projects • LCG-ATLAS • Interests • Willingness to share is there • Goals for 2008 • Get the current system into production. • Experience, lessons learned from previous projects

  15. UniBe – LHEP • What is there in terms of resources (compute, data, instruments) • 34 worker nodes on SLC4, NorduGrid frontend, 10TB, and two more fileservers with 22TB • Rely on the IT services (Ubelix) • Current projects • LCG-ATLAS • Interests • Increase visibility our network, and the visibility of our work • Access to Grid services that may be needed in the future, so that we don’t need to run them ourselves • Access to more resources if possible, sharing own resources as possible (depends on effort as well) • Goals for 2008 • Have the Swiss computing infrastructure ready for data taking this year to ATLAS • Experience, lessons learned from previous projects

  16. HES-SO EIG • What is there in terms of resources (compute, data, instruments) • 3 people working half-time. 300CPU windows/linux available 24/7 • Current projects • Development of XtremWebCH • Porting / developing applications on this platform (GAMESS, CONES, Phylip, Neuroweb) • Interests • Provide these resources to all applications (life science and medical in particular) • Integration of XtremWebCH with other mw, interoperability • Goals for 2008 • Extensions to XtremWebCH • Bridging with gLite • Experience, lessons learned from previous projects

  17. USI • What is there in terms of resources (compute, data, instruments) • a few large servers • one small Apple cluster (16CPU) • Current projects • EU-IP RESERVOIR (Virtualization, SLA) • JOpera grid workflow tool (http://www.jopera.org) • Interests • Use available SwiNG resources • SwiNG for real-world requirements for research • Grid middleware research: Meta-middleware, Grid workflows • Integrate JOpera with more low-level Grid middleware (Condor already supported) • Integrate JOpera with data management middleware • Goals for 2008 • Expand hardware resources at USI • Transfer research into "production" • Start SWITCH/AAA collaboration projects • Develop the JOpera Swiss user community • Experience, lessons learned from previous projects

  18. CSCS • What is there in terms of resources (compute, data, instruments) • Some service nodes and a few WNs (Biogrid + seed) • Current projects • LCG / EGEE • PRACE • SEPAC • Interests • Establish a national network to access the CSCS resources • Establishment of a high-throughput infrastructure in switzerland • Goals for 2008 • 25WN 2x2, 18TB, Suse • Successfully run the projects • Start the establishment of the network • Experience, lessons learned from previous projects

  19. SIB - VitalIT • What is there in terms of resources (compute, data, instruments) • 400CPU dedicated to life sciencies • 37TB storage, to be increased to 100TB by end of january • DB services (mysql, oracle), public DB repositories for genomics, .. • Dedicated systems with large memory • NodruGrid installed, gLite tried (constrained by OS) • LUSTRE • Current projects • Swiss Bio Grid • EMBRACE • Interests • DataGrid – storage services to be provided to our partners • Propagate the DB repository to other sites • Goals for 2008 • Accommodate high-throughput sequencing on the infrastructure – increase storage by a large amount • Upgrade infrastructure further • Experience, lessons learned from previous projects • Grid protocols are very firewall unfriendly

  20. Experience / Discussions • Current Middleware knowledge • Globus and globus-based • Globus 4 • edg, gLite • NorduGrid • Condor • Unicore • Desktop Grids • BOINC • United Devices • XtremWebCH • Interoperability / Interoperation is deemed essential between various MW • Already working for some, like Condor/NorduGrid, gLite/NorduGrid • User Groups – definition is nontrivial • Scientists (focusing on publications) • Developers of services • Maintainers and supporters of resources • Applications • Predefined validated set of applications • User-compiled custom code • Combination of both • KISS principle – Keep It Simple and Stupid • For overall design of basic Grid services • File catalog, hierarchies • Authentication and authorization – either in the catalog, propagating a token or at the site that then needs to be synchronized. • Monitoring • Pilot Jobs and overlay grids

  21. Project Ideas • Central Services: • Comparaison of resource brokers • Comparaison of IS • Monitoring, site functional tests • VO support • Push vs Pull architectures • Site specific issues: • Installation of different middlewares at the same site • Compatibilities, problems • Recommendations on LRMS • Use of AAI attributes for authorization • Data Management Issues • Comparaison of different solutions • Identify user communities and their requirements

  22. File Management Project • Design the necessary components to provide distributed file access on the Grid. • Store files • Access files

  23. 0. One Grid software • One selected middleware • To be installed everywhere • Choice needs to be made

  24. 1. Infrastructure dependent projects • Every institution makes its choice • Minimal common elements • Independent infrastructures

  25. 2. Centralization at the top • Sites are independent • Standardization effort at the top • Independent projects at the site • Common project at the top

  26. 3. Common decentralized approach • Support of different stacks at the site • Standardization at the top as much as possible • Coordinated projects at site and top

  27. 4. Common layer at the bottom • Common layer at every site • X may be very thin, not necessarily excluding G, N • In addition site specific / infrastructure specific components • Pull vs push architecture

  28. Data Management Project • Project Idea: • Select one data management solution • Install at a number of sites for supporting various Vos • Could be used together with a grid middleware or independent of a given middleware • Technical evaluation of DM solutions • Gathering user requirements from scientific communities • Implementation as a project between different institutions • as AAA/SWITCH in the project competition • Stakeholders: Scientists, IT services

  29. Subgroups • Data Management:Can Tuerker, Wibke Sudholt, Peter Kunszt, Chad La Joie, Christoph Witzig, Alessandro Usai, Roberto Fabbrettihttp://www.doodle.ch/vgivau2akspintqx • Assemble requirements and initial design for DM services • Grid Operations Team GOT: Micheala Thiemard, Pascal Jermini, Wibke Sudholt, Christian Bolliger, Sigve Haug, Sergio Maffioletti, Placi Flury Alessandro Usai http://doodle.ch/ytpnscqvgyhrah47 • Define the initial operating infrastructure pragmatically today • Define operational procedures and processes as possible • Grid Architecture Team GAT: Chad LaJoie, Peter Kunszt, Wibke Sudholt, Can Tuerker, Derek Feichtinger, Christoph Witzig http://doodle.ch/pyp4s64vbdsqaisc • Define the future Grid architecture for SwiNG

More Related