1 / 22

The UK NGS utilising heterogeneous middleware

The UK NGS utilising heterogeneous middleware. David Wallom Technical Director, UK NGS Associate Director-Innovation, Oxford eResearch Centre VP Community, OGF. Overview. Our Mission and Goals Connecting to create integrated e-infrastructure Membership Experiences….

elda
Download Presentation

The UK NGS utilising heterogeneous middleware

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The UK NGS utilising heterogeneous middleware David Wallom Technical Director, UK NGS Associate Director-Innovation, Oxford eResearch Centre VP Community, OGF

  2. Overview • Our Mission and Goals • Connecting to create integrated e-infrastructure • Membership • Experiences…

  3. NGS Mission and Goal To enable coherent electronic access for UK researchers toall computational and data based resources and facilities required to carry out their research, independent of resource or researcher location. Goal: • To enable a UK wide integrated production quality e-infrastructure • Enable expansion to all Higher Education Institutes and UK based research institutions • Supporting cutting edge research • To deliver core services and support • Support research computing groups within universities and research organisations to help them support users • Highlight Collaborative research in key communities • Integrate with international infrastructures supporting UK participation in international projects with EU and US collaborations

  4. Impact • Improve accessibility to local and national resources • ‘Use once Use anywhere’ • Support the Share/Trade/Buy/Sell of resources • Facilitate collaboration nationally and internationally

  5. UK e - Infrastructure get common access, tools, information, Users nationally supported services, through NGS HECtoR Regional and HEIs Campus grids Community Grids Integrated internationally LHC VRE, VLE, IE ISIS TS2

  6. National and International Project Engagement • CCPs (4 & b) • STFC CLF • MOTT-2 • NSCCS* • NanoCMOS • DSR (analysing requirements) • NeISS • DiRAC* • ELIXIR* • LifeWatch • CLARIN* • GridPP* • SKA* • SDSS

  7. NGS and US Collaboration • Open Science Grid • MoU signed to share technology, and give enhanced support to bridging user communities (Structural Biology taken as exemplar) • XSEDE • NGS provided Letter of Commitment during proposal stage • MoU in negotiation • Shared vision of open standards based infrastructure (supported on both sides OGF standards demonstration platform • Assist communities with HPC time to utilise services • Engage with Campus Champions program to share best practice

  8. Institutional Membership • Personnel • Appointment of an institutional Campus Champion • Resource Exchanging • Commit institutional research computing resources to join the NGS • Nomination of Collaboration Board Member • Partner • Supporting access by a significant body of NGS users • Affiliate • Supporting only internal users

  9. Services EXAMPLES Facility AHE ARGUS EBI DB Lang Corpa Institutional Portal Community Specific Services Community Shared Services/Site Level Services Data Compute WMS NGS Portal Cloud Central Authn Authz Accnt Helpdesk Training Monitor Discovery Security Central Services

  10. NGS Site Level Services • Description of the different solutions to service requirements challenges • A set of different modules describing a particular function, each with a number of solutions which are grouped together as profiles • Supported interfaces are community driven, we will not/cannot dictate what we make available. • Computational interfaces: VDT, gLite, Globus 4, GridSAM • Data Services • Databases (Oracle), SRB, global filesystems, Service access to structured data (OGSA-DAI)

  11. Community shared computational resources Requirements • Support for MPI • Integration with existing production systems • Stable interfaces for production system • Most UK HEI systems are not institutionally based therefore have high utilisation by single communities • Communities requiring inter-institutional collaboration/access are a minority • Single integrated service, not requiring separate systems for CE, SE, BDII, accounting, monitoring etc. • Simple ‘automated’ installation • Integration with NGS central services, AAA, Monitoring, Service Discovery • Some support for access through higher level access services such as WMS, P-Grade, Nimrod-G, Condor-G

  12. Solution • VDT • Supported! • Simple installer (PacMan) • Globus 2.4, known stable interfaces • Developed our own documentation... • http://www.ngs.ac.uk/sites/default/files/file/VDT-INSTALL.pdf • Developed automated installer

  13. Customisations/Enhancements • Support for VOs • Support for pool accounts and integration with MOST known schedulers • Integration with LCAS/LCMAPS

  14. LCAS/LCMAPS • NGS needed something better than the gridmapdir pool account patches to allow for correct VO/project accounting to work. • gLite had LCAS/LCMAPS plugins, but they didn't work with the “vanilla” Globus libraries from VDT. • Patches were developed to the LCAS/LCMAPS gLite plugins to work with “vanilla” Globus

  15. Customisations/Enhancements • Support for VOs • Support for pool accounts and integration with MOST known schedulers • Integration with LCAS/LCMAPS • Accounting • Development of OGF UR generator and RUS publisher • http://vidar.ngs.manchester.ac.uk/rus.html • Information publishing through GLUE schema • Significant work to get publishing in conformance with gLite washing service

  16. Knock-on effects • Required customised WMS system to support community utilisation of all available NGS partner resources • Multiple national BDII services and increased support requirement to partnership program to get member resources installed and working • Able though to develop NGS Uniform Execution Environment to allow different systems to present application in a uniform location to the user community

  17. Moving Forward • Standards are essential (current ones, new ones, any) • IGE must be compatible with other offerings at an interface level to enable to utilise higher level tools as well as make central services operations possible • IGE must support all known LRM systems with native integrated MPI support that’s works first time, everytime • It must consider itself as a product development activity not a project

  18. Thank You http://www.ngs.ac.uk support@grid-support.ac.uk david.wallom@oerc.ox.ac.uk

  19. Diverse User Community www.ngs.ac.uk 28 member institutes 33 heterogeneous resources 15,000 processing cores > 75 applications • In the last 12 months • 4,629,127 CPU hrs used • 888,862 jobs ran • 2nd largest e-Science CA • 22,121 certificates issued • 4,911 active currently

More Related