1 / 10

Stephen Booth s.booth@ed.ac.uk

Stephen Booth s.booth@ed.ac.uk. GridSafe Overview. Grid-SAFE. JISC funded project to build general purpose accounting/monitoring solution. http://gridsafe.forge.nesc.ac.uk/ Builds on accounting subsystem from SAFE user administration system used by HPCx/HECToR Challenges:

elmo
Download Presentation

Stephen Booth s.booth@ed.ac.uk

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Stephen Booth s.booth@ed.ac.uk GridSafe Overview

  2. Grid-SAFE JISC funded project to build general purpose accounting/monitoring solution. http://gridsafe.forge.nesc.ac.uk/ Builds on accounting subsystem from SAFE user administration system used by HPCx/HECToR Challenges: Need to work with wide variety of different local policies. Need to work with both grids and local HPC resources. One solution won’t fit all potential users Build kit of parts Pre-built solutions for common deployment scenarios. Key aims Modular design, individual functions can be deployed independently Behaviour can be customised using plug-ins to implement different service policies.

  3. Overview

  4. Stand-alone accounting portal Initial development phase concentrated on Accounting module Reporting module Can be deployed together as stand-alone accounting portal Consumes and stores accounting data in variety of formats Generates on demand reports. Target users: Local Campus HPC resources. Grids with centralised accounting. Requirements Java / apache-tomcat / mysql Current status New D4 release of code and documentation August 24th Code deployed for ECDF cluster in Edinburgh Development team can assist with installation/configuration.

  5. Accounting Module Accounting data is stored in relational DB UsageRecords are treated as a collection of properties Code does not mandate formats. Users can standardise on one format if they wish. Plug-in parser modules handle different types of input data. OGF-UR SGE PBS EGEE JobManager Etc. Plug-in policy modules augment these allowing site local customisation Generate additional properties such as Charge Generate derived properties defined as expressions over other properties. Decrement budgets Link records to other tables (Person, Project etc.)‏ Generate Aggregates

  6. Report generation module Grid-safe uses XML templates to define reports Can generate unified reports over multiple data tables containing different types of data Tables/charts Parameterised reports (e.g. to select user or project). Support reports in multiple formats PDF HTML CSV Performance of report generation a particular issue Utilise database effectively. Old data can be moved to separate tables. Support for Daily aggregates For ECDF 51 million records -> 35 thousand aggregates

  7. Person SGERecord Institute UnixGroup School Project DailyAggregate Queue ClassificationPolicy • Converts selected fields from raw accounting data into references to separate database table. • Reduces data footprint. • Augmenting information can be added to these tables. • Example from ECDF

  8. OGF-UR support OGF-UR XML is supported as an interchange format Parser plug-in to parse OGF-UR Export module to format internal data as OGF-UR Grids may only want to use only this Format Various grid communities seem to interpret OGF-UR differently and/or make additional requirements beyond that in the schema Required fields Different charging models Different global username models OGF-UR spec allows extensions. Specification will also evolve over time. Parser/exporter highly configurable to support variations/extensions.

  9. Web Services • RUPI • Current proposal from OGF RUS-WG • Web service for the upload of XML usage record. • Grid-SAFE has an implementation of the current upload service (RUPI). • RUQI • Currently working on a proposal for a Query specification • Aims • Easy to implement in different code bases. • Provide sufficient functionality for efficient report generation. • Long term aim to provide reporting portal that can query any system that implements this interface.

  10. Where next • Need more early adopters. • Can add features/capabilities as requested • Possible candidates • Instrumentation of Job Prologue/Epilogue • Useful for systems without an accounting logfile to parse. • Would also allow run-time budget checks. • Storage accounting. • There is existing code from HPCx/HECToR we could cross-port.

More Related