1 / 14

DGAS

DGAS. May, 25th Bologna Andrea Guarise – guarise@to.infn.it. EGEE is a project funded by the European Union under contract INFSO-RI-508833. Introduction. A generic Grid accounting process involves many subsequent phases that can be divided in:

lecea
Download Presentation

DGAS

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DGAS May, 25th BolognaAndrea Guarise – guarise@to.infn.it EGEE is a project funded by the European Union under contract INFSO-RI-508833

  2. Introduction A generic Grid accounting process involves many subsequent phases that can be divided in: • Metering: collection of usage metrics on computational resources. • Accounting: storage of such metrics for further analysis. • Usage Analysis: Production of reports from the available records. • Pricing: Assign and manage prices for computational resources. • Billing: Assign a cost to user operations on the Grid and charge them. In this presentation we give a quick overview of DGAS, the accounting middleware of the EGEE project.

  3. Accounting &/vs. Monitoring Is accounting nothing but monitoring? • Accounting is about tracing single jobs, file requests, etc. It is about single transactions (and their logic aggregations) associated to accounts. • Monitoring can use accounting information. But not only. • Get/archive Usage Record for a particular job. • How many jobs has a given User submitted? • On which CEs did a given user submit his jobs? • Jobs submitted/executed per VO, per Resource, per Site. • Does a VO use more resources than it provides? Accounting • What is the actual CPU load on a CE? • What is the average QWT on a queue? • What is the storage occupancy on a SE? • Are services on a CE up&running? Monitoring Economic Accounting • Has the user enough credits to submit a job? • Get account balance. • Credits spent/earned per VO, per resource • Does a Vo spend more than it earns?

  4. DGAS The Data Grid Accounting System was originally developed within the EU Datagrid Project and is now being maintained and re-engineered within the EU EGEE Project. The Purpose of DGAS is to implement Resource Usage Metering, Accounting and Account Balancing (through resource pricing) in a fully distributed Grid environment. It is conceived to be distributed, secure and extensible. The system is designed in order for Usage Metering, Accounting and Account Balancing (through resource pricing) to be indipendent layers. Account balancing, resource pricing, (billing) accounting data Usage accounting Usage Analysis usage records Usage Metering

  5. DGAS: Metering Usage Metering on Computing Elements is done by lightweight sensors installed on the Comuting Elements. These sensors parse PBS/LSF/Torque event logs to built Usage Records that can be passed to the accounting layer. For a reliable accounting of resource usage (essential for billing) it is important that the collected data is unequivocally associated to the unique grid ID of the user (certificate subject/DN), the resource (CE ID) as well as the job (global job ID). A process, completely transparent to the Grid User collects the necessary information needed by the Accounting. These, and the corresponding metrics are sent via an encrypted channel to the Accounting System signed with the user credentials.

  6. Metering: Gianduia

  7. DGAS: Accounting The usage of Grid Resources by Grid Users is registered in appropriate servers, called Home Location Registers (HLRs) that manages both users and resources accounts. In order to achieve scalability, accounting records can be stored on an arbitrary number of independent HLRs. At least one HLR per VO is foreseen, although a finer granularity is possible. Each HLR keeps the records of all grid jobs submitted or executed by each of its registered users or resources, thus being able to furnish usage information with many granularity levels: Per user or resource, per group of users or resources, per VO. Accounting requires usage metering, but not necessarily resource pricing and billing.

  8. Balancing and Resource Pricing Resource pricing is done by dedicated Price Authorities (PAs) that may use different pricing algorithms: manual setting of fixed prices, dynamical determination of prices according to the state of a resource. In order to achieve scalability, prices can be established by an arbitrary number of independent PAs. At least one PA per VO is foreseen (VOs will want to retain control on the pricing of their resources). Price algorithms are dynamically linked by the PA server and can be re-implemented according to the resource owners' needs. The job cost is determined (by the HLR service) from resource prices and usage records. Account balancing is done by exchanging virtual credits between the User HLR and the Resource HLR.

  9. DGAS accounting architecture A simplified view of DGAS within the WMS context.

  10. WMS PA 2 HLR 1 CE CE HLR 2 Example of economic accounting VO 1 VO 2 Check of economic authorization Job Flow Usage Metering Usage Accounting Account Balancing/Resource Pricing

  11. DGAS deployment DGAS architecture allows flexible deployment schemas. The deployment schema that we suggest to start with is to have: • A User HLR per VO (or more for VOs with large amount of users.) • It stores the Information and Usage Records for the users of the VO. The user specifies the address of his HLR in the job JDL. • One or more Resource HLRs, these can be one per site, but it is not mandatory, there can be HLR that stores information for many sites. • These store the Information and Usage Records for the resources registered. The site manager specifies the address of the resource HLR with a configuration file on the CE node. • The Gianduiametering system installed and configured on the Computing Elements. • One or more Price Authorities (only if Economic Accounting is desired). There should be one PA for every Resource HLR.

  12. HLR 4 HLR 1 HLR 2 HLR 3 HLR 5 CE CE CE CE CE CE DGAS deployment VO1 site1 site2 VO2 Job1 UR1 UR1 UR2 Job2 VO3 site3 Aggregatesite accounting APEL UR2

  13. DGAS on gLite and INFNGrid Since February DGAS is being intensively tested on the gLite prototype testbed. DGAS is also officially part of the INFNGrid 2.4.0 distribution. Since the beginning of May is being installed on the production testbed and is currently getting information from around 15 sites in Italy. Up to now the records are collected for jobs submitted to only one of the three Italian RB, this is needed in the deployment phase to test the system configuration. gLite testbed daily Jobs

  14. References • Further information and documentation about DGAS can be found at: http://www.to.infn.it/grid/accounting

More Related