Cloud Usage with CloudStack

1. Cloud Usage with CloudStack Aug 2014 Chandra Prakash Mishra

2. In next 45 minutes … • Agenda • CloudStack Usages • Cloudstack configuration to enable Usage Capture • Usage Data Generation Process • Usage Type • Usage Event • Usage Records • How can Access Usage Data • Use Cases

3. UsageRequirements • Meter their cloud's consumption and monetize the cloud offerings by charging for usage • Report on the cloud’s resource usage for monitoring/capacity planning • Existing billing system, just need access to the usage data • Organizations tend to have different billing criteria/charging models and require access to usage data for their specific needs Footer – Apply across document

4. UsageinCloudStacktoday • CloudStack does not offer any native usage monitoring or billing capability • Root Admin can view current System Capacity • Domain Admin and Users can view VM statistics Footer – Apply across document

5. Usage Server • CloudStack includes a Usage Server that create summary usage records for the various resources consumes in cloudstack. • The usage server is an optional, separately installed component called “cloudstack-usage”. • The usage records are stored in a separate database called “cloud_usage”. • Its Runs once a day by default .It can be configured to run multiple times per day and it’s behavior is controlled by the following global configuration settings. enable.usage.server : Whether the usage is active ? usage.stats.job.aggregation.range: Usage is aggregated for specified number of minutes. e.gdefault:1440 usage.stats.job.exec.time : Time when the usage server will start processing. Default: 00.15 usage.execution.timezone : The timezone to use for usage.stats.job.exec.time. Default: timezone of the management server usage.aggregation.timezone : The timezoneto use for Cloudstack usage stats aggregation (default GMT) Footer – Apply across document

6. CloudStack Event • Usage events are stored in usage_event table. Any resource for which usage has to be tracked needs to log usage events when the the resource is created/allocated and when its deleted/released. • Network usage is the only exception which doesn’t use usage events. Once these events are logged usage server will process them and generate Cloudstack usage records. Description of various columns in usage_event table: • type – Event type as described in cloudstack-oss/api/src/com/cloud/event/EventTypes.java • account_id – Id of the account which owns the resource • zone_id – Data center to which this resource belongs to • resource_id – Id of the resource. For VM usage this will be VmId, for template templateIdetc • resource_name – Name of the resource. • offering_id – Id of Service Offering in case of Vm usage, Disk offering in case of volume usage etc.. • template_id – Id of the template used for a volume / Vm.  • size – Size of the resource. Volume Size or Template size etc. • resource_type - Additional type attribute for a resource. Could be used for hypervisor_type etc. • processed – Is set to 0 by default. This field is updated to 1 when the usage server completes processing of this event. Footer – Apply across document

7. Usage Record Generation • The usage server schedule a job at the time specified by usage.stats.job.exec.time parameter .This job query cloud databases cloud.usage_event table and generate usage records in the cloud_usage database table cloud_usage as follows • CloudStack events have been logged into cloud.usage_event table . • The cloudstack_usage job gets a list of the latest usage events. • Inserts these events into cloud_usage.usage_eventstable. • Parses cloud_usage.usage_event and populates Helper tables . • Helper table data used to populate cloud_usage.cloud_usagetable with aggregation range wise • data . Footer – Apply across document

8. Usage Type

9. Usage Records • Currently there is no UI which displays usage records. API call listUsageRecords fetches usage records. Usage Records show the amount of resources consumed by guest instances. • Allocated & Running VMs • Network Usage • IP Address • Disk Volume • Template, ISO and Snapshot • Load Balancer or Port Forwarding • Network Offering • VPN User Footer – Apply across document

10. Vm Usage Record Format • For running and allocated virtual machine usage, the following fields exist in a usage record: • account – name of the account • accountid – ID of the account • domainid – ID of the domain in which this account resides • zoneid – Zone where the usage occurred • description – A string describing what the usage record is tracking • usage – String representation of the usage, including the units of usage (e.g. 'Hrs' for VM running time) • usagetype – A number representing the usage type (see Usage Types) • rawusage – A number representing the actual usage in hours • virtualMachineId – The ID of the virtual machine • name – The name of the virtual machine • offeringid – The ID of the service offering • templateid – The ID of the template or the ID of the parent template. The parent template value is present when the • current template was created from a volume. • usageid – Virtual machine • type – Hypervisor • startdate, enddate – The range of time for which the usage is aggregated; see Dates in the Usage Record Footer – Apply across document

11. Usage Record Example Allocated VM Every time a VM is created VM.CREATE event is logged. Similarly VM.DESTROY event is logged when VM is destroyed. Corresponding entries in cloud.usage_event table would like the following: +———————-——————-——-———–————-———–———–—-———-+| id | type | account_id | created | zone_id | resource_id | resource_name | offering_id | template_id | size | processed |+———————-——————-——-———–————-———–———–—-———-+| 11 | VM.CREATE | 2 | 2011-02-09 11:15:27 | 1 | 5 | i-2-5-KK | 7 | 2 | NULL | 0 || 12 | VM.DESTROY | 2 | 2011-03-09 10:16:18 | 1 | 5 | i-2-5-KK | 7 | 2 | NULL | 0 |+———————-——————-——-———–————-———–———–—-———-+ Duration between VM.CREATE and VM.DESTROY is reported as Allocated VM usage. Running VM Usage for Running VM is implemented similar to allocated VM usage. Events VM.start and VM.stop are tracked here instead. +———–———-——————-——-———–————-———–———–—-———-+| id | type     | account_id | created             | zone_id | resource_id | resource_name | offering_id | template_id | size | processed |+———–———-——————-——-———–————-———–———–—-———-+| 12 | VM.START |          2 | 2011-02-09 11:16:18 |       1 |           5 | i-2-5-KK      |           7 |           2 | NULL |         0 || 15 | VM.STOP  |          2 | 2011-02-09 12:00:24 |       1 |           5 | i-2-5-KK      |           7 |           2 | NULL |         0 |+———–———-——————-——-———–————-———–———–—-———-+ Duration between VM.START and VM.STOP is reported as Running VM usage. Footer – Apply across document

12. How we can access usage data Footer – Apply across document

13. CloudStack API • The usage records can be accessed through the CloudStack API . • This is best done using the authentication port 8080 and requires both the API Key and Secret Key and for the API request to be signed • CloudStackprovides the following Root Admin API calls and responds with XML and JSON responses: • listUsageTypes – lists available Usage Types • listUsageRecords- provides Usage records for a date range • generateUsageRecords- asynchronous usage record generation Footer – Apply across document

14. ListUsageRecords API • The base API request is: • http://<HOST>:<8080>/client/api?command=listUsageRecords&startdate=y • yyy-MM-dd&enddate=yyyy-MM-dd&signature • Optional request parameters: Footer – Apply across document

15. Example response from listUsageRecords • <listusagerecordsresponse> • <count>1816</count> • <usagerecord> • <account>user5</account> • <accountid>10004</accountid> • <domainid>1</domainid> • <zoneid>1</zoneid> • <description>i-3-4-WC running time (ServiceOffering: 1) (Template: 3)</description> • <usage>2.95288 Hrs</usage> • <usagetype>1</usagetype> • <rawusage>2.95288</rawusage> • <virtualmachineid>4</virtualmachineid> • <name>i-3-4-WC</name> • <offeringid>1</offeringid> • <templateid>3</templateid> • <usageid>245554</usageid> • <type>XenServer</type> • <startdate>2009-09-15T00:00:00-0700</startdate> • <enddate>2009-09-18T16:14:26-0700</enddate> • </usagerecord> • </listusagerecordsresponse> Footer – Apply across document

16. CloudMonkeyCLI • CloudMonkeyis a CLI tool for CloudStack and can be used as an interactive shell or from within a shell script to call the API. • It can output the Usage Data in both JSON and Tabular form to a file or pipe it to another application for further processing. • Filtering can be used to limit the result set. • It supports argument passing and shell automation. • The host, port, apikey and secretkey can be configured on setup. Footer – Apply across document

17. CloudMonkeylist usagerecords The command to retrieve Usage Records is: cloudmonkeylist usagerecordsdomainid=7d8707bf-e6ad-11e3-a7e8-f245a5b3ba0f startdate=2014-08-10 enddate=2014-08-11 accountid=efda79de-ff01-4de2-bbe9-56b0ec366a07 Footer – Apply across document

18. SQL • The usage records can be also accessed by directly querying the cloud_usage.cloud_usageMySQL table in CloudStack • Usage Data can be exported in a CSV format from a SELECT query • The SQL query can become complex if the ID fields in the data need to be de-referenced • Typically multiple passes of the query results may be required to extract the required information Footer – Apply across document

19. CloudStackMessage Queue • CloudStack 4.1 started publishing events onto a message queue Uses RabbitMQ as the message broker, but likely to work with other AMQP- based brokers . • Use case: A third-party cloud usage solution can subscribe to CloudStack • events and generate usage data which can be consumed by their billing software . • For more details Please refer this link • https://cwiki.apache.org/confluence/display/CLOUDSTACK/EventBus Footer – Apply across document

20. Customer Use Cases Footer – Apply across document

21. MS Excel • MS Excel • MS Excel can be used to analyseCloudStack Usage Records • The usage records are imported from the cloud_usage table into MS Excel via an MySQL ODBC connection • Either PivotTables or reports can be used to present the data • This is convenient and great for integration testing between CloudStack and external systems Footer – Apply across document

22. Splunk • Splunkis an operational intelligence tool that was initially used for collecting and indexing infrastructure logs • We then connected splunkto the CloudStack Usage database and automated the collection and indexing of Usage Data • Setup feature-rich dashboards and the capability to drill down into the Usage Data for BI reporting Footer – Apply across document

23. Amysta Fore more details please refer this link: https://www.youtube.com/watch?v=ZA-VPb7WAy8 http://www.slideshare.net/buildacloud/apaliaamysta-cloud-usage-metering-and-billing Provides usage visibility, cost control and IT billing for Private and Hybrid Clouds Dashboard Integrated in the CloudStack UI Pricing, Alerting, Reporting, Capacity MgmtAmysta consolidates public cloud expenses e.g. AWS and private cloud consumption in one dashboard view Footer – Apply across document

24. SunGard Cloud Portal Footer – Apply across document

25. Ubersmith's Fore more details please refer this link : http://www.ubersmith.com/kbase/index.php?_m=knowledgebase&_a=viewarticle&kbarticleid=161&nav=0 Ubersmith'sCloudStack Billing Service Module allows you to have Ubersmith bill for CloudStack usage. Footer – Apply across document

26. Thank You!