OpenStack QA tooling & How to Use it for Production Cloud Testing

OpenStack QA tooling & How to Use it for Production Cloud Testing Ghanshyam Mann, Aug 24, 2019

Agenda Who I am What is OpenStack QA? OpenStack QA Tools OpenStack QA Tools: Used for Production Cloud Testing OpenStack Cloud: What All to Test Tempest & Patrole tests: Simultaneous Multiple Run Cloud Health Node: Deployment Model Cloud Health Node: Dashboard View

Who I am • Ghanshyam Mann • IRC: gmann • Open Source Developer, NEC • Involved in OpenStack since 2012 • Full time OpenStack Upstream Developer since 2014 • Member of OpenStack Technical Committee (TC) • PTL of OpenStack QA • Active Contributors in Nova, FC SIG, Infra and various others projects. • https://ghanshyammann.com/about/

What is OpenStack QA? Official Mission Statement: “Develop, maintain, and initiate tools and plans to ensure the upstream stability and quality of OpenStack, and its release readiness at any point during the release cycle.”

OpenStack QA Tools

OpenStack QA Tools: Used for Production Cloud Testing • Patrole is a security validation tool for verifying the OpenStack Cloud’s Role-Based Access Control • OpenStack Integration Test Suite Patrole Tempest & its Plugins OpenStack-Health Dashboard Stackviz • Stackviz is a visualization utility to help analyze the performance of DevStack setup and Tempest test runs • The OpenStack • Health dashboard • provides a view of the • status of all the tests running in • your CI

OpenStack Cloud

What All to Test Role Based Access Control API Upgrade services status Non-frequent scenario Integrated Functionality

OpenStack QA Tools Tempest Patrole API tests RBAC testing Scenario tests Positive and Negative testing of each API policy Cover Nova, Cinder, Glance, Neutron, keystone, Swift Cover Nova, Cinder, Glance, Neutron, keystone Other Services tests in *-tempest-plugin

OpenStack QA Tools: Tempest • Overview of gate tests • Every proposed code needs to be passed via multiple gate tests • pep8 (coding style) • python 2/3 (unit test) • tempest-*(integration test) • 1,000+ gate test cycles run in most working days • Each gate test cycle consists of 1,000+ tests • That means 1,000,000+ tests run in a day

OpenStack QA Tools: Tempest • Tempest - OpenStack Integration Test Suite • This is a set of integration tests to be run against a live OpenStack cluster. • Tempest is designed to be useful for a large number of different environments. This includes gating commits to OpenStack and testing the production Cloud in many large organization. • Providing the stable framework to write Tempest like test with tempest plugins either on upstream or downstream

OpenStack QA Tools: Tempest • API tests • Scenario tests • Tempest Plugins: • 59 plugins • https://docs.openstack.org/tempest/latest/plugin-registry.html

OpenStack QA Tools: Tempest • Tempest can run with dynamic credential or pre defined credential(account.yaml) • tempest.confto tell Tempest about your Cloud and testing preferences. • Tempest CLIs for easy to use interfaces:

OpenStack QA Tools: Patrole • Patrole - RBAC Integration Tempest Plugin • Patrole is a set of integration tests to be run against a live OpenStack cluster. It has a battery of tests dedicated to validating the correctness and integrity of the cloud’s RBAC implementation. • Patrole is tempest plugin and based on same design principle as Tempest • Patrole override the set of configured roles to tests the policy access.

OpenStack QA Tools: Patrole Nova "os_compute_api:servers:create" POST compute/v2.1/servers/ Call OpenStack APIs with overridden Role oslo.policy direct check with overridden role Role: ‘member’ Compare results

Tempest & Patrole tests run: multiple workspace https://docs.openstack.org/tempest/latest/overview.html#quickstart Cloud Health node workspace-1 tempest.conf tempest_lock .stestr log-dir lock-dir workspace-2 tempest.conf tempest_lock .stestr log-dir workspace-n tempest.conf tempest_lock .stestr log-dir tempest run --workspace workspace-1 tempest init workspace-1 tempest run --workspace workspace-2 Installed Tempest, tempest plugins & Patrole

OpenStack QA Tools+ Zuul Jobs • Zuulv3 Jobs • Add playbook to setup and run the Tempest+plugins • Standard setup of playbooks involve • pre-run • run • post-run • Tempest tests will run as part of those playbooks

OpenStack Health Dashboard Subunit stream (Tempest, plugins tests, etc.) subunit2sql O-H API server DB subunit2sql

Cloud Health Node: Deployment Model Cloud Heath node Openstack-health Dashboard Tooling Tempest Patrole workspace-n workspace-1 workspace-2 *-tempest-plugin Run test & Fetch Results Run test & Fetch Results Run test & Fetch Results OpenStack Site 1 OpenStack Site 2 OpenStack Site n

Cloud Health Node: Deployment Model Cloud Heath node No need to upgrade the health node. Until explicitly need new tests. Openstack-health Dashboard Tooling Tempest Patrole workspace-n workspace-1 workspace-2 *-tempest-plugin Run test & Fetch Results Run test & Fetch Results Run test & Fetch Results OpenStack Site 1 OpenStack Site 2 OpenStack Site n Site 1 is upgraded to new version (Rocky>Stein) Site 3 is upgraded to new version (Rocky>Stein) Site 2 is upgraded to new version (Rocky>Stein)

Cloud Health Node: Dashboard View Openstack-health Dashboard

Reference https://docs.openstack.org/tempest/latest/ https://docs.openstack.org/patrole/latest/ https://docs.openstack.org/stackviz/latest/ https://opendev.org/openstack/openstack-health http://status.openstack.org/openstack-health/#/

Q&A? Thank you

OpenStack QA tooling & How to Use it for Production Cloud Testing