1 / 22

From 2 to 1000’s DevOps at UNH

Learn how the University of New Hampshire implemented DevOps practices, including containerization, process automation, test automation, and security automation, to improve their web infrastructure. Discover the benefits and tools used in this transformation.

jeanettec
Download Presentation

From 2 to 1000’s DevOps at UNH

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. David Blezard Chris Schneider Academic Technology From 2 to 1000’sDevOps at UNH

  2. Container sets for each website from shared code, but having isolated function Create dev, test, production as needed VMs with separate functions - DB vs apps vs…Separate environments - dev/test/prod Shared resources for DB, apps, load balancing, dev/test/prod for 100’s of websites 2011 2019 2014 UNH Web Infrastructure

  3. Previous State • Handful of VM’s and running multiple stacks as Apache VirtualHosts • Coldfusion • PHP • Drupal – multiple versions • Wordpress • Static HTML • User SFTP access • Directory Layout

  4. Pain Point Examples • Development Staff • Time to deploy changes is too long • Time for site setup too long • Client Services Staff • Timelines for clients complicated with deployment requirements • Two week site launch lead time - security testing and bug fixes • Operations and IT Staff • Manual security testing • Manual Configuration and setup • Issues with consistency over time • Shared hosting means system upgrades have potentially HUGE impacts

  5. Four Primary Areas of Focus • Development Independence (For developers and IT staff) • Process Automation • Test Automation • Security Automation

  6. UNH’s DevOps Manifesto • Increasing the speed with which software can get updated, including freeing ourselves from the one-day-per-week-only rollouts • Removing repetitive, non-productive tasks from developer and system administrator workloads primarily via automation • Increasing value to our clients by delivering more utility, higher quality, more consistency, more quickly Any efforts, tools, techniques, processes, or communications means that can accomplish these goals are possible within the scope of this overall effort.

  7. Development Independence • Refactor or create processes to allow the development staff to develop locally • Templated builds based that are standardized via Drupal profiles • Significantly reduces site provisioning time • Refactor or create processes to allow IT staff to test and update systems independently • Reduces the impact to development and client services staff • Quicker turnaround on critical updates and configuration changes • Easy rollbacks • All of this depends on Containerization and Process Automation

  8. Process Automation • Define repetitive tasks in code and execute from a job controller • Allows permissions to be maintained using a least privileged model • Allows certain administrative tasks to be executed by development teams • Database provisioning • Site definitions and provisioning • Scheduling certain maintenance operations • Scripts are stored in our SCM system for versioning and deployment via CI pipelines. • Script development follows a similar development model to our general Drupal development model.

  9. Testing • User Interface tests • ADA • Compliance Testing • ADA • FERPA • Stress Tests • JMeter • Reporting on all of the above • Log aggregator/Visualizer • Kibana • Testing tools reporting features as project artifacts • Testing depends on Containerization and Automation

  10. Security • Application security • Container security • Stack security • Follows a similar set of reporting options as general testing does • Depends on Containerization and Automation

  11. Thedev ops Tool Kit • Hosting – docker on top of vmware • Orchestration by Docker Swarm • Job Control • Rundeck • Combined Source Control and CI/CD Engine • git via gitlab • Issue tracking • Logging • Syslog • logspout • graylog • Monitoring and metrics • Elk stack – Elastisearch, logspout, and graphana

  12. Hosting • Docker and docker swarm • Docker is an orchestration Alternative to prodcuts such as kubernetes • Native to docker • API interface with a number of libraries in popular languages for programmatic maintenance • Manages private networks for routing internal traffic • VMWare • Swarm nodes are VM’s • Using VM affinity decreases chances of swarm node failures

  13. Source Control • GitLab • Use branches to control deployment to the various environments • Protect branches to prevent unauthorized merge requests and deployments from happening without proper testing within pipelines • Pipelines are defined in yaml and are kept with the individual project

  14. CI/CD Engine • GitLab • Uses pipelines defined on a per project basis for moving code through the various environment and testing stages. • Notifications of pipeline failure or success • Use of Secret Variables to store application and pipeline configuration data such as database credentials, SSH keys,configuration data, etc.

  15. Issue, Request, and Bug Tracking • TeamDynamics • UNH Offical Project Management and Service Desk Ticket System • GitLab • Use Issues to track problems and feature requests directly in the SCM system. • Pin resolutions to specific commits and merge requests

  16. Job Control • RunDeck • Centralized Job and Task Management • User and Group Access Management • Job Logging and Log Retention • Based on Quartz Engine, which allows for down to the second scheduling • Web API for integration with other systems and processes

  17. Log Aggregator and Analysis • GreyLog • User and Group Management to restrict log access on a need to know basis • Uses industry standard syslog by default, supports GELF and various other input protocols. • Can filter inputs into Streams by keywords on log fields. • Provides notifications and alerts, with support for Slack, Teams, Email, and others via plugins

  18. Monitoring • Monitis • External Monitoring of Production Sites • Prometheus • Gathers metrics on Application Perfomance • Gathers metrics on Swarm performance • Cadvisor • Gathers metrics on docker container operations • LogStash • Parses and forwards logs via pipelines to an aggregator or visualizer • Graphana • Front end visualize for graphing data from Prometheus

  19. General Testing • JMeter • Industry standard tool for various types of testing • BeeHat • Automated User Interface Testing • Mobile Testing • Browser Testing • Manual Testing • User Inteface • Features • Process Flow

  20. Security and Compliance Testing • Accunetix Scans • ADA Scans • FERPA scans

  21. Real-World Benefits • Drastically reduced setup time for Devs • Working with non-trusted Devs • Much more frequent releases (weekly) • Fixed cascading crashes by isolation of apps/sites • Security benefits – Got audit off our back

  22. Questions

More Related