Introduction to DevOps and Tools: Containers, .Net Core, NuGet, Xamarin, and more

1. DevOps and Tools: Containers, .Net Core, NuGet, Xamarin, and more… Software Modeling and Analysis (CSE 681) Fall 2018

2. Overview • DevOps Concepts and Ideas • Tools • Container Technology • Concepts and Ideas • Docker • Relationship to DevcOps • Demonstration • Survey of .Net Technologies • .Net framework (studied in CSE 681) • Net Core • Universal Windows Platform (UWP) • Xamarin (Mono) • .Net Standard • Demo .Net Core with Docker • Use of Nuget • Time Permitting • Demo Xamarin (shared code: Android, UWP) • Demo ASP.Net Core REST service • Lists of References

3. DevOps? What’s is that…? • Portmanteau for “Development and Operations” – DevOps • a culture and set of processes with an objective • Optimize the efficiency of software development process within an organization • develop, test, integrate, deploy, deliver software faster and more efficiently than through traditional approaches • It’s popular movement…but not completely new idea! • … grew out of the Agile Manifesto [1] • iterative/incremental approach focused on collaboration, customer feedback, and small, rapid releases. • At organizational level! • breaking down communication silos between teams • Who’s in? • Just about everybody: Amazon, Microsoft, Google, Facebook, Twitter Adobe, …., …., https://www.amazon.com/s/ref=nb_sb_noss_1?url=search-alias%3Daps&field-keywords=devops https://aws.amazon.com/devops/?nc2=type_a https://azure.microsoft.com/en-us/services/devops/?&OCID=AID736750_SEM_uI9fty9d https://devops.com/news/

4. DevOps Overview • Main ideas Grew out of the disconnects that naturally arise between (people) • Miscommunication, misaligned goals and objectives manifest as conflict and inefficiency. • compound to become increasing complex as the size of an organization increases (as development teams increase in size) • Evolved out of approaches to overcome these problems by focusing on the communication, collaboration, and integration between software development and information technology personnel [1] • Goal is to highlight the break downs and bottlenecks (the Silos) that impede the development process [1] 1] - “Secure DevOps Process and Implementation” Software Engineering Institute (SEI), Carnegie Mellon University, January 11, 2017, https://www.sei.cmu.edu/training/P116.cfm

5. Breaking Down the silos! Silos inhibit collaboration and reinforce the waterfall methodology A cultural movement in as much as it is a development cycle [1] “…that’s not my Job!” “That’s is my Job, stay out!...” (Ownership ideology) Everyone becomes a first class citizen and works collaboratively across the disparate areas: Requirements->Design->Implementation->verification>maintenance e.g. Operations (security staff) should be involved in requirements and design phases else domain expertise is lost! • [2] - http://www.agiletestingframework.com/devops-getting-old-deja-vu-feeling/ [3] - http://www.agilebuddha.com/agile/demystifying-devops

6. DevOps Principles Overview • DevOps (Development and Ops)– out growth of Agile manifesto [1] • Agile embraces continuous change, DevOps embraces continuous test/integration/delivery/deployment • How? Break down the silos sub teams (Dev, Ops, Test, Q/A). Everyone becomes a first class citizen. • Principles • Collaboration • Use tools that influence collaboration (within and between teams), enforce policy • Infrastructure as Code (IaC) • entire environment (servers/links/platforms/apps/configuration) all represented as scripts stored in version control • Tools: Open stack (Heat), AWS (Cloud Formation), Docker, Chef, Puppet, Vagrant • Automation • Script as much manual processing as possible! • Continuous Integration/Test/Delivery/Deployment • Tools: Jenkins/Team City/AWS DevOps, Docker • Monitoring: • Cloud watch (AWS), Ceilometer (Openstack), VictorOPs

7. DevOps Core Principles [1] • Collaboration • Efficiency (communication and processes) between disparate roles, both within teams between teams in an organizations • Infrastructure as Code (IaC) • Script and version control as much of infrastructure as possible, principally for capturing the disparate environment details (.e.g. development, testing, production, etc.) • Goal: (repeatable, recoverable, reusable, sharable) • Automation • Automate human (error processes) wherever possible • Goal: Increase frequency of testing, continuous (consistent/uniform) integration, delivery, and deployment processes • Monitoring • Data to needed to inform development/operational decisions • policies and priorities are domain dependent • Is the System working? Performance? Are users consuming the system as expected Source: https://www.sei.cmu.edu/education-outreach/courses/course.cfm?coursecode=V38

8. Parity: Automate (Standardize) Processes • Integration: merging disparate system artifacts is challenging • Often done manually – (slow and error prone) == complex and expensive • Can’t automate everything, but … big changes have security implications that require review etc.) • Continuous Integration (automate what makes sense…) • Automate/standardize submission process (get artifacts) into the various environments (testing, staging…) • Greater visibility: everybody gets notified of issue(s) sooner: developers, testers (QA), Op (Security), Management, etc. • fail fast and fix fast == increase efficiency. • Continuous Delivery • Changes pushed often to customer/mock production environment • perhaps staging is configured as such) • Give everybody (customer/stake holders) visibility • Continuous Deployment • Changes are pushed continuously (often) into production • Domain specific • doesn’t mean this makes sense • does mean the process for deployment is automated, consistent, reliable, and repeatable. • *Goal Process/Environment Parity to achieve frequent/regular testing • Make small, incremental changes

9. Tools that support/enable DevOps • Not a single tool (cross-functional concerns) • multiple tools (tool chains) that fit into one or more categories reflective of the development and delivery process… • Code – code development and review, source code management tools, code merging • Build – continuous integration tools, build status • Test – continuous testing tools that provide feedback on business risks • Package – artifact repository, application pre-deployment staging • Release – change management, release approvals, release automation • Configure – infrastructure configuration and management, infrastructure as code tools • Monitor – applications performance monitoring, end-user experience [3] • https://en.wikipedia.org/wiki/DevOps • https://www.quora.com/What-are-the-more-efficient-combinations-of-Devops-tools-to-use-in-a-web-project

10. DevOps Toolbox https://hostadvice.com/blog/devops-toolbox-jenkins-ansible-chef-puppet-vagrant-saltstack/

11. Ultimate DevOps Tool list # 1 https://devopedia.org/devops

12. Ultimate DevOps Tool list # 2 https://goo.gl/images/zH2Pys

13. What is a Container? “Containerization” • Operating-system-level-virtualization – isolated (“sandboxed”) process instances • share the OS kernel, no guest OS, no hyperviser • Container Instances are built from distinct OS images • Runs natively in Host OS, versus VM runs virtually in full blown guest (via hypervisor) • Uses kernel namespaces ( for isolation) and cgroups (control groups) for resource management. • Images consists only of the application and the library dependencies required to run the app • Lightweight (efficient), flexible, portable execution environment • Decreased code footprint lends to enhanced security • Foundational component of microservices architectures • Not a new idea…remember (BSD jail, Unix chroot), LXC (Linux Containers) • https://en.wikipedia.org/wiki/Operating-system-level_virtualization

14. https://goo.gl/images/FCgPFY

15. Images and Layers • Containers are built from layered images (think .exe image versus process) • Each layers is an application dependency (read only) • Union filesystem (Overlay FS – OverlayFS, AUFS, BtrFS storage driver) • CoW (copy-on-write) https://docs.docker.com/v17.09/engine/userguide/storagedriver/imagesandcontainers/#images-and-layers

16. Layer Image (Concept)

17. What is Docker? • An open platform developing, shipping and running applications using container technology • ship, testing, and deploy code quickly • reduce the time between code development and running in production. • Docker Engine is a client-server application with these major components: • A serverwhich is a type of long-running program called a daemon process (the dockerd command) • A REST API which specifies interfaces that programs can use to talk to the daemon and instruct it what to do. • A command line interface (CLI) client (the docker command). • Registry (Docker Hub) • https://docs.docker.com/engine/docker-overview/

18. Container is not a Virtual machine! https://docs.docker.com/get-started/#images-and-containers

19. Benefits and Risks • Benefits • Flexible: Even the most complex applications can be containerized. • Lightweight: Containers leverage and share the host kernel. • Interchangeable: You can deploy updates and upgrades on-the-fly. • Portable: You can build locally, deploy to the cloud, and run anywhere. • Scalable: You can increase and automatically distribute container replicas. • Stackable: You can stack services vertically and on-the-fly. • Risks -> Security is the big one • Subject of on going discussion.. • Docker Engine (daemon itself), registry (.e.g. docker hub), intrinsic security of kernel • Limited ability to reason effectively across abstraction boundaries • https://docs.docker.com/get-started/ • https://docs.docker.com/engine/security/security/

20. Example Scenario… • Developers write code locally and shares their work with their colleagues using Docker containers. • Use Docker to push their applications into a test environment and execute automated and manual tests. • When developers find bugs, they can fix them in the development environment and redeploy them to the test environment for testing and validation. • When testing is complete, getting the fix to the customer is as simple as pushing the updated image to the production environment. https://docs.docker.com/engine/docker-overview/#docker-engine

21. Flexible OS Stack (Run Linux or Windows) Windows OS Guest Linux OS Guest QEMU: Machine Hardware Configuration 1 QEMU: Machine Hardware Configuration 2 Image Layers QEMU (Bin) • Update respective layers to extend new machine hardware and Operating System type Base Image: Bins/Libs /Dependencies Docker Engine • Running container instance: build from image • Shares host OS with other containers Linux Virtual Machine XenServer Hypervisor

22. Hypervisor NIC (eth0) X VM_NET (eth0) Virtual Machine (VM) Host X tap1 tap0 X X X br0 Container 1 Container 2 tap0 tap0 eth0 eth0 X X X X br0 br0 X X QEMU QEMU X Hardware Emulation (x86 Intel) Hardware Emulation (x86 Intel) X NIC E1000 NIC qemu-system-x86_64 -m 8192 -vnc :2 –net nic,vlan=0,macaddr=00:80:7F:31:41:72,model=e1000 -net tap,vlan=0,ifname=tap0,script=/etc/qemu-ifup -usbdevice tablet [windows-image.qcow2] | [android-image.qcow2]

23. Physical interface NIC (eth0) X VM_NET (eth0) X bridge0 tap0 X X X X X X X X X X Ubuntu 14.04 base image Container 2 Container 3 Container 4 Container 1 Container 5 Container 6 Container 9 Container 7 Container 8 Virtual Machine (VM) Node: Mint 17.1 Layer 2: Windows 7 Windows image Layer 1: Firebox Layer 1: Qemu Firefox image Base image: Ubuntu 14.04 Base image: Ubuntu 14.04

24. Demo Docker Containers

25. What didn’t we discuss about Docker? • Orchestration tools and Microservices • Docker machine • install Docker Engine on virtual hosts, and manage the hosts with docker-machine commands • local machines (Mac or Windows) • company network (data center) • Public cloud providers like Azure, AWS https://docs.docker.com/machine/overview/

26. 2. Docker Compose • tool for defining and running multi-container Docker applications • Define YAML formatted file to configure your application’s services. • Execute the file as (single command) to create and start all the services from your configuration • This is microservices! • Transient, lightweight services composed of cooperative containers! • This is DevOps • Automation (repeatable and reproducible) • Infrastructure as Code (IaC) • Environment Parity https://docs.docker.com/compose/overview/

27. 3. Docker Swarm • Cluster management integrated with Docker Engine • Decentralized design • Declarative service model • Scaling • Multi-host networking • Service discovery • Load balancing • Security • TLS mutual authentication and encryption to secure communications • Rolling updates • At rollout time you can apply service updates to nodes incrementally https://docs.docker.com/engine/swarm/#feature-highlight

28. Cloud Orchestration • Open stack (leading open source on-premise (private cloud) -- IaaS • Template based (YMAL format) • describes the infrastructure for a cloud application in a text file that is readable and writable by humans, and can be checked into version control, diffed • Infrastructure resources that can be described include: servers, floating ips, volumes, security groups, users, etc. • primarily manages infrastructure, but the templates integrate well with software configuration management tools such as Puppet and Chef. • Seeks for compatibility with AWS (Cloud formation) • i.e. Portable cloud infrastructure provisioning • Easy to support containers • Built in container service: Openstack Magnum • https://github.com/openstack/heat-templates/blob/master/hot/F18/WordPress_Native.yaml • https://wiki.openstack.org/wiki/Heat • https://wiki.openstack.org/wiki/Magnum

29. Mike Metralhttps://getcarina.com/docs/best-practices/container-technologies-orchestration-clusters/

30. Microservices (Orchestration) Main goal: provide automated container management as well as guarantees for multi-container services and container engines. • Marathon with Apache Mesosphere(Scheduler) and DC/OS • Multiple physical node scaling (treated as one machine) • Kubernetes (Google Inc.) • Large scale service oriented design • “Self-Healing” services and Load balancing • Swarm (Docker) • Standard basic clustering tool • Native Docker API for third party tool integration https://docs.mesosphere.com/overview/, https://github.com/coreos/fleet/blob/master/Documentation/fleet-k8s-compared.md

31. Google Docs (An Example) Alice Bob Kubernetes Master/Proxy Kubernetes Agent Kubernetes Agent Kubernetes Node Kubernetes Node Persistent Storage Shared Hardware = Service Pod = Container

32. A survey of .Net Implementations • Microsoft actively develops and maintains four (4) distinct .Net implementations • .Net framework • Current version 4.7.x • Windows (desktop) applications. Includes Windows specific APIs, such those the utilized with Winforms, WPF, and WCF. • .Net Core • Cross platform (Windows, Linux, macOS) • Current version 2.1 • server and Cloud (microservices), service oriented (Web) applications ASP.NET Core • Version 3 will include support for Winforms and WPF (Windows only) • Universal Windows Platform (UWP) • building modern, touch-enabled Windows applications and software for the Internet of Things (IoT). unify the different types of devices that you may want to target, including PCs, tablets, phablets, phones, and even the Xbox. • provides many services, such as a centralized app store, an execution environment (AppContainer), • C# and VB.NET, the .NET APIs are provided by .NET Core

33. Survey of .Net Implementations (cont.) Xamarin (Mono) • Often use cases where a small runtime is required • Used for Xamarin applications on Android, Mac, iOS, tvOS and watchOS and is focused primarily on a small footprint. • also powers games built using the Unity engine. • Xamarin Native and Xamarin Forms: https://docs.microsoft.com/en-us/xamarin/#pivot=platforms&panel=Android • .Net standard (replacement for portable class libraries (PCL) ) – latest version: 2.0 • An API (BCL) standard implemented by all .Net implementations • enables portability across different .NET implementations, effectively allowing code to run everywhere. .Net standard Source: https://docs.microsoft.com/en-us/dotnet/standard/components Advice Source: https://docs.microsoft.com/en-us/dotnet/standard/choosing-core-framework-server .Net Native: https://docs.microsoft.com/en-us/dotnet/framework/net-native/

34. .Net Implementations https://social.msdn.microsoft.com/Forums/vstudio/en-US/7035edc6-97fc-49ee-8eee-2fa4d040a63b/what-are-differences-between-net-framwork-net-standard-and-net-core?forum=clr

35. .Net Implementations -- Architecture Overview https://blogs.msdn.microsoft.com/cesardelatorre/2016/06/27/net-core-1-0-net-framework-xamarin-the-whatand-when-to-use-it/

36. Xamarin Native Versus Xamarin Forms

37. Xamarin Architecture https://docs.microsoft.com/en-us/xamarin/android/internals/architecture

38. Prototype: UWP, Android, iOS

39. What is .Net Core? • open-source, general-purpose .Net platform maintained by Microsoft and the .NET community on GitHub • Cross-platform: Windows, macOS and Linux • Consistent across architectures: multiple architectures, including x64, x86, and ARM. • Command-line tools: command-line tools for local development and in continuous-integration scenarios. • Flexible deployment: install machine wide or install side-by-side • include with app, Designed for containerization • Compatible: .NET Core is compatible with .NET Framework, Xamarin and Mono, via .NET Standard • Open source: MIT and Apache 2 licenses. https://docs.microsoft.com/en-us/dotnet/core/about

40. Core CLI (command line interface) • Cross platform toolchain for developing .Net applications • Principle interaction component for • end users, higher level tools (IDE such as Visual Studio), other orchestration environments ( e.g. Docker ) • a NuGet client (stay tuned) • CLI Command structure: the driver (“dotnet”), following by a command (“verb”) • Driver performs two functions: (1) run a command, (2) run a framework dependent app • “dotnet new console” -- create a new console app • “dotnet build --output /build_output“ -- build output • “dotnet /build_output/my_app.dll“ -- run the app https://docs.microsoft.com/en-us/dotnet/core/tools/?tabs=netcore2x

41. NuGet • Package Manager for .Net • enables developers to create, share, and consume code • Microsoft supported mechanism for sharing .Net code as “packages” • A "packages" contains compiled code (as DLLs) along with other content needed in the projects that consume these packages • defines how packages for .NET are created, hosted, and consumed • A single zip file with a .nupkg extension • Contains a compiled code (.dll), Related files, and descriptive manifest • Dependency Management • Builds and maintains dependency graph on behalf of the project • Distributed with Visual Studio • Can host packages privately in the cloud (such as on Azure DevOps), • https://docs.microsoft.com/en-us/nuget/what-is-nuget

42. NuGet Model Source: https://docs.microsoft.com/en-us/nuget/what-is-nuget

43. The Basic process.. • “dotnet new console” • Creates an up-to-date project: “Hello.csproj” • The project files specifies everything need to restore dependencies, and build the program • Uses Nuget tools: MSBuild: (Creation/Consumption – restore ) as part of CLI driver “dotnet” • OutputType tag specifies the project type: library or executable (console) • TargetFramework tag specifies the .Net implementation being targeted. • As of .net Core 2.0. each of the CLI driver command: “new”, “build”, “run” implicitly run “restore” (i.e. dotnet restore) which performs the following operations using NuGet • analyzes the .csproj file • downloads the file dependencies (or grabs from your machine cache) • writes the obj /project.assets.json file • the package reference lists https://docs.microsoft.com/en-us/dotnet/core/docker/docker-basics-dotnet-core https://docs.microsoft.com/en-us/nuget/reference/msbuild-targets

44. Demonstrate .Net Core with Docker Visual Studio Tools for Docker https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/docker/visual-studio-tools-for-docker?view=aspnetcore-2.1 • create .Net core project • Add -> container orchestration support

45. Sources [1] - “Secure DevOps Process and Implementation” Software Engineering Institute (SEI), Carnegie Mellon University, January 11, 2017, https://www.sei.cmu.edu/training/P116.cfm [2] - http://www.agiletestingframework.com/devops-getting-old-deja-vu-feeling/ [3] - http://www.agilebuddha.com/agile/demystifying-devops/ [4] - https://github.com/openstack/heat-templates/blob/master/hot/F20/WordPress_Native.yaml [5] - https://docs.mesosphere.com/overview/ [6] - https://github.com/coreos/fleet/blob/master/Documentation/fleet-k8s-compared.md [7] - https://en.wikipedia.org/wiki/OpenStack