Cloud Native NetflixOSS Services on Docker

1. Cloud Native NetflixOSS Services on Docker Andrew Spyker (@aspyker) SudhirTonse (@stonse)

2. Agenda • Introduction • NetflixOSS, Cloud Native with Operational Excellence, and IBM Cloud Services Fabric • Docker Local Port • Docker Cloud Port

3. About Andrew @aspyker ispyker.blogspot.com • IBM - Cloud Performance Architecture and Strategy • How did I get into cloud? • Performance led to cloud scale, led to cloud platforms • Created Mobile/Cloud Acme Air • Cloud platforms led to NetflixOSS, led to winning Netflix Cloud Prize for best sample application • Also ported to IBM Cloud - SoftLayer • Two years focused onIBM Cloud Services Fabric and Operations • RTP dad that enjoys technology as well as running, wine and poker

4. About Sudhir • Manages the Cloud Platform Infrastructure team at Netflix • Many of these components have been open sourced under the NetflixOSS umbrella. • Sudhiris a weekend golfer and tries to make the most of the wonderful California weather and public courses.

5. NetflixOSS on Github • NetflixOSS is what it takes to run a cloud service and business with operational excellence • netflix.github.io • 40+ OSS projects • Expanding every day • Focusing more on interactive mid-tier server technology today

6. NetflixOSS Categorized OR Other IaaS

7. Netflix OSS – Application Container/Services App Instance Eureka Server(s) Hystrix Dashboard NoSQL (Cassandra) Persistence Astyanax NetflixOSS Library Function Functional Reactive Programming Monitoring Logging Distributed Caching Resiliency/Fallback RPC (Routing/LB) Distributed Co-ordination (Zookeeper) REST Framework/Bootstrapping/DI Metrics Blitz4J Curator Karyon/Governator Servo Hystrix RxJava Turbine Ribbon/Eureka EVCache Properties/Configuration Archaius Service Requests IPC (smart LB) Data Access/Caching Cassandra Metrics Dashboard Config/Insights

8. Elastic, Web and Hyper Scale Doing This Not Doing That Source: Programmableweb.com 2012

9. Elastic, Web and Hyper Scale Durable Storage Load Balancers … … … … Front end API (browser and mobile) Booking Service Temporal caching Authentication Service

10. HA and Automatic Recovery Feeling This Not Feeling That

11. Highly Available Service Runtime Recipe Eureka Server(s) Micro service Implementation Eureka Server(s) Eureka Server(s) Web App Front End (REST services) Execute auth-service call Call “Auth Service” Ribbon REST client with Eureka Hystrix App Service (auth-service) Fallback Implementation Karyon

12. IaaS High Availability … … … … … … … … … … … Region (Dallas) DAL01 … … … … … Datacenter (DAL06) DAL05 Global Load Balancers Eureka Local LBs Web App Auth Service Booking Service Cluster Auto Recovery and Scaling Services

13. Only proof is testing!Chaos Testing … … … … … … … … … … … Region (Dallas) … … … … … DAL06 ✗ Datacenter (DAL05) DAL01 Global Load Balancers Eureka Local LBs Web App Auth Service Booking Service ✗ Cluster Auto Recovery and Scaling Services Chaos Gorilla Videos: bit.ly/noss-sl-blog, http://bit.ly/sl-gorilla

14. Continuous Delivery Reading This Not This

15. ContinuousDelivery … … v Baked to SoftLayer Image Templates (or AMI’s) Continuous Build Server Cluster v1 Canary v2 Cluster V2

16. Operational Visibility If you can’t see it, you can’t improve it

17. Operational Visibility Incidents Metric/Event Repositories Uptime Hystrix/Turbine Servo … … LogStash/Elastic Search/Kibana Web App Auto Service

18. Current IBM Cloud Services Fabric 7. Uptime Service 2. Global Load Balancers … 3. Region (us-south-1) 3. Datacenter (DAL01) – Fabric services are clustered across 3 DC’s … … … … … … 3. Datacenter (DAL05) – Apps are clustered across 3 DC’s Currently VM based 8. Logstash Kibana 1. Eureka Datacenter (DAL06) Devops 5. Asgard Service Your front end service A service you depend on Your mid tier service 2. Local LB Service Code and Image Build 6. Imaginator Service Tested baseimages /w agents Your built code 4. Cluster Auto Recovery and Scaling Services

19. Agenda • Introduction • Docker Local Port • Lessons Learned • Open Source • Docker Cloud Port

20. Demo Start Start demo loading here

21. Docker “Local” Setup … … … … … Region (docker-local) Docker-local-1b … … … … Docker-local-1c Service Discovery (Eureka) Datacenter (docker-local-1a) Load Balancer (Zuul) Users Devops Console (Asgard) Acme Air Web App Acme Air Auth Service Devops (admin) Cassandra Web App Auth Service Cluster Auto Recovery & Scaling Service (Microscaler) Skydock SkyDNS Node Blue and green boxes are container instances

22. Why Dockerfor our work? • Because we could, actually … • To show Netflix cloud platform as portable to non-VM clouds • Help with NetflixOSS understanding inside of IBM • Local Testing – “Cloud in a box” more production like • Developers able to do larger scale testing • Continuous build/test tool systems able to run at “scale” • Public Cloud Support • Understand how an container IaaS layer could be implemented • So far, proof of concept, you can help continue • More on that later (hint open source!)

23. Two Service Location Technologies? Eureka Server(s) Eureka Server(s) Micro service Implementation Eureka Server(s) Web App Front End (REST services) Execute auth-service call Call “Auth Service” Ribbon REST client with Eureka App Service (auth-service) Karyon DockerHost SkyDNS Skydock Eureka Auth Service Micro Service Docker Daemon Event API

24. Service Location Lessons Learned • Both did their job well • SkyDNS/SkyDock for container basic DNS • Must be careful of DNS caching clients • Eureka for application level routing • Interesting to see the contrasts • Intrusiveness (Eureka requires on instance/in app changes) • Data available (DNS isn’t application aware) • Application awareness (running container != healthy code) • Points to value in “above IaaS” service location registration • Transparent IaaS implementations struggle to be as application aware • More information on my blog http://bit.ly/aws-sd-intr

25. Instance Auto Recovery / Scaling • Auto scaling performs three important aspects • Devops cluster rolling versions • Auto recovery of instances due to failure • Auto scaling due to load • Various NetflixOSS auto scalers • For NetflixOSS proper – Amazon Auto Scaler • For SoftLayer port – RightScale Server Arrays • For Docker local port – we implemented “Microscaler”

26. Microscaler Agent Architecture Dockerhost Web App i001 Web App i002 Auth Service i001 Microscaler Agent Microscaler Auth Service i002 Microscaler RESTor CLI • OSS at http://github.com/EmergingTechnologyInstitute/microscaler • Microscaler service, agent are containers • Microscaler has CLI remote client and REST interface • Note: • No IBM support, OSS proof of concept of auto scaler needed for local usage • Works well for small scale Docker local testing Docker Remote API

27. Microscaler CLI/REST usage • Login CLI: • mslogin --target <API URL> --user user01 --key key • Login REST: • curl-X POST -H "Content-Type: application/json" -d '{"user":“user01","key":“key01"}' http://localhost56785/asgcc/login • {"status":"OK","token":"a28e7079-db0b-4235-8b9b-01c229e02e9a“} • Launch Config CLI: • msadd-lconf --lconf-name lconf1 --lconf-image-id cirros --lconf-instances-type m1.small --lconf-key key1 • Launch Config REST: • curl -X POST -H "Content-Type: application/json" -H "authorization: a28…e9a" -d '{"name":"mylconf","image_id":”img1","instances_type":"m1.small","key":"keypair"}' http://locahost:56785/asgcc/lconfs • {"status":"OK”} • ASG CLI: • msadd-ms --ms-name asg1--ms-availability-zones docker01,docker02 --asg-launch-configuration lconf1 --asg-min-instances 1 --asg-max-instances 3 --asg-scale-out-cooldown 300 --asg-scale-in-cooldown 60 --asg-no-load-balancer--asg-domain docker.local.io • msstart-ms --ms-name asg1 • ASG REST: • curl -X POST -H "Content-Type: application/json" -H "authorization: a28…e9a" -d '{"name":”asg1","availability_zones":[”az1"],"launch_configuration":”lconf1","min_instances":1,"max_instances":3}' http://localhost:56785/asgcc/asgs • {"status":"OK“} • curl -X PUT -H "Content-Type: application/json" -H "authorization: a28e…e9a” http://localhost:56785/asgcc/asgs/myasg/start • {"status":"OK”}

28. Working with the Docker remote API • Microscaler and Asgard need to work against the “IaaS” API • Docker remote API to the rescue • Start and stop containers, query images and containers • Exposed http://172.17.42.1:4243 to both • Could (should) have used socket • Be careful of security once you do this • Found that this needs to easily configurable • Boot2docker and docker.iodefault to different addresses • Found that current API isn’t totally documented • Advanced options not documented or shown in examples • Open Source to the rescue (looked at service code) • Need to work on submitting pull requests for documentation

29. Region and Availability Zones • Coded Microscaler to assign availability zones • Via user_data in an environment variable • Need metadata about deployment in Docker eventually? • Tested Chaos Gorilla • Stop all containers in a single availability zone • Tested Split Brain Monkey • Jepsen inspired, used iptables to isolate Docker network • Eureka awareness of availability zones not there yet • Should be an easy change based on similar SoftLayer port

30. Image management • Docker and baked images are kindred spirits • Using locally built images - Easy for a simple demo • Haven’t yet pushed the images to dockerhub • Considering Imaginator(Aminator) extension • To allow for Docker images to be built as we are VM’s • Considering http://www.packer.io/ • Or maybe the other way around? • Dockerfiles for VM images?

31. Using Docker as an IaaS? • We do all the bad things • Our containers run multiple processes • Our containers use unmanaged TCP ports • Our containers run and allow ssh access • Good • Get all the benefits of Docker containers and images • Only small changes to CSF/NetflixOSS cloud platform • Bad • Might not take full advantage of Docker • Portability, container process optimizations, composability • Considering more Docker centric approaches over time

32. Where can I play with this? # on boot2docker or docker.io under virtual box Ubuntu git clone http://github.com/EmergingTechnologyInstitute/ acmeair-netflixoss-dockerlocal cd bin # please read http://bit.ly/aa-noss-dl-license ./acceptlicenses.sh # get coffee (or favorite caffeinated drink), depending on download speed ~ 30 min ./buildsimages.sh # this is FAST! – but wait for about eight minutes for cross topology registration ./startminimum.sh # Route your network from guest to docker network (http://bit.ly/docker-tcpdirect) ./showipaddrs.sh # Look at the environment (Zuul front end, Asgard console, Eureka console, etc.) Browse to http://172.17.0.X All Open Source Today!

33. Docker “Local” Setup … … … … … Region (docker-local) Docker-local-1b … … … … Docker-local-1c Service Discovery (Eureka) Datacenter (docker-local-1a) Load Balancer (Zuul) Users Show demo here Devops Console (Asgard) Acme Air Web App Acme Air Auth Service Devops (admin) Cassandra Web App Auth Service Cluster Auto Recovery & Scaling Service (Microscaler) Skydock SkyDNS Node Blue and green boxes are container instances

34. Agenda • Introduction • Docker Local Port • Docker Cloud Port • Lessons Learned

35. Docker Cloud on IBM SoftLayer DAL05 Datacenter DAL06 Datacenter Dockerhost Registry Zuul Eureka Cassandra Microscaler Asgard Skydock API Proxy SkyDNS Dockerhost Dockerhost Web App i002 Web App i004 Auth Service i002 Auth Service i004 Skydock Microscaler Agent Web App i001 Web App i003 Auth Service i001 Microscaler Agent Auth Service i003 Skydock Docker Remote API Docker Remote API SoftLayer Private Network

36. Networking • Docker starts docker0 bridge to interconnect single host instances • We assigned the subnet of the bridge to be a portable subnet within our SoftLayer account within a VLAN • We routed all traffic to the actual private interface • This allows network to work seamlessly • Between datacenters • Across hardware firewall appliances • To external load balancers • To all other instances (VM’s, bare metal) in SoftLayer • This allowed for easy networking between multiple Docker hosts

37. Docker API and Multi-host • Once you have multiple Docker hosts • You have multiple Docker remote API’s • Wrote “API Proxy” to deal with this • Not the best solution in the world, but worked • Considering how this works with existing IaaS API • Single SoftLayer API handles bare metal, virtual machines • How to keep the API Docker compatible • Maybe other more Docker centric approaches coming?

38. Image Management • Currently using standard Docker private registry • Considering how this could be integrated with SoftLayer Image management system • Use optimized cross datacenter distribution network • Expose Docker layered versions through console • Again, important to not lose Docker value in image transparency and portability

39. Docker Cloud on IBM SoftLayer DAL05 Datacenter DAL06 Datacenter Dockerhost Registry Zuul Eureka Cassandra Microscaler Asgard Skydock API Proxy SkyDNS Demos 1-1 today or tomorrow at Jerry’s session Dockerhost Dockerhost Web App i002 Web App i004 Auth Service i002 Auth Service i004 Skydock Microscaler Agent Web App i001 Web App i003 Auth Service i001 Microscaler Agent Auth Service i003 Skydock Docker Remote API Docker Remote API SoftLayer Private Network

40. Questions?