200 likes | 292 Views
Josh Holtzman from Xoom (Acq, PayPal) discusses Xoom's migration from their monolith to microservices at the 2017 Microservices Practitioner Summit.
E N D
Microservices are the Future! … and always will be Josh Holtzman, Director, Infrastructure Engineering Xoom, a PayPal Service 31 January 2017
Xoom.com A little history • Digital remittance • Founded in 2001 • Acquired bluekite.com in 2014 • Acquired by PayPal in 2016
Xoom.com A little history (translated) • Digital remittance Highly regulated environment • Founded in 2001 • Acquired bluekite.com in 2014 • Acquired by PayPal in 2016
Xoom.com A little history (translated) • Digital remittance Highly regulated environment • Founded in 2001 16 years of code and data • Acquired bluekite.com in 2014 • Acquired by PayPal in 2016
Xoom.com A little history (translated) • Digital remittance Highly regulated environment • Founded in 2001 16 years of code and data • Acquired bluekite.com in 2014 Polyglot code and persistence • Acquired by PayPal in 2016
Xoom.com A little history (translated) • Digital remittance Highly regulated environment • Founded in 2001 16 years of code and data • Acquired bluekite.com in 2014 Polyglot code and persistence • Acquired by PayPal in 2016 New rules
Throwing down the gauntlet Break up the monolith(s) • Decouple teams • Reduce time to build and deploy • Understand our resource needs • Scale appropriately
Microservices to the rescue Challenges and risks • Programming paradigms and idioms • Service discovery • Monitoring • Performance • Infrastructure as code • Build and deployment pipeline • Data ownership
Programming paradigms and idioms • Network operations • Circuit breakers • Aggressive timeouts • Retries • Throttles • API designs • RPC vs REST • Batch operations • Response code granularity • Contracts • Packaging • Metadata • Management uniform
Service discovery The service-proxy solution Host App B App A Service-proxy Zookeeper • Custom, local, layer seven load balancers • Zookeeper back-end • Apache Curator • Registration, health checks, and routing • Service Portal • Integrating with linkerd.io
Monitoring Grafana and InfluxDB • Define required measurements • persistence operations • remote calls • service endpoints • 3rd party service endpoints • Define metric types • gauges • counters • histograms • Standard naming scheme • Self-service dashboards • Time series explosion
Performance Throughput and response latency • Additional network latency has been offset by: • Reduced contention on datastores • Limiting the scope of database transactions • Optimization through observability • Throughput has improved dramatically • Latency distribution is wider • Latency sensitive APIs are deployed nearby
Infrastructure as code • TDD isn’t just for applications • Terraform and Packer for host provisioning on AWS and Vsphere • Puppet and Ansible acceptance testing using beaker • Network gear • Standardize app packaging • Docker • Contracts for deployment • Application control plane
Build and deploy pipeline • Git-flow • Branch per feature • Docker-flow • Container per branch • Seed jobs • Build job per branch • Automated and self service deployments • Dev and QA teams can choose branches to deploy and test • Fidelity of environments • Environment fidelity ∝ automation success
Data ownership • Hard problem • Start eliminating cross-domain joins now • Two years on, we are just now migrating the last auth-server client from tables to APIs • Analytics becomes more complicated
Current status • ~100 distinct microservicesacross 3 production data centers • Most new features are developed as microservices • Monoliths still exist, but are being chipped away
Lessons learned • Measure everything, and be prepared to scale your monitoring system • Application packaging contracts and delivery pipelines are mandatory • Staff a tooling team for build, test, and deployment automation • Enroll your network operations team • The infrastructure and culture we built in order to move to microservices has paid off • Elimination of the monoliths isn’t that important
Josh Holtzman jholtzman@paypal.com