Case Study: Applying Automated Optimization Tooling to Microservice Environments that Scale Safely at Ancestry.com and the Learnings

In 2018, Ancestry migrated into Amazon’s public cloud. Hundreds of applications that were hosted on physical hardware were migrated to the cloud onto internally supported Kubernetes environments. This established a microservice architecture for many applications at Ancestry. In 2020, Ancestry’s cloud expenses goals were set at $12M in Operating expenses savings over 2 years. To overcome the problem of scale, an automated right-sizing effort needed to take place. Ancestry purchased a tool called Opsani. Opsani allowed the team to use dynamic canary testing in combination with a CanaryAdvisor to act as the AI agent. Opsani was integrated into Ancestry’s continuous integration and continuous delivery platform to automate right-sizing and help reach cost optimization targets with limited resources. After a successful proof of concept was developed by the DNA Matches agile team, there was a company-wide integration of Opsani into Ancestry’s CI/CD platform. The tool would need to scale out continuous optimization efforts across many of the teams utilizing Kubernetes and virtual machine infrastructure. This case study details the experience and knowledge gained from scaling automation optimizations using an AI agent with more than 200 individual applications across 30 agile teams. The challenges introduced by the rate of changes in the system and monitoring feedback delays began to affect the customer experience with the product. As teams adopted optimization changes at higher rates, new guard rails were introduced to control the release cycles and ensure stability for paying customers.