Case Study: MaestroQA

We spoke with Amrisha Sinha who is responsible for DevOps and Infrastructure Automation at MaestroQA, a Shipyard customer that provides QA software for the best customer success teams. Amrisha shared the challenges of managing build pipelines during a period of high growth for the company and their success with implementing Shipyard to manage and handle ephemeral environments for their QA process.

The Challenge

Maestro has been experiencing rapid growth in its development organization over the last few years. As the team increased in size, so did the need for a more robust QA team and process.

“Customer success is in our DNA—I knew I needed to get ahead of any problems and increase our understanding throughout the entire build pipeline. We knew our developers were writing good code and shipping quality software, but we didn’t always have the test data to back that up.”

In September of 2020, Amrisha started building out a QA team alongside the growing development team and put them in charge of building out the test suite and resolving any subsequent failures found in pre-production. At this time, open pull requests were merged to a single, production-like staging environment, and their Cypress tests would run in CircleCI prior to code being merged to production. When the development team was smaller, this might look like 2-3 pull requests per day. As Maestro’s development team grew in size, this looked more like 8-9 pull requests being merged onto a single staging environment at one time.

In the event of failing tests, it became increasingly difficult to diagnose which of the pull requests actually caused the failure. Maestro’s build pipeline became increasingly stalled on these hard-to-diagnose failures that often resulted in the dreaded cherry-pick deploy.

The Solution

With only a single staging environment for running end-to-end tests, Amrisha knew she had to solve this budding problem.

“I was in the process of trying to solve this problem for my QA team when I learned about Shipyard. I was considering using ephemeral environments for running end-to-end tests earlier in the pipeline but knew it’d take at least 4 months to build myself with ongoing maintenance.”

With Shipyard, Amrisha could instrument her ideal solution—spinning up ephemeral environments to run tests against on every pull request— in far less time with less resources and little to no maintenance cost. Shipyard handles the creation of full stack ephemeral environments on every pull request and interacts with CircleCI via Shipyard’s CircleCI Orb to run end-to-end tests against the production-like environments.