Case Study: MaestroQA

Learn how Amrisha from MaestroQA uses Shipyard’s ephemeral environments to deploy faster and with confidence, testing before merge during a period of massive company growth

Amrisha Sinha on how Shipyard saves her 10-20% of time she was spending on pre-prod

"I used to spend 10-20% of time on preprod and helping QA things out..I no longer even think about it!
Customer success is in our DNA—I knew I needed to get ahead of any problems and increase our understanding throughout the entire build pipeline."

- Amrisha Sinha

MaestroQA uses Shipyard and CircleCI for E2E testing on every pull request to deliver quickly with confidence

We spoke with Amrisha Sinha who is responsible for DevOps and Infrastructure Automation at MaestroQA, a Shipyard customer that provides QA software for the best customer success teams. Amrisha shared the challenges of managing build pipelines during a period of high growth for the company and their success with implementing Shipyard to manage and handle ephemeral environments for their QA process.

The Challenge

Maestro has been experiencing rapid growth in its development organization over the last few years. As the team increased in size, so did the need for a more robust QA team and process.

“Customer success is in our DNA—I knew I needed to get ahead of any problems and increase our understanding throughout the entire build pipeline. We knew our developers were writing good code and shipping quality software, but we didn’t always have the test data to back that up.”

In September of 2020, Amrisha started building out a QA team alongside the growing development team and put them in charge of building out the test suite and resolving any subsequent failures found in pre-production. At this time, open pull requests were merged to a single, production-like staging environment, and their Cypress tests would run in CircleCI prior to code being merged to production. When the development team was smaller, this might look like 2-3 pull requests per day. As Maestro’s development team grew in size, this looked more like 8-9 pull requests being merged onto a single staging environment at one time.

In the event of failing tests, it became increasingly difficult to diagnose which of the pull requests actually caused the failure. Maestro’s build pipeline became increasingly stalled on these hard-to-diagnose failures that often resulted in the dreaded cherry-pick deploy.

The Solution

With only a single staging environment for running end-to-end tests, Amrisha knew she had to solve this budding problem.

“I was in the process of trying to solve this problem for my QA team when I learned about Shipyard. I was considering using ephemeral environments for running end-to-end tests earlier in the pipeline but knew it’d take at least 4 months to build myself with ongoing maintenance.”

With Shipyard, Amrisha could instrument her ideal solution—spinning up ephemeral environments to run tests against on every pull request— in far less time with less resources and little to no maintenance cost. Shipyard handles the creation of full stack ephemeral environments on every pull request and interacts with CircleCI via Shipyard’s CircleCI Orb to run end-to-end tests against the production-like environments.


Shifting Left

Almost all of Maestro’s test suite, including end-to-end tests, is run through CircleCI against Shipyard environments, on every pull request. This allows developers to get quick feedback on failing tests related to their code changes, shifting the responsibility for results to the developer. Developers find out if they have introduced a regression or need to update a failing test while they are still actively developing the feature, instead of having to wait for hours or days before realizing the problem.

Hardening The Test Suite

By running end-to-end tests on every pull request, the team is able to diagnose flaky vs failing tests earlier in the process. And, with tests running more often, it’s easier to spot which tests may be inherently problematic. With Shipyard, tests are run in an isolated way, per code change—and much more often— giving a clearer view of when to rewrite or remove tests. To date they’ve reduced the number of tests by 25% (from 90 to 67) and now run the full test suite multiple times an hour, instead of a few times a day.

More Time For What’s Most Important: Production

With QA environments handled automatically by Shipyard, Amrisha and her team are able to focus more on the management of production environments.

Shipyard Newsletter
Stay in the (inner) loop

Hear about the latest and greatest in cloud native, container orchestration, DevOps, and more when you sign up for our monthly newsletter.