Most software teams work multiple non-production environments, each one increasing in complexity as they approach production. Test environments fill in the gap between development and staging environments: they enable developers with realistic testing so they can spend more time fixing bugs, and less time waiting for available infrastructure.
While these two environment definitions might vary between orgs, there are still some common distinguishers between test environments and staging.
What is a test environment for?
A test environment (sometimes referred to as a QA environment) is used to verify new software features before they hit production. Test environments are for checking, both manually and with automated tests, that each new feature:
- Doesn’t break other features
- Functions as designed
There are several different levels to test at within this environment, including (but not limited to):
- Frontend/UI behaviors
- Database operations
- Network configurations
Test environments are most effective when they’re used earlier on in the deployment pipeline. This way, engineers can find out ASAP when a feature needs work, and bring it back to development.
What is a staging environment for?
Staging environments are the final quality check before production. Out of all an org’s environments, they’ll most closely resemble production: they have the same services, or at least mocks, and a production-like database. That being said, they should still be 100% isolated from production.
Because they’re so complex and similar to production, staging environments are very expensive to host and maintain. They also can be fragile. Because of this, most companies only have a few staging environments available.
Staging allows engineers to test their features amongst existing features; they’ll also run their automated test suite against this environment to catch any bugs that might appear from dev/prod parity (staging can be too different from the environment preceding it).
Where do test environments fit in the development pipeline?
Traditional, “minimal” development pipelines often just have local dev environments deploy directly to staging environments. This is an environment anti-pattern:
- There are often days between staging deploys, unnecessarily increasing dev lead time
- Everyone needs to test their code, so the staging queue adds up quickly
- Testing features == trying to break them. Broken staging is an inconvenience to everyone
- Features should be tested in isolation, then against other features. This can’t really happen in staging
In order to make sure features get tested thoroughly and in isolation, teams should incorporate test environments at the pull request level. This means right after a developer opens a PR, a test environment should provision, reflecting the latest code changes.
At this point, the app’s CI workflow should also trigger, pointing automated tests at the live environment. Once the tests have executed, the developer will see what fails, which lets them return to the inner dev loop and pick up where they left off.
Test environments should minimize the amount of human review needed on a feature; they can (and should) accommodate full E2E tests and realistic database operations. This way, developers can loop in peers for code review, design review, and UAT after all automated checks are passed.
This is where staging environments come in: once a test environment fulfills its purpose by ensuring a feature is unbreakable, tested, and functions as designed, the feature will be staging-ready.
What should I only test in staging?
Test environments are pretty effective for the majority of tests and validations. However, since they aren’t always infrastructure-identical to production, there are a few things that should merit a staging environment.
For many teams, external APIs aren’t integrated until staging. If you have a feature that interfaces with a third-party API in any capacity, you should delay its most critical testing until it hits staging.
Another shortcoming of “overly lightweight” test environments can be data. If your test environment doesn’t have a representative database with realistic data, your E2E tests won’t reveal enough about your new feature’s stability. This can undermine the value of your test environments, as they’re incredibly useful for shifting E2E tests earlier (where possible).
We advocate that teams architect their test environments as close to staging and production as possible, since this allows each feature to get comprehensive isolated testing. Doing so means that most of your testing will be complete by staging, which helps you estimate how much iteration and deployment lead time a given feature might require. Some features might be too costly or complex to host in test environments, however, we suggest finding the tradeoff between test environment incremental costs and the productivity that they enable.
When is a staging environment not enough?
Smaller teams can often get away with just a staging environment (or two), since they’re rolling out features at a slower rate. These teams might be able to get isolated testing per feature without separate test environments.
The decision to adopt test environments usually comes from the engineering manager: how is your deployment lead time impacted by environment wait times? If the answer is “by more than a day or two”, your process isn’t optimized to your team. Test environments can take a lot of the lift off of staging, as you can repeatedly “break” and verify features, without necessarily worrying about impacting other team members.
We generally see teams hit 7-10+ developers when they run into staging bottlenecks that throttle their deployment velocity. This makes test environments a logical roadmap decision.
What if you’re a small team with a fragile staging setup? You might adopt test environments earlier on, even before you start to encounter staging bottlenecks. Test environments are much safer for experimentation and risks — ultimately, software stability is only guaranteed by taking risks (not from walking on eggshells).
Production-like test environments: shift testing left
At Shipyard, we offer a simple solution to staging bottlenecks: ephemeral, near-production test environments. What makes our offering stand out against traditional test environments?
- Data snapshotting: you can seed data into your PR environments, and reset your database after running tests
- GitOps automation: your environments respond to major code changes. They provision on PRs, rebuild on commits, and spin down upon merges
- Kubernetes orchestration: we run all test environments in Kubernetes clusters, so you get near-production application behavior
We’ll transpile your Docker Compose into Kubernetes manifests, so you can start testing and previewing on every PR without the engineering overhead. Try it free for yourself, and watch your development velocity improve.
