Engineering teams are improving their Claude Code workflows every day, but making Claude iterate and test its own work is still a challenge. Agents rarely get a complex feature right after a one-shot, and they usually need human intervention to give them the feedback they need to iterate. A feature might take ~10 edits to get right, many of which are obvious after running simple smoke or UI tests. More recently, some devs have been trying to solve this using the Ralph loop.
When you give agents access to real infra and give them rules to self-verify, you can get stronger outputs and your agent can work longer autonomously.
Why local dev isn’t enough for agents
The modern application is complex and layered. It’s difficult to isolate certain components without fundamentally changing an app’s behavior. It’s also quite difficult to approximate production behavior locally.
Agents can test single services or components of an app locally, but that won’t give an accurate representation of what happens when a code change is deployed to customers. Your agent can iterate and review as much as it wants locally, but you’ll have to keep in mind that their review doesn’t mean much when their environment ≠ production.
“But it works on my agent’s machine”
Agent-written code is a review burden because it often misses important context about an app or codebase. The best engineering teams need someone who knows the codebase inside out, and can recognize when an agent’s solution needs work. Obviously, that alone won’t replace rigorous testing and QA, but it’s an equally-important quality gate.
Agents can’t always get this right, but they can get closer to the real solution when given the right tools.
An agent might get something working locally then mark it ready for review. For the sake of example, let’s say it’s something as simple as a React button:
- A coding agent builds a React button component
- Tests it locally in Storybook/standalone → it works perfectly
- Changes get merged into full application → it breaks
- A dev finds out why it breaks:
- The button needs a backend endpoint that doesn’t exist yet
- Redis cache key conflicts with the existing data flow
- Postgres schema doesn’t support the new feature’s data model
- CORS, auth, rate limiting only exist in full stack
Unless the agent has a platform in which it can see how code changes respond in a real environment, devs need to pick up the slack here. The devs end up not saving much time vs. without a coding agent, because they have to:
- understand the code that the agent wrote, and why
- figure out what went wrong, and write new code and/or new prompts for the agent
- check their solution in staging
Where ephemeral environments come in
Engineering teams rely on ephemeral environments when they need to test new features. Teams need remote environments where they can run automated tests, get product/design review, and do manual QA. Realistically, this doesn’t work on staging due to PR volume bottlenecks and overall risk of “breaking” staging. These environments are GitOps-enabled, in that they automatically respond to code changes, e.g. an environment will spawn when someone opens a PR, and rebuild to reflect the latest commit. This keeps environment management low maintenance.
Agents run into the exact same problems (and perhaps even faster). You’d never give your agent access to staging either, given the security and stability risks. Ephemeral environments are secure sandboxes for agents to self-validate their code changes.
In order for this to work, your agent will need a way to interface with the environments, either through CLI or MCP. From there, you can incorporate these actions into your agentic dev loop and get closer to continuous building and validation.
A more continuous agentic dev loop
In our (simple) example from earlier, an agent wrote code for a React button and tested it locally. The button broke in staging because the agent wasn’t able to verify then fix the code.
After giving your coding agent access to an ephemeral environment, it can get into a stronger feedback loop and generate a more complete, working feature (or at least get pretty close).
Here’s what that feedback loop might look like:
- Agent adds button to React UI
- Runs tests locally, validates → button renders
- Pushes to GitHub, ephemeral env spawns
- Clicks button in full app → 404 error (no endpoint)
- Agent adds Python endpoint
- Pushes again, environment rebuilds
- Agent clicks button → 500 error (missing Postgres columns)
- Agent adds migration
- Pushes again, env rebuilds
- Clicks button → CSV downloads
Without an ephemeral environment, the agent wouldn’t have delivered past the second step here. You can define a workflow loop for your agents to follow, and they can apply such a pattern to every code snippet they write.
How do you trust the loop?
Right now, devs accept that agents can write good quality production code. It does take a combo of strong prompting and usually a few revisions. The validation part of the pipeline is where things get complicated: non-deterministic outputs are understandably difficult to verify.
Agent self-validations only work if you assign smart rules/guidelines, and keep these consistent for every iteration. You might have a pipeline like this:
- reset state
- run smoke tests
- run E2E tests
- read logs
And once you bring in test infrastructure (like ephemeral environments) you can ensure that these checks are run against a complete and consistent environment after every iteration.
Setting up an iteration pipeline like this will shift human validation left, so that it’s reserved for larger-scale revisions, instead of obvious bugs that your agent can catch on its own.
More effective agentic dev
Coding agents are built for more than just writing code: with the right prompting and plugins, they can pick up tasks further down in the SDLC, like testing, QA, and other validations. When you set up systems for them to self-validate their work, you can get longer agentic dev loops and closer-to-correct outputs. Most of the time, these validations catch trivial bugs that would’ve been found later by your dev team.
You can give your agents full-stack environments so they can self-validate and correct their code, before your team even sees it. Try Shipyard and set up your agentic dev pipeline (or have your agent do it for you).
