AI Debugging Harness
An AI-assisted debugging system I built at Ploomber to reduce failed deployment support work.
Ploomber helped users deploy Python applications such as Streamlit and Dash without needing to manage their own infrastructure. That was useful when things worked, but when a deployment failed, the support burden could become expensive very quickly.
The idea behind the debugging harness was simple: instead of asking a human to inspect every failed deployment, let an AI system investigate the failure, reproduce it, attempt a fix in a sandbox, and return both the patch and a readable explanation.
The Idea
When a deployment failed, have an agent inspect the logs, debug the problem and suggest a fix. For this, we made the following loop:
- Inspect the deployment failure
- Form a hypothesis
- Modify the project
- Build and deploy it to a sandbox
- Observe the result
- Repeat until the deployment succeeded or it ran out of attempts
The Harness
This was not a chat wrapper around Claude Code. We built a complete execution harness around the time Claude Code was released because it could not easily be adapted to our deployment workflow, and we also wanted to explore the space ourselves.
One of the hardest parts was getting the model to reliably follow the structure required for tool calling. We spent a lot of time shaping prompts and outputs so the agent would invoke tools in the right order and with the right arguments. Looking back, a model specifically optimized for tool use and function calling would have been extremely valuable.
When a deployment failed, the harness gave the agent access to:
- User deployment logs
- Deployment status
- The user’s repository inside an isolated environment
- Internet access to read documentation
- A list of known fixes
- Code editing capabilities
- The Ploomber CLI to build projects and deploy instances
Everything happened inside a sandbox (which was simply an application deploy on the Ploomber platform since this is what our platform was made for), allowing the agent to safely experiment without affecting the user’s actual deployment.
The Hard annoying Part
Users were not deploying into fully custom environments. They were building on top of ours.
That meant many failures were not traditional application bugs. A project might work perfectly on a developer’s machine while failing on the platform because it relied on an undeclared package, a different working directory, a missing compiler, or assumptions about how the application started.
The harness had to reason about two things simultaneously:
- What the user’s application was trying to do
- What the platform expected in order to build and run it correctly
Most deployment failures happened somewhere in the gap between those two.
Where it was used
The agent started automatically as soon as a deployment failed. The user did not have to ask for help or even know the system existed.
If the harness managed to resolve the issue, it could return both the patch and a human-readable explanation describing what had gone wrong and how it was fixed.
The goal was to improve the user’s first experience with the platform. We noticed that users who successfully deployed their application within their first couple of attempts were much more likely to continue using the platform and eventually consider a subscription.
Evaluating the Harness
One advantage of this project was that evaluation was straightforward.
We had collected a good set of real deployment failures over time. By mocking deployment logs, status, errors, and infrastructure responses, we could replay those failures through the harness and verify whether it produced the expected fix.
The evaluation did not stop at comparing generated patches. We also rebuilt the modified project and attempted to deploy it using the same Ploomber CLI. A fix was only considered successful if the project could actually be built and deployed.
Having a deterministic regression suite made it much easier to iterate on prompts, tool definitions, and agent behavior without accidentally making the system worse.
Outcome
The system reduced support tickets related to failed deployments by around 80%.
And for us, it eliminated a large amount of repetitive support work. Instead of manually helping users with missing dependencies, incorrect configuration, missing compilers, or startup issues, the harness could often resolve the problem.