# How to ship an agent change without breaking it

Source: https://dev.definable.ai/blog/ship-agent-changes-safely
Published: 2026-10-15 · Definable team

Evals before publish, approvals on risky actions and a trace for every run: the three checks that make an agent safe to change.

Changing an agent is easy. A new instruction, a different model, one more app. Knowing the change didn't break something it used to do well is the hard part. Three things in Definable exist for exactly that, and they work best together.

## 1. Run evals before you publish

An eval is a set of cases: a message, and what a good answer has to do. Beside **Publish** there's a **Run evals first** switch, on by default. With it on, publishing runs every case on your draft and shows what changed before the draft goes live.

Each case is a real run, with the same model, the same cost and the same trace as production. Any tool that isn't safe to repeat, like sending an email, answers from a fixture instead, so an eval never touches the world.

A case can check that the answer:

- **contains** a phrase, matches a **regex** or is **exactly** a value
- is **valid JSON**
- **used** a tool, **never** used one, or used tools **in order**
- finished in **fewer steps** than a limit, or **under a budget**
- passes a **judge** model's reading of it

A case passes at a score of 0.7, and every eval run is compared with a baseline, so you see what got better as well as what got worse. One eval run takes up to 50 cases.

> Model answers vary. The same unchanged agent can score a little differently twice, so read a small drop as a nudge to look, not proof of a regression.

## 2. Put risky actions behind approval

Some actions shouldn't happen without a person: sending an email to a customer, issuing a refund, deleting a record. List those tools and app actions under **Needs approval** in the agent's settings.

When the agent wants to use one, the run pauses.

- **In the app,** the person sees a card with **Approve** and **Deny**.
- **On a channel,** the agent's message ends with "Reply YES to approve or NO to deny".

A waiting run has no deadline, and it carries on with the version of the agent it started on, even if you publish a new one meanwhile.

## 3. Read the trace

Every run has a trace, built live while it runs and kept after it ends. It shows each model call with its tokens, cost and time to first token, each tool call with what went in and what came out, and the moments the run waited on a person. When people vote on an answer, the votes land there too.

When an eval case fails, open its trace first. It usually shows the exact step where the agent went wrong, which is the step to fix.

## Put together

1. Make the change on a draft.
2. Publish with **Run evals first** on, and read what changed.
3. Keep anything that can't be undone behind **Needs approval**.
4. Spot-check a few traces from real traffic after it's live.

None of this slows you down much, and it turns "I think it still works" into "I checked".