Mindset docsOperatorsTest behavior
Test behavior
Proves every step of a script works, checks how an agent behaves across many runs, and compares a change before you make it.
Skills for your coding agent
See it running: Trip planner, Store assistant, Ask your data, Guided intake, Help widget
Ask your AI client#
Write acceptance tests for the invoice exceptions agent: it calls the comparison function, its supplier query names the invoice and PO numbers, and it never registers a query when the invoice matches. Run them, then compare it on a cheaper model.What you get back#
Two kinds of test, and your AI client runs both.
| Test parameters | Acceptance tests | |
|---|---|---|
| Proves | Every step of a script or function runs, and every operation it calls works | The agent behaves as it must, across many runs |
| Writes | Real, within a list an admin agreed | Refused before they leave |
| Result | Verified, or each step says why not | Pass or fail against a threshold, with a recorded reason |
| Blocks going live | Yes, over untested operations | No. You read the result and decide |
The go-live gate and what makes an operation lapse are on Versions, testing and going live.
Test parameters#
Keep one set per path through the script, so every branch is taken: an invoice that matches its PO, one with a difference the contract allows, and one it doesn't, which registers a supplier query. A step that wasn't exercised says why: Branch not taken, Not reached, Refused, Write not agreed or Failed. Branch not taken means you need another case.
A go-ahead covers that exact write list and is recorded with who gave it. If the list changes, you're asked again. Tested status is kept per environment, so run them in Production too.
Acceptance tests#
A test is a situation plus criteria. Some are checked by code (invokesTool, doesNotInvokeTool, toolParametersContain, answerContains, respondsWithin), others judged by a model (doesNotHallucinate, staysInCharacter, responseGroundedInKnowledge, satisfiesAssertion). A refused write still counts as a call, so test what the agent tried to send with toolParametersContain.
Results are statistical. A judged criterion runs 5 times by default and passes when the lower bound of a 95% confidence interval on its pass rate clears 0.5, which at 5 runs means 5 out of 5. You can raise the runs (up to 200) or the threshold.
A simulated user tests a conversation instead of one answer: a second model with a persona and a goal, such as a supplier who wants the query dropped without sending a delivery note. The whole conversation is scored.
An accepted test can't be edited or deleted. Amend replaces it and Retire stands it down, each with a reason, so the bar only moves where people can see it.
Compare a change before you make it#
An experiment runs your accepted tests against a changed setup, a different model or part of the system prompt, beside the current one. Each criterion comes back Improved, Regressed or No detectable difference, with cost and speed for each. Nothing touches the live agent, and there is no way to split live traffic between two versions.
What you decide#
- What the agent must always do, and at least one thing it must never do.
- Where test writes land.
- How much to spend. Tests run the agent many times, and the judge and simulated user are model calls too.
- Whether a version is good enough to activate. Failing acceptance tests don't stop it.
Done when#
- Every script and function the agent runs reads Verified in the environment it goes live in.
- The agent has accepted tests for what it must do and must not do, and they run daily.
- You can say which passed on the version you last activated.
In the Studio#
Test parameters run from the agent's Script tab, or a function's Preview tab, with Run. Acceptance tests live in Optimize → Acceptance tests: describe a test to the Optimize Agent, Dry-run the draft, Accept it, then Run all. Under Scheduled runs, Run these tests daily runs the set at a UTC time. Optimize → Experiments compares a change, and Optimize → Runs shows what each test run cost.