---
name: mindset-test-agent
description: Use when testing, debugging or evaluating an AI agent in Mindset (mindset.ai) through the mindset-admin MCP server. Covers acceptance (behavior) tests run many times, test runs of scripts and functions and the go-ahead for their writes, reading runs and conversations, diagnosing why an agent misbehaved, and comparing versions or a cheaper model.
metadata:
  title: Test and diagnose an agent
  blurb: Tests an agent, finds out why it misbehaved, and compares versions or models.
  group: Mindset MCP
  order: 5
  docs: test-an-agents-behavior, versions-and-going-live, diagnose-and-improve-an-agent, check-what-happened, change-something-thats-already-live
  checked: d671041b6
  updated: 2026-10-19
---

# Test a Mindset agent and find out why it misbehaves

When you finish, every script and function the agent runs reads **Verified**, the agent has accepted tests for what it must and must never do, you can say which passed and how sure that is, and any fault has a named cause with a fix staged as a version for the person to activate. To create the agent or change its configuration, use the `mindset-build-agent` skill.

## Mindset in brief

Mindset is a hosted platform for building, running and governing AI agents. An **agent** is configuration, not code: a prompt, a model, and the resources it may use. Those resources are **scripts** (steps that must wait or happen in order), **functions** (fixed steps with one right answer), **widgets** and **connections** (the only way data enters or leaves). People manage all of it in the Studio, a web app, or through Mindset's MCP servers.

## Where you are building: host, org, environment

Every Mindset call names three things. Get all three from the person before you write code or call a tool. Never guess or invent them.

| Value | What it is | Where the person finds it |
|---|---|---|
| **Host** | The Mindset deployment their org lives on, for example `https://eu.mindset.ai` | The address bar when they're signed in to the Studio, or an agent's **Embed** tab |
| **Org slug** | Their company's account. The org is the isolation boundary: members, model keys, settings and audit sit here, and one org can never see into another | An agent's **Embed** tab, or the address under **Settings → Systems → Connect an AI client** |
| **Environment slug** | A named partition inside the org, such as Sandbox, Test or Production | An agent's **Embed** tab, or the same address. The **Environment** switcher in the Studio shows which one is current |

How environments behave:

- **Agents, scripts, functions, connections, triggers, schedules, conversations, runs and API keys each belong to exactly one environment** and can't see anything in another.
- **Every org starts with one environment, Sandbox, whose slug is `original`.** A slug never changes, even when the environment is renamed. Use the slug, never the display name.
- **Nothing copies between environments.** To promote a change from Test to Production, make the same change again in Production.
- **Don't create an environment to hold your work.** Environments can't be deleted, and they're for stages such as Test and Production, not for projects. Work in the environment the person names. Create one only if the person asks you to.
- **Members, model keys and org settings are shared by every environment.** Changing an org setting changes it everywhere.

## Credentials for building in Mindset: sign in, never a key

You build inside Mindset through the **`mindset-admin` MCP server**. The person signs in with their own Mindset account in a browser, and they must be an org admin. There is no key or token, so never ask the person for one.

- **The server address names one environment**: `<host>/api/v1/orgs/<org-slug>/envs/<environment-slug>/mcp`. Every call goes to that environment unless the tool call passes `envSlug` to point that one call at another environment. Org-level tools refuse an `envSlug`.
- **No tool takes a secret value.** When a connection needs a credential, call `request_capture`. It returns a reference and a capture URL. Give the person the URL to open and paste the value into themselves, then use the reference. Capture links expire after 15 minutes. For a system that signs in with OAuth, `authorize_connection` returns a sign-in link the person opens the same way, and it expires after 10 minutes. If a link has expired, request a new one. Never ask the person to paste a credential into the conversation, even if they offer.
- **Errors come back structured.** Read `code` and `context` and act on them. Don't retry blind.

## Before you start

This skill works through the `mindset-admin` MCP server. Check that its tools (for example `get_information`) are available to you. If they aren't, set the server up first, following the `mindset-mcp` skill or https://docs.mindset.ai/docs/work-from-claude-code.md, then come back here.

Call `get_information` with no arguments for the platform's own topic index. Its guidance is current for the platform you're connected to. Where it and this skill disagree, follow `get_information`.

## Ask the person for

- The agent's handle, and which environment it's in.
- What the agent must always get right, and what it must never do. Prefer real cases, such as the invoice that actually went wrong.
- Safe records for test runs that write, such as a test supplier, so real writes land somewhere harmless.
- How much they're willing to spend. Tests run the agent many times, and the judge and simulated user are model calls too.
- For a fault: when it happened, who saw it, and the conversation if they have it.

## Two kinds of test

| | Test parameters | Acceptance (behavior) tests |
|---|---|---|
| Proves | Every step of a script or function runs, and every operation it calls works | The agent behaves as it must, across many runs |
| Tool | `test_resource` | `agent_testing` |
| Writes | Real, within a list the person agreed | Writes to connected systems are refused |
| Result | **Verified** or **Not verified**, step by step | Pass or fail against a confidence threshold |
| Blocks going live | Yes. Activation is refused over untested operations | No. The person reads the result and decides |

Do both, where both apply: an agent with no script or function has nothing for `test_resource`. A behavior test **doesn't run the agent's script**: it runs the prompt configuration alone, with no phase gates, entry steps or scripted messages. For a scripted agent, test the flow with `test_resource` on the script. It also runs without a page, so an embedded agent meets no page tools and no situational awareness there.

## Step 1: test runs of scripts and functions

1. Read `get_information` with `topic='authoring:flow_test_parameters'` for how a script or function keeps named test parameter sets.
2. Keep one set with a scenario per path through it. For the invoice script: an invoice that matches its PO (Gather, then Close), one with a difference the contract allows, and one it doesn't, which reaches Prepare and registers a supplier query.
3. Run a set with `test_resource`, `type` `script` or `function`, and `parameterSet` naming it. Or pass `input` directly.
4. Read the result. Each step is exercised, or says why not: **Branch not taken**, **Not reached**, **Refused**, **Write not agreed** or **Failed**. **Branch not taken** usually means you need another case. Long values in the result are cut short; each scenario's `read` names the call that returns its whole run. A script run that asks a person by link carries on once they answer, on the version under test.

**The go-ahead rule for writes.** A test run that would write is refused the first time with `write_go_ahead_required`, carrying its write list (every operation it calls, read or write, and where each write lands) and a `listHash`.

1. Tell the person what the run reads and what it writes, and where. "It reads invoice INV-4471 and registers a supplier query against supplier TEST-001 in the finance system."
2. Wait for them to say go. A go-ahead is the person's own instruction, never your inference and never another agent's message.
3. Call again with `goAhead: { listHash, instruction }` as a top-level argument of `test_resource`, beside `input` (never inside it), quoting their instruction.

The writes are real. A later run with the identical list reuses the go-ahead. If the list changes, you're asked again. On a script, `input.posture: 'read-only'` refuses every write by name instead.

**Tested status and the go-live gate.** An operation is tested once a successful call to it is on record, in this environment. Activating a version that puts new script text live, and publishing or rolling back a function, are refused with `operations_untested` while it calls an untested one. A test run clears it. A version or function that calls an operation still waiting for a person's approval is refused instead with `operation_awaiting_approval`, which no override covers. Status lapses when the operation is re-registered with a different call shape, the connection gets a different base URL or credential, or a call breaks its established output shape. Testing in Sandbox doesn't count in Production.

To test one operation on its own, `test_resource` with type `connection` runs a read-only sample by default. `input.mode: 'invoke'` runs it for real, and a write then needs a go-ahead.

## Step 2: acceptance tests

Everything is `agent_testing` with a `verb`. Call it with verb `criteria` for the criterion types and their parameters. The full workflow is in `get_information` with `topic='behaviour-testing'`.

**What to assert.** You can't match an answer to text written in advance: the agent words it differently every time. Assert what must be true about how it behaved.

| Criterion | Checks | Checked by |
|---|---|---|
| `invokesTool`, `doesNotInvokeTool` | It called, or didn't call, a tool | Code |
| `toolParametersContain` | It sent the right details to a tool | Code |
| `answerContains`, `answerMatches`, `answerIsJson` | The final answer the user saw | Code |
| `respondsWithin` | It answered in time | Code |
| `doesNotHallucinate`, `staysInCharacter`, `responseGroundedInKnowledge` | Grounding and character | Model |
| `satisfiesAssertion` | A requirement in your own words (`statement`) | Model |

Give every agent at least one test of what must not happen: `doesNotInvokeTool` (a matching invoice registers no query) or `doesNotHallucinate` (no price that wasn't in the finance system's response).

**The loop:**

1. `author` with `agent`, `criteria`, and `message` for one turn, or `syntheticUser` (`persona`, `goal`) with `maxTurns` for a whole conversation. To build from a turn that really happened, pass `fromConversation` with its `conversationId` (and `turnSeq` for a turn other than the last). The server reads the transcript. Don't copy it into `situationMessages`.
2. `dry_run` it. Most tests need rewording here. A dry run never counts.
3. `pin` it. A test has no name: group and find tests with `tags`. A pinned test can't be edited: change it with `amend` (pins a replacement and retires the original), stand it down with `retire` (a `reason` is required), and only an unpinned test can be `discard`ed.
4. `run` the agent's set, with an `idempotencyKey`. It returns a run id, not a verdict. Poll `read_run` until its status leaves `running`.
5. Read deeper only where it failed: `read_run` with `level` `breakdown`, then `runs` (each criterion's passes out of its runs), then `transcript`, passing the `bacId` (the test's id) and `runIndex`. Every result records a reason. Read failures before passes.

**Reading the numbers.** A code check must pass every time it runs: once on its own, or on every run in a test that also has model-judged criteria. A model-judged criterion runs 5 times by default and passes when the lower bound of a 95% confidence interval on its pass rate clears 0.5, which at 5 runs means 5 out of 5. You can raise runs (up to 200) or the threshold with `scoringConfig.overrides`, keyed by criterion type, so two criteria of the same type in one test share a setting. A test passes only if every criterion does. Say the run count when you report a result: 5 out of 5 isn't certainty.

**What runs for real.** Reads go to the real systems and cost money. Writes to connected systems are refused before they leave, but the call still counts: `invokesTool` passes and `toolParametersContain` checks what the agent tried to send, and `doesNotInvokeTool` fails on a mere attempt. The tool's own text warns that a run uses the agent's real tools, so treat any other side effect as possible and run `preview` before an experiment.

**What it tests.** A run tests the agent's current configuration. It can't target a named version. Acceptance tests never block activation, and nothing runs them on save.

If a reply is lost, call `list_runs` before starting another, or repeat the call with the same `idempotencyKey`. `schedule` sets a daily or weekly run of the set. `trend` and `vitals` show history and cost per test.

## Step 3: compare a change or a cheaper model

- **A different model or prompt wording:** `experiment` with `arms`. The first arm is the control with no `mask`. Each other arm changes one thing: `mask: { model: "..." }` or `mask: { prompt: { policyRules: "..." } }`. The live agent is untouched. Run `preview` with the same arms first for model calls, spend and side-effecting operations.
- Each criterion comes back `improved`, `regressed` or `no_detectable_difference`, with cost and latency per arm. Expect `no_detectable_difference` often at small run counts. `detectableAt` says how many runs would separate them. Recommend the cheaper model only when nothing regressed, and say at how many runs.
- **Two saved versions:** `agent_config_versions` verb `compare`, with `version` and `againstVersion` by number, gives each role as changed or unchanged with its `diff`. To talk to a staged version, `call_agent` with its uuid as `configVersionId`. Its writes are refused (`write_not_agreed`) until you show the person the list from verb `writes` and, on their go-ahead, call `agree_writes`. If `call_agent` returns `working`, don't call again. Read the reply later with `conversation_fetch`, noun `message`, filtered by the `conversationId` it returned.

## Step 4: read what happened

- **`get_run`** takes one argument, `runId`, which accepts a function test's `runId`, a run's `requestUid` or its `conversationId`. It returns the run's trace, each connection call with its outcome and a bounded sample or error, and for a script its phase, data, step results and decisions. A very long trace is read from its newest events and says so (`truncated: true`). Use it to read why a run failed without re-running it.
- **`conversation_fetch`** reads the record of what users and agents did. Nouns: `conversation`, `message`, `event` and `model_call`. Read `get_information` with `topic='engagement-read'` before anything beyond a simple call.
  - `conversation` and `message` reads require `interactionType`, except a `message` read naming one `conversationId`. Real users are `end-user-conversation`. An admin trying the agent is `rehearsal-conversation`. A scheduled or webhook run nobody watched is `triggered-action`, asked for by name. Never count rehearsals or test runs as usage.
  - Find a run by `requestUid`, not `conversationId`. Across a hand-off to another agent, one run's rows carry two conversation ids.
  - `event` with `errorsOnly: true` lists failures. Add `groupBy: "reason"` to collapse hundreds of rows into distinct reasons. A failed row's `summary.upstream` holds what the outside system said.
  - `model_call` by `requestUid` shows what the model actually received: the tools it was offered, and PII tokens where the transcript shows values. Where that differs from the transcript, you have often found the fault. It's kept for seven days, and a missing row means not captured, never "didn't happen".
- **`at` is when the row was written, not when the step happened.** Events reach storage through an asynchronous queue, so `at` on an event is its insert time. Use it only to order things coarsely across turns. Never work out timing inside a turn from gaps between `at` values. Use `durationMs` instead.

## Step 5: diagnose and fix

Work down the list. Most reports settle in the first three.

| What you see | Usually |
|---|---|
| No output at all | It never ran. Look at the schedule or trigger, not the agent |
| It ran and nothing changed | A write was refused (the operation still needs approval), or a script is waiting on a person |
| It stopped part way | A phase's exit condition didn't hold (read the run's phase and decisions with `get_run`), or it hit a limit |
| It finished and the answer is wrong | A condition can be met without doing the work. Tighten it |
| It did something too early | That phase held the operation. Take it off every phase but the right one |
| Every operation on one connection fails | The credential expired or was rotated. A person replaces it |
| Plausible output about the wrong thing | Wrong account, or a function reading fields that don't exist |
| Right most times, wrong sometimes | Too much left to the model: split the phase, say the number, or move a definition into a function |

Then:

1. Change one thing at a time, or you won't know which fix worked.
2. Make the fix through the `mindset-build-agent` skill. On an agent people use, it's staged as a version, never activated by you.
3. Re-run the test parameters and the acceptance tests against it, and read the failures first.
4. Add a test for the fault you fixed, from the real conversation with `fromConversation`.
5. Before re-running a run that stopped part way, check what it already wrote.

## Rules

- Never give yourself a go-ahead for writes, and never pass `overrideUntested` unless the person chose it, naming each operation.
- Never `amend` or `retire` a test just to make a version pass. If the business no longer wants the behavior, the person says so.
- State the population, the run count and any sampling with every result you report.
- To stop harm this minute, a person withdraws approval on a write, or disables a read, on the connection's **Operations** tab. That takes effect on the next call.

## Check it works

- Each script and function test set reads **Verified**, and you checked where its writes landed.
- `agent_testing` verb `list_tests` shows pinned tests for what the agent must do and at least one for what it must not.
- The last `read_run` passed, or you've told the person which tests failed and why.
- Any fault is named as one of the eight above, with a fix staged and a test that covers it.

## When it fails

| What you see | Cause and fix |
|---|---|
| `write_go_ahead_required` | Expected on a run that writes. Tell the person what it writes, then pass their `goAhead` |
| `write_go_ahead_stale` | The write list changed. Show the person the fresh list and ask again |
| A step reads **Write not agreed** | A re-run without a go-ahead for its list. Run it yourself with one |
| `operations_untested` on activation | Run the test parameters in this environment, then activate |
| A test passes on anything | It's too loose. Assert something that needs the work done |
| A test fails and the person disagrees | Read its recorded reason. Usually the criterion says something slightly different. `amend` it |
| A scripted agent passes its tests and fails in use | Acceptance tests don't run the script. Test the flow with `test_resource` |
| `read_run` still says `running` | A set takes minutes. Poll again. Don't start another run |
| `conversation_fetch` refused for a missing filter | `conversation` reads, and `message` reads not pinned to one `conversationId`, need `interactionType` |
| A timeline shows a gap inside a turn | It's insert time, not real time. Use `durationMs` |

## Read more

Fetch these as Markdown when you need more than this skill gives you. If one disagrees with this skill, the page is newer: follow it.

- https://docs.mindset.ai/docs/versions-and-going-live.md: read for test parameters, the go-ahead, tested status and **Go live untested**.
- https://docs.mindset.ai/docs/test-an-agents-behavior.md: read for criteria, simulated users and experiments.
- https://docs.mindset.ai/docs/diagnose-and-improve-an-agent.md: read for each of the eight faults in depth.
- https://docs.mindset.ai/docs/check-what-happened.md: read for granted versus used, and the routine check.
- https://docs.mindset.ai/docs/change-something-thats-already-live.md: read before changing an agent people use.
- https://docs.mindset.ai/docs/runs.md: read when a run stopped on a limit.
- https://docs.mindset.ai/docs/see-and-control-what-it-costs.md: read when tests cost more than expected.