Optimize, Acceptance tests in Mindset: the tests for one agent and their results

Outcome

Prove a change is better before anyone meets it.

Tests built from real conversations. Compare a change with the live version first.

7 of 10
Speaker notes

A test is a situation plus criteria, some checked by code and some judged by a model, run many times because agents vary. Save a failing conversation as a test so the failure stays fixed. An experiment runs the tests against a changed setup, such as a cheaper model, beside the current one and reports improved, regressed or no difference. Tests run offline: Mindset does not split live traffic between two versions.