
Outcome
Prove a change is better before anyone meets it.
Tests built from real conversations. Compare a change with the live version first.
Speaker notes
A test is a situation plus criteria, some checked by code and some judged by a model, run many times because agents vary. Save a failing conversation as a test so the failure stays fixed. An experiment runs the tests against a changed setup, such as a cheaper model, beside the current one and reports improved, regressed or no difference. Tests run offline: Mindset does not split live traffic between two versions.