> ## Documentation Index > Fetch the complete documentation index at: https://developers.telnyx.com/llms.txt > Use this file to discover all available pages before exploring further. # eval ```bash theme={null} telnyx-acp eval telnyx-acp eval --local telnyx-acp eval evals/refunds.jsonl greets telnyx-acp eval --ci --out results.json telnyx-acp eval --compare results.json ``` | Flag | Meaning | | - | - | | `--local` | Target the running `dev` instead of the deployed agent. | | `--ci` | Exit 1 when any case fails or cannot run. A case whose every check was skipped does not fail. | | `--out ` | Write the results as JSON, the file `--compare` reads. | | `--compare ` | Say which cases newly fail, were fixed, are new, were not run, or changed outcome (`error` ↔ `fail`) since that results file. | | `--concurrency ` | Cases run at once, 4 by default, 1 to 32. | | `--judge-model ` | The model that scores `judge` and `rubric` checks. The agent's own model by default. | `eval` runs the project's saved test conversations against the agent and scores each reply. Every case runs in a fresh conversation, `eval--`, so cases never share memory or history, and each turn goes through the same send-and-wait path as [invoke](https://developers.telnyx.com/docs/agents/cli/invoke.md). With arguments, each one is a case file (a path ending in `.jsonl` or `.json`) or a case name; with none, every file under `evals/` runs. [create](https://developers.telnyx.com/docs/agents/cli/create.md) writes `evals/basic.jsonl` with one case, so a new project has something to run: ```json theme={null} {"name":"greets","input":"Hello","expect":{"regex":"/\\b(hello|hi|hey|help|welcome|greetings)\\b/i"}} ``` ## Cases Cases live in `evals/`: a `.jsonl` file holds one case per line (blank lines and lines starting with `//` are skipped), and a `.json` file holds an array of cases. ```jsonl theme={null} {"name": "hours", "input": "What are your opening hours?", "expect": {"contains": ["9", "5"], "maxMs": 8000}} {"name": "refund-policy", "input": ["Hi", "I want a refund for order 123"], "expect": {"judge": "Explains the 30-day refund policy and asks for the order email; does not promise a refund."}} {"name": "uses-search", "input": "What is the weather in Paris today?", "expect": {"tool": "web_search", "notContains": "I cannot"}} ``` | Key | Meaning | | - | - | | `name` | The case's name, unique across all files. Without one, the file's name and the line number, `basic-3`. | | `input` | One message, or a list of turns for a multi-turn conversation. A turn is a string or `{"text": "…", "expect": {…}}`. | | `expect` | The checks on the last turn's reply. A turn can carry its own `expect` instead, to score an earlier reply. | Parsing is strict: an unknown key, an empty value, or a regular expression that does not compile is an error that names the file and the line, `evals/basic.jsonl:3: unknown check expect.contain`, and nothing runs. A case with nothing to check is refused too. ## Checks Every check given in an `expect` must hold. | Check | Holds when | | - | - | | `contains` | The reply contains each string, ignoring case. A string or a list. | | `notContains` | The reply contains none of them, ignoring case. | | `regex` | The reply matches a JavaScript regular expression. `/pattern/flags` keeps its flags; anything else is the pattern itself, so escape slashes (`\/usr\/`) to match them. | | `maxMs` | The reply arrived within this many milliseconds of sending the turn. Under `--local` it is measured to the reply's own timestamp; against a deployed agent to within about 0.2 seconds of it, the interval at which `eval` reads the agent. | | `tool` | The run called each named tool. Read from the run's [trace](https://developers.telnyx.com/docs/agents/cli/traces.md). | | `noTool` | The run called none of them. Read from the trace too. | | `judge` | An LLM judge, given the conversation so far and the reply, says the reply satisfies the criterion. | | `rubric` | A list of `{"name": "…", "criteria": "…"}`, each scored by the judge on its own and reported by name, the same shape as the AI Assistants [tests](https://developers.telnyx.com/api-reference/assistants/create-a-new-assistant-test.md) rubric. | The judge runs on Telnyx inference as the signed-in account, the same way `dev` runs the agent's model, and answers pass or fail with one sentence of why. A model that needs its own provider key reads it from the environment or from the project's `.env`. The conversation and the reply go to the judge as data it is told never to follow, so a reply cannot rewrite its own criterion. ## Outcomes | Case | When | | - | - | | `pass` ✓ | Every check passed, or passed and some were skipped. | | `fail` ✗ | At least one check failed. The line names each failed check and why. | | `error` ! | The case could not be scored: the agent was unreachable, a turn timed out or waited on an approval, or the judge gave no verdict after a retry. The judge's problem is never counted against the agent. | | `skipped` – | Every check was skipped. | A `tool` or `noTool` check is skipped, not failed, when the agent keeps no trace of the run, or was deployed before traces existed: `tool web_search skipped: needs traces`. ```text theme={null} ✓ greets 2.1s ✓ hours 3.4s ✓ refund-policy 6.2s ✗ uses-search 4.1s tool web_search — no tool was called 3/4 passed · 1 failed · p50 3.4s · judge meta-llama/Llama-3.3-70B-Instruct ``` `eval` exits 0 whatever the cases' outcomes unless `--ci` is given. It exits 2 on a usage error, before any case runs: a case file that does not parse, a missing `evals/`, an unknown case name, an unreadable `--compare` file. It exits 1 when the agent cannot be reached at all (not deployed, or no `dev` running under `--local`). Under `--json` the envelope's `data` holds the agent, the target (`dev` or `deployed`), the judge model, a `summary` (`total`, `passed`, `failed`, `errors`, `skipped`, `skippedChecks`, `p50Ms`, `p95Ms`), and every case with each turn's reply, run id, time and checks. ## In CI Deploy, then fail the job when any case fails: ```yaml theme={null} - name: Evals env: TELNYX_API_KEY: ${{ secrets.TELNYX_API_KEY }} run: | npm install -g @telnyx/acp telnyx-acp eval --ci --out eval-results.json - uses: actions/upload-artifact@v4 if: always() with: name: eval-results path: eval-results.json ``` A later run with `--compare eval-results.json` says what changed since: ```text theme={null} 3/4 passed · 1 failed · p50 2.9s newly failing: refund-policy fixed: uses-search ```