eval

telnyx-acp eval
telnyx-acp eval --local
telnyx-acp eval evals/refunds.jsonl greets
telnyx-acp eval --ci --out results.json
telnyx-acp eval --compare results.json

Flag	Meaning
`--local`	Target the running `dev` instead of the deployed agent.
`--ci`	Exit 1 when any case fails or cannot run. A case whose every check was skipped does not fail.
`--out <file>`	Write the results as JSON, the file `--compare` reads.
`--compare <file>`	Say which cases newly fail, were fixed, are new, were not run, or changed outcome (`error` ↔ `fail`) since that results file.
`--concurrency <n>`	Cases run at once, 4 by default, 1 to 32.
`--judge-model <id>`	The model that scores `judge` and `rubric` checks. The agent’s own model by default.

eval runs the project’s saved test conversations against the agent and scores each reply. Every case runs in a fresh conversation, eval-<case>-<nonce>, so cases never share memory or history, and each turn goes through the same send-and-wait path as invoke. With arguments, each one is a case file (a path ending in .jsonl or .json) or a case name; with none, every file under evals/ runs. create writes evals/basic.jsonl with one case, so a new project has something to run:

{"name":"greets","input":"Hello","expect":{"regex":"/\\b(hello|hi|hey|help|welcome|greetings)\\b/i"}}

Cases

Cases live in evals/: a .jsonl file holds one case per line (blank lines and lines starting with // are skipped), and a .json file holds an array of cases.

{"name": "hours", "input": "What are your opening hours?", "expect": {"contains": ["9", "5"], "maxMs": 8000}}
{"name": "refund-policy", "input": ["Hi", "I want a refund for order 123"], "expect": {"judge": "Explains the 30-day refund policy and asks for the order email; does not promise a refund."}}
{"name": "uses-search", "input": "What is the weather in Paris today?", "expect": {"tool": "web_search", "notContains": "I cannot"}}

Key	Meaning
`name`	The case’s name, unique across all files. Without one, the file’s name and the line number, `basic-3`.
`input`	One message, or a list of turns for a multi-turn conversation. A turn is a string or `{"text": "…", "expect": {…}}`.
`expect`	The checks on the last turn’s reply. A turn can carry its own `expect` instead, to score an earlier reply.

Parsing is strict: an unknown key, an empty value, or a regular expression that does not compile is an error that names the file and the line, evals/basic.jsonl:3: unknown check expect.contain, and nothing runs. A case with nothing to check is refused too.

Checks

Every check given in an expect must hold.

Check	Holds when
`contains`	The reply contains each string, ignoring case. A string or a list.
`notContains`	The reply contains none of them, ignoring case.
`regex`	The reply matches a JavaScript regular expression. `/pattern/flags` keeps its flags; anything else is the pattern itself, so escape slashes (`\/usr\/`) to match them.
`maxMs`	The reply arrived within this many milliseconds of sending the turn. Under `--local` it is measured to the reply’s own timestamp; against a deployed agent to within about 0.2 seconds of it, the interval at which `eval` reads the agent.
`tool`	The run called each named tool. Read from the run’s trace.
`noTool`	The run called none of them. Read from the trace too.
`judge`	An LLM judge, given the conversation so far and the reply, says the reply satisfies the criterion.
`rubric`	A list of `{"name": "…", "criteria": "…"}`, each scored by the judge on its own and reported by name, the same shape as the AI Assistants tests rubric.

The judge runs on Telnyx inference as the signed-in account, the same way dev runs the agent’s model, and answers pass or fail with one sentence of why. A model that needs its own provider key reads it from the environment or from the project’s .env. The conversation and the reply go to the judge as data it is told never to follow, so a reply cannot rewrite its own criterion.

Outcomes

Case	When
`pass` ✓	Every check passed, or passed and some were skipped.
`fail` ✗	At least one check failed. The line names each failed check and why.
`error` !	The case could not be scored: the agent was unreachable, a turn timed out or waited on an approval, or the judge gave no verdict after a retry. The judge’s problem is never counted against the agent.
`skipped` –	Every check was skipped.

A tool or noTool check is skipped, not failed, when the agent keeps no trace of the run, or was deployed before traces existed: tool web_search skipped: needs traces.

✓ greets           2.1s
✓ hours            3.4s
✓ refund-policy    6.2s
✗ uses-search      4.1s  tool web_search — no tool was called

3/4 passed · 1 failed · p50 3.4s · judge meta-llama/Llama-3.3-70B-Instruct

eval exits 0 whatever the cases’ outcomes unless --ci is given. It exits 2 on a usage error, before any case runs: a case file that does not parse, a missing evals/, an unknown case name, an unreadable --compare file. It exits 1 when the agent cannot be reached at all (not deployed, or no dev running under --local). Under --json the envelope’s data holds the agent, the target (dev or deployed), the judge model, a summary (total, passed, failed, errors, skipped, skippedChecks, p50Ms, p95Ms), and every case with each turn’s reply, run id, time and checks.

In CI

Deploy, then fail the job when any case fails:

- name: Evals
  env:
    TELNYX_API_KEY: ${{ secrets.TELNYX_API_KEY }}
  run: |
    npm install -g @telnyx/acp
    telnyx-acp eval --ci --out eval-results.json
- uses: actions/upload-artifact@v4
  if: always()
  with:
    name: eval-results
    path: eval-results.json

A later run with --compare eval-results.json says what changed since:

3/4 passed · 1 failed · p50 2.9s
newly failing: refund-policy
fixed: uses-search

Get Started

Runtime Platform

Managed Harness

CLI

Reference

For AI Agents

Cases

Checks

Outcomes

In CI

​Cases

​Checks

​Outcomes

​In CI

Cases

Checks

Outcomes

In CI