Evals

Score what the agent did on a turn: tools called, answer text, optional LLM judge.

An eval is a runtime test whose assertions are about behavior, not wiring.

Use it when you care that the model called searchDocs with a sensible query, or that the answer named a record. Stub tools so a bad CRM cannot fail the eval. Use a real model when you are scoring tool choice; use a mocked model when you are writing the eval itself.

import { runAgentEval } from "@pai/test-utils";
import { assistantAgent } from "./assistant-agent";

await runAgentEval({
  agent: assistantAgent,
  identity: {
    identity: { userId: "eval-user", workspaceId: "eval-workspace" },
  },
  run: async (t) => {
    t.stubTools({
      unhandled: "fail",
      tools: {
        searchDocs: async ({ input }) => ({
          hits: [{ title: "Identity", url: "/guides/identity" }],
        }),
      },
    });

    await t.send("Where is tenancy documented?");

    t.completed();
    t.noToolErrors();
    t.calledTool("searchDocs")
      .inputIncludes({ query: "tenancy" })
      .succeeded();
    t.answer.includes("Identity");
  },
});

Assertions apply to the latest send(). Setup history cannot accidentally pass calledTool.

An agent whose tools declare capabilities, such as the sandbox tools, needs their bindings in runAgentEval({ capabilities }), even when you stub those tools.

Driver

Call	Meaning
`t.send(text)`	One user turn. Waits until idle unless you pass `{ until: "blocked" }`.
`t.calledTool("searchDocs")`	That tool ran this turn. Chain `.inputIncludes`, `.outputIncludes`, `.succeeded()`, `.failed()`, `.times(n)`.
`t.tool("searchDocs")`	Same chain when the call might be optional.
`t.didNotCallTool("deleteRecord")`	Must not have run.
`t.usedNoTools()`	Text-only turn.
`t.maxToolCalls(n)`	Cap how many tool calls this turn made.
`t.answer.includes("…")`	Final assistant text. Also `.equals`, `.matches`, `.excludes`.
`t.completed()` / `t.waiting()` / `t.failed()`	Run lifecycle.
`t.noToolErrors()`	No tool part in `output-error`.
`t.satisfies(label, fn)`	Custom assertion on the turn context.
`t.stubTools({ … })`	Must run before the first `send`.
`t.judge`	Rubric / pass-fail / structured check — below.
`t.state`	Latest `ThreadState`, or `null` before the first `send`.
`t.current`	This turn's run, messages, parts, answer, and tools.

Approvals

runAgentEval does not expose a thread handle for action.submit. Assert that the turn blocked:

await t.send("Publish the draft", { until: "blocked" });
t.waiting();

To submit the approval and continue, use a runtime test (waitForPending + action.submit).

LLM judge

When the answer is fuzzy ("names a specific agency"), pass a judge model and ask for a rubric. The judge is an AI SDK language model, not a provider id string.

import { openai } from "@ai-sdk/openai";

await runAgentEval({
  agent: assistantAgent,
  judge: openai("gpt-4.1-mini"),
  run: async (t) => {
    await t.send("Find an observability agency.");
    t.judge.rubric("The answer names a specific agency.");
    t.judge.passFail("The answer does not invent a company that was not in the tool output.");
  },
});

t.judge.rubric(criteria).soft() still runs the check but does not fail the test.

t.judge.check(name, { schema, prompt, assert }) is the escape hatch for a structured verdict.

Judge calls are extra model spend. Prefer calledTool / answer.includes when the behavior is discrete.

Mocked vs real model

Goal	Model
Write the eval, CI on every PR	queued mock model
Score tool choice against a live model	the agent's real model, stubbed tools

runAgentEval has no model option. Pass the mock as defineAgent({ model }):

import { createTestUtils, runAgentEval } from "@pai/test-utils";

const utils = createTestUtils(assistantAgent);
const model = utils.model.queue((m) => [
  m.response([m.toolCall("searchDocs", { query: "tenancy" })]),
  m.response([m.text("See the identity guide.")]),
]);

await runAgentEval({
  agent: defineAgent({
    name: "assistant",
    model,
    identity: z.object({
      userId: z.string(),
      workspaceId: z.string(),
    }),
    instructions: "Help with this workspace.",
    tools: { searchDocs },
  }),
  run: async (e) => {
    e.stubTools({
      unhandled: "fail",
      tools: {
        searchDocs: async () => ({
          hits: [{ title: "Identity", url: "/guides/identity" }],
        }),
      },
    });
    await e.send("Where is tenancy documented?");
    e.calledTool("searchDocs").succeeded();
    e.answer.includes("Identity");
  },
});

runAgentEval still runs a real PAI runtime either way.

Runtime tests for wiring without eval assertions
runAgentEval

Evals

Driver

Approvals

LLM judge

Mocked vs real model

Next

On this page