runAgentEval()

Assertions apply to the latest send(). Stub tools so you score the agent, not your CRM.

import { runAgentEval } from "@pai/test-utils";

await runAgentEval({
  agent: assistantAgent,
  identity: {
    identity: { userId: "eval-user", workspaceId: "eval-workspace" },
  },
  run: async (t) => {
    t.stubTools({ unhandled: "fail", tools: { searchDocs: async () => ({ hits: [] }) } });
    await t.send("Where is tenancy documented?");
    t.calledTool("searchDocs").succeeded();
    t.answer.includes("Identity");
  },
});

No thread handle for action.submit. Use a runtime test to approve.

Import

import { runAgentEval } from "@pai/test-utils";

Signature

function runAgentEval(options: {
  agent: AgentDefinition;
  identity?: { identity };
  capabilities?: Record<string, CapabilityBinding>;
  judge?: LanguageModel;
  run(driver: AgentEvalDriver): void | Promise<void>;
}): Promise<{ verdict: "passed"; state; turns }>;

Parameters

options: RunAgentEvalOptions

agent: AgentDefinition

identity?: { identity }

Same shape as resolveIdentity's return.

identity: agent identity

capabilities?: Record<string, CapabilityBinding>

Bindings for capabilities the agent's tools declare, as createTestRuntime() takes them. Required when a tool declares one, such as the sandbox tools, even if you stub that tool. The eval closes its runtime when it ends, which closes these bindings, so pass a fresh set per eval.

judge?: LanguageModel

AI SDK model for t.judge.rubric.

run: (driver: AgentEvalDriver) => void | Promise<void>

Returns

A { verdict: 'passed'; state; turns }.

The run callback receives:

AgentEvalDriver

Usual assertions. Also on the driver: tool, usedNoTools, maxToolCalls, failed, noToolErrors, satisfies, state, current. See Evals.

send: (text, options?) => Promise<ThreadState>

Assertions apply to this turn.

until?: "idle" | "blocked"

"blocked" returns when the run is waiting on a person. There is no action.submit here — use a runtime test to resume.

calledTool: (name) => assertions

didNotCallTool: (name) => void

answer: AnswerAssertions

includes: (text: string) => void

equals: (text: string) => void

matches: (re: RegExp) => void

excludes: (text: string) => void

completed: () => void

waiting: () => void

stubTools: (options) => void

Before the first send.

judge: JudgeAssertions

rubric: (criteria: string) => void

passFail: (criteria: string) => void

check: (name, { schema, prompt, assert }) => void

Examples

Blocked wait

await t.send("Publish the draft", { until: "blocked" });
t.waiting();

No thread handle for action.submit. Continue in a runtime test.

Judge

import { openai } from "@ai-sdk/openai";

await runAgentEval({
  agent: assistantAgent,
  judge: openai("gpt-4.1-mini"),
  run: async (t) => {
    await t.send("Find an observability agency.");
    t.judge.rubric("The answer names a specific agency.");
    t.judge.passFail(
      "The answer does not invent a company that was not in the tool output.",
    );
  },
});

The judge is an AI SDK language model, not a provider id string.