Driver

The driver form is the eval API. It uses normal TypeScript control flow, so multi-turn scenarios, intermediate assertions, human-input simulation, and branching stay natural.

Run In Your Test Runner

Agent evals should run inside the app's existing test runner. Vitest, Jest, or Node test already know how to load env vars, start containers, seed databases, apply module mocks, resolve TypeScript aliases, and run app fixtures. PAI should not replace that setup.

await runAgentEval({
  agent: myAgent,
  run: async (t) => {
    await t.send("Approve the award recommendation.");

    t.completed();
    t.didNotCallTool("approveAwardRecommendation");
    t.answer.matches(/which job brief|job brief id/i);

    await t.send("Use job-telemetry-platform.");

    t.calledTool("getJobBrief").inputIncludes({
      jobBriefId: "job-telemetry-platform",
    });
  },
});

The driver API should feel like a test harness, not a config language. Use it when the eval has more than one step or when assertions need to happen before the whole scenario is over. The it(...) name is the eval case name.

Common per-case inputs include identity, seeded memory, uploaded files, fixture data, locale, current time, feature flags, model settings, max steps, timeouts, and retry policy. Those should be explicit fields or setup hooks, not hidden side channels.

Pass identity and clientData to evals when agent runtime context should be derived the same way it is in production:

await runAgentEval({
  agent: myAgent,
  identity: {
    identity: { id: "eval-user", workspaceId: "ws-1" },
  },
  clientData: {
    locale: "en-AU",
  },
  run: async (t) => {
    await t.send("Load the active brief.");
    t.completed();
  },
});

Use overrideRuntimeContext for focused evals that need a precise runtime-context fixture and are not testing the app's runtime-context builder:

await runAgentEval({
  agent: myAgent,
  identity: {
    identity: { id: "eval-user", workspaceId: "ws-1" },
  },
  overrideRuntimeContext: {
    workspace: { id: "ws-1", plan: "enterprise" },
    locale: "en-AU",
  },
  run: async (t) => {
    await t.send("Compare the shortlisted proposals.");
    t.calledTool("compareProposals");
  },
});

Driver Flow

t.send(...) runs one agent turn and waits for the run to settle far enough for assertions. Assertions after that call are scoped to that current turn.

await t.send("Load the active brief.");
t.calledTool("getJobBrief");

await t.send("Now compare the proposals.");
t.calledTool("compareProposals");
t.didNotCallTool("getJobBrief");

The second t.calledTool(...) checks the second send only. Initial state and prior turns are still available to the agent as conversation context, but they do not satisfy current-turn assertions.

Most assertions are synchronous because they register checks against the current turn. Judge assertions are also registered synchronously; the runner awaits the judge model calls after the driver finishes.

Reading State

The driver should expose stable views for the latest turn so tests do not reach into runtime internals:

await t.send("Compare Northstar and Orbit Labs.");

const answer = t.answer.text;
const compareCalls = t.tools.byName("compareProposals");
const pendingActions = t.pendingActions.all();
const messages = t.messages.text();
const state = t.state.current;

Use these reads when a one-off product assertion is clearer than a named matcher. Prefer the named matchers for common behavior because they can produce better failure messages.

Initial State

Use initialState when the prompt under evaluation depends on prior thread history. It should be a factory so the builder can be typed from the agent contract.

await runAgentEval({
  agent: myAgent,
  initialState: (s) => [
    s.userMessage("Load job-telemetry-platform."),
    s.assistantResponse([
      s.tool("getJobBrief", {
        jobBriefId: "job-telemetry-platform",
      }).result({
        title: "Warehouse telemetry platform",
        budget: 420000,
      }),
      s.text("Loaded the warehouse telemetry platform brief."),
    ]),
  ],
  run: async (t) => {
    await t.send("Now compare Northstar and Orbit Labs.");

    t.calledTool("compareProposals").inputIncludes({
      jobBriefId: "job-telemetry-platform",
    });
    t.answer.includes("Northstar");
  },
});

The initial-state builder describes existing thread history. It should not execute tools or call a model. It creates a starting transcript that the next t.send(...) can build on.

Useful initial-state builders include:

s.userMessage("...");
s.assistantResponse([s.text("...")]);
s.tool("getJobBrief", input).result(output);
s.tool("compareProposals", input).error(error);

This mirrors the mock-model response builders, but the meaning is different: mock-model builders describe future model output; initial-state builders describe thread history that already exists before the eval starts.