Runtime Tests

A runtime test proves PAI wiring: the model called a tool, the tool appeared on the thread, a pending action showed up, a second send ran.

The model is usually mocked so the test can force a path. Tools can be real (with fake services) or stubbed.

import {
  createTestUtils,
  createTestRuntime,
  stubTools,
} from "@pai/test-utils";
import { assistantAgent } from "./assistant-agent";

const t = createTestUtils(assistantAgent);

const model = t.model.queue((m) => [
  m.response([m.toolCall("searchDocs", { query: "tenancy" })]),
  m.response([m.text("See the identity guide.")]),
]);

const agent = stubTools(assistantAgent, {
  unhandled: "fail",
  tools: {
    searchDocs: async () => ({
      hits: [{ title: "Identity", url: "/guides/identity" }],
    }),
  },
});

await using runtime = createTestRuntime({ agent, model });
const client = runtime.client({
  identity: { userId: "test-user", workspaceId: "test-workspace" },
});
const thread = client.thread(client.newThreadId());
const run = await thread.send("Where is tenancy documented?");
const state = await run.waitUntilIdle();

expect(t.getTool(state, "searchDocs").output?.hits[0]?.title).toBe("Identity");

createTestUtils(agent) binds helpers to your contract so m.toolCall("searchDocs", …) is typed. unhandled: "fail" means every model tool call must have a stub — leftover calls fail the test instead of hitting the network.

Stub vs real tools

You care about	Do this
PAI projected the call onto the thread	`stubTools`
The real `execute` plus fake services	pass the agent unchanged, mock `runtimeContext` services
A nested `tools` map / code mode child	runtime test only — not `runTool`

Prefer the agent's real runtimeContext with a test identity. Use overrideRuntimeContext only when you want to skip that builder:

await using runtime = createTestRuntime({
  agent: assistantAgent,
  model,
  overrideRuntimeContext: {
    workspaceId: "test-workspace",
    docs: fakeDocs,
    permissions: { canDelete: false },
  },
});

Wait for a tool or an approval

waitUntilIdle() is enough when the run finishes. If you need a tool part or a pending action in the middle:

const tool = await t.waitForTool(thread, "searchDocs");
expect(tool.state).toBe("output-available");

const action = await t.waitForPending(
  thread,
  t.actionName("publishPost", "approve"),
);
await action.submit({ approved: true });

The synchronous form is a separate import, not on t:

import { findPending } from "@pai/test-utils";

findPending(thread, state, t.actionName("publishPost", "approve"));

A second send

Use the same thread handle. A second send while a run is active throws ThreadBusyError.

await (await thread.send("Now make it shorter")).waitUntilIdle();

Evals when the question is "did the model choose well?"
Tool tests when PAI wiring is not the point

Runtime Tests

Stub vs real tools

Wait for a tool or an approval

A second send

Next

On this page