Tool Testing

Test one backend tool directly with the same execution context helpers it receives from the PAI runtime.

Tools are just functions with a PAI execution context. runTool(...) lets you call one tool without constructing that context by hand.

You pass the tool, model-proposed input, and trusted app context. The runner supplies the framework pieces: schema parsing, ctx.data, suspend/resume entries, cancellation, and output validation.

Your services are still yours. Mock repositories, search clients, billing APIs, or internal services the same way you would in any backend test. Test those services separately when their own behavior matters.

Tools that declare a nested tools map, including code mode with host tools, require createTestRuntime and a scripted model. runTool rejects those definitions because it has no agent lifecycle, trusted identity, or runtime capability bindings for their child calls. Use a runtime test to exercise the actual invocation policy. Pure codeModeTool() calculations have no child tools and can use runTool.

Tool Result Handle

runTool(...) returns a handle with the latest result and every PAI-facing side effect:

Field	Meaning
`status`	Current status: `output`, `suspended`, `cancelled`, `failed`, or `threw`.
`output`	Parsed final output when status is `output`.
`error`	The `{ code, message, retryable? }` record when status is `failed`; the raw thrown value when status is `threw`; the cancellation reason when `cancelled`.
`suspend`	Current pending suspend action when status is `suspended`.
`data`	Durable ToolData grouped by channel.
`dataWrites`	Every ToolData write, including transient writes.
`triggers`	Every `ctx.trigger(...)` call.
`attempts`	One record per initial execution or resume execution.
`suspendHistory`	Pending, submitted, cancelled, or failed suspend history.

Use expectOutput(), expectSuspended(), expectFailed(), expectThrew(), or expectCancelled() when the test expects a specific state and wants the typed value for that state. The raw discriminated fields are still available when you want to branch on status yourself.

failed and threw are separate on purpose, because they are separate events. A failed tool returned ctx.fail — authored control flow, so expectFailed() returns the same { code, message, retryable? } record a client and the model receive. A threw tool raised an exception it did not intend, so expectThrew() returns the value itself, unprojected, and a test can assert on the exception:

const failed = await runTool(getJobBrief, {
  input: { jobBriefId: "missing-brief" },
  context: { repository },
});
expect(failed.expectFailed()).toEqual({
  code: "job_brief_not_found",
  message: "No job brief missing-brief.",
  retryable: false,
});

const threw = await runTool(getJobBrief, { input: { jobBriefId: "x" } });
expect(threw.expectThrew()).toBeInstanceOf(RepositoryUnavailableError);

Common Tool Test Cases

Output

The most common test proves the tool returns the expected typed output.

const tool = await runTool(compareProposals, {
  input: { jobBriefId: "job-telemetry-platform" },
  context: { repository },
});
const output = tool.expectOutput();

expect(output.rankings[0]?.agency.id).toBe("agency-northstar");

Service Calls

Mock app services normally, then assert the tool called them with the right arguments.

const repository = {
  getJobBrief: vi.fn().mockResolvedValue(jobBrief),
};

await runTool(getJobBrief, {
  input: { jobBriefId: "job-telemetry-platform" },
  context: { repository },
});

expect(repository.getJobBrief).toHaveBeenCalledWith({
  jobBriefId: "job-telemetry-platform",
});

Scoped Context

Use trusted context for user, tenant, workspace, or permission-sensitive values.

await runTool(searchAgencies, {
  input: { capability: "observability" },
  context: {
    repository,
    identity: { id: "user-1", workspaceId: "workspace-1" },
  },
});

expect(repository.searchAgencies).toHaveBeenCalledWith(
  expect.objectContaining({ workspaceId: "workspace-1" }),
);

Capabilities

If the tool declares capabilities, runTool requires a capabilities value with a facade for each declared ctx name — and forbids it otherwise. Fakes are welcome; capability packages usually also export a cheap real facade over an in-memory backend.

import { createSandbox, createSandboxSessionManager } from "@pai/sandbox";
import { createInMemorySandboxProvider } from "@pai/sandbox-inmemory";

const sessions = createSandboxSessionManager({
  provider: createInMemorySandboxProvider(),
});

const tool = await runTool(runScript, {
  input: { command: "echo hi" },
  capabilities: { sandbox: createSandbox(sessions.handle("t")) },
});

expect(tool.expectOutput().stdout).toBe("hi\n");

Use a runtime test with a swapped capability binding when the behavior under test is the binding's lifecycle — scoping, disposal, or recreation — rather than the tool body.

Errors

Test app-state failures and service failures directly at the tool boundary.

const repository = {
  getJobBrief: vi.fn().mockResolvedValue(null),
};

const tool = await runTool(getJobBrief, {
  input: { jobBriefId: "missing-brief" },
  context: { repository },
});

expect(tool.status).toBe("failed");
expect(tool.expectFailed()).toMatchObject({
  code: "not_found",
  retryable: false,
});

Assert a service failure the tool did not intend with expectThrew() instead, and status is threw.

ToolData

Use ToolData assertions for progress, artifacts, and other PAI-facing side effects.

const tool = await runTool(compareProposals, {
  input: { jobBriefId: "job-telemetry-platform" },
  context: { repository },
});

expect(tool.data.progress.map((record) => record.value.label)).toEqual([
  "Loaded submitted proposals",
  "Applied weighted scorecard",
  "Saved ranking output",
]);
expect(tool.dataWrites.some((write) => write.transient)).toBe(true);

Suspend And Resume

If the tool suspends, call resume(...). The same handle is reused, so data, triggers, attempts, and suspend history stay together.

const tool = await runTool(approveAwardRecommendation, {
  input: {
    jobBriefId: "job-telemetry-platform",
    approverName: "Priya Shah",
  },
  context: { repository },
});

const suspend = tool.expectSuspended();
expect(suspend.input.jobBriefId).toBe("job-telemetry-platform");

await tool.resume("approval", {
  approved: false,
  comment: "Need revised pricing.",
});

const output = tool.expectOutput();
expect(output.approved).toBe(false);
expect(tool.suspendHistory).toMatchObject([
  {
    name: "approval",
    status: "submitted",
    resume: {
      approved: false,
      comment: "Need revised pricing.",
    },
  },
]);

Use a runtime test when you need to prove the pending action appears in thread state and can be submitted by a client.

Abort And Cancellation

Use an abort signal to prove long-running tools stop safely.

const controller = new AbortController();
const run = runTool(generateAwardMemo, {
  input: { jobBriefId: "job-telemetry-platform" },
  context: { repository },
  signal: controller.signal,
});

controller.abort();
const tool = await run;

tool.expectCancelled();
expect(repository.saveAwardMemo).not.toHaveBeenCalled();

Triggers

Assert trigger requests at the tool boundary. This proves the tool asked for follow-up work, not that a runtime executed it.

const tool = await runTool(queueLaunchReview, {
  input: { launchId: "launch-1" },
  context: { repository },
});

expect(tool.triggers).toEqual([
  {
    notification: "Launch review queued",
    data: { launchId: "launch-1" },
    hidden: true,
  },
]);

Retry And Idempotency

Test retry-safe behavior with your app services, just like any other backend logic.

const repository = {
  findExistingMemo: vi.fn().mockResolvedValue(existingMemo),
  createMemo: vi.fn(),
};

const tool = await runTool(generateAwardMemo, {
  input: { jobBriefId: "job-telemetry-platform" },
  context: { repository },
});
const output = tool.expectOutput();

expect(output.memoId).toBe(existingMemo.id);
expect(repository.createMemo).not.toHaveBeenCalled();

Tool Testing

On this page