Tool Stubbing

Tool stubs are useful when a test is not about the tool body itself. Runtime tests use them to isolate PAI wiring. Agent evals use them to isolate model decision-making from databases, external APIs, emails, queues, and destructive side effects.

Stub Selected Tools

stubTools(agent, { tools }) returns a copy of the agent with selected tool execute functions replaced.

import { stubTools } from "@pai/test-utils";
import { myAgent } from "./agent";

const searchAgencies = vi.fn(async ({ input }) => ({
  agencies: [
    {
      id: "agency-northstar",
      name: "Northstar Engineering",
      matchedCapability: input.capability,
    },
  ],
}));

const agent = stubTools(myAgent, {
  tools: {
    searchAgencies,
  },
});

The tool name, schema, description, prompt exposure, and visibility rules stay the same. Only the execution function changes.

Because the replacement is a normal Jest/Vitest mock, use the test framework to control it:

searchAgencies.mockResolvedValueOnce({ agencies: [] });
searchAgencies.mockClear();

expect(searchAgencies).toHaveBeenCalledWith(
  expect.objectContaining({
    input: { capability: "observability", region: "North America" },
  }),
);

Fail Unexpected Tool Calls

Use unhandled: "fail" when every tool call should be explicit in the test.

const agent = stubTools(myAgent, {
  unhandled: "fail",
  tools: {
    searchAgencies: async () => ({ agencies: [] }),
  },
});

If the model or mocked response calls any other tool, the test fails clearly. This is usually the right default for orchestration tests and evals.

The stub's failure is a deliberate ctx.fail, so the tool error names the tool it caught rather than the generic redacted message, under its own code:

expect(t.getTool(state, "approveShipment").error).toEqual({
  code: "tool_unexpected_call",
  message: "Unexpected tool call in test: approveShipment",
  retryable: false,
});

Keep Real Tools When They Matter

Do not stub a tool when the tool/runtime integration is the behavior under test. Keep the real tool and replace external dependencies the same way you would in ordinary backend tests: module mocks, fake repositories, sandbox clients, fixture data, or a prebuilt test agent.

import { testAgent } from "./agent.test-fixtures";

await using runtime = createTestRuntime({
  agent: testAgent,
  model,
});

The choice is simple: stub a tool when its body would add noise or risk; keep the real tool when its ToolData, metadata, suspend behavior, or service calls are part of what the test proves.