Tool Stubs

Control tool behavior in real-model agent evals without calling unsafe or flaky dependencies.

Agent evals usually stub tools. The goal is to test model behavior, not send emails, mutate production data, charge a card, or depend on a flaky search index.

Tool stubs should be typed from the agent contract:

t.stubTools({
  unhandled: "fail",
  tools: {
    getJobBrief: async ({ input }) => ({
      id: input.jobBriefId,
      title: "Warehouse telemetry platform",
      budget: 420000,
    }),
    compareProposals: async () => ({
      rankings: [{ agencyId: "agency-northstar", score: 94 }],
    }),
  },
});

unhandled: "fail" should be the default for CI-style evals. Unexpected tool calls are important model behavior and should fail clearly.

Repeated Calls

Some evals need a tool to return different fixture data across calls. A plain closure keeps that intent explicit without adding another mocking API.

let searchCall = 0;

t.stubTools({
  unhandled: "fail",
  tools: {
    searchAgencies: async ({ input }) => {
      searchCall += 1;

      if (input.capability === "observability") {
        return { agencies: observabilityAgencies };
      }

      if (input.capability === "data pipelines") {
        return { agencies: dataPipelineAgencies };
      }

      throw new Error(`Unexpected capability: ${input.capability}`);
    },
  },
});

Use a test-framework mock if the test needs call introspection or reset helpers. stubTools(...) only needs an async function with the same shape as the tool implementation.

Forbidden Tools

Use a forbidden stub when calling a tool would be unsafe or would mean the model made the wrong decision.

t.stubTools({
  unhandled: "fail",
  tools: {
    notifyAwardApprover: async () => {
      throw new Error("notifyAwardApprover should not be called in this eval.");
    },
  },
});

That keeps the dangerous parts controlled while still allowing assertions about what the model tried to do.

Tool Stubs

Repeated Calls

Forbidden Tools

On this page