Agent Testing

Test PAI agents and tools through the same public backend surfaces your app uses in production.

PAI tests should prove public app behavior with the unstable boundaries made deterministic. Keep the runtime, thread state, and tool projection real when they are part of the behavior being tested. Mock the model, external services, or tools only when that boundary is not what the test is about.

Use React Testing for React renderers, hooks, client tools, and browser-visible UI.

Agent Test Categories

Most agent and tool tests fall into four categories. Pick the smallest category that proves the behavior you care about.

Category	What It Proves	Typical Shape
Tool tests	One tool handles input, trusted context, service calls, ToolData, metadata, errors, cancellation, and suspend/resume correctly.	Run one tool in isolation. Mock repositories, external APIs, and internal services.
Runtime tests	The PAI runtime wires the agent correctly: lifecycle hooks, tool visibility, thread state, pending actions, ToolData projection, storage, and multi-turn flow.	Use a mock model that calls real or stubbed tools through a real runtime.
Evals	The real model chooses the right tool, extracts sensible arguments, asks for clarification, or refuses unsafe requests.	Use a real model with stubbed tools or fake services so the test focuses on model behavior.
Full integration smoke tests	The whole backend path still works with real model calls and real tools against fixture or sandbox data.	Run a small number of end-to-end scenarios with production-like dependencies or emulators.

These categories separate the three things that are easy to conflate:

tool behavior: did the tool do the right domain work?
runtime behavior: did PAI execute and project the tool correctly?
model behavior: did the model decide to do the right thing?

Tool Tests

Use tool tests when the behavior belongs to one backend tool. The model and agent loop are not involved.

const result = await runTool(compareProposalsTool, {
  input: { jobBriefId: "job-telemetry-platform" },
  context: { repository: fakeRepository },
});
const output = result.expectOutput();

expect(output.rankings[0]?.agency.id).toBe("agency-northstar");
expect(result.data.progress.map((record) => record.value.label)).toEqual([
  "Loaded submitted proposals",
  "Applied weighted scorecard",
  "Saved ranking output",
]);

This is the right boundary for schema validation, trusted context, service mocks, domain side effects, ToolData writes, metadata, trigger calls, failures, and direct suspend/resume behavior.

Runtime Tests

Use runtime tests when behavior depends on PAI's runtime machinery. The model is usually mocked so the test can force a specific path.

const model = t.model.queue((m) => [
  m.response([
    m.toolCall("compareProposals", {
      jobBriefId: "job-telemetry-platform",
    }),
  ]),
  m.response([m.text("Northstar is ranked first.")]),
]);

const agent = stubTools(myAgent, {
  unhandled: "fail",
  tools: {
    compareProposals: async () => ({
      rankings: [{ agency: { id: "agency-northstar" } }],
    }),
  },
});

await using runtime = createTestRuntime({ agent, model });
const client = runtime.client({
  identity: { id: "test-user", workspaceId: "test-workspace" },
});
const thread = client.thread(client.newThreadId());
const run = await thread.send("Compare proposals");
const state = await run.waitUntilIdle();

expect(t.getTool(state, "compareProposals").output?.rankings[0]?.agency.id).toBe(
  "agency-northstar",
);

This is the right boundary for lifecycle hooks, active-tool filtering, runtime context mapping, storage, thread state, pending actions, custom data projection, commands, queueing, and multi-turn workflows.

Tools can be real with fake services, or stubbed when the test only cares about runtime orchestration.

Evals

Use agent evals when the question is whether the real model chooses the right behavior. Stub tools so the test does not depend on expensive or dangerous side effects.

Examples:

the model calls searchAgencies before asking for a recommendation;
the model asks for clarification instead of guessing missing budget data;
the model does not call an approval or publish tool without confirmation;
the model refuses data the current user should not receive.

They should assert tool choice, arguments, refusal/clarification behavior, and final answer behavior.

Full Integration Smoke Tests

Use full integration tests sparingly. They prove the production-shaped path still works, but they are slower and less deterministic.

Run a small set of critical scenarios with:

a real model;
real tools;
fixture, sandbox, or emulator-backed data;
external services disabled, sandboxed, or explicitly fake-safe.

These tests are useful for release confidence, not for covering every branch.

What To Mock

Boundary	Default
LLM provider	Mock with `t.model.queue(...)` or `t.model.imperative()`.
Tools	Use real tools in tool and runtime tests; stub tools in agent evals.
External APIs, email, billing, search, queues	Prefer app-level dependency injection; use tool stubs when replacing the whole tool body is the right test.
PAI runtime, threads, runs, pending actions	Use the real in-process runtime.
Storage and files	Use in-memory or app fixtures for most tests; use real providers only for focused smoke/provider tests.
Prompt quality	Use agent evals; do not use real LLMs for runtime invariants.

Recommended Structure

src/
  agent/
    agent.ts
    agent.test.ts
    tools/
      compare-proposals/
        compare-proposals-tool.ts
        compare-proposals-tool.test.ts
  test/
    agent-test-utils.ts

Keep shared test setup small. It should create the typed helpers and common identity/database fixtures, not hide the testing primitive being used.

// src/test/agent-test-utils.ts
import { createTestUtils } from "@pai/test-utils";
import type { MyContract } from "../agent";

export const t = createTestUtils<MyContract>();

export const testIdentity = {
  identity: { id: "test-user", workspaceId: "test-workspace" },
};

Then tests should call the primitive directly:

await using runtime = createTestRuntime({ agent, model });
const client = runtime.client(testIdentity);
const thread = client.thread(client.newThreadId());
const run = await thread.send("Compare proposals");
const state = await run.waitUntilIdle();
const tool = t.getTool(state, "compareProposals");

expect(tool.output?.rankings[0]?.agency.id).toBe("agency-northstar");
expect(tool.data.progress.map((record) => record.value.label)).toEqual([
  "Loaded submitted proposals",
  "Applied weighted scorecard",
  "Saved ranking output",
]);

Tool Testing: isolated backend tool execution, ToolData, metadata, triggers, cancellation, failure, and suspend/resume.
Tool Stubbing: replace selected tool bodies while keeping the agent contract intact.
Runtime Testing: tools, workflows, model mocks, suspend/resume, typed runtime helpers, and scoped runtime behavior.
Evals: real-model tests for tool choice, argument extraction, clarification, refusal, and final answer behavior.
Agent Harness: drive a live agent headlessly from a CLI — interactive debugging and reverification, built for coding agents. See how it works for the process and result contracts, and use cases for the debugging workflows.