Evals

Agent evals test app-agent behavior with the real model. They should read like ordinary test cases: define the input, control unsafe dependencies, run the agent, and evaluate the observed output.

Runtime tests answer "did PAI wire this path correctly?" by mocking the model. Agent evals answer "did the model make a good decision?" by keeping the model real and controlling the rest of the world.

import { runAgentEval } from "@pai/test-utils";
import { describe, it } from "vitest";
import { judgeModel, myAgent } from "./agent";

describe("engineering procurement agent evals", () => {
  it("finds qualified agencies", async () => {
    await runAgentEval({
      agent: myAgent,
      judge: judgeModel,
      run: async (t) => {
        t.stubTools({
          unhandled: "fail",
          tools: {
            searchAgencies: async ({ input }) => ({
              agencies: [
                {
                  id: "agency-northstar",
                  name: "Northstar Engineering",
                  capability: input.capability,
                  region: input.region,
                },
              ],
            }),
          },
        });

        await t.send(
          "Find observability agencies in North America and recommend one.",
        );

        t.completed();
        t.calledTool("searchAgencies").inputIncludes({
          capability: "observability",
          region: "North America",
        });
        t.answer.includes("Northstar");
        t.judge.rubric("The answer explains why the agency matches the need.", {
          minScore: 0.8,
        });
      },
    });
  });
});

judge is an AI SDK LanguageModel, not a provider/model id string. Create it in your app or test setup with the provider package you already use, then pass the model object to PAI. This keeps @pai/test-utils provider agnostic and lets your normal env loading, credentials, gateway setup, and model selection live in app code.

The test runner owns process setup. runAgentEval(...) owns the agent-specific work: create the test runtime, apply tool stubs, drive the case, capture the final state and trace, and evaluate assertions. When an assertion fails, it should fail the test with the assertion label, reason, and relevant tool calls or messages.

Sections

When Not To Use Evals

Use a different test boundary when the question is not about model behavior:

Agent evals should stay focused on whether the real model made an acceptable decision for a realistic scenario.