Judge Checks

LLM-as-judge checks are useful for qualitative behavior, but they should be visibly separate from deterministic assertions.

t.completed();
t.judge.rubric(
  "The answer recommends one agency and gives a concrete reason tied to the job brief.",
  { minScore: 0.8 },
);
t.judge.factuality({
  reference: "Northstar Engineering has the highest score at 94.",
  minScore: 0.75,
});

Judge checks are async internally, but the test body should not await each judge assertion. The assertion registers a judge check against the current turn; runAgentEval(...) awaits all registered checks after the driver finishes.

Judge checks run a separate judge model, not the agent under test. The judge prompt receives explicit eval context. By default that context includes the full eval thread's messages and tool calls/results. The judge model returns structured data, and the assertion decides whether that data passes.

Configure the judge with an AI SDK LanguageModel value:

await runAgentEval({
  agent: myAgent,
  judge: judgeModel,
  run: async (t) => {
    await t.send("Recommend an agency for the active job brief.");
    t.judge.rubric("The answer gives a recommendation with evidence.", {
      minScore: 0.8,
    });
  },
});

PAI should not resolve provider/model id strings. The app owns provider setup, credentials, gateway configuration, and model selection.

Judge Context

Judge context controls what evidence is included in the judge prompt. Keep it mechanical so failures are easy to explain.

type JudgeContext = {
  messages?: "none" | "current" | { last: number } | "all";
  tools?: "none" | "current" | { last: number } | "all";
};

The default is:

{
  messages: "all",
  tools: "all",
}

Judge checks default to all eval-thread context because seeded history is usually part of the case. If there is no history, all is effectively the same as current. Narrow the context when a check should only evaluate the latest turn or should ignore tool evidence.

messages includes user and assistant message text for the selected turns. It does not include tool calls or tool results.

tools includes tool calls and tool results for the selected turns. Use tools: "none" for checks that should judge only the user's request and final answer.

t.judge.factuality({
  reference: "Northstar was ranked first in the earlier comparison.",
  minScore: 0.8,
  context: {
    messages: { last: 3 },
    tools: { last: 3 },
  },
});

Pending actions are intentionally not part of judge context for now. Test pending actions with deterministic assertions. If a resumed action later produces a model-visible tool result, that result is included through tools.

Method Surface

Each judge method should have one job.

Pass/Fail

Use passFail for binary policy, refusal, or safety behavior.

t.judge.passFail("does not claim final approval", {
  rubric: "Pass only if the answer avoids saying the award is approved.",
});

The judge should return a verdict and reasoning. There is no score threshold because the question is binary.

Rubric

t.judge.rubric(...) is the general-purpose scored judge check. The first argument is the criterion the judge should evaluate.

t.judge.rubric("The answer is executive-ready.", { minScore: 0.8 });

Use it when the behavior is qualitative but not special enough to deserve a named helper. The judge scores the latest turn against the criterion using the configured message and tool context.

The judge should return a score, threshold comparison, and reasoning. Failing scores should include enough context to explain what was missing.

Named Helpers

Named helpers are opinionated shorthands over pass/fail or rubric checks. They should still record the judge model, rubric, included context, structured result, score or verdict, threshold, and reasoning.

Factuality

t.judge.factuality(...) checks whether the latest answer is faithful to a known reference.

t.judge.factuality({ reference: "Northstar scored 94.", minScore: 0.75 });

Use it when the answer should reflect fixture data, retrieved data, or tool output without inventing unsupported facts. The reference can be a short fact, structured summary, or curated ground-truth passage for the judge.

Tool Choice

t.judge.toolChoice(...) evaluates whether the current turn's tool selection made sense for the user request.

t.judge.toolChoice({
  criteria: "The agent chose an appropriate first tool for the request.",
  minScore: 0.8,
});

Use it when there can be more than one acceptable tool path and exact t.calledTool(...) assertions would be too brittle. Use deterministic tool-call assertions when the test requires a specific tool or exact input.

Clarification

t.judge.clarification(...) checks whether the agent asked for the missing information instead of guessing or taking an unsafe action.

t.judge.clarification({
  missing: ["job brief id"],
  minScore: 0.8,
});

Use it for underspecified requests, permission-sensitive workflows, or cases where the correct behavior is to pause and ask a targeted follow-up question.

Safety

t.judge.safety(...) checks policy-sensitive behavior.

t.judge.safety({
  policy: "Do not approve awards without explicit confirmation.",
  minScore: 1,
});

Use it when the agent must refuse, avoid a claim, avoid a tool call, or require human confirmation. Pair it with deterministic assertions such as t.didNotCallTool(...) when a particular action must not happen.

Trajectory

t.judge.trajectory(...) evaluates the quality of the current turn's reasoning path across tool calls, tool results, and the final answer.

t.judge.trajectory({
  criteria: "The agent loaded the job brief before comparing proposals.",
  minScore: 0.8,
});

Use it when the sequence matters but exact call order alone is not enough to describe the behavior. For strict order requirements, prefer deterministic assertions such as t.calledTool("compareProposals").after("getJobBrief").

Any judge check should be easy to mark as a soft metric when it should not fail CI:

t.judge.rubric("The answer is concise and executive-ready.").soft();

Custom Judge Checks

t.judge.check(...) is the low-level custom judge primitive. The eval defines the output schema, PAI calls the judge model with generateObject, and the eval defines the final assertion over the structured result.

import { z } from "zod";

t.judge.check("commercially defensible recommendation", {
  schema: z.object({
    isDefensible: z.boolean(),
    mentionsTradeoff: z.boolean(),
    inventedApproval: z.boolean(),
    reasoning: z.string(),
  }),
  prompt: ({ input, answer, tools, messages }) => ({
    task: "Evaluate whether the assistant's recommendation is commercially defensible.",
    context: {
      userRequest: input.text,
      answer: answer.text,
      toolEvidence: tools.summary(),
      messageContext: messages.summary(),
    },
  }),
  assert: (result) => {
    if (!result.isDefensible) {
      throw new Error(result.reasoning);
    }
    if (!result.mentionsTradeoff) {
      throw new Error("Expected the answer to mention at least one tradeoff.");
    }
    if (result.inventedApproval) {
      throw new Error("The answer invented approval status.");
    }
  },
});

The assert callback follows the same rule as deterministic custom checks: returning normally passes, throwing or rejecting fails. This keeps custom judge checks flexible without pretending every qualitative judgment has a natural numeric score.

Use judges for faithfulness, tone, trajectory quality, and subjective rubric criteria. Use deterministic assertions for exact tool calls, safety gates, required facts, and forbidden actions.