Assertions

Assert run status, answer text, tool calls, tool results, pending actions, and custom deterministic behavior.

Assertions should produce structured failures with the assertion name, reason, and relevant messages or tool calls. Use deterministic assertions by default.

Assertions default to the latest t.send(...). Initial state and previous turns do not satisfy t.calledTool(...), t.tool(...), t.answer, or pending action assertions. That keeps evals focused on the prompt being evaluated.

Custom assertion callbacks pass when they return normally or resolve. They fail when they throw or reject. The callback can use any test framework assertion, but it does not need to.

Run Status

t.completed();
t.failed();
t.waiting();
t.noRuntimeErrors();
t.noToolErrors();

Use these for broad outcome checks: the run finished, failed, parked on a pending action, or avoided runtime/tool errors.

Answer Text

t.answer.includes("Northstar");
t.answer.excludes("Orbit Labs is approved");
t.answer.matches(/which job brief|job brief id/i);
t.answer.equals("I need a job brief id before approving an award.");
t.answer.satisfies("mentions a recommendation", (answer) => {
  if (!/recommend/i.test(answer.text)) {
    throw new Error("Expected the answer to recommend an agency.");
  }
});

t.answer is the latest assistant answer after the most recent t.send(...). Use answer assertions for required facts, forbidden claims, clarification questions, and refusal language.

Tool Calls

t.calledTool("getJobBrief");
t.calledTool("getJobBrief").inputIncludes({
  jobBriefId: "job-telemetry-platform",
});
t.calledTool("getJobBrief").inputEquals({
  jobBriefId: "job-telemetry-platform",
});
t.calledTool("compareProposals").after("getJobBrief");
t.calledTool("searchAgencies").times(2);
t.tool("searchAgencies").at(0).inputIncludes({ capability: "observability" });
t.tool("searchAgencies").at(1).inputIncludes({ capability: "data pipelines" });
t.didNotCallTool("approveAwardRecommendation");
t.usedNoTools();
t.maxToolCalls(2);

Tool assertions should be typed from the agent contract. Use them for tool choice, argument extraction, ordering, count, and forbidden actions.

Tool Results

t.tool("compareProposals").inputIncludes({
  jobBriefId: "job-telemetry-platform",
});
t.tool("compareProposals").outputIncludes({
  rankings: [{ agencyId: "agency-northstar" }],
});
t.tool("compareProposals").inputSatisfies("uses active job brief", (input) => {
  if (input.jobBriefId !== "job-telemetry-platform") {
    throw new Error("Expected the active job brief.");
  }
});
t.tool("compareProposals").outputSatisfies("has a ranked winner", (output) => {
  if (!output.rankings?.[0]?.agencyId) {
    throw new Error("Expected at least one ranked agency.");
  }
});
t.tool("compareProposals").succeeded();
t.tool("compareProposals").failed();

Use tool-result assertions when the model behavior depends on stubbed tool output or when the answer should reflect a specific tool result. They make the behavior chain explicit: the model called the tool, the tool returned expected data, and the answer used that data.

inputIncludes(...) and outputIncludes(...) use deep partial matching: objects match by subset, arrays match when they contain matching items, and primitives match exactly. Use inputEquals(...) or outputEquals(...) when the full payload matters.

Pending Actions

Use pending-action assertions when the correct behavior is to ask a human, client, or browser to do something before continuing.

await t.send("Prepare the award memo and ask me to approve it.");

t.waiting();

const approval = await t.pendingAction(
  "approveAwardRecommendation",
  "approval",
);

approval.input.matches({
  jobBriefId: "job-telemetry-platform",
});

await approval.submit({
  approved: true,
  comment: "Approved for the pilot.",
});

This asserts that the agent suspended on the expected action, verifies the action input, and then simulates the human or client response so the eval can continue.

State And Custom Checks

t.messages.includes("Loaded the warehouse telemetry platform brief.");
t.satisfies(
  "compares once and recommends from evidence",
  ({ answer, tools, messages }) => {
    if (tools.byName("compareProposals").length !== 1) {
      throw new Error("Expected one comparison tool call.");
    }
    if (!messages.text().includes("Northstar")) {
      throw new Error("Expected Northstar to appear in the current turn.");
    }
    if (!/recommend/i.test(answer.text)) {
      throw new Error("Expected a recommendation.");
    }
  },
);

Use custom checks when the assertion needs product-specific logic. The callback should receive stable eval views such as answer, tools, and messages, not raw runtime internals.

Good eval assertions usually check selected tool names, extracted arguments, tool order when order matters, absence of dangerous tool calls, final answer facts, clarification/refusal behavior, and stable structured outputs.

Avoid overconstraining exact prose or every intermediate step. Eval assertions should lock in product behavior, not one lucky model trajectory.