Agent Harness

The agent harness runs your real PAI app and lets a coding agent operate it the way a user would: send messages, read what the tools actually did, answer approvals, attach files, act as any tenant, read runtime diagnostics, and reload code — every command returning one JSON document.

It is built for one audience. No human is expected to use it, which is why there is no interactive mode, no pretty output, and no prose help: the CLI describes itself in machine-readable form, and every result is structured for a program to branch on.

It exists for one loop — reproduce → diagnose → edit → reload → reverify — run autonomously. The test categories prove behavior in CI; the harness is how an agent investigates behavior that only appears at runtime: a tool that fails for one tenant's data, an approval prompt showing the wrong value, a note that silently never saved.

It ships as @pai/harness with the pai-harness bin. A worked example lives at apps/harness-demo — three agents, approval-gated tools, several tenants, one deliberately broken persona, an app-defined command — which is what every example below runs against.

Quickstart

# Once per repository: install the skill, so a coding agent knows this exists.
pnpm --silent harness install-skill

# Start the app. Returns only when it is ready to serve.
pnpm --silent harness start --app apps/harness-demo/pai.ts --port 0

# Orient: who you can act as, and what exists.
pnpm --silent harness personas                              # ~550 B
pnpm --silent harness agents --names                        # ~550 B

# Read state and verify a tool with no model call.
pnpm --silent harness call support listOrders --as globex

# Exercise the agent itself, then answer what it asks for.
pnpm --silent harness send support "Refund ord-1001, it broke"
pnpm --silent harness respond support <threadId> --submit '{"approved": true}'

# Edit code, then make the barrier explicit and reverify.
pnpm --silent harness restart
pnpm --silent harness send support "Re-run the failing case"

pnpm --silent harness stop

That is the whole loop. Everything else on this page explains why each step is shaped the way it is.

Tell the agent what this repository needs. The packaged skill is generic, and in a real repository the first two steps are not: which entry to start, what has to be running before it, what each persona represents. Write those into .claude/skills/pai-harness/SETUP.md — the skill reads it before anything else, and install-skill never writes or replaces it, so it survives every upgrade. That separation is the point: repo-specific facts added to SKILL.md itself would make the next install-skill refuse the file as modified, and --force would throw them away. install-skill reports whether one exists yet.

How the agent invokes it. In a consumer repository the harness is a dev dependency, so the bin is at node_modules/.bin/pai-harness rather than on PATH. install-skill detects that and rewrites the skill's command examples to match, so the agent runs what it reads and never has to guess at pnpm exec, npx, or a script name. pnpm exec pai-harness and a "harness": "pai-harness" script both work too; results come from stdout, and a package manager's echo line goes to stderr, so only merging the two (2>&1) breaks parsing.

Start with the schema

The first call an agent should make is not documentation, it's the tool:

pnpm --silent harness schema --summary      # ~4 KB: every command, one line each
pnpm --silent harness schema --command send # ~4 KB: one command's full contract
pnpm --silent harness schema                # ~50 KB: everything, when you need it

schema emits the complete contract: commands, positionals, flags with types and accepted values, what each result contains, the exit codes, every stable error code, and the enumerations results use (outcomes, troubleKinds, recoveryHints). --summary also carries the invocation rules — how to keep stdout parseable, how to tell a stale read from a fresh one, and how to avoid sharing a daemon with another session. <command> --help returns just that command's entry.

That catalog is the same declaration the argument parser itself uses, so a documented flag and an accepted flag cannot drift apart. This page explains the why; the schema is the authority on the what.

What an agent needs

call verifies tools; send verifies agents

send is how you test an agent: whether it picks the right tool with the right arguments and reports the outcome honestly. Every send is a real model call, and the model's choice sits between you and the tool.

call removes it. It executes one tool with the input you specify, through the same identity resolution a real request uses, and returns what the tool returned:

pnpm --silent harness call billing listInvoices --as default
pnpm --silent harness call billing payInvoice \
  --input '{"invoiceId":"inv-2202"}' --resume '{"approved": true}'

That covers the two questions send answers expensively and unreliably: what does this tool actually do with this input, and what does the app's data look like now — the second being otherwise unreachable, since reading state meant persuading a model to call a read tool and hoping it chose the right one.

It resolves a tool the way the runtime does, through PAI's own registration resolution rather than by reading the entry's shape. That matters because { tool, enabled, mapContext } is a first-class registration for any reusable tool: call builds the agent's runtime context, applies the registration's mapContext projection, and evaluates its enabled predicate for the resolved identity — so a result reflects what that identity would actually get.

Two honest limits. It is not a durable run: no runId, nothing in thread state, nothing that survives a reload — side effects on your own stores are real, which is what makes it useful for assertions. And a tool it cannot honestly execute is refused with call_unsupported rather than faked: one that declares capabilities, one whose ctx.context nothing derives, or one that delegates to another agent, since running that starts a child agent's durable run. Those need send. A tool the agent's own registration disables for this identity is refused too, with reason: "disabled_for_identity" — and there send is no way through either, because the model is not offered the tool at all. That refusal is often the answer to "the agent ignores my tool".

Two ways to reach an app

start boots the app: the harness imports one entry, forks a worker per generation, and owns the reload barrier — which is what lets restart promise that a reverification ran against the code you just wrote.

serveHarness attaches to an app that is already running, for when booting is not possible or the app's own bootstrap is what makes it work — a CommonJS graph that only resolves under its bundler, connections its startup opens, middleware its routes rely on:

if (process.env.PAI_HARNESS === "1") {
  const { serveHarness } = await import("@pai/harness");
  await serveHarness({ pai, personas, around, sourcePaths: ["packages/server/src"] });
}

Every command then works unchanged — they all discover the endpoint from the same file a supervisor writes — and results carry daemon.mode: "attached".

The trade is the barrier: restart and stop refuse (restart_unsupported, stop_unsupported) because the process is the app's, not the harness's. Pass sourcePaths and results report stale: true whenever the running process predates your last edit, so a read from old code is never mistaken for a read of new code.

When the app's tools need its middleware

A tool that reads async-local state — an authenticated GraphQL context, a tenant transaction, a provider context for cost attribution — has nothing to read when the runtime is driven directly. Export around and the app wraps every operation the way its own middleware would:

export const around: HarnessAround = async (run, { identity, agent }) => {
  await connect();
  return runWithAuth(userIdOf(identity), run);
};

It receives the resolved identity, so the scope is built for exactly the tenant the operation runs as, and it wraps agent turns, call, and invoke alike.

The app entry contract

The harness boots the module you point it at. Export your createPai registry, plus optional personas — named identity fixtures:

import type { HarnessPersonas, ThreadContinuity } from "@pai/harness";

export const pai = createPai({
  agents: { support: supportAgent },
  scopeKey: (identity) => identity.workspaceId,
});

export const personas: HarnessPersonas = {
  default: () => ({ identity: { id: "dev", workspaceId: "acme" }}),
  globex: () => ({ identity: { id: "dev-2", workspaceId: "globex" }}),
};

// What a fresh worker means for existing threads.
export const threadContinuity: ThreadContinuity = "reset";

An app built on createAgentRuntime, or wired for a web framework, exports no pai — write a small entry of your own that does, and put the personas there too. Without personas, every command needs identity inline: --identity '<json>' , matching that agent's own schema.

Personas are the answer to "my agent sits behind auth middleware." In PAI, middleware exists only to derive { identity } — plain data. The harness skips the derivation and injects identity directly, entering the same runtime code path your endpoint would one step later. Factories may be async and do real derivation. harness personas resolves them — ~550 B, no tool schemas — so an agent never has to read your app source to learn what a persona means.

clientData is separate: it is per-request data a UI would attach, so it rides on the command (--client-data '{"locale":"en-AU"}') rather than living in a persona.

The agent sees what a person would

Anything a UI renders, the transcript describes. toolCalls[] and tool transcript entries carry data — the durable channels a tool writes with ctx.data.write(...), which is how it streams rows, progress, or a document beside the answer, and which no return value contains. Attachment and file entries name their mediaType, filename, and size rather than appearing as a bare kind. The thread's title and its queue of messages sent but not yet run are both reported. Every entry carries its messageId, so an entry you can read is an entry you can act on — that id is what regenerate --message takes.

That last one is the difference between watching a conversation and having one: regenerate re-runs a user message, --replace edits it first, and both are destructive exactly as in a UI — everything after that message is discarded before the new run starts.

What the exit code means

send blocks until the turn reaches a durable boundary and reports which one via outcome and the exit code:

• 0 — completed, or blocked on a pending action (the result carries the actionId, the payload the approver would see, and the resume JSON schema);
• 1 — the target run failed, was cancelled, or hit a step limit;
• 2 — the command itself was wrong: always { code, message, retryable, details } with a stable code, and details.recovery naming the next command whenever one exists;
• 124 — timeout.

Exit 0 means the run reached a boundary, not that nothing failed inside it. A tool that errored while the agent recovered gracefully still exits 0, which is why every turn result carries troubles[]. For the same reason, a thread's status returns to idle the moment a run ends — including one that failed, whose failure is in error on the same result.

Failure messages are redacted; the id is the way back

PAI persists a client-safe message for every failure — "The agent could not complete this request. Please try again." — and keeps the real error, its stack, and its cause for the app's own diagnostics, joined to it by an errorId. That is the right default for a browser and useless on its own for debugging, so every place the harness reports a failure carries the id: run.error.errorId on a turn, errorId on each entry in troubles[], and details.errorId when a command fails.

For a harness session the app's diagnostics are the harness log, so the real failure is one command away:

pai-harness send support "the failing case"     # error: "…Please try again.", errorId: PAI-4f2c…
pai-harness logs --grep PAI-4f2c --around 20    # the exception, with frames

--around is not optional in practice. An app's console output arrives one record per line, so a thrown object spans a dozen records and only one of them holds the id — grep it alone and you get the id back and nothing about the cause.

A deliberate failure is the exception to the rule, and it needs no id: ctx.fail() text is not redacted, so the message you are reading is already the tool's own account of what happened. A failed action still is redacted and carries no id, so the text supplied to respond --fail is gone rather than recoverable. That gap, and the others this tool works around, are listed in follow-ups/harness-core-gaps.md.

Working alongside another session

One control-plane directory holds one supervisor, and every command discovers it from the working directory. Everyone working in a checkout therefore shares one daemon, which is usually what you want: booting a real app is the expensive part, and a second start for the same app, port, and watch configuration hands back the running one rather than a second copy of a large process.

What sharing costs is that stop ends it for everyone, restart interrupts a run in flight, and under threadContinuity: "reset" a restart discards every thread including someone else's. status names the app, when it started, and that continuity, so both commands can be checked before rather than explained after — and stop reports what it took down.

Claim your own when you need a different app, port, or watch configuration than the one already running, which is exactly when start refuses:

export PAI_HARNESS_DIR=.pai-my-branch

Every command in that session then uses its own discovery file, lock, and log. Name it after the branch or task rather than something random: a supervisor is a detached process, and stopping one requires knowing the directory it wrote itself into. ls -d .pai* finds every control plane in a checkout, including one abandoned by a session that has ended.

A collision otherwise surfaces as daemon_unreachable or daemon_not_found on a command that worked a moment earlier, which reads like a crash rather than someone else's stop.

Why an agent gets further with this than with scripts

• It closes the runtime-observation gap. Coding agents read and edit code well but are blind to runtime behavior. This gives them a runtime they can converse with.
• Tool truth is separate from model prose. answer is what the agent said; toolCalls is what executed. When they disagree, that is the bug — a single command catches an approval that reported success while doing nothing.
• Nothing needs parsing or polling. One JSON document per command, turns that settle at durable boundaries, and errors that name the next command to run in CLI vocabulary.
• Multi-tenant testing is two adjacent commands — the hardest thing to exercise through a UI.
• Reloads are provable. Fresh-process reloads plus generation stamps mean a fix is verified against the new code, not the old.
• Token cost is a design constraint. Discovery narrows (--names, --agent, --tool), transcripts filter (--items, --errors, --last), and logs page by cursor, so an agent pays for what it asked for.
• Model calls are a design constraint too. call verifies tools and reads state without one, so the expensive commands are spent only on the questions that genuinely need a model.
• A degraded daemon still answers reads. After a failed reload the last-good worker keeps serving inspection, with daemon.status and daemon.reloadError on every result so a read is never mistaken for fresh code. Only a worker that actually died (worker_exited) leaves nothing to read.

Limits

• Every send is a real model call: seconds, and real provider cost. Use the harness to investigate and CI tests to cover.
• createPai() defaults to in-memory storage, so a reload resets threads. Declare threadContinuity honestly; use durable storage when threads must survive a code change.
• .env is deliberately not hot-reloaded — stop and start after editing it.
• Reload transforms TypeScript without typechecking it: a syntax error fails loudly, a type error loads happily. Run your own typecheck when verifying.
• Tool stubbing and state seeding do not exist here. When a scenario needs a fabricated dependency, use a runtime test.
• The composed prompt is not observable. The harness shows what an agent said and did, not the instructions it received — no PAI surface exposes them today. If behavior only makes sense as a prompt problem, read the agent's instructions in source and check whether a lifecycle hook rewrites them.
• Capture approval payloads when you see them. pending[] in a turn result is the only place a durable suspend payload appears; once the action resolves, no surface can reproduce what the approver saw. call <tool> --input … can re-raise the same suspension deterministically, which is usually the faster way to inspect one.
• restart discards threads when threadContinuity is "reset". That is honest behavior of an in-memory store, not a harness quirk — but it is the command the loop tells you to run after every edit, so capture the before-half of a comparison first. The result says threadsDiscarded: true when it happened, and durable storage avoids it entirely.
• Runs longer than 30 seconds are fragile. The runtime claims a 30-second run lease and never renews it, so any send to the same thread after that window treats the still-executing run as dead: it ends failed with "lease expired before completion". Even a send rejected as thread_busy triggers it. Let a slow turn settle before touching the thread again.

Where to go next

How it works covers the process model, the full result contract, and the failure taxonomy. Use cases walks through the debugging workflows this was built for.