Use Cases

Each section is a workflow the harness was built for, with the commands that carry it out. Examples use a support/billing app with orders, approval-gated refunds, and invoices — substitute your own agents. Every command is shown with pnpm --silent harness; installed consumers call the pai-harness bin directly.

Reproduce a tenant-scoped bug

Bugs that only appear for one customer's data are the hardest to chase through a UI: you need that customer's session. With injected identity, the same question asked as two personas is two commands, and the tool output makes the answer unambiguous.

pnpm --silent harness send support "Search my orders for a keyboard" --as globex

"toolCalls": [
  {
    "tool": "searchOrders",
    "state": "output-available",
    "input": { "query": "keyboard" },
    "output": { "matches": [{ "orderId": "ord-1001", "amountUsd": 129 }] }
  }
]

ord-1001 belongs to another workspace, so the tool is not filtering by scope. Confirm the boundary by asking a scoped tool the same thing:

pnpm --silent harness send support "List my orders" --as globex   # → only globex orders

Two sends localize the defect to one tool, before reading a line of source. Every result also echoes the identity it ran under, so the evidence is self-contained: the tenant and the leaked row sit in one document, and a saved result can be attributed later without shell history.

Once you suspect a specific tool, drop the model entirely — call fixes the input, so a difference between tenants can only come from the tool:

pnpm --silent harness call support searchOrders --input '{"query":"ord-1001"}' --as globex
pnpm --silent harness call support searchOrders --input '{"query":"ord-1001"}' --as default

That is also the fastest way to prove a fix: rerun both personas after the edit, with no model call and no nondeterminism in between.

See exactly what an approver sees

When a tool suspends for human input, the harness surfaces the suspend payload verbatim — the same data your approval UI would render — plus the JSON Schema of a valid response:

pnpm --silent harness send support "Refund ord-1001, it arrived broken"

"outcome": "blocked",
"pending": [
  {
    "actionId": "act_…",
    "tool": "refundOrder",
    "suspendName": "approval",
    "payload": {
      "orderId": "ord-1001",
      "amountUsd": 12900,
      "prompt": "Approve refund of $12900 for ord-1001 (Mechanical keyboard)?"
    },
    "resumeSchema": { "type": "object", "properties": { "approved": { "type": "boolean" } } }
  }
]

A $129 keyboard is being approved as $12900 — a cents/dollars bug that is invisible in the agent's prose and only visible in the payload. Approving is then mechanical, because the schema is right there:

pnpm --silent harness respond support <threadId> --submit '{"approved": true}'

Save this result if the payload is your evidence. pending[] is the only place the suspend payload appears: once the action resolves it is gone from every surface, including thread --raw. Redirect the JSON to a file, or copy the payload into your notes, before you respond.

Exercise the paths nobody clicks

Happy paths get tested; denials, cancellations, and tool failures do not. All three are single commands, and each returns the resulting tool output:

pnpm --silent harness respond billing <threadId> --submit '{"approved": false}'   # rejected
pnpm --silent harness respond billing <threadId> --cancel --reason "wrong invoice" # withdrawn
pnpm --silent harness respond billing <threadId> --fail "approver service down"    # errored

This is where the separation between answer and toolCalls earns its keep. A rejected void that reports success looks like this:

"answer": "The invoice remains unchanged and has not been voided.",
"toolCalls": [
  { "tool": "voidInvoice", "state": "output-available", "output": { "voided": true } }
]

The prose is reassuring and wrong: the resume handler ignored approved and voided anyway. Confirm the damage through the app's own read path, which is the strongest possible evidence:

pnpm --silent harness send billing "List my invoices"   # → inv-2201 status: "void"

An audit pass worth running on any approval-gated agent: for every suspending tool, drive --submit approved, --submit rejected, --cancel, and --fail, and check the durable state after each.

Find the bug that leaves no trace in a thread

The hardest failures are the ones the app handled: a tool that caught its own error and returned a success-shaped result, or a write that silently no-oped. Thread state shows nothing wrong, because nothing reported anything wrong. What the app logged is the only evidence:

pnpm --silent harness logs --level error --last 20

"entries": [
  { "at": "2026-07-27T10:19:55.738Z", "level": "error", "source": "app",
    "stream": "stderr", "generation": 4, "pid": 73255,
    "message": "[notes] persist failed: note exceeds column limit" }
],
"nextCursor": "8f3c1e7a-…:4821",
"session": "8f3c1e7a-…"

logs reads the daemon's diagnostics — your tools' console.*, unhandled rejections, provider SDK warnings, supervisor events — as records rather than text, filterable by --level, --source, --generation, and --grep.

Three properties matter for a loop. First, nextCursor is opaque and identifies the log session as well as the position: pass it back as --cursor and you get only what arrived since, and a log that was replaced comes back as restarted: true rather than silently skipping a fresh run's first bytes. A cursor from a different daemon is reported as cursorSession, so "my daemon restarted" and "I am reading the wrong endpoint" stay distinguishable. Second, one read is bounded to 4 MB, and a clamped window says so with skippedOlderOutput plus windowStartByte — so "no matching records" and "the matches are older than this window" are never confused. Third, it reads the log file directly, so it still works when the daemon has crashed — which is exactly when you need it.

--source supervisor is the process timeline: generations becoming ready, reloads starting, promotions, reload failures, and a worker that exited unexpectedly with its code and signal. That is where "did the code change under me, did the worker die?" is answered.

The division to remember: thread state holds run and tool failures; logs holds everything the process printed. If a send looked clean but the outcome was wrong, check logs before re-reading the transcript.

Vary the request, not just the caller

Scope and user cover "who is asking". clientData covers "how they are asking" — the per-request data your UI would attach, which tools and instructions may branch on:

pnpm --silent harness send support "Where is my order?" \
  --as globex --client-data '{"locale":"fr-CA","surface":"mobile"}'

It composes with either identity mode, so one persona can be exercised across many request shapes without defining new fixtures.

Attaching a file is part of the request too, so send takes one:

pnpm --silent harness send support "Here is my receipt" --attach ./receipt.pdf
pnpm --silent harness send support "And this screenshot" \
  --attach ./shot.bin=image/png --attach-id file_9f2c

The bytes upload through the app's own file provider before the send, so the agent receives a fileId exactly as it would from a browser upload, and the thread shows an attachment entry with its mediaType, filename, and byteSize. Media type comes from the extension; name it as <path>=<mediaType> when the extension cannot say. An app that configures no files provider is refused with attachments_unsupported and nothing is sent — a missing provider is a fact about the app, not a failed upload.

Follow work an agent delegated to a subagent

A subagent's child thread is created hidden, so the child agent's thread listing looks empty even after real delegated work ran — and --has-error would skip a failed child entirely. --include-hidden is the harness's own privileged read:

pnpm --silent harness threads researcher --include-hidden --has-error
# → items[].parentThreadId links each child back to the turn that caused it

The parent's tool call also carries the child's threadId and runId in its output, so thread researcher <childThreadId> reads the delegated conversation directly. Use the listing when you do not have a parent turn to read — a child that failed before returning leaves no output to extract an id from.

Exercise what the app exposes to its own clients

A command is a typed server action an agent exposes on a thread — pushing a system message, escalating a case, firing a run from a webhook. It is a real client surface, so it needs testing like any other, and invoke is how:

pnpm --silent harness agents --agent ops                 # commands[] with their inputSchema
pnpm --silent harness invoke ops <threadId> pageOperator \
  --input '{"incident":"disk full","wakeAgent":true}'

Commands run with a trusted thread handle, which makes invoke the only way to reach a trigger from this tool. A command that starts a run does not have to report its id — its return type is its own contract — so read the thread afterwards rather than relying on activeRunId, which is only a snapshot and is absent once a fast run has finished.

Refusals separate the three causes: a rejected input comes back as invalid_arguments with the validation issues naming the field, an unknown name as unknown_command listing what the agent registers, and a command whose own code threw as command_failed. Because commands are first-touch operations in the runtime, an unknown thread id is refused unless you pass --create, so a typo cannot quietly bring a thread into being.

Sweep for damage after a session

Once a few things have gone wrong, ask which threads need attention rather than remembering ids. Availability returns to idle after a run ends, so use the error query, and sweep across tenants in one call:

pnpm --silent harness threads support --has-error --all-personas

(Add --include-pending when you also want each thread's outstanding approvals, with their payloads and resume schemas, in the same call.)

"items": [
  {
    "threadId": "…",
    "persona": "globex",
    "troubles": [
      {
        "kind": "tool_error",
        "runId": "…",
        "tools": [{ "tool": "applyPromoCode", "error": { "message": "Promo code welcome10 is not valid." } }]
      }
    ]
  }
],
"skippedPersonas": [
  { "persona": "legacyAdmin", "code": "invalid_identity", "message": "The legacy admin directory is offline." }
]

Every run in each thread is inspected, so a failure masked by a later successful run still shows up, and skippedPersonas keeps a broken identity source from masquerading as "nothing found."

Each row names the persona it was listed under, and rows are never merged across personas: storage is partitioned by scope, so two tenants can each own a different thread with the same id, and collapsing them would report one tenant's evidence under both names. When --limit withholds threads the result says hasMore: true, because --status and --has-error filter the page that was returned — a filtered sweep of page one is not a sweep of everything.

Read troubles as evidence rather than a verdict. Once you have added guards, your own deliberate rejections — a refused double refund, a declined approval — appear here in exactly the same shape as accidents. A sweep that returns entries after a fix is normal; check the tool error and decide.

Reverify without testing stale code

The risk in any edit-and-retry loop is measuring the old code. The harness makes that impossible rather than unlikely:

# edit the tool…
pnpm --silent harness restart
# → { "generation": 5, "previousGeneration": 4, "sourceReloadsSinceLastBarrier": 1 }
pnpm --silent harness send support "Search my orders for a keyboard" --as globex
# → { …, "daemon": { "generation": 5 } }

Because each reload is a fresh process, edits to transitively imported modules take effect. Because every result is generation-stamped, an observed behavior can be tied to an exact version of your source. And a broken edit fails loudly instead of silently keeping the old worker:

pnpm --silent harness send support "Anything"
# → exit 1, { "code": "reload_failed", "message": "support.ts:13:42: ERROR: Unexpected \";\"",
#             "details": { "activeGeneration": 7, "attemptedGeneration": 8 } }

Fix the file, restart, continue.

Learn an unfamiliar agent app

Before driving an app you did not write, ask it what it is:

pnpm --silent harness schema --summary   # every command, one line each
pnpm --silent harness personas          # who you can act as, and what each sees
pnpm --silent harness agents --names    # which agents and tools exist
pnpm --silent harness status            # app entry, watch paths, generation, continuity

Both are cheap and answer different questions: agents --names (~500 B) says what exists, agents --agent support --tool refundOrder (~1.5 KB) gives one tool's schemas, and the unfiltered manifest (~9 KB for three agents) is there when you genuinely want everything. personas resolves each one to its { identity } for about 500 bytes, so learning that globex means workspaceId: "globex" never requires opening the app source — and a persona whose factory throws is reported with its own error rather than omitted.

thread <agent> <threadId> then returns the conversation as structured entries, and filters keep the read small:

harness thread support <id> --items tool --last 3

"transcript": [
  { "kind": "tool", "tool": "refundOrder", "state": "output-available",
    "input": { "orderId": "ord-1001" },
    "output": { "refunded": true, "amountUsd": 129 },
    "runId": "…" }
],
"totalEntries": 6,
"truncated": true

truncated and totalEntries mean a slice is never mistaken for the whole conversation. Unresolved approvals appear in pending[]; an approval already answered leaves only its tool result, so a transcript alone will not tell you an approval happened.

Control long, queued, or runaway runs

Slow tools and busy threads have first-class handling instead of dead ends. First, learn the ids — a send only reports its threadId once the turn settles, so while a run is still executing you ask the listing for them:

pnpm --silent harness threads ops --status running --include-state
# → items[].threadId, and items[].activeRunId for the executing run

--include-state is what turns a thread listing into a control surface; without it a row tells you a thread is running but not which run. Then:

# A second message while a run is executing is rejected, with the way forward.
pnpm --silent harness send ops "status?" --thread <threadId>
# → exit 2, thread_busy, recovery: ["resolve_pending_action","send_with_queue","send_with_steer"]
#   details.activeRunId also names the run you are waiting on

pnpm --silent harness send ops "status?" --thread <threadId> --queue   # → outcome "queued" + runId
pnpm --silent harness run ops <threadId> <runId> --wait                # follow that exact run later
pnpm --silent harness run ops <threadId> <runId> --stop --reason "wrong input"

A queued message is visible where a person would see it — thread reports queue[] with each waiting turn's queueItemId, runId, and messages in the same shape the transcript uses, so a message reads the same before and after it runs. --stop covers each non-terminal state: a running or waiting run is cancelled, and a queued one is recalled so it never starts.

Two things to know about thread ids. send without --thread creates a new thread and reports its id, and send --thread <id> is restore-only, so a typo never silently starts a fresh conversation. For a scripted scenario, choose the id up front with --new-thread <id>, which fails if the id already exists — no id-capture plumbing needed.

--stop settles to the run's durable terminal outcome rather than returning a racy snapshot. It cancels a running or waiting run, including the waiting run's pending tool tail; a run that already ended is refused rather than reported as a stop. A queued run that was admitted a moment before the recall lands is refused too, with reason: "already_admitted". A stop also only asks a run to abort — a tool that never checks ctx.signal keeps going, and the stop ends in turn_timeout with the run still running. Restart the worker to reclaim it.

Assert what the app's data actually is

The hardest disagreement to see is between what a tool reported and what it did. A payment tool can return {"paid": true, "invoiceId": "inv-2202"} while settling a different row — the agent then reports success honestly, troubles is empty, and the exit code is 0, because nothing failed. Only the store knows.

pnpm --silent harness call billing payInvoice \
  --input '{"invoiceId":"inv-2202"}' --resume '{"approved": true}'
pnpm --silent harness call billing listInvoices     # → which row actually changed?

The read runs in the worker process, against the same module instances your agents use, so this is your app's real state and not a copy. --resume drives the approval-gated path deterministically, which matters because that branch — the code that runs after approval — is where money-movement bugs live and is the hardest to reach through a model.

Retry or reword a request, as a person would

A user who gets a bad answer retries it, or rewords the question. Both are runs over an existing conversation, so both report exactly like a send:

pnpm --silent harness thread support <threadId> --items text   # messageId per entry
pnpm --silent harness regenerate support <threadId> --message <messageId>
pnpm --silent harness regenerate support <threadId> \
  --message <messageId> --replace "Refund ord-1001, not ord-1002"

regenerate is destructive exactly as in a UI: everything after that user message is discarded before the new run starts. Omit --message to retry the message whose run failed — the UI's retry button — and if nothing failed, the command says so rather than picking a message for you.

This is also the cheapest way to test prompt sensitivity: the same thread, the same tools, one clause changed.

Promote a reproduction into a regression test

The harness finds and confirms the bug; a test keeps it fixed. Once a reproduction is understood, mirror it in a runtime test with a mock model so CI reproduces it in milliseconds and without provider cost:

const model = t.model.queue((m) => [
  m.response([m.toolCall("voidInvoice", { invoiceId: "inv-2201" })]),
  m.response([m.text("The invoice was not voided.")]),
]);

await using runtime = createTestRuntime({ agent: billingAgent, model });
const client = runtime.client(testIdentity);
const thread = client.thread(client.newThreadId());
const run = await thread.send("Void inv-2201");

const pending = await run.waitForPending({
  toolName: "voidInvoice",
  suspendName: "approval",
});
await pending.submit({ approved: false });
const state = await run.waitUntilIdle();

// The bug: a rejected approval voided the invoice anyway.
expect(t.getTool(state, "voidInvoice").output?.voided).toBe(false);

Use an eval instead when the behavior you fixed was the model's decision — which tool it chose, whether it asked for clarification — rather than your code's.

Let a coding agent run the whole loop

Everything above is designed to be driven without a human reading the output; the harness overview lists the properties that make that safe. What is worth writing down is the loop itself.

A practical charter for an agent handed a ticket:

0. If anyone else may be working in this checkout, export PAI_HARNESS_DIR=… before start. Sharing a supervisor means either session can stop the other's, and the symptoms read like a crash rather than a collision.
1. schema --summary, then agents --names and status — learn the contract and confirm which generation you are testing.
2. Reproduce with send under the persona named in the ticket. Redirect the JSON to a file: pending[] payloads and tool errors are your evidence, and payloads do not survive the action resolving.
3. Compare answer against toolCalls[]. Disagreement between them is a bug before you have read any source.
4. Read the app source to diagnose, edit, then restart — the barrier — and confirm the new daemon.generation in the next result.
5. Rerun the exact reproduction command. Also rerun it under a second persona if the fix touched scope handling.
6. Finish with threads <agent> --has-error --all-personas per agent to catch collateral damage — adding --include-hidden for any agent that receives delegated work, whose threads are otherwise invisible to the sweep — check logs --level error for failures the app swallowed, and run your project's typecheck: reload strips types without checking them.

Two failure modes to code against explicitly: an exit code of 0 means the run succeeded, not that nothing broke inside it — read troubles[], which carries the same taxonomy as threads --has-error — and any *_outcome_unknown or a turn_timeout from a mutation means inspect, never retry.