Observability

PAI runtime telemetry records spans for runs, model steps, tool preparation and execution, suspended work, and runtime failures. Use it when you need traceable agent execution rather than application-level logging.

Install

pnpm add @pai/observability

For Langfuse export:

pnpm add @pai/langfuse

Configure Telemetry

// server/telemetry.ts
import { createPaiOpenTelemetry } from "@pai/observability";

export const telemetry = createPaiOpenTelemetry({
  serviceName: "assistant-api",
  contentCapture: {
    recordRunInputs: false,
    recordRunOutputs: false,
    recordModelOutputs: false,
    recordToolInputs: false,
    recordToolOutputs: false,
    recordToolData: false,
    recordErrorDetails: false,
  },
});

Pass it to the PAI runtime registry:

const pai = createPai({
  agents: { main: assistantAgent },
  telemetry,
  scopeKey: (identity) => identity.workspaceId,
  telemetryAttributes: ({ identity, userKey }) => ({
    "app.user.id": userKey,
    "app.workspace.id": identity.workspaceId,
  }),
});

createPaiHonoReceiver({ pai, resolveIdentity });

Only enable content capture after reviewing what may leave the app boundary. Run inputs and outputs, model outputs, tool inputs and outputs, and incremental tool data can contain user content, retrieved documents, instructions, or application payloads. Each object-form setting is independent and defaults to false.

Error detail is a separate privacy boundary. PAI records safe error code, diagnostic reference, operation, and status metadata by default. Original exception messages and stacks require an explicit trusted error-detail policy; set contentCapture.recordErrorDetails: true only when that sink may retain them. Enabling individual model or tool capture fields does not enable raw error details. The contentCapture: true shorthand deliberately enables every PAI-owned content field, including raw error details.

PAI can record the bounded user input and final user-visible assistant text on the owned pai.run.execute span with recordRunInputs and recordRunOutputs. recordModelOutputs separately records post-transform model-step text or tool calls on pai.model.step; results returned later by local tools stay behind the tool-output setting. PAI does not label the run's user message as each model step's input, because later steps may actually receive tool results or lifecycle-patched context. Detailed provider-native telemetry remains independently configured through the AI SDK or provider integration you use.

Use onRuntimeError and onHttpError for application-level logging and error reporting. OpenTelemetry is not required to learn why a request failed. See Error Handling.

Langfuse

import { createLangfuseSpanProcessor } from "@pai/langfuse";
import { createPaiOpenTelemetry } from "@pai/observability";

const langfuse = createLangfuseSpanProcessor("env");

export const telemetry = createPaiOpenTelemetry({
  serviceName: "assistant-api",
  contentCapture: {
    recordRunInputs: true,
    recordRunOutputs: true,
    recordModelOutputs: true,
  },
  spanProcessors: langfuse ? [langfuse] : [],
});

Run payload capture populates Langfuse trace input and output fields directly from the root span. Model-output capture additionally populates each model observation's output. Enable only the fields your Langfuse project is approved to retain.

For "env", set PAI_LANGFUSE_PUBLIC_KEY and PAI_LANGFUSE_SECRET_KEY. PAI_LANGFUSE_BASE_URL defaults to http://localhost:7320, and PAI_LANGFUSE_OTLP_TRACES_ENDPOINT overrides the derived OTLP endpoint.

Runtime Identity

Use userKey and telemetryAttributes to attach application identity without coupling traces to raw identity objects.

const pai = createPai({
  agents: { main: assistantAgent },
  telemetry,
  userKey: (identity) => identity.userId,
  scopeKey: (identity) => identity.workspaceId,
  telemetryAttributes: ({ identity, userKey, threadId }) => ({
    "app.user.id": userKey,
    "app.workspace.id": identity.workspaceId,
    "app.thread.id": threadId,
  }),
});

createPaiHonoReceiver({ pai, resolveIdentity });

The hook can also resolve labels asynchronously from stable identity ids:

telemetryAttributes: async ({ identity, userKey }) => {
  const labels = await directory.getLabels({
    workspaceId: identity.workspaceId,
    userId: identity.userId,
  });
  return {
    "app.user.id": userKey,
    "app.workspace.id": identity.workspaceId,
    "app.workspace.name": labels.workspaceName,
  };
},

PAI resolves the hook once per execution, after the run is claimed and lease protection is running. The root run span and that execution's model, tool, title, and reasoning-summary spans share one snapshot. A resumed execution resolves again, even though its logical runId stays the same. Queued sends resolve when they execute using their captured identity, and concurrent executions keep separate snapshots.

An asynchronous lookup runs beside the work PAI does to start an execution — building runtime context, prepareRun, thread reads, generation start — and is awaited only at the span that needs its labels. A lookup that finishes inside that window costs no run latency; a slower one delays the first model step by whatever it has left, up to one second, once per execution. Nothing other than a span waits for it: lease renewal, queue admission, durable finalization, and the run span's end are never held behind enrichment. PAI does not deduplicate across executions, so memoize inside your resolver if the lookup is expensive: a delegating agent resolves once per child execution.

Label under a prefix your application owns. pai. is reserved for runtime correlation, and keys returned under it are dropped — app.workspace.name works, pai.workspace.name is discarded and named in pai.telemetry.attributes.reserved on every span of that execution so the mistake is visible in the trace rather than silent.

clientData is absent when the executing agent declares no client-data schema, which a delegated sub-agent run can hit even in a registry whose other agents declare one, so guard it rather than assuming the shared registry type.

Attributes remain in memory for the execution and its associated background work. They are not added to identity, queued context, run records, or client messages. Keep identity limited to the trusted fields needed for application behavior; directory labels needed only for telemetry can be looked up here. The callback receives detached identity and client-data values, and PAI copies returned attribute arrays per span. Only explicitly returned fields leave through the telemetry provider, so choose labels appropriate for that destination.

Enrichment is scoped to executions. Reads, control operations, standalone pending-tool-data writes, and receiver-owned spans such as pai.http.operation do not resolve the hook — a client tool can write tool data on every progress update, so PAI never puts an application lookup in front of that path. Those spans still carry PAI's correlation fields and the pai.user.id derived from identity; add labels to them from your own telemetry provider if you need them.

Disabled telemetry skips the hook entirely. A synchronous throw or rejected promise omits app attributes and records pai.telemetry.attributes.error: "resolver_failed". A span waits at most one second for enrichment; expiring records "resolver_timeout" and proceeds without app attributes. Run cancellation also stops waiting. Lease acquisition and renewal are never held behind enrichment. That second is measured from the moment a span needs the labels, not from when resolution started, so a lookup slower than the budget is still used if PAI's own setup outlasted it. It is a backstop against a resolver that never settles, not a tuning dial, and it bounds only PAI's wait: use application-owned query timeouts or cancellation where appropriate. Late results are ignored, and a synchronous callback that blocks the JavaScript event loop cannot be preempted by a timer.

The root span starts before enrichment to preserve execution timing, and its app attributes are attached by the first span that needs them, or before the run span ends if nothing did. They are therefore available to exporters but not to a sampler that decides at root-span creation, and a run whose lookup had not finished by the time the run ended carries runtime correlation only. PAI keeps its own correlation fields for agent, thread, run, tool, and model work, and because pai. is reserved, application attributes can never override them.

pai.run.id remains stable for the logical run across execution episodes, including a resume after a pending action. Run spans also record pai.run.initiator, pai.run.admission.mode, and pai.run.input.count, so direct, queued, steered, triggered, regenerated, and merged-input work can be grouped without inspecting captured content.

PAI does not persist telemetry span or trace context in run records. A resumed execution can therefore begin a new trace while retaining the same pai.run.id correlation attribute. This keeps telemetry observational and prevents an exporter or telemetry provider from becoming part of durable run correctness.

The owned pai.run.execute span includes pai.run.input and pai.run.output only with the corresponding capture settings. Failed runs omit pai.run.input so its final visibility read cannot delay durable failure handling. The owned pai.model.step span includes safe model/provider identity, tool counts, finish reason, and token usage by default; recordModelOutputs adds the post-transform step result as pai.model.output. Detailed provider-native model spans and wire prompts are outside PaiTelemetryProvider; configure those independently through the model or AI SDK integration in use.

When input capture is enabled, PAI reads the run's complete ordered inputMessageIds set and records only transcript-visible inputs. Hidden lifecycle context does not leak into pai.run.input, even when its provider-facing role is "user".

Telemetry Cannot Affect Application Work

Telemetry is observational. A provider, exporter, span, context operation, or cleanup failure can cause missing diagnostics, but cannot fail a request, strand an admitted run, replace an application error, or repeat a model, tool, or storage callback.

Custom PaiTelemetryProvider implementations must enter withSpan and withContext callbacks synchronously, exactly once, before the provider method returns. They must preserve the callback's exact result or rejection. Provider-owned failures must not escape. PAI also isolates telemetry-owned failures and falls back to no-op instrumentation at runtime boundaries.

Do not put authorization, persistence, retries, or other application semantics inside a telemetry provider.