Self-hosting the sandbox runtime

Package: @pai/gke-sandbox-infra

Provide an existing cluster, a verified runtime image in your registry and the caller identities/networks you trust. PAI's Terraform module installs a closed runtime template and its warm pool, namespace permissions and a 20-pod quota for each application. Your infrastructure owns the cluster, cloud IAM, routing, registry and capacity choices. Deploy the application using your existing tools; PAI does not require a separate application-hosting module.

Before installation

The runtime requires a configured Kubernetes provider, gVisor scheduling, an enforcing NetworkPolicy dataplane and an Agent Sandbox controller serving agents.x-k8s.io/v1beta1 and extensions.agents.x-k8s.io/v1beta1. On GKE that is the Agent Sandbox add-on on a control plane at 1.36.3-gke.1767000 or later. Every v1beta1 claim names a SandboxWarmPool, so the module installs one per application even when it keeps no pods warm. The CRDs must exist before the first plan, because kubernetes_manifest types each object against its CRD. Closed egress does not require FQDN NetworkPolicy.

Your infrastructure supplies the installer permissions, caller authentication, cluster API access and private routes to the daemon. The installer manages namespaces, ResourceQuotas, Roles, RoleBindings and Agent Sandbox custom resources, and reads CustomResourceDefinitions (on GKE, container.customResourceDefinitions.get and .list). It creates no Service, Secret or NetworkPolicy: the Agent Sandbox controller writes each template's NetworkPolicy. Your infrastructure also enables cloud APIs and supplies node image-pull permissions. Create this foundation before the runtime's provider initializes; the module contains no cluster, Google account, IAM or registry resources.

Build the runtime image

Keep your sandbox Dockerfile, tools and scripts in your application repository. Extend PAI's published Node 24 + sandboxd base, replacing PAI_BASE_IMAGE@sha256:BASE_DIGEST with the FROM reference in the publisher's summary:

FROM PAI_BASE_IMAGE@sha256:BASE_DIGEST
USER root
RUN apt-get update && apt-get install -y --no-install-recommends python3 \
    && rm -rf /var/lib/apt/lists/*
COPY scripts/ /opt/my-app/scripts/
USER 1000

The base supports linux/amd64 and linux/arm64 and includes the daemon entrypoint, Node, home and workspace settings. Inherit those settings and restore USER 1000 after installing your tools. Builders need Artifact Registry read access and Docker authentication to the private PAI base registry. Your final image is published in your registry; sandbox nodes only pull that final image.

The pod mounts an empty volume over /workspace, so anything the image puts there is hidden at runtime. Install Node packages outside it and link them at /node_modules, which Node finds from any script under /workspace:

USER root
WORKDIR /opt/my-app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev && ln -s /opt/my-app/node_modules /node_modules
WORKDIR /workspace
USER 1000

Install @pai/gke-sandbox-infra as a development dependency for its verification and local-cluster tools. From your application repository:

docker build --platform linux/amd64 -t my-registry/my-app-runtime:build .
bash node_modules/@pai/gke-sandbox-infra/scripts/verify-runtime-image.sh my-registry/my-app-runtime:build
# Publish the verified image through your registry workflow, then use its digest.

The verifier checks the daemon contract. Verify your own tools in your build process and own final-image scanning and publication. Update the base digest deliberately and rebuild your image when adopting PAI runtime updates. The packaged runtime/Dockerfile remains available to build the same base yourself.

Each application's image is the required digest-pinned final image. Keep its base aligned with your @pai/* release: the provider requires its exact daemon version in /v1/metadata and rejects missing, empty or unreadable metadata. This check does not attest image provenance.

For local development, build a native image and select it explicitly:

docker build -t my-app-runtime:local .
PAI_LOCAL_IMAGE=my-app-runtime:local \
  bash node_modules/@pai/gke-sandbox-infra/scripts/local-cluster.sh up

The installed helper creates or reuses kind and prints the app's connection settings. No PAI checkout is needed. See the package README for namespace, kubeconfig and lifecycle options. kind tests the mechanics without gVisor; acceptance still needs the real deployment checks below.

Install for an application

The module binds existing Google service accounts, so the application runs as one: on Cloud Run, on a VM, or in a pod that uses Workload Identity. Configure the Kubernetes provider for the sandbox cluster, then call the module with the application's identity and its verified dedicated source CIDR:

provider "kubernetes" {
  config_path    = pathexpand(var.kubeconfig_path)
  config_context = var.kube_context
}

module "sandbox" {
  source = "./path-to-installed-package/terraform"

  applications = {
    chat = {
      image      = var.runtime_image
      allow_from = [var.caller_cidr]
      consumers  = { server = var.caller_service_account_email }
    }
  }
}

Review a saved plan and apply. This creates namespace pai-sandbox-chat, a template and a warm pool both named pai-node-runtime, zero warm pods, deny-all egress and a quota of 20 pods. The caller CIDR reaches only TCP 8080/9090.

Your infrastructure supplies the caller's Google IAM, credentials, cluster API reachability, private pod routes and firewall rules for TCP 8080/9090. The module binds the supplied email as a Kubernetes User; it creates no Google account or key. runtime/contract.json provides the daemon ports and workspace root for external firewall configuration before the runtime exists.

Deploy the application with the module's outputs and the target cluster's endpoint and base64-encoded CA. Its Google identity alone does not identify which cluster to reach:

import { createGkeSandboxProvider } from "@pai/sandbox-gke";

const provider = createGkeSandboxProvider({
  namespace: "pai-sandbox-chat", // application_namespaces["chat"]
  cluster: {
    endpoint: process.env.PAI_GKE_SANDBOX_ENDPOINT!,
    caCertificate: process.env.PAI_GKE_SANDBOX_CA_CERTIFICATE!,
  },
});

The provider uses ADC for the explicit cluster and dials each sandbox pod directly, the default pod-ip transport. Without cluster it uses KUBECONFIG, ~/.kube/config or the pod's service-account token instead. consumers authorizes Kubernetes operations; allow_from admits daemon traffic. Neither replaces the other. Close the provider when the application host shuts down to release its connections; this does not delete active sandboxes.

Daemon connectivity is separate from Kubernetes access: verify the source Pod/Node/SNAT address the sandbox policy actually sees and determine which other workloads can use it. A CIDR allows all sources in that range. Kubernetes claim RBAC does not authenticate direct daemon requests; a daemon token does. Set the application's daemon_token_sha256 to the token's SHA-256 digest and the provider's daemonToken to the token, and the daemon refuses every call without it except its health check. The provider's port-forward transport cannot reach GKE gVisor sandboxes.

The package's terraform/examples/self-hosted supplies the root variables for this example. It uses local state; configure your team's remote backend before a shared deployment. If copying the starter outside the package, replace its relative module source with the installed package's terraform directory or an immutable Git source that keeps the package's runtime/ directory beside terraform/. Configure private repository access if that source needs it.

Defaults and overrides

Quota omission or null uses the default, while any supplied map replaces it. Include pods in that map to retain a pod limit. Warm-pool pods count toward the quota and are billed while idle. When the template changes, for example with a new image, the pool replaces its idle pods at once, so no claim adopts a pod running the previous daemon.

Size each sandbox

resources uses Kubernetes requests and limits. Each accepts cpu, memory and "ephemeral-storage" as Kubernetes quantity strings. Omitted fields retain these defaults, even when another field is overridden:

Omitted or null resources, requests and limits objects use the defaults. Omit an individual quantity to keep its default; a null quantity is invalid. Requests must not exceed their corresponding limits, which Kubernetes checks during pod admission. The cpu and memory requests must not be bare whole numbers such as "1": kubernetes_manifest sends one as an integer, which GKE's sandbox admission policy denies at apply, so the module rejects it at plan. Write 1000m for one CPU. The same policy requires a memory request of at least 1Gi per CPU requested.

For example, inside an application:

resources = {
  limits = { memory = "2Gi" }
}

This raises each sandbox's memory limit while retaining the default requests, CPU limit and storage limit. quota instead bounds the whole application namespace, including claimed and warm sandboxes.

workspace_size and tmp_size cap the /workspace and /tmp emptyDir volumes. They remain ephemeral; these settings allocate no persistent disks. Increasing a volume cap does not raise the pod's overall ephemeral-storage limit, set by its single runtime container's resources.limits["ephemeral-storage"]. Allow space for both volumes and runtime logs when changing that limit. Sizing changes affect new sandboxes and replacement warm pods; already claimed sandboxes keep their current pod.

Choose the toolchain

For a different toolchain, build an image on top of the runtime and give it its own application:

applications = {
  data = {
    image      = var.python_runtime_image
    warm_pool  = 1
    allow_from = [var.caller_cidr]
    consumers  = { server = var.caller_service_account_email }
  }
}

Configure that application's provider with namespace pai-sandbox-data. Each application has exactly one template, so a caller that needs two toolchains uses two applications and two namespaces.

PAI owns the container, daemon ports and arguments, the /workspace root, gVisor, non-root execution, read-only root filesystem, dropped capabilities and absence of a sandbox service-account token. Every template's NetworkPolicy denies all egress, DNS included, and admits only allow_from on the daemon ports. There is no open-network mode. Cluster policy must not broaden these defaults: NetworkPolicies are additive.

Release the image from your own deploy

By default Terraform rolls out every change to an application's image. When the application's own deploy should release its image instead, set ignore_image_changes. Terraform then uses image only when it creates the template and keeps the live image on every later apply, so changing image affects only templates created afterwards:

applications = {
  app = {
    image                = var.starting_image # used only to create the template
    ignore_image_changes = true
  }
}

The deploy replaces only the runtime container's image with a digest-pinned reference, using a JSON patch that first checks the container name:

kubectl -n pai-sandbox-app patch sandboxtemplate pai-node-runtime \
  --type=json --field-manager=app-release -p '[
    {"op":"test","path":"/spec/podTemplate/spec/containers/0/name","value":"runtime"},
    {"op":"replace","path":"/spec/podTemplate/spec/containers/0/image","value":"my-registry/my-app-runtime@sha256:..."}]'

Grant the deploy's identity get and patch on that template; the consumer Role covers claims only. Kubernetes RBAC cannot limit a patch to one field, so that identity can change any part of the template: keep it to a trusted deploy, or add an admission policy that lets it change only the runtime image. A recreated template, for example after its namespace is deleted, starts from image again until the next release.

A Terraform apply that overlaps a release can fail with a field manager conflict naming the deploy's manager; plan and apply again. Opting in an existing application moves its template to a new resource address: add a moved block from module.sandbox.kubernetes_manifest.template["app"] to module.sandbox.kubernetes_manifest.app_released_template["app"] in the same change. Without it, Terraform deletes the template and creates it again under the new address, and that apply can fail partway.

Upgrades and acceptance

Outputs are application_namespaces and sandbox_operator_role. Each application needs a distinct namespace, because the provider names a bound claim from its binding key alone and claim names are namespace-scoped. Independent deployments need an explicit isolation decision before sharing one namespace.

A template update reaches newly claimed sandboxes; running sandboxes keep their pod. Destroying this module removes its namespaces and ephemeral sandboxes; destroy runtime resources before deleting the external cluster so its API remains reachable.

An installation from a release that used Agent Sandbox v1alpha1 and a Helm chart cannot upgrade in place. With the release you run today, destroy the runtime while the cluster still serves v1alpha1. Then move the cluster to v1beta1, rebuild the runtime image from the new release, and apply the new module using the existing state backend after the old runtime resources have been destroyed. Preserve unrelated resources in a shared root's state. The package's terraform/README.md migration section lists every step, maps each old input to its replacement, and covers the removed dashboard input.

Offline validation checks the template's isolation literals in mocked Terraform plans, the runtime contract and package contents. Local setup requires kind v0.24.0 or later with its default node image and networking (tested with v0.33.0), which enforces the fixture's closed NetworkPolicy. Recreate older or custom clusters unless their policy enforcement has been verified. The fixture has no gVisor. Test provider probe/create/command/file operations, a denied caller and denied egress from the actual application identity/network before accepting a GKE deployment. A successful plan or probe alone cannot prove those paths.