Skip to content
Learn Agent Substrate

Learn Agent Substrate

How agent-substrate works - visualized end to end.
Start with the topology Resume-actor walkthrough View on GitHub

What is Substrate?

Agent Substrate is a Kubernetes-native runtime for workloads that spend most of their time idle (AI agents, sandboxed environments, stateful services, etc). We run a lot of these, and giving each one its own pod means paying for capacity that sits there waiting. Substrate decouples an actor’s lifecycle from the pod that runs it, so a small pool of pre-started gVisor or micro-VM workers can host far more actors than there are pods. It does that by packing many actors onto each worker, suspending the idle ones to snapshots, and restoring them when a request shows up.

Each actor lives in an atespace (Substrate’s tenancy boundary), so its identity is the pair (atespace, name). You’ll see that pair everywhere in this atlas, e.g. in the ate-target-actor routing header, in storage keys, in credentials, and in every API call. The diagram below puts the whole system on one whiteboard.

Substrate topology - whiteboard view A client sends a request to atenet-router naming the actor in the ate-target-actor header. The router asks ate-api-server, backed by PostgreSQL, to resume the actor, then tunnels the request over mTLS CONNECT to atunnel in the worker pod. atelet drives the ateworker and moves snapshots to GCS or S3 through its objectstore-plugin. Outbound traffic leaves through atunnel and the atenet-egress gateway. CONTROL PLANE DATA PLANE · per worker node SNAPSHOT STORAGE EGRESS Client HTTP request ate-target-actor header atenet-router AgentGateway (preferred) or Envoy (install default) parks requests during resume ate-api-server gRPC + workflows placement · golden snapshots PostgreSQL atespaces · templates actors · workers · tags leases ate-controller WorkerPool → worker pods worker syncer · SandboxConfig atelet DaemonSet drives the worker Worker pod · gVisor pool of N pods ateworker drives runsc atunnel in / out actors × N one sandbox each GCS / S3 actor snapshots zstd-compressed atenet-egress policy · credentials AgentGateway or Envoy External APIs what the actor dials ResumeActor (AgentGateway caches it) mTLS CONNECT → atunnel Run · Checkpoint · Restore via objectstore-plugin worker-for-actor mTLS CONNECT

The whole system on one whiteboard. Click any box to drill in. For the wiring-diagram version with every port, see System topology.

Where to go next

System topology

The static architecture, with every binary, port, and connection. Open →

Resume actor

The most interesting flow, since it touches every component. Restoring an actor from a snapshot, end to end. Open →

ateapi internals

Control plane API: RPCs, workflows, PostgreSQL leases, scheduling. Open →

Concepts glossary

Quick definitions for the domain terms, e.g. actor, atespace, worker, actor identity, snapshot, WorkerPool, ActorTemplate. Open Actor →

How to read this atlas

Most pages are built around Mermaid diagrams, with sequence diagrams for flows, state diagrams for lifecycles, and flowcharts for topology. The boxes are clickable in the hero diagram above and in the Mermaid diagrams on the deeper pages, so try clicking ate-api-server above to drill into its internals.

We stick to the components, resources, and flows here and leave the line-by-line code to the source. If you want the current implementation, browse the repository and the docs listed below.