System topology

Substrate is a control plane / data plane split that runs on Kubernetes but deliberately routes around the K8s scheduler for the hot path. Actor assignment to pre-warmed worker pods happens in ateapi in milliseconds, not in kube-scheduler in seconds.

In three pictures

The whole-picture hero on the Overview is the elevator pitch. This page zooms in on the three regions that matter most - what’s in each plane, and how a request actually flows.

1 · Control plane

flowchart LR
  KAPI[(K8s API)]
  ATECTRL["atecontroller<br/>(Deployment)<br/>CRD reconciler"]
  ATEAPI["ateapi<br/>(Deployment)<br/>gRPC :443"]
  REDIS[("Redis / Valkey<br/>actors · workers · locks · atespaces")]

  ATECTRL -- "reconciles<br/>ActorTemplate · WorkerPool" --> KAPI
  ATEAPI -- "watches worker pods" --> KAPI
  ATEAPI <--> REDIS

  click ATEAPI "/components/ateapi/" "ateapi"
  click ATECTRL "/components/atecontroller/" "atecontroller"
  click REDIS "/components/storage/" "Storage"

Stateless workloads, declarative inputs. atecontroller owns the CRDs; ateapi is the operational brain backed by Redis.

2 · Data plane (one worker node)

flowchart LR
  ATELET["atelet<br/>(DaemonSet)<br/>gRPC :8085"]

  subgraph WPOD["Worker pod (gVisor pool shown)"]
    direction TB
    ATEOM["ateom-gvisor<br/>gRPC over Unix socket"]
    RUNSC[["runsc<br/>(gVisor runtime)"]]
    WL[["Actor workload"]]
    ATEOM -- exec --> RUNSC
    RUNSC -- manages --> WL
  end

  GCS[("GCS / S3<br/>snapshots (zstd)")]

  ATELET -- "Run · Checkpoint · Restore" --> ATEOM
  ATELET <-- "upload / download" --> GCS

  click ATELET "/components/atelet/" "atelet"
  click ATEOM "/components/ateom-gvisor/" "ateom-gvisor"
  click WPOD "/components/workers/" "Workers"
  click GCS "/components/storage/" "Storage"

Every node runs one atelet DaemonSet pod and many worker pods. atelet is the single component on the node that touches object storage.

There are two sandbox runtimes, chosen per-pool by a SandboxClass: gvisor (shown above - ateom-gvisor shelling out to runsc) and microvm (ateom-microvm, a kata + cloud-hypervisor micro-VM needing /dev/kvm).

A WorkerPool is single-class: its spec.sandboxClass is one value, so every worker pod in the pool runs that one runtime. And a gVisor checkpoint (a runsc image) is a completely different format from a micro-VM snapshot (a cloud-hypervisor memory image) - you can’t restore one on the other. So the scheduler treats sandbox class as a hard gate: an actor whose ActorTemplate is gvisor is only ever placed on gvisor workers, and its snapshots only move between workers of that same class. An actor is effectively pinned to one runtime family for life. (See ActorTemplate and SandboxConfig for how class is declared and resolved.)

Whichever runtime a pool uses, atelet drives it over the same Run · Checkpoint · Restore gRPC on a Unix socket. See ateom-gvisor; the micro-VM sibling ateom-microvm gets its own deep-dive in the Components section.

3 · Request flow

flowchart LR
  Client([External client])
  DNS["atenet DNS<br/>(CoreDNS Corefile mgr)"]
  ROUTER["atenet router<br/>L7 proxy :8080/:8443<br/>ExtProc :50051"]
  ATEAPI["ateapi"]
  WORKLOAD[["Worker workload"]]

  Client -- "1. resolve name.atespace.actors…ate.dev" --> DNS
  DNS -- "2. router ClusterIP" --> Client
  Client -- "3. HTTP request" --> ROUTER
  ROUTER -- "4. ResumeActor (per request)" --> ATEAPI
  ATEAPI -- "5. Actor{pod IP}" --> ROUTER
  ROUTER -- "6. rewritten :authority → worker pod" --> WORKLOAD

  click DNS "/components/atenet/" "atenet DNS"
  click ROUTER "/components/atenet/" "atenet router"
  click ATEAPI "/components/ateapi/" "ateapi"
  click WORKLOAD "/components/workers/" "Workers"

The thing that lets Substrate scale: routing is resolved at the request moment, not pre-baked. No worker-IP DNS records, no sidecars, no scheduler involvement. For the full sequence (including the cold-restore case), see Resume actor.

What each component is for

Component	Kind	Role
ateapi	Deployment	Source of truth for actor & worker state. Serves Control and SessionIdentity gRPC services. Owns the suspend/resume workflows.
atecontroller	Deployment	Watches the K8s API. Reconciles WorkerPool (→ Deployments of worker pods) and ActorTemplate (→ golden snapshot bootstrap).
atenet	Deployments (one binary, two subcommands)	atenet router runs an L7 proxy + ExtProc and does the per-request dance to find the right worker. atenet dns programs CoreDNS so <actor_name>.<atespace>.actors.resources.substrate.ate.dev resolves. Each subcommand is typically deployed as its own Deployment.
atelet	DaemonSet	Lives on each worker node. Receives gRPC calls from ateapi. Pulls images, manages OCI bundles, talks to ateom over a Unix socket, ships snapshots to GCS/S3.
ateom	In-pod helper	Lives inside every worker pod. Tiny gRPC server on a Unix socket that drives the sandbox runtime. Two implementations: ateom-gvisor (shells out to runsc) and ateom-microvm (kata + cloud-hypervisor).
Worker pods	Deployment-spawned (from WorkerPool)	Pre-warmed gVisor or micro-VM sandboxes (one class per pool). Each can host one actor at a time; the sandbox (with its pause process) is created on assignment, not at pod start.
Redis / Valkey	StatefulSet	Holds actor records, worker records, and the per-actor distributed locks that serialize workflows.
GCS / S3	External	Holds the checkpoint images (RAM + sentry state + lazy-load pages), zstd-compressed.
podcertcontroller	Deployment	Polyfill for the upstream K8s Pod Certificate signers feature. Not on the hot path.

Ports at a glance

Service	Port	Protocol
ateapi	443	gRPC (TLS)
atelet	8085	gRPC
atenet router (L7 proxy)	8080 / 8443	HTTP / HTTPS
atenet ExtProc	50051	gRPC (ext_proc)
atenet xDS	18000	gRPC (xDS control plane)
ateom-gvisor	Unix socket	gRPC - /var/lib/ateom-gvisor/ateoms/{podUID}/ateom.sock

How traffic actually gets to an actor

The short version: there is no per-pod sidecar. atenet’s centralized L7 proxy is the single dataplane proxy, and ExtProc consults ateapi on each request to discover (and if needed, resume) the actor’s worker.

See the request path flow for the full sequence.

The hot path bypasses kube-scheduler

The big architectural bet: when an actor needs to wake up, ateapi picks a pre-warmed worker out of Redis and tells atelet to restore into it. The Kubernetes scheduler is never on the critical path. That’s how Substrate gets sub-second resume times against 30× more actors than pods.

Go deeper

Each component box above links to its own page. For the current implementation, browse the source and the authoritative docs/ (architecture, glossary, API guide).