Resume actor (end-to-end)

This is the flow to understand if you want to understand Substrate. It touches DNS, the L7 proxy, ExtProc, ateapi’s workflow engine, the Redis store, atelet, GCS, ateom-gvisor, and runsc restore. Six components and two data stores collaborate on a single HTTP request.

The setup

• The actor was created earlier (status SUSPENDED, with a LastSnapshot pointing at a checkpoint in GCS).
• A pool of worker pods is pre-warmed and idle (actor_id == "").
• A client wants to hit the actor.

The full sequence (cold path)

sequenceDiagram
  autonumber
  participant C as Client
  participant DNS as CoreDNS<br/>(atenet DNS)
  participant E as L7 proxy<br/>(atenet router)
  participant X as ExtProc<br/>(atenet router)
  participant A as ateapi
  participant R as Redis
  participant L as atelet DaemonSet<br/>(on assigned worker's node)
  participant G as GCS / S3
  participant O as ateom-gvisor<br/>(worker pod)
  participant W as Worker workload

  C->>DNS: A? actorId.actors.resources.substrate.ate.dev
  DNS-->>C: atenet-router ClusterIP
  C->>E: HTTP request, :authority=actorId.actors.resources.substrate.ate.dev

  E->>X: ext_proc: ProcessRequest (headers)
  Note over X: handleRequestHeaders()<br/>extracts actorId

  X->>A: ResumeActor(actorId)<br/>(via singleflight dedupe)

  rect rgb(240,240,255)
    Note over A,R: Workflow engine runs under<br/>lock:actor:{actorId} (30s TTL, 28s workflow timeout)

    A->>R: LoadActorForResume → fetch actor + template
    A->>R: AssignWorker: pick random idle worker
    A->>R: UPDATE worker (actor_id=X), UPDATE actor (RESUMING + ateom pod ref)

    Note over A,L: Dial atelet on the *node* hosting<br/>the assigned worker pod (atelet pod IP : 8085)
    A->>L: Restore(snapshotURI, ateom pod, runsc cfg)
    L->>G: Download checkpoint.img + pages.img + pages_meta.img (zstd, parallel)
    G-->>L: bytes
    L->>O: RestoreWorkload (gRPC over Unix socket)
    O->>O: exec runsc restore -background -direct -detach
    O-->>L: ok (returns immediately, pages lazy-load)
    L-->>A: Restore ok

    A->>R: FinalizeRunning: UPDATE actor (RUNNING)
  end

  A-->>X: Actor{ ateom_pod_ip, status=RUNNING }
  X-->>E: HeaderMutation: :authority := pod_ip:80

  E->>W: HTTP request, :authority=pod_ip:80
  W-->>E: HTTP response
  E-->>C: HTTP response

What’s happening in each step

1–3. DNS to the front door

The client resolves actorId.actors.resources.substrate.ate.dev. The atenet DNS controller programs CoreDNS so this name pattern always returns the ClusterIP of the atenet router service - not the worker IP. This is deliberate: the worker IP isn’t known until ExtProc consults ateapi, and it may change between requests if the actor moves between workers.

internal/dns/dns.go:71-88 · internal/dns/corefile.go:42-59

4–5. L7 proxy + ExtProc

The L7 proxy accepts the request on :8080 and applies a single ext_proc filter that streams headers to ExtProc on :50051. ExtProc extracts the actor ID from the :authority header and decides where the request should go.

cmd/atenet/internal/app/router/extproc.go:125-175 (handleRequestHeaders lives in extproc.go; extproc_in.go only holds the requestMetadata / parseActorID helpers.)

6. ResumeActor (with singleflight)

ExtProc calls ateapi.ResumeActor(actorId). To avoid stampedes when 50 concurrent requests hit a cold actor, ExtProc wraps the call in a singleflight.Group so only the first call goes through; the rest piggyback on its result.

cmd/atenet/internal/app/router/resumer.go:29-93

7–9. ateapi runs the resume workflow

ateapi acquires lock:actor:<actorId> in Redis (30s TTL, 28s workflow timeout) and steps through:

1. LoadActorForResume - fetch actor + ActorTemplate.
2. AssignWorker - random shuffle over idle workers in the target WorkerPool; update Redis: worker becomes assigned, actor goes RESUMING.
3. CallAteletRestore - pick which restore strategy to use, then resolve the atelet DaemonSet pod that runs on the same node as the assigned worker (informer indexed by node name) and dial that atelet’s pod IP on :8085. atelet is a per-node DaemonSet, not a per-pod sidecar - one atelet serves every worker pod scheduled to its node.
4. FinalizeRunning - set actor to RUNNING.

cmd/ateapi/internal/controlapi/workflow_resume.go:35-293

The restore strategy priority (lines 213–265):

if actor.LastSnapshot exists                  → restore from it
else if template.GoldenSnapshot exists & !boot → restore from golden snapshot
else                                          → atelet.Run (boot from scratch)

10–14. atelet does the heavy lifting

What ateom-gvisor actually is

A worker pod runs exactly one container - and that container is ateom-gvisor, the small gRPC server in cmd/ateom-gvisor/. ateom-gvisor is not itself sandboxed; it runs as PID 1 in the worker pod with normal container privileges. Its job is to receive RPCs from atelet and exec runsc (the gVisor runtime), which spawns a separate gVisor sandbox. The actor’s workload - pulled from the actor’s OCI image - runs inside that sandbox, not alongside ateom-gvisor. The pod hosts at most one such sandbox at a time. See ateom-gvisor for the RPC surface.

Where the actor’s state actually lands: on the node, not in the pod. atelet’s Restore handler downloads the three snapshot files from GCS in parallel and zstd-decompresses them onto the node’s own filesystem, under /var/lib/ateom-gvisor/actors/{ns}:{name}:{id}/restore-state/. The files are owned by the host - they’re not written into either pod’s container filesystem.

How does the worker pod then see those bytes? Through a hostPath bind mount. Both pods on the node - the atelet DaemonSet pod and every worker pod - mount the same node directory /var/lib/ateom-gvisor into their own filesystem at the same path. A hostPath volume is a slice of the host’s filesystem grafted into the pod’s mount namespace, so when atelet writes a file there, ateom-gvisor (inside the worker pod) reads exactly the same bytes - no copy, no network, same inode.

This is also why the gRPC channel from atelet to ateom-gvisor can be a unix socket: the socket file /var/lib/ateom-gvisor/ateoms/<podUID>/ateom.sock sits in the same shared hostPath, so both ends address it by the same path. atelet opens it and calls RestoreWorkload; ateom-gvisor, listening on the same path inside the worker pod, accepts the call and proceeds with the restore.

Two pods on one node, talking through the node’s own filesystem. The shared hostPath is how atelet “delivers” the snapshot files and where it finds the unix socket to ateom-gvisor. Note the purple gVisor sandbox inside the worker pod - that’s a separate runsc-managed sandbox that ateom-gvisor spawns; the actor’s OCI image runs in there, not in ateom-gvisor itself.

The pieces worth noticing in that picture:

• atelet is per-node, not per-pod. One DaemonSet pod handles every worker pod scheduled to that node. It does not run as a sidecar.
• ateom-gvisor and the actor live at different layers. ateom-gvisor is the pod’s container (PID 1, unsandboxed). The actor runs in a runsc-managed gVisor sandbox that ateom-gvisor spawns - a sibling process tree, not a child of the ateom-gvisor binary. When you read “exec runsc restore” in the code, that’s the moment the sandbox comes into existence.
• The three snapshot files are downloaded in parallel via an errgroup and streamed through zstd on the way to disk (ategcs.FetchLocalFileFromGCSWithZstd).
• The unix socket path is conventional, not negotiated. atelet composes it from the worker pod’s UID (/var/lib/ateom-gvisor/ateoms/<podUID>/ateom.sock) and dials it directly. ateom-gvisor inside the worker pod is listening on the same path because it sees the same mount.
• OCI bundle goes there too. Before calling RestoreWorkload, atelet writes a fresh config.json + rootfs (unpacked from the actor’s image) into the hostPath so runsc restore finds everything in one place.

cmd/atelet/main.go:414-474

15–16. ateom-gvisor runs runsc restore -background -direct

This is where the magic happens. The -background flag enables demand paging.

What “demand paging” means

A normal “eager” restore would have to copy the entire pages.img file into the sandbox’s RAM before the workload could execute a single instruction - for a multi-GB snapshot that’s seconds of wall clock spent just shuffling bytes.

Demand paging flips that: runsc brings the sentry up immediately with pages.img mapped but not yet resident in memory. The very first time the workload reads or writes a given memory page, the CPU takes a page fault; runsc catches the fault, reads just that one page out of pages.img, hands it to the workload, and execution continues. Pages that the workload never touches are never loaded at all.

This is the same trick the Linux kernel uses for mmap’d files and for loading executables - it’s just being applied to the RAM image of a checkpointed process.

So -background returns control as soon as the sentry is up; actual memory pages stream in lazily, on fault, as the workload touches them. -direct skips some gVisor security restrictions for snapshot data. -detach returns immediately.

Without -background, restore would have to map every page of pages.img before the workload can run a single instruction. With it, the sentry is “up” the moment the metadata is loaded; the first HTTP request page-faults its own working set in.

A few things to note about this trick:

• The first request pays a small page-in tax, but only for the pages it actually touches - usually a tiny fraction of resident memory.
• Subsequent requests are warm. Once a page is in, it stays in until the actor suspends again, so steady-state latency is whatever the workload’s normal latency is.
• -detach is what makes the RPC return. Without it, RestoreWorkload would block until the sandboxed process exited. Combined with -background, atelet gets its ok back in milliseconds and ateapi can finalize the actor to RUNNING.

cmd/ateom-gvisor/runsc.go:134-158

This is the trick that makes resume fast: we don’t wait for the entire snapshot to be paged in before serving the request - only what the workload actually needs to handle this request.

17–18. ExtProc rewrites :authority

When ResumeActor returns, the Actor object now carries ateom_pod_ip. ExtProc emits a header-mutation response telling the proxy to overwrite :authority with <pod_ip>:80. The proxy then forwards the request to the worker.

cmd/atenet/internal/app/router/extproc_out.go:36-44

19–22. The workload responds

From the workload’s perspective, this is a plain old HTTP request arriving at its listener. It has no idea it was suspended five seconds ago.

What if there are no idle workers?

ateapi’s AssignWorkerStep retries 5 times with exponential backoff (10ms initial). If no worker is available after that, it returns codes.FailedPrecondition, which ExtProc maps to HTTP 503 ServiceUnavailable.

cmd/ateapi/internal/controlapi/workflow_resume.go:134-141 · cmd/atenet/internal/app/router/errors.go:66-71

What about the warm path?

If the actor is already RUNNING, the workflow is a no-op: ResumeActor returns the current Actor (with current ateom_pod_ip) and ExtProc proceeds straight to step 17. Same code path, just no atelet / GCS / runsc work.

State transitions during this flow

stateDiagram-v2
  [*] --> SUSPENDED: CreateActor
  SUSPENDED --> RESUMING: ResumeActor<br/>(AssignWorker)
  RESUMING --> RUNNING: FinalizeRunning
  RUNNING --> SUSPENDING: SuspendActor
  SUSPENDING --> SUSPENDED: FinalizeSuspended<br/>(worker released)
  SUSPENDED --> [*]: DeleteActor
  note right of RUNNING
    The only state in which
    a worker is assigned.
  end note

Why this design is interesting

• No K8s scheduler on the hot path. Worker pods are pre-warmed; ateapi picks one in microseconds via a Redis lookup.
• No per-pod sidecar. A single L7-proxy fleet handles all routing.
• Lazy-paging restore. The workload is “back” the instant runsc restore -background returns; pages come in on demand.
• Per-actor distributed lock. Two concurrent Resume calls for the same actor don’t race - the second one waits or gets Aborted.

Related

• Suspend actor - the reverse flow (Cut 2).
• Actor lifecycle - full state machine (Cut 2).
• ateapi internals - the workflow engine in detail.