Worker lifecycle

A worker in Substrate is a pre-warmed sandbox pod - a member of a WorkerPool Deployment, sitting around with the ateom runtime (ateom-gvisor or ateom-microvm) running, waiting for an actor to be restored into it.

Workers are simpler than actors: just two states (plus “gone”).

States

Idle vs. assigned is expressed by a single Assignment message on the worker record: no assignment = idle; an assignment = hosting one actor.

stateDiagram-v2
  [*] --> Idle: pod gets a PodIP<br/>(create worker record)
  Idle --> Assigned: ResumeActor:<br/>assign worker
  Assigned --> Idle: SuspendActor:<br/>Finalize → SUSPENDED
  Assigned --> [*]: pod deleted<br/>(syncer cleanup)
  Idle --> [*]: pod deleted<br/>(syncer cleanup)

  note right of Idle
    assignment == nil
    Eligible for any actor whose
    sandbox class + selectors
    this worker satisfies.
  end note

  note right of Assigned
    assignment = { actor_template, actor }
    Hosting one actor,
    scoped by (atespace, name).
    Released back to Idle on suspend.
  end note

Worker record in Redis

A worker is just a JSON proto at key worker:<ns>:<pool>:<pod> (unchanged):

Field	Idle worker	Assigned worker
worker_namespace, worker_pool, worker_pod	always set	always set
ip, worker_pod_uid, version	always set	always set
node_name	always set	always set
sandbox_class	cached from the pool (gvisor | microvm)	same
labels{}	cached from the pool’s labels	same
assignment	unset (idle)	{ actor_template, actor }

The old flat actor_id / actor_namespace / actor_template fields are gone. assignment.actor_template is a KubeNamespacedObjectRef{namespace, name} and assignment.actor is an ObjectRef{atespace, name} - so the assignment is scoped by the actor’s (atespace, name) identity, not a bare id. The cached sandbox_class and labels are what the scheduler matches eligibility against without re-reading the WorkerPool.

The worker-pool syncer keeps this fresh

flowchart LR
  K[K8s pod informer] -- Add/Update<br/>(PodIP assigned) --> C[Create worker record<br/>assignment=nil<br/>+ cache node_name,<br/>sandbox_class, labels]
  K -- pod gone<br/>(delete/soft-delete) --> R[dead-worker recovery<br/>if assignment set:<br/>actor → SUSPENDED]
  R --> D[Delete worker record]
  C -.-> RED[(Redis)]
  D -.-> RED
  R -.-> RED

On Add/Update the syncer reads the pod’s WorkerPool to cache the pool’s sandboxClass and labels onto the worker record (and refreshes them if they change). The release-on-death handler runs before the worker record is deleted, and only acts when the worker has an assignment (and the actor still points back at this pod), so the actor is restored to SUSPENDED while the worker record is still readable. A CRASHED actor is left CRASHED, not forced back to SUSPENDED. Eligibility to create a worker is keyed on the pod having a PodIP, not on the Ready condition.

How workers get assigned

When ResumeActor runs, the assign-worker step reads the in-memory worker cache, filters to eligible idle workers, and does a randomized shuffle to pick one. If none are available it retries a few times with exponential backoff (10ms initial). After that it fails with a “no free workers” precondition error, which the router maps to a 503.

There is no WorkerPoolRef anymore - eligibility is selector-based. A worker is eligible for an actor when all of these hold:

1. Sandbox class matches - the worker’s cached sandbox_class equals the template’s sandboxClass (snapshots are not portable across classes).
2. Template gate - the worker’s labels satisfy the template’s workerSelector.
3. Actor selector - the worker’s labels also satisfy the actor’s own worker_selector.

Both selectors are AND’d. If the actor’s latest_snapshot_info is a local snapshot pinned to specific node VMs, the candidate set is further restricted to workers on one of those nodes.

This is the bypass-the-K8s-scheduler trick: pod allocation is just a filter plus a shuffle over the cached worker set, not a scheduler decision.

How workers get released

When SuspendActor finishes, the finalize step clears the worker’s assignment (sets it to nil) and writes it back to Redis, and clears the actor’s worker_pool_name and ateom_pod_namespace/ateom_pod_name/ateom_pod_ip pointers. The worker is immediately idle again and eligible to host a different actor.

What happens when a worker pod dies

The syncer’s pod-delete handler fires on pod removal. If the worker had an assignment, the syncer forces the actor back to SUSPENDED without producing a snapshot (the actor’s existing latest_snapshot_info, if any, is still valid) and clears its pod pointers and worker_pool_name. A CRASHED actor keeps its status. Then the worker record is deleted.

This is a recovery path, not a planned transition. It is best-effort: the optimistic version check means a concurrent SuspendActor/ResumeActor can win and this attempt is dropped.

Known race

If a worker pod dies and a concurrent SuspendActor is in flight, the release-on-death and the suspend workflow can both touch the actor. Today this is best-effort - the worse outcome is a benign double-write, not data loss.

Pre-warming: where workers come from

The WorkerPool CRD owns a Deployment of N replicas. atecontroller reconciles changes to that CRD into Deployment spec changes. Pods come up, start the pool’s ateom runtime (ateom-gvisor or ateom-microvm), get a PodIP, and the syncer registers them as Idle workers - caching the pool’s sandboxClass and labels onto each record.

flowchart LR
  WP["WorkerPool CRD<br/>(replicas: 10)"] --> CTR[atecontroller]
  CTR --> DEP[K8s Deployment<br/>10 worker pods]
  DEP --> P1[Pod 1: ateom runtime]
  DEP --> P2[Pod 2: ateom runtime]
  DEP --> PN[...]
  P1 -- has PodIP --> SYNC[worker-pool syncer]
  P2 -- has PodIP --> SYNC
  PN -- has PodIP --> SYNC
  SYNC -- create worker record --> RED[(Redis: Idle worker)]

Worker pools are isolation boundaries

A WorkerPool is no longer named directly by an ActorTemplate. Instead an actor lands on a pool whose sandboxClass matches the template’s and whose labels satisfy both the template’s workerSelector and the actor’s own worker_selector. This selector-based eligibility is the unit at which:

• You separate sandbox runtimes: a gvisor pool and a microvm pool never share workers (snapshots are not portable across classes).
• You have different worker base images / node placement per pool.
• You scale capacity independently per workload class, and target templates at a pool with labels.

Related

• Actor lifecycle - the dual state machine on the actor side.
• Resume actor · Suspend actor
• WorkerPool · Workers