Skip to content
Learn Agent Substrate

Worker lifecycle

A worker in Substrate is a pre-warmed gVisor sandbox pod - a member of a WorkerPool Deployment, sitting around with ateom-gvisor running and a pause container, waiting for an actor to be restored into it.

Workers are simpler than actors: just two states (plus “gone”).

States

stateDiagram-v2
  [*] --> Idle: pod gets a PodIP<br/>(syncer.CreateWorker)
  Idle --> Assigned: ResumeActor:<br/>AssignWorker step
  Assigned --> Idle: SuspendActor:<br/>FinalizeSuspended step
  Assigned --> [*]: pod deleted<br/>(syncer cleanup)
  Idle --> [*]: pod deleted<br/>(syncer cleanup)

  note right of Idle
    actor_id == ""
    Eligible to host any
    actor from its WorkerPool.
  end note

  note right of Assigned
    actor_id == "{some-id}"
    Hosting one actor.
    Released back to Idle on suspend.
  end note

Worker record in Redis

A worker is just a JSON proto at key worker:<ns>:<pool>:<pod>:

FieldIdle workerAssigned worker
worker_namespace, worker_pool, worker_podalways setalways set
ip, worker_pod_uid, versionalways setalways set
actor_id""actor ID
actor_namespace""actor’s namespace
actor_template""template name
cmd/ateapi/internal/store/ateredis/ateredis.go:40-80

The WorkerPoolSyncer keeps this fresh

flowchart LR
  K[K8s pod informer] -- AddFunc<br/>(PodIP assigned) --> C[CreateWorker<br/>actor_id=""]
  K -- DeleteFunc<br/>(pod gone) --> CHK{worker had<br/>actor_id?}
  CHK -- yes --> R[releaseActorOnDeadWorker<br/>actor → SUSPENDED]
  CHK -- no --> D[DeleteWorker]
  R --> D
  C -.-> RED[(Redis)]
  D -.-> RED
  R -.-> RED

releaseActorOnDeadWorker runs before DeleteWorker so the actor is restored to SUSPENDED while the worker record is still readable. Eligibility to create a worker is keyed on pod.Status.PodIP != "", not on the Ready condition.

cmd/ateapi/internal/controlapi/syncer.go:28-192

How workers get assigned

When ResumeActor runs, AssignWorkerStep does a randomized shuffle over idle workers in the target WorkerPool (matching the actor’s ActorTemplate.WorkerPoolRef) and picks one. If none are available it retries up to 5 times with exponential backoff (10ms initial). After that: FailedPrecondition.

This is the bypass-the-K8s-scheduler trick: pod allocation is just a Redis shuffle, not a scheduler decision.

cmd/ateapi/internal/controlapi/workflow_resume.go:78-158

How workers get released

When SuspendActor finishes, FinalizeSuspendedStep zeros out the worker’s actor fields and writes it back to Redis. The worker is immediately eligible to host a different actor.

cmd/ateapi/internal/controlapi/workflow_suspend.go:174-239

What happens when a worker pod dies

The syncer’s DeleteFunc fires on pod removal. If the worker was hosting an actor, the syncer forces the actor back to SUSPENDED without producing a snapshot (the actor’s existing LastSnapshot, if any, is still valid). Then the worker record is deleted.

This is a recovery path, not a planned transition.

Pre-warming: where workers come from

The WorkerPool CRD owns a Deployment of N replicas. atecontroller reconciles changes to that CRD into Deployment spec changes. Pods come up, start ateom-gvisor, get a PodIP, and the syncer registers them as Idle workers.

flowchart LR
  WP["WorkerPool CRD<br/>(replicas: 10)"] --> CTR[atecontroller]
  CTR --> DEP[K8s Deployment<br/>10 worker pods]
  DEP --> P1[Pod 1: ateom-gvisor]
  DEP --> P2[Pod 2: ateom-gvisor]
  DEP --> PN[...]
  P1 -- has PodIP --> SYNC[WorkerPoolSyncer]
  P2 -- has PodIP --> SYNC
  PN -- has PodIP --> SYNC
  SYNC -- CreateWorker --> RED[(Redis: Idle worker)]
internal/controllers/workerpool_controller.go:52-176

Worker pools are isolation boundaries

Actors of a given ActorTemplate can only be assigned to workers from the pool referenced by template.WorkerPoolRef. Different pools never share workers. This is the unit at which:

  • You can have different worker base images per pool.
  • You can scale capacity independently per workload class.