Skip to content
Learn Agent Substrate

Golden snapshot bootstrap

An ActorTemplate says “this is what an actor of this kind looks like” - container image(s), entrypoint, env vars, sandbox class, and a workerSelector. By itself that’s just a manifest. (The template no longer names a worker pool: placement is selector-based, so an ActorTemplate and a WorkerPool are decoupled - see WorkerPool.)

The interesting part is how Substrate produces the golden snapshot that later lets actors of this template come up in milliseconds. It does not build the snapshot statically from the image. Instead:

How a golden snapshot gets created atecontroller drives the ActorTemplate through four phases: it creates a throwaway 'golden' actor, resumes it (cold-boot on a real worker pod), waits 20 seconds while the workload actually runs, then suspends it so runsc checkpoints the sandbox and atelet uploads the snapshot files to GCS. Future actors of this template restore from that GCS snapshot. How a golden snapshot gets made It's not built. It's run: a throwaway actor really boots on a real worker, then its memory is captured. ① ActorTemplate applied image, env, entrypoint, target WorkerPool - just a Kubernetes manifest at this point Status.Phase = Initial ② atecontroller creates a throwaway "golden" actor No special API - the reconciler calls the same CreateActor + ResumeActor RPCs a client would call. Phase = ResumeGoldenActor ③ REAL worker pod from the WorkerPool - same code path as any actor gVisor sandbox runsc create + runsc start actor workload - boots from scratch from the actor's OCI image runs for ≈ 20 seconds consuming real CPU + real RAM then: SuspendActor → ateom-gvisor exec's runsc checkpoint sandbox memory + state freeze to disk Phase = WaitGoldenActor → Ready ④ GCS bucket snapshot of the run checkpoint.img.zstd pages.img.zstd pages_meta.img.zstd uploaded by atelet from the worker node ⑤ ActorTemplate .Status .GoldenSnapshot = URI .Phase = Ready upload stamp URI AFTER READY - what every later actor of this template gets to skip No more Phase ①–④. The GCS snapshot is the new starting line. client calls CreateActor + ResumeActor ateapi picks restore strategy: GoldenSnapshot → pull from GCS runsc restore -background on a fresh worker pod live in milliseconds no cold boot, no 20s wait

The bootstrap really does occupy a worker pod for the warmup window - that pod is the one a regular actor would have gotten. After Ready, every later actor of this template skips ①–④ entirely and restores from the external snapshot.

Phase machine

The reconciler drives the template through four phases, tracked in Status.Phase. PhaseInitial is the empty-string zero value, so a freshly applied template starts there.

stateDiagram-v2
  [*] --> PhaseInitial
  PhaseInitial --> PhaseResumeGoldenActor: CreateAtespace(ate-golden) +<br/>CreateActor()<br/>(throwaway "golden" actor)
  PhaseResumeGoldenActor --> PhaseWaitGoldenActor: ResumeActor()<br/>(cold boot, stamp<br/>TakeGoldenSnapshotAt = now + warmup)
  PhaseWaitGoldenActor --> PhaseReady: requeue fires,<br/>SuspendActor()
  PhaseReady --> [*]: template usable

PhaseFailed is declared in the CRD type but the reconciler never assigns it - errors return up and trigger a normal requeue.

Sequence

sequenceDiagram
  autonumber
  participant K as ActorTemplate CRD
  participant CT as atecontroller
  participant A as ateapi
  participant W as Worker (golden actor)
  participant G as GCS / S3

  K-->>CT: ActorTemplate created (PhaseInitial)

  rect rgb(235,245,235)
    Note over CT: PhaseInitial → PhaseResumeGoldenActor
    CT->>A: CreateAtespace(ate-golden)<br/>(idempotent; ignores AlreadyExists)
    CT->>A: CreateActor(golden actor into ate-golden, template=this)
  end

  rect rgb(235,245,235)
    Note over CT: PhaseResumeGoldenActor → PhaseWaitGoldenActor
    CT->>A: ResumeActor(golden)
    A->>W: boot from scratch (no snapshot, no GoldenSnapshot yet)
    A-->>CT: RUNNING (ResumeActor waits on readyz)
    CT->>CT: stamp Status.TakeGoldenSnapshotAt = now + warmup<br/>(warmup = 0 if every container has a readyz probe)
  end

  rect rgb(245,245,235)
    Note over CT: PhaseWaitGoldenActor
    CT->>CT: if now < TakeGoldenSnapshotAt:<br/>return ctrl.Result{RequeueAfter: remaining}<br/>(controller-runtime requeues - not a blocking sleep)
  end

  rect rgb(235,235,245)
    Note over CT: PhaseWaitGoldenActor → PhaseReady
    CT->>A: SuspendActor(golden)
    A->>W: checkpoint
    W->>G: upload snapshot files (parallel)
    A-->>CT: SUSPENDED, latest_snapshot_info.external.snapshot_uri_prefix
    CT->>K: Status.GoldenSnapshot = URI prefix<br/>Status.Phase = PhaseReady<br/>Ready condition = True
  end

What “Ready” means

Once ActorTemplate.Status.GoldenSnapshot is set:

  • Any subsequent CreateActor + ResumeActor for an actor of this template will be restored from the golden snapshot - unless the actor has its own newer snapshot. The resume workflow picks the restore strategy in this order: the actor’s own latest_snapshot_info (local or external) if set, else the template’s GoldenSnapshot (when ResumeActor was not called with boot=true), else a cold boot from the template spec.
  • The result: new actors of this template come up in lazy-page-restore time, not cold-boot time.

The warmup wait - why?

The reconciler stamps Status.TakeGoldenSnapshotAt = now + warmup after resuming and defers the suspend until that time. It’s not a blocking sleep: PhaseWaitGoldenActor returns ctrl.Result{RequeueAfter: remaining}, so controller-runtime re-fires reconcile later.

The warmup is now a fallback, not a fixed 20 seconds for everyone:

  • If every container in the template declares a readyz HTTP probe, the warmup is 0. ResumeActor already blocks until each container’s readyz reports 200 (both runtimes wait for every readiness probe before Run / Restore returns), so the workload is already initialized by the time the controller gets RUNNING back - no extra wait is needed.
  • Otherwise (any container lacks a probe, or the template declares no containers), it falls back to a default 20 s warmup as a coarse “give the workload time to initialize” budget.

If you rely on the fallback and your workload needs longer (e.g. downloading a model), the resulting golden snapshot will be of a partially-initialized process - and that’s what every actor of this template will restore from. Declaring a readyz probe (or baking the heavy initialization into your container image) is the way to make this deterministic.

What about the workload running during the wait?

Yes - the golden actor is occupying a real worker pod, eating real resources, for the whole warmup window (see the callout at the top of the page). The reconcile goroutine is not blocked during the wait; the work-queue holds the item until RequeueAfter elapses, so atecontroller is free to handle other reconciles in the meantime.

If an eligible WorkerPool is sized tightly, this matters: bootstrap consumes a slot. Operationally you’ll want to either provision an extra worker for the bootstrap window, or accept that during warmup one fewer worker is available for traffic. (Which pool the golden actor lands on is selector-based, like any other actor of the template.)

After Ready

The golden actor record stays in Redis (in the ate-golden atespace) with status=SUSPENDED. Once a template reaches PhaseReady the reconciler is a no-op for it: the ActorTemplate spec is immutable after creation, so there is no in-place “refresh” - you create a new template (with a new image) to get a new golden snapshot.

Failure paths

The reconciler does not transition to PhaseFailed on errors - it simply returns the error up to controller-runtime, which requeues with backoff. The phase stays at whatever it was (PhaseInitial / PhaseResumeGoldenActor / PhaseWaitGoldenActor), and the next reconcile retries the same step.

  • CreateAtespace / CreateActor fails → stays in PhaseInitial, requeued (AlreadyExists on the atespace is ignored, not an error).
  • ResumeActor fails (no eligible workers, image pull error) → stays in PhaseResumeGoldenActor, requeued.
  • SuspendActor fails (checkpoint or upload error), or the suspended actor comes back without an external snapshot → stays in PhaseWaitGoldenActor, requeued.

The golden actor record may be left dangling on a worker if a reconcile errored mid-flow - cleanup is the operator’s responsibility for now.