Skip to content
Learn Agent Substrate

Golden snapshot bootstrap

An ActorTemplate says “this is what an actor of this kind looks like” - image, entrypoint, env vars, worker pool. By itself that’s just a manifest.

The interesting part is how Substrate produces the golden snapshot that later lets actors of this template come up in milliseconds. It does not build the snapshot statically from the image. Instead:

How a golden snapshot gets created atecontroller drives the ActorTemplate through four phases: it creates a throwaway 'golden' actor, resumes it (cold-boot on a real worker pod), waits 20 seconds while the workload actually runs, then suspends it so runsc checkpoints the sandbox and atelet uploads the snapshot files to GCS. Future actors of this template restore from that GCS snapshot. How a golden snapshot gets made It's not built. It's run: a throwaway actor really boots on a real worker, then its memory is captured. ① ActorTemplate applied image, env, entrypoint, target WorkerPool - just a Kubernetes manifest at this point Status.Phase = Initial ② atecontroller creates a throwaway "golden" actor No special API - the reconciler calls the same CreateActor + ResumeActor RPCs a client would call. Phase = ResumeGoldenActor ③ REAL worker pod from the WorkerPool - same code path as any actor gVisor sandbox runsc create + runsc start actor workload - boots from scratch from the actor's OCI image runs for ≈ 20 seconds consuming real CPU + real RAM then: SuspendActor → ateom-gvisor exec's runsc checkpoint sandbox memory + state freeze to disk Phase = WaitGoldenActor → Ready ④ GCS bucket snapshot of the run checkpoint.img.zstd pages.img.zstd pages_meta.img.zstd uploaded by atelet from the worker node ⑤ ActorTemplate .Status .GoldenSnapshot = URI .Phase = Ready upload stamp URI AFTER READY - what every later actor of this template gets to skip No more Phase ①–④. The GCS snapshot is the new starting line. client calls CreateActor + ResumeActor ateapi picks restore strategy: GoldenSnapshot → pull from GCS runsc restore -background on a fresh worker pod live in milliseconds no cold boot, no 20s wait

The bootstrap really does occupy a worker pod for ≈ 20 seconds - that pod is the one a regular actor would have gotten. After Ready, every later actor of this template skips ①–④ entirely and restores from the GCS snapshot.

Phase machine

stateDiagram-v2
  [*] --> Initial
  Initial --> ResumeGoldenActor: CreateActor()<br/>(throwaway "golden" actor)
  ResumeGoldenActor --> WaitGoldenActor: ResumeActor()<br/>(cold boot, stamp TakeGoldenSnapshotAt = now + 20s)
  WaitGoldenActor --> Ready: requeue fires,<br/>SuspendActor()
  Ready --> [*]: template usable

PhaseFailed is declared in the CRD type but the reconciler never assigns it - errors return up and trigger a normal requeue.

internal/controllers/actortemplate_controller.go:70-155

Sequence

sequenceDiagram
  autonumber
  participant K as ActorTemplate CRD
  participant CT as atecontroller
  participant A as ateapi
  participant W as Worker (golden actor)
  participant G as GCS / S3

  K-->>CT: ActorTemplate created (Initial)

  rect rgb(235,245,235)
    Note over CT: Phase: ResumeGoldenActor
    CT->>A: CreateActor(golden, template=this)
    CT->>A: ResumeActor(golden)
    A->>W: boot from scratch (no LastSnapshot, no GoldenSnapshot yet)
    A-->>CT: status=RUNNING
  end

  rect rgb(245,245,235)
    Note over CT: Phase: WaitGoldenActor
    CT->>CT: stamp Status.TakeGoldenSnapshotAt = now + 20s<br/>return ctrl.Result{RequeueAfter: 20s}<br/>(controller-runtime requeues - not a blocking sleep)
  end

  rect rgb(235,235,245)
    Note over CT: Phase: Ready
    CT->>A: SuspendActor(golden)
    A->>W: checkpoint
    W->>G: upload checkpoint.img + pages.img
    A-->>CT: status=SUSPENDED, LastSnapshot=URI
    CT->>K: Status.GoldenSnapshot = URI<br/>Status.Phase = Ready
  end

What “Ready” means

Once ActorTemplate.Status.GoldenSnapshot is set:

  • Any subsequent CreateActor + ResumeActor for an actor of this template will be restored from the golden snapshot (priority 2 in the resume workflow’s strategy selection, see workflow_resume.go:230-248).
  • The result: new actors of this template come up in lazy-page-restore time, not cold-boot time.

The 20-second wait - why?

The reconciler waits 20 seconds between resume and suspend to give the workload time to initialize. It’s not a blocking sleep: the reconciler stamps Status.TakeGoldenSnapshotAt = now + 20s and returns ctrl.Result{RequeueAfter: 20s}, so controller-runtime re-fires reconcile later. This is intentionally crude: the controller has no protocol for “ready” callbacks from arbitrary user code, so it just budgets a fixed window.

If your workload needs longer (e.g. downloading a model), the resulting golden snapshot will be of a partially-initialized process - and that’s what every actor of this template will restore from. You’d typically build the heavy initialization into your container image instead.

internal/controllers/actortemplate_controller.go:112

What about the workload running during the wait?

Yes - the golden actor is occupying a real worker pod, eating real resources, for the whole 20-second window (see the callout at the top of the page). The reconcile goroutine is not blocked during the wait; the work-queue holds the item until RequeueAfter elapses, so atecontroller is free to handle other reconciles in the meantime.

If your WorkerPool is sized tightly, this matters: bootstrap consumes a slot. Operationally you’ll want to either provision an extra worker for the bootstrap window, or accept that the first 20 seconds after applying a new template will see one fewer worker available for traffic.

After Ready

The golden actor record stays in Redis but with status=SUSPENDED. It can be re-resumed and re-suspended to refresh the snapshot (e.g. if the template’s image changes). Today this happens via re-reconcile when the spec changes.

Failure paths

The reconciler does not transition to PhaseFailed on errors - it simply returns the error up to controller-runtime, which requeues with backoff. The phase stays at whatever it was (Initial / ResumeGoldenActor / WaitGoldenActor), and the next reconcile retries the same step.

  • CreateActor fails → stays in Initial, requeued.
  • ResumeActor fails (no workers, image pull error) → stays in ResumeGoldenActor, requeued.
  • SuspendActor fails (checkpoint or upload error) → stays in WaitGoldenActor, requeued.

The golden actor record may be left dangling on a worker if a reconcile errored mid-flow - cleanup is the operator’s responsibility for now.