Skip to content

Snapshot

A snapshot is what makes Substrate actors portable. It’s a persisted representation of an actor’s state at the moment it was checkpointed, so the actor can later be restored onto a (potentially different) worker instead of cold-booting. The exact bytes depend on the sandbox runtime, but the actor record refers to a snapshot through one uniform message: SnapshotInfo.

SnapshotInfo: external or local

SnapshotInfo is a oneof data with two variants - a snapshot is either uploaded to object storage or kept on the node:

VariantMessageWhere it livesSet by
ExternalExternalSnapshotInfo{ snapshot_uri_prefix }Object storage (GCS/S3) at a URI prefixSuspendActor
LocalLocalSnapshotInfo{ snapshot_prefix, node_vms_with_local_snapshots[] }On the worker node VM(s) listed in node_vms_with_local_snapshotsPauseActor

An actor’s latest_snapshot_info field carries whichever of these was most recently produced. (There is no longer a SnapshotType enum or a last_snapshot string - those are gone.)

Snapshot scope: what goes in

Independently of where a snapshot lands, the ActorTemplate’s snapshotsConfig decides what it contains, via the SnapshotScope enum:

ScopeContentsOn resume
FullProcess memory plus the filesystem delta on top of the OCI image (which includes any attached DurableDir volumes, since they live inside rootfs)Restores hot from the memory image
DataOnly the contents of snapshot-supporting volumes (currently DurableDir); process memory and the rest of rootfs are discardedCold-boots from the OCI image with DurableDir contents restored

Scope is chosen per trigger in SnapshotsConfig:

  • onPause (default Full) - what a Pause captures and keeps on the node.
  • onCommit (default Full) - what a Suspend captures and uploads.

onCommit must be a subset of onPause (enforced by a CEL validation rule on the CRD): if onPause is Full, onCommit may be Full or Data; if onPause is Data, onCommit must be Data.

What’s on disk

atelet drives whichever sandbox runtime a pool uses over the same Checkpoint/Restore gRPC, and treats the checkpoint contents as opaque - it ships exactly the files the runtime wrote, each zstd-compressed (regular files use a sparse-extent zstd format that skips holes; magic ATESPRSE), plus a manifest.json it writes beside them to pin the sandbox binary versions and file list.

  • gVisor (ateom-gvisor, runsc checkpoint/restore): a checkpoint.img holding memory, sentry state, and filesystem deltas, plus any pages images runsc emits. ateom reports exactly the files runsc wrote rather than a hardcoded list, so resume can stream RAM pages in lazily instead of reading the whole image up front.
  • micro-VM (ateom-microvm, cloud-hypervisor): a config.json, a state.json, and a sparse memory-ranges file - the guest’s writable rootfs upper lives in guest RAM, so process memory and rootfs writes are both captured by the memory snapshot.

Snapshots are not portable across sandbox classes - a gVisor snapshot can only be restored into a gVisor worker, and likewise for micro-VM.

Where snapshots are produced and consumed

atelet is the only component on a node that touches object storage. On checkpoint it collects the files ateom wrote and either leaves them on the node (Pause) or uploads them zstd-compressed to the external prefix (Suspend). On restore it downloads/decompresses (or reads the local copy) onto the worker, then has ateom restore against them. See the suspend flow and resume flow.

Golden vs. latest snapshot

KindStored whereWhen created
Golden SnapshotOn the ActorTemplate statusOnce, when the template is created, from a throwaway “golden” boot (see the bootstrap flow)
Last / latest snapshotOn the Actor record (latest_snapshot_info)Every time SuspendActor (or PauseActor) finishes for that specific actor

The resume workflow prefers the actor’s latest snapshot if one exists; otherwise it falls back to the template’s golden snapshot; otherwise it cold boots. (ResumeActor with boot=true skips the golden snapshot and cold boots deliberately.)

How external snapshots are addressed

By URI prefix. The prefix comes from ActorTemplate.spec.snapshotsConfig.location (e.g. gs://my-bucket/some/prefix); ateapi appends a per-snapshot segment. The actor record stores just this prefix in ExternalSnapshotInfo.snapshot_uri_prefix; atelet appends the individual file names (each with a .zstd suffix) plus manifest.json when reading or writing.

Why zstd, not gzip?

Better ratio at comparable speed, and decompression is fast - important because decompress happens on the critical path of resume.