Snapshot
A snapshot is what makes Substrate actors portable. It’s a persisted
representation of an actor’s state at the moment it was checkpointed, so the
actor can later be restored onto a (potentially different) worker instead of
cold-booting. The exact bytes depend on the sandbox runtime, but the actor
record refers to a snapshot through one uniform message: SnapshotInfo.
SnapshotInfo: external or local
SnapshotInfo is a oneof data with two
variants - a snapshot is either uploaded to object storage or kept on the node:
| Variant | Message | Where it lives | Set by |
|---|---|---|---|
| External | ExternalSnapshotInfo{ snapshot_uri_prefix } | Object storage (GCS/S3) at a URI prefix | SuspendActor |
| Local | LocalSnapshotInfo{ snapshot_prefix, node_vms_with_local_snapshots[] } | On the worker node VM(s) listed in node_vms_with_local_snapshots | PauseActor |
An actor’s latest_snapshot_info field carries whichever of these was most
recently produced. (There is no longer a SnapshotType enum or a
last_snapshot string - those are gone.)
Snapshot scope: what goes in
Independently of where a snapshot lands, the ActorTemplate’s
snapshotsConfig decides what it contains, via the SnapshotScope enum:
| Scope | Contents | On resume |
|---|---|---|
Full | Process memory plus the filesystem delta on top of the OCI image (which includes any attached DurableDir volumes, since they live inside rootfs) | Restores hot from the memory image |
Data | Only the contents of snapshot-supporting volumes (currently DurableDir); process memory and the rest of rootfs are discarded | Cold-boots from the OCI image with DurableDir contents restored |
Scope is chosen per trigger in SnapshotsConfig:
onPause(defaultFull) - what a Pause captures and keeps on the node.onCommit(defaultFull) - what a Suspend captures and uploads.
onCommit must be a subset of onPause (enforced by a CEL validation rule
on the CRD): if onPause is Full, onCommit may be Full or Data; if
onPause is Data, onCommit must be Data.
What’s on disk
atelet drives whichever sandbox runtime a pool uses over the same
Checkpoint/Restore gRPC, and treats the checkpoint contents as opaque -
it ships exactly the files the runtime wrote, each zstd-compressed (regular
files use a sparse-extent zstd format that skips holes; magic ATESPRSE), plus
a manifest.json it writes beside them to pin the sandbox binary versions and
file list.
- gVisor (
ateom-gvisor,runsc checkpoint/restore): acheckpoint.imgholding memory, sentry state, and filesystem deltas, plus any pages imagesrunscemits. ateom reports exactly the filesrunscwrote rather than a hardcoded list, so resume can stream RAM pages in lazily instead of reading the whole image up front. - micro-VM (
ateom-microvm, cloud-hypervisor): aconfig.json, astate.json, and a sparsememory-rangesfile - the guest’s writable rootfs upper lives in guest RAM, so process memory and rootfs writes are both captured by the memory snapshot.
Snapshots are not portable across sandbox classes - a gVisor snapshot can only be restored into a gVisor worker, and likewise for micro-VM.
Where snapshots are produced and consumed
atelet is the only component on a node that touches object storage. On checkpoint it collects the files ateom wrote and either leaves them on the node (Pause) or uploads them zstd-compressed to the external prefix (Suspend). On restore it downloads/decompresses (or reads the local copy) onto the worker, then has ateom restore against them. See the suspend flow and resume flow.
Golden vs. latest snapshot
| Kind | Stored where | When created |
|---|---|---|
| Golden Snapshot | On the ActorTemplate status | Once, when the template is created, from a throwaway “golden” boot (see the bootstrap flow) |
| Last / latest snapshot | On the Actor record (latest_snapshot_info) | Every time SuspendActor (or PauseActor) finishes for that specific actor |
The resume workflow prefers the actor’s latest snapshot if one exists;
otherwise it falls back to the template’s golden snapshot; otherwise it cold
boots. (ResumeActor with boot=true skips the golden snapshot and cold
boots deliberately.)
How external snapshots are addressed
By URI prefix. The prefix comes from ActorTemplate.spec.snapshotsConfig.location
(e.g. gs://my-bucket/some/prefix); ateapi appends a per-snapshot segment. The
actor record stores just this prefix in ExternalSnapshotInfo.snapshot_uri_prefix;
atelet appends the individual file names (each with a .zstd suffix) plus
manifest.json when reading or writing.
Why zstd, not gzip?
Better ratio at comparable speed, and decompression is fast - important because decompress happens on the critical path of resume.
Related
- Suspend actor · Resume actor
- Golden snapshot bootstrap
- Storage - the GCS/S3 layer.