Snapshot

A snapshot is what makes a Substrate actor portable. It’s a persisted capture of an actor’s state at the moment it was checkpointed, so the actor can later be restored onto a (potentially different) worker instead of cold-booting. The bytes depend on the sandbox runtime, but the actor record refers to every snapshot through the same Snapshot message.

Snapshot generations

Each suspend mints a new generation. The actor’s status.snapshots holds its active and in-progress generations (at most a handful), and status.lastAssignedGeneration only ever counts up. The table below lists the fields each generation carries.

Field	Meaning
uuid	Server-generated name, used for the local checkpoint directory and the object-storage path
generation	Monotonic counter, starting at 1
owner	ACTOR (taken from this actor) or TAG (borrowed from a tag)
fidelity	What it holds: VOLUMES, ROOTFS, or MEMORY
actorTemplateUid	The template whose sandbox the state was captured under
locality	The node that holds the snapshot’s files, while it does
durableSnapshot	The object-storage copy: a URI and a status, PENDING or COMPLETED
sandboxRuntime	The sandbox class and runtime version that took it

When we resume an actor, the resume restores the newest generation that is usable, meaning one that is still on a node or whose durable copy is COMPLETED.

Two copies: node-local and durable

A snapshot can exist in two places at once, and most of the suspend and resume behavior follows from which copy is where.

Copy	Where	Recorded as
Node-local	On the node that took it, under atelet’s /var/lib/ate directory	locality = <node>, and the actor’s assignedNode
Durable	In object storage (GCS or S3)	durableSnapshot.status = COMPLETED

So when does each copy show up, and when does the local one go away?

• With an asynchronous suspend (the default), the checkpoint lands on the node and SuspendActor returns with the durable copy PENDING. atelet uploads it in the background (--snapshot-upload-delay, 5 minutes by default) and reports it, which marks it COMPLETED. The actor stays attached to that node, so its next resume restores in place with no download.
• With a synchronous suspend (DURABILITY_MODE_SYNCHRONOUS, kubectl ate suspend actor --mode=sync), atelet uploads before SuspendActor returns, so the durable copy is already COMPLETED, and then tries to keep a local copy as well.
• On detach, which happens after --snapshot-detach-delay (30 minutes by default) or during a resume when the attached node has no room, atelet uploads the snapshot if it still needs to and deletes the local files. The snapshot’s locality and the actor’s assignedNode are cleared, and the actor can resume on any eligible node.

A node-local copy is only trusted on the boot that wrote it, so after a node reboot, atelet ignores it and falls back to the durable copy.

Why keep a local copy at all?

An agent that is suspended and resumed again within minutes (the common pattern for a conversational agent) never pays for an upload and a download on its critical path. The durable copy is still made, just off that path.

Fidelity: what a snapshot holds

The template’s snapshotConfig.preferredFidelity sets the ceiling for what a snapshot captures, and each level in the table below includes the ones above it.

Fidelity	Contents	On resume
VOLUMES	Durable volumes (durableDir) only	Cold-boots the containers from the OCI image with the volumes restored
ROOTFS	Volumes plus root filesystem changes since boot	Cold-boots with the rootfs writes back in place
MEMORY (default)	Volumes, rootfs changes, and process memory	Resumes where it left off

The ceiling is best-effort, and the system may package less when a layer cannot be captured. A restore can also use less than the snapshot holds, which happens in the two cases below.

• after the actor was repointed at a different template, only the volumes are restored;
• if no eligible worker still offers the sandbox runtime that took a memory snapshot, the restore drops to ROOTFS rather than fail.

What’s on disk

atelet drives each sandbox runtime over the same Checkpoint/Restore gRPC and treats the files the runtime writes as opaque. What ends up in those files depends on the runtime.

• With gVisor (ateworker-gvisor), a MEMORY snapshot is a runsc checkpoint of the sandbox plus separate filesystem bundles for the root filesystem and the durable volumes. ROOTFS and VOLUMES snapshots save only the bundles they need, without memory.
• With a micro-VM (ateworker-microvm, Cloud Hypervisor), the VM is paused, a MEMORY snapshot takes a VM memory snapshot, and the durable volumes and the rootfs upper layer are each shipped as a tar.

Next to the files, atelet writes a manifest.json that records the sandbox class, the sandbox runtime assets, the actor’s identity, the file lists, and the fidelity. In object storage each file is stored zstd-compressed (sparse files in a format that skips holes), and the manifest is written last as the commit marker, so a snapshot without a manifest is incomplete.

A worker that is asked to checkpoint an actor it already checkpointed (e.g. a retried suspend after a lost response) reuses the completed checkpoint instead of taking another, and clears any partial files left by an interrupted one. atelet commits a local checkpoint by staging the files and moving the staged directory into place with one rename.

Snapshots are not portable across sandbox classes, so a gVisor snapshot only restores into a gVisor worker (and likewise for micro-VM), and the scheduler enforces this.

Who touches object storage

Neither atelet nor ateapi talks to object storage for snapshots directly. Each runs an object-store plugin sidecar (cmd/objectstore-plugin, backed by GCS or S3) and calls it over a Unix socket, and the two plugins do different jobs.

• atelet’s plugin uploads and downloads a snapshot’s files, authenticated per transfer with a credential scoped to the actor;
• ateapi’s plugin copies whole snapshots (for tags) and deletes them.

How durable snapshots are addressed

Durable snapshots are addressed by URI prefix under a storage location, which is ate-api-server’s --snapshot-location, or the template’s snapshotConfig.storageLocation when the PerTemplateSnapshotLocation preview gate is on. Every object sits under its owner’s prefix, as the layout below shows.

<location>/atespaces/<atespace>/actors/<actor-uid>/snapshots/<uuid>   an actor's snapshot
<location>/atespaces/<atespace>/tags/<tag-uid>                        a tag's only snapshot

Who owns a snapshot decides when it gets cleaned up. A synchronous suspend releases the durable snapshot it replaced, while an asynchronous one keeps the previous one until a newer one is durable, and DeleteActor deletes the actor’s whole prefix. A borrowed snapshot lives under its tag’s prefix, so an actor never collects it.

Tags

What if we want many actors to start from the same snapshot? That’s what a tag is for. A tag is an atespace-owned, stable name for its own copy of one durable snapshot. CreateTag (kubectl ate create tag) takes the snapshot a SUSPENDED actor holds (waiting for its upload first if it is still pending) and copies it under the tag’s prefix, so suspending or deleting the actor afterwards cannot collect it.

• Scope ATESPACE lets the tag seed actors in its own atespace; PUBLISHED lets actors in any atespace use it, still addressed as <atespace>/<tag>.
• A tag only seeds actors of the template it was taken under.
• CreateActor with a tag borrows the snapshot rather than copying it, and the actor’s first suspend moves it onto its own.
• DeleteTag deletes the copy. Actors still borrowing it can no longer resume, and an atespace that holds tags cannot be deleted.

The template’s golden snapshot is a tag too (see Golden snapshot bootstrap). Because a tag’s snapshot is shared by many actors, atelet keeps it in a node-local snapshot cache after the first download on that node.

Related

• Suspend actor · Resume actor
• Golden snapshot bootstrap
• ActorTemplate - snapshotConfig.
• Storage - the object-storage layer.