Storage
Substrate has two persistent stores. They serve completely different needs and have completely different access patterns.
Redis / Valkey - hot path
The source of truth for actor and worker state. Every routing decision
reads from here. The deployed backend is ValKey 9.1, run as a TLS
cluster; the cluster reloads its own TLS certificate on an interval
(tls-auto-reload-interval, 12h in the shipped config) so pod-certificate
rotations take effect without a restart.
flowchart LR API[ateapi] -- read/write --> R[(Redis / Valkey)] SYNC[Worker-pool syncer] -- write --> R WF[Workflow engine] -- read/write/lock --> R R -- worker updates --> WC[In-memory worker cache] click API "/components/ateapi/" "ateapi"
Keyspace
| Key pattern | Value | Notes |
|---|---|---|
actor:<atespace>:<name> | Actor proto (JSON) | Keyed by (atespace, name) |
atespace:<name> | Atespace proto (JSON) | Global-scoped |
worker:<ns>:<pool>:<pod> | Worker proto (JSON) | One per Ready worker pod |
lock:actor:<atespace>:<name> | Workflow-instance ID | 30s TTL, used by ateapi workflows |
To speed up scheduling, ateapi also fronts the worker keys with an in-memory worker cache that streams changes over Redis pub/sub and relists periodically. See ateapi.
Why Redis (and not the K8s API)?
The hot path needs sub-millisecond reads and atomic compare-and-set for locks. Etcd-backed K8s API isn’t built for that. Substrate uses K8s for declarative state (CRDs) and Redis for high-frequency operational state.
Locks
lock:actor:<atespace>:<name> is set with a 30-second TTL whenever a workflow
starts on an actor. The workflow itself runs under a context that times out at
28 seconds (TTL minus 2s of padding). Two concurrent operations on the same
actor cannot both succeed - the second returns Aborted.
Snapshots: external vs. local
A snapshot is the memory + sandbox state that lets a stopped actor come back to
life. There are now two storage locations, modeled as a oneof on
SnapshotInfo (Actor.latest_snapshot_info):
SnapshotInfo variant | Where it lives | Backs which state |
|---|---|---|
ExternalSnapshotInfo { snapshot_uri_prefix } | Object storage (GCS or S3) | SUSPENDED - actor released from its worker |
LocalSnapshotInfo { snapshot_prefix, node_vms_with_local_snapshots[] } | On the worker node’s disk | PAUSED - snapshot kept on the node VM for a fast resume |
SuspendActor uploads to external storage; PauseActor keeps the
snapshot local on the node and records which node VMs hold it. What each
transition captures is governed by the template’s
snapshotsConfig (onPause, onCommit, each a SnapshotScope of Full or
Data). See Snapshot.
GCS / S3 - external snapshots
flowchart LR
AT[atelet] -- zstd upload<br/>(parallel) --> G[("GCS or S3")]
AT -- zstd download<br/>(parallel) --> G
click AT "/components/atelet/" "atelet"
Object layout
The external snapshot prefix is built by ateapi from the per-template config, not a hardcoded bucket path:
<ActorTemplate.spec.snapshotsConfig.location>/<actorName>/<RFC3339-timestamp>-<random>/ <snapshot-file>.zstd (one per file runsc actually wrote) manifest.json (sandbox binaries + the snapshot file list)spec.snapshotsConfig.location is whatever the template provides (e.g.
gs://my-bucket/some/prefix); ateapi appends /<actorName>/<ts>-<rand> to form
the per-snapshot directory.
atelet no longer ships a hardcoded set of image files. The sandbox runtime
(e.g. ateom-gvisor via runsc) reports exactly the files it produced, and
atelet uploads precisely that set, each compressed with a sparse-extent zstd
format (magic ATESPRSE) that skips file holes, plus a manifest.json so a
later Restore on any node is self-describing. For a gVisor checkpoint this set is
typically checkpoint.img (always) plus lazy-paged RAM images such as
pages.img / pages_meta.img when produced; the micro-VM runtime writes CH’s
memory-ranges set. The sparse format is applied uniformly to both.
Both the checkpoint upload and the restore download run in parallel.
Why compress?
Network bandwidth between worker nodes and object storage is the bottleneck on cold resume. zstd buys a large reduction on these RAM-image files. The CPU cost is paid once on suspend, hidden behind storage I/O on restore.
Why GCS or S3?
Both back-ends implement the same internal object interface. The choice is environmental - GCS in GCP, S3 in AWS - selected at atelet startup, not per snapshot.
Node-local snapshots
For a PAUSED actor, atelet keeps the snapshot on the worker node’s disk
instead of uploading it, under a per-actor directory
(rooted under /var/lib/ateom-gvisor/..., plus the snapshot_prefix), again
beside a manifest.json. Because the bytes never
leave the node, resuming a paused actor on the same node avoids the object-store
round trip entirely; LocalSnapshotInfo.node_vms_with_local_snapshots records
which nodes hold a copy.
Who reads each store
| Component | Redis | GCS / S3 | Node-local snapshot |
|---|---|---|---|
| ateapi | read + write | never | never |
| atecontroller | never (via ateapi) | never | never |
| atelet | never | read + write | read + write |
| atenet | never (via ateapi) | never | never |
| ateom | never | never | reads files atelet stages |
This is a deliberate isolation: only ateapi touches Redis, only atelet touches object storage and node-local snapshot files. Everything else goes through them.
Related
- ateapi internals - Redis is its private database.
- atelet - the GCS/S3 and node-local snapshot client.
- Snapshot - what the snapshot files actually are.