Skip to content
Learn Agent Substrate

atelet

atelet is the node-side workhorse. One pod per worker node (it’s a DaemonSet), exposing a gRPC server that ateapi calls into for every Run / Checkpoint / Restore. atelet doesn’t make decisions - it executes.

It is runtime-agnostic: it drives whichever sandbox a worker pod runs (ateom-gvisor or ateom-microvm) over the same Unix-socket gRPC contract.

What it does

flowchart LR
  API[ateapi] -- gRPC :8085 --> AT[atelet]

  subgraph AT_internals["atelet process"]
    direction TB
    SRV["gRPC server"]
    PULL["Image puller<br/>(pre-stage OCI bundles)"]
    OCI["OCI bundle builder"]
    OBJ["GCS / S3 client<br/>(zstd compress/decompress)"]
    SOCK["Unix socket gRPC client<br/>→ ateom"]
  end

  SRV --> PULL
  SRV --> OCI
  SRV --> OBJ
  SRV --> SOCK

  SOCK -- "Run /<br/>Checkpoint /<br/>Restore" --> OM[ateom<br/>in worker pod]
  OBJ <--> GCS[("GCS / S3")]

  click API "/components/ateapi/" "ateapi"
  click OM "/components/ateom-gvisor/" "ateom"
  click GCS "/components/storage/" "Storage"

The gRPC service

Three RPCs, each called by ateapi during the corresponding workflow. Every request’s path components (atespace, actor name, container names, target ateom UID, snapshot URI prefix) are validated up front, and an invalid request is rejected as an invalid-argument error.

RPCCallerWhat atelet does
Runateapi resume workflow (cold boot fallback)Fetch sandbox assets, reset actor dirs, build OCI bundles, issue the Run call to ateom
Checkpointateapi suspend / pause workflowRe-fetch the recorded sandbox assets, issue the Checkpoint call to ateom, then ship exactly the files ateom reports (zstd-compressed) locally or to GCS/S3
Restoreateapi resume workflowFetch sandbox assets, download the snapshot files (parallel), decompress to a local volume, issue the Restore call to ateom

Sandbox assets (not a hardcoded runsc download)

The runtime binaries are no longer a baked-in “download runsc” step. The Run request carries a sandbox-assets description: a sandbox_class plus a per-architecture set of content-addressed files (URL + sha256). atelet projects that onto the node’s architecture, then fetches each asset:

  • The download is streamed to a temp file (hashing as it goes, verified against the expected sha256 and capped) and atomically renamed into the shared content-addressed cache - never buffered whole in memory.
  • Public buckets (gVisor’s runsc in gs://gvisor) are tried anonymously first, then the cluster’s own object-storage client.
  • gVisor needs a single runsc asset; the micro-VM runtime needs several (e.g. cloud-hypervisor, kata-kernel, kata-image).

atelet also records the asset set on the node at Run/Restore (sandbox-assets.json under the actor dir) so a later Checkpoint - whose request no longer carries the sandbox config - can re-fetch the identical binaries and pin them into the snapshot manifest.

Actor identity file

Before building the OCI bundles, atelet writes the actor’s own name to actor-id in the per-actor identity dir and bind-mounts that dir read-only at /run/ate in each application container. The write is atomic, and the dir is regenerated on every resume, so the value is correct even when the actor was restored from a shared golden snapshot (whose checkpointed process environment would otherwise carry the golden actor’s identity).

How atelet talks to ateom

Each worker pod has a host-bind-mounted Unix socket at:

/var/lib/ateom-gvisor/ateoms/<pod-uid>/ateom.sock

atelet dials this socket (over unix://) and gets a gRPC client to that specific pod’s ateom. No TCP, no service discovery - the pod UID in the filesystem path is the addressing scheme. The base path is shared by atelet and ateom.

Snapshot upload / download

atelet is the only component that talks to object storage; the same interface fronts both GCS and S3.

ateom reports the exact set of files it wrote (rather than atelet assuming a fixed list), so atelet ships precisely that set: gVisor’s checkpoint image files, or the micro-VM’s cloud-hypervisor snapshot set. Every on-disk snapshot file is uploaded with a sparse-extent zstd format (magic ATESPRSE) that compresses only the populated extents and skips the holes - so a mostly-zero RAM image (whether a gVisor pages.img or a micro-VM memory-ranges) only costs its resident pages. Snapshots can be kept node-local (for Pause) or uploaded to external storage (for Suspend), keyed under the prefix supplied by the ActorTemplate. Restore downloads happen in parallel; zstd compression is applied on the atelet side to keep network bytes down.

Local filesystem layout

atelet manages a tree on each node:

PathContents
/var/lib/ateom-gvisor/static-files/runsc-<sha>Content-addressed sandbox asset binaries
/var/lib/ateom-gvisor/actors/<actor-uid>/Per-actor working tree (keyed by the actor’s server-assigned UID)
.../identity/The actor-id file, bind-mounted read-only at /run/ate
.../sandbox-assets.jsonRecorded sandbox asset set (survives Run→Checkpoint)
.../bundles/<container>/OCI bundles per container
.../checkpoint-state/Output of a checkpoint
.../restore-state/Input to a restore (downloaded files staged here)

checkpoint-state and restore-state are kept separate so a future restore doesn’t trample a checkpoint in progress.

Why a DaemonSet, not a sidecar?

atelet is heavy: it caches sandbox binaries, holds object-storage credentials, and image-pulls. Running one per worker pod would multiply those costs by the worker count. One atelet per node, shared by all worker pods on that node, is much cheaper.