Skip to content
Learn Agent Substrate

atecontroller

atecontroller is the Kubernetes-native half of the control plane. Everything declarative about Substrate goes through it: WorkerPool and ActorTemplate are the CRDs it reconciles, and reconciling them produces real K8s resources (Deployments) and ateapi state (golden snapshots).

What it owns

It runs exactly two reconcilers - one per CRD:

flowchart LR
  subgraph CTRL["atecontroller process"]
    direction TB
    WPR["WorkerPool reconciler"]
    ATR["ActorTemplate reconciler"]
  end

  K8s[(K8s API)] --> WPR
  K8s --> ATR

  WPR -- SSA apply --> DEP["K8s Deployment<br/>(N worker pods)"]
  ATR -- CreateAtespace /<br/>CreateActor /<br/>ResumeActor /<br/>SuspendActor --> API[ateapi]

  click API "/components/ateapi/" "ateapi"

There is no atespace or SandboxConfig reconciler here. SandboxConfig is a cluster-scoped CRD, but ateapi consumes it directly (via a lister) when scheduling - atecontroller does not reconcile it.

WorkerPool reconciler

apiVersion: ate.dev/v1alpha1
kind: WorkerPool
metadata:
  name: default
spec:
  replicas: 10
  ateomImage: ghcr.io/.../ateom-gvisor:vX
  sandboxClass: gvisor

The reconciler:

  1. Reads desired state from WorkerPool.spec.
  2. Server-side-applies a K8s Deployment named <workerpool-name> (same name as the pool; owning it, so pod changes re-trigger reconcile).
  3. Updates WorkerPool.status.replicas with the Deployment’s actual replica count.

That’s it. No data plane involvement, no Redis writes. The pods themselves get registered in Redis by ateapi’s worker-pool syncer once they go Ready.

Crucially, a WorkerPool is decoupled from ActorTemplate: templates no longer reference a pool. Instead, placement is selector-based - the scheduler ANDs a template’s workerSelector with the actor’s own worker_selector and matches against pool labels (and sandboxClass).

ActorTemplate reconciler

apiVersion: ate.dev/v1alpha1
kind: ActorTemplate
metadata:
  name: my-agent
spec:
  sandboxClass: gvisor
  workerSelector: { matchLabels: { pool: default } }
  containers: [...]
  pauseImage: ...
  snapshotsConfig: { location: gs://my-bucket/actors }

This one is more interesting - it runs a golden snapshot bootstrap as a small phase machine (ActorTemplate.Status.Phase), described in Golden snapshot:

stateDiagram-v2
  [*] --> PhaseInitial
  PhaseInitial --> PhaseResumeGoldenActor: CreateAtespace(ate-golden)<br/>+ CreateActor()
  PhaseResumeGoldenActor --> PhaseWaitGoldenActor: ResumeActor()<br/>(stamp TakeGoldenSnapshotAt)
  PhaseWaitGoldenActor --> PhaseReady: requeue fires, SuspendActor()
  PhaseReady --> [*]

Two things worth calling out:

  • The golden actor lives in the reserved ate-golden atespace. In PhaseInitial the reconciler first ensures that atespace exists (CreateAtespace, tolerating AlreadyExists), then creates a golden actor there whose name is a fresh UUID (Status.GoldenActorID).
  • The warmup wait is readyz-gated. The reconciler stamps Status.TakeGoldenSnapshotAt and requeues (rather than blocking on a sleep) before it suspends the golden actor to capture the snapshot. That delay is 20s only as a fallback: if every container in the template declares a readyz probe, the delay is 0, because ResumeActor already blocked until the workload reported readiness. A template with no containers keeps the 20s default.

The reconciler never assigns a PhaseFailed; on error it just returns and lets controller-runtime requeue. The result: every ready ActorTemplate has a Status.GoldenSnapshot URI (read from the suspended golden actor’s latest_snapshot_info.external.snapshot_uri_prefix) that new actors of this template can restore from in milliseconds.

Why is this a separate process from ateapi?

It uses the Kubernetes informer cache and has reconcile-loop semantics - the standard operator pattern. ateapi is stateless gRPC, optimized for QPS. Different concerns, different scaling profiles. (The manager is constructed without leader election, so if you scale this to multiple replicas they will all reconcile concurrently.)