ateapi

ateapi is a stateless gRPC server backed by Redis/Valkey. It is the source of truth for actor and worker state - every other component either calls it (atenet) or is called by it (atelet).

What it owns

flowchart LR
  subgraph ateapi["ateapi process"]
    direction TB
    GRPC[gRPC server :443]
    CTL[Control service<br/>actor + atespace RPCs]
    SID[SessionIdentity service<br/>MintJWT / MintCert]
    WF[Workflow engine<br/>resume / suspend / pause]
    SYNC[Worker-pool syncer<br/>watches K8s worker pods]
    WCACHE[In-memory worker cache]
    STORE[Redis store adapter]
  end

  GRPC --> CTL
  GRPC --> SID
  CTL --> WF
  WF --> STORE
  WF --> WCACHE
  SYNC --> STORE
  SYNC --> WCACHE

  CLIENT[atenet ExtProc] -.->|ResumeActor| CTL
  CLI[kubectl-ate] -.->|all RPCs| CTL
  K8s[(K8s API)] -.->|watch worker pods| SYNC
  STORE -.-> REDIS[(Redis / Valkey)]
  WF -.->|gRPC| ATELET[atelet on target node]

  click ATELET "/components/atelet/" "atelet"
  click CLIENT "/components/atenet/" "atenet"
  click REDIS "/components/storage/" "Storage"

Resource identity: (atespace, name)

Every Substrate resource now carries a common ResourceMetadata (atespace, name, uid, version, create_time, update_time), and an actor’s identity is the tuple (atespace, name) - the old bare actor_id is gone. Requests reference a resource with an ObjectRef{atespace, name}, or carry the resource body itself (e.g. CreateActorRequest{actor: Actor} where the identity lives in actor.metadata). See the Atespace concept for the Create/Get/List/Delete + ObjectRef/ResourceMetadata conventions.

The Control gRPC service

RPC	Request → Response	Notes
GetActor	GetActorRequest{actor: ObjectRef} → Actor	Fetch one actor record
CreateActor	CreateActorRequest{actor: Actor} → Actor	Identity in actor.metadata.{atespace,name}; created with status SUSPENDED
UpdateActor	UpdateActorRequest{actor: ObjectRef, worker_selector: Selector} → UpdateActorResponse{actor}	Updates placement; takes effect on next resume
SuspendActor	SuspendActorRequest{actor: ObjectRef} → SuspendActorResponse{actor}	Running → Suspended; uploads snapshot to external storage
PauseActor	PauseActorRequest{actor: ObjectRef} → PauseActorResponse{actor}	Running → Paused; keeps the snapshot on the node VM
ResumeActor	ResumeActorRequest{actor: ObjectRef, boot: bool} → ResumeActorResponse{actor}	Suspended/Paused → Running. boot=true skips the golden snapshot and cold-boots
DeleteActor	DeleteActorRequest{actor: ObjectRef} → Actor	Only a SUSPENDED or CRASHED actor may be deleted; returns the deleted record
ListWorkers	ListWorkersRequest{page_size, page_token} → ListWorkersResponse{workers, next_page_token}	Paginated
ListActors	ListActorsRequest{atespace, page_size, page_token} → ListActorsResponse{actors, next_page_token}	Paginated; optional atespace scopes the scan
CreateAtespace	CreateAtespaceRequest{atespace: Atespace} → Atespace	Atespace CRUD (see below)
GetAtespace	GetAtespaceRequest{atespace: ObjectRef} → Atespace
ListAtespaces	ListAtespacesRequest{page_size, page_token} → ListAtespacesResponse{atespaces, next_page_token}	Paginated
DeleteAtespace	DeleteAtespaceRequest{atespace: ObjectRef} → Atespace	Rejects FailedPrecondition if any actors remain

Actor status is one of SUSPENDED, RESUMING, RUNNING, SUSPENDING, PAUSING, PAUSED, or CRASHED. PAUSING, PAUSED, and CRASHED are recent additions that go with the pause/crash flows.

The destructive DebugClear (drop all data from the database) no longer lives on Control - it was split out into a separate Debug service, which opens the door to gating it independently. Note that today the server always registers the Debug service; there is no flag to disable or access-control it separately yet.

Atespaces

Atespaces are the tenancy boundary an actor is created into - Substrate-native records stored in Redis, not Kubernetes objects. An Atespace is global-scoped (its metadata.atespace is empty; its identity is metadata.name). DeleteAtespace refuses (FailedPrecondition) to remove an atespace that still holds actors. See Atespace.

The SessionIdentity gRPC service

A second service on the same server.

Workloads call these from inside their sandbox to mint session-scoped credentials that survive worker migration:

• MintJWT(MintJWTRequest) - an OIDC-discovery-compatible JWT whose subject is apps/{appid}/users/{userid}/sessions/{sessionid} and which carries an ate.dev extension claim with the app/user/session IDs.
• MintCert(MintCertRequest) - an X.509 cert (returned as a DER chain, leaf-first) scoped to the same app/user/session.

The workload authenticates either with its K8s service-account bearer token (the pod identity) or with an mTLS client cert; ateapi checks that the credential belongs to the pod currently mapped to the requested session, then signs.

The workflow engine

All actor state transitions go through a small generic workflow engine that runs a workflow as a list of idempotent steps. Each workflow:

1. Acquires lock:actor:<atespace>:<name> in Redis with a 30s TTL. The workflow runs under a context that times out at 28s (TTL minus 2s of padding) so the lock cannot be released after it has already expired.
2. Runs its steps in order - each step first checks whether it is already complete (so a retried workflow fast-forwards), then reads/writes Redis and may call atelet over gRPC.
3. Returns Aborted to the caller if the lock is already held.

flowchart LR
  IN[RPC handler] --> LOCK{Acquire<br/>lock:actor:atespace:name}
  LOCK -- failed --> ABORT[return Aborted]
  LOCK -- ok --> S1[Step 1]
  S1 --> S2[Step 2]
  S2 --> S3[Step 3]
  S3 --> S4[Step N]
  S4 --> UNLOCK[Release lock]
  UNLOCK --> OUT[return Actor]

The steps for each workflow:

Workflow	Steps
Resume	load the actor → assign a worker → tell atelet to restore → finalize as Running
Suspend	load the actor → mark it Suspending → tell atelet to suspend → finalize as Suspended
Pause	load the actor → mark it Pausing → tell atelet to pause → finalize as Paused

The worker-pool syncer and worker cache

The hot path needs to answer “give me a free worker in pool X” fast. That answer comes from Redis, fronted by an in-memory worker cache. A worker-pool syncer keeps Redis in sync with actual K8s state, and the cache keeps an in-process view current:

• Watching worker pods: pod gets a PodIP → a worker record (idle, with no assignment) is written to Redis. (Registration is keyed on the pod having an IP, not on the Ready condition.)
• Pod removal: pod is gone → if the worker held an assignment, its actor is reset to SUSPENDED, then the worker record is deleted from Redis.
• The worker cache streams worker updates from the store and periodically relists to recover from missed events, exposing an O(1) view for scheduling.

Known race

If a pod dies and a concurrent SuspendActor is in flight, the actor can get released twice. Best-effort cleanup only - tracked upstream.

The Redis keyspace

Key	Value	Notes
actor:<atespace>:<name>	Actor proto (JSON)	Keyed by (atespace, name)
atespace:<name>	Atespace proto (JSON)	Global-scoped
worker:<ns>:<pool>:<pod>	Worker proto (JSON)	Keyed by pod identity
lock:actor:<atespace>:<name>	Lock owner ID	30s TTL

Authentication

Incoming gRPC auth is selected by --auth-mode:

• mtls (default) - client identity comes from transport-level mTLS.
• jwt - additionally requires a Kubernetes ServiceAccount Bearer token on every RPC, verified against the cluster’s OIDC issuer in a gRPC interceptor. This mode is a stopgap that will be dropped once Pod Certificates are enabled by default in the minimum supported Kubernetes version.

Related

• Resume actor flow - sees the workflow engine in action.
• Atespace - the tenancy boundary in resource identity.
• System topology - how ateapi connects to everything else.
• Storage - Redis/Valkey and GCS/S3 details.