Skip to content
Learn Agent Substrate

Actor lifecycle

Every actor is in exactly one of seven states at any moment. That state is stored in the actor’s row in ateapi’s PostgreSQL store and changed by ateapi’s workflows, each of which holds a per-actor lease while it runs. The ActorState values are ACTOR_STATE_RESUMING, RUNNING, SUSPENDING, SUSPENDED, CRASHED, DELETING, and REVERTING (plus the unused UNSPECIFIED zero value), and we can see in the diagram below how an actor moves between them.

The states

stateDiagram-v2
  [*] --> SUSPENDED: CreateActor
  SUSPENDED --> RESUMING: ResumeActor<br/>(worker assigned)
  RESUMING --> RUNNING: Run / Restore done
  RUNNING --> SUSPENDING: SuspendActor
  SUSPENDING --> SUSPENDED: checkpoint done,<br/>worker released

  RESUMING --> CRASHED: restore or boot failed
  SUSPENDING --> CRASHED: checkpoint failed
  RUNNING --> CRASHED: sandbox died,<br/>worker pod gone

  RUNNING --> REVERTING: RevertActor
  CRASHED --> REVERTING: RevertActor
  SUSPENDED --> REVERTING: RevertActor<br/>(node-local snapshot)
  REVERTING --> SUSPENDED: at durable snapshot

  SUSPENDED --> DELETING: DeleteActor<br/>(any state)
  DELETING --> [*]

  note right of RUNNING
    Has a worker assignment
    and a live sandbox.
  end note

  note left of SUSPENDED
    The idle state. May still be
    attached to a node that holds
    its latest snapshot.
  end note

DeleteActor is accepted from every state, but the diagram draws only the SUSPENDED edge to keep it readable.

What each state means

StateWorker assigned?Snapshot activityTypical exit
SUSPENDEDNo (may be attached to a node)Latest snapshot is node-local, durable, or bothResumeActor
RESUMINGYes (just assigned)Restoring the latest snapshot, or cold-booting if there is noneFinalize to RUNNING
RUNNINGYesn/a (live)SuspendActor
SUSPENDINGYes (about to release)A new snapshot generation is being writtenFinalize to SUSPENDED
CRASHEDNoKeeps its snapshots, including any partial oneRevertActor or DeleteActor
REVERTINGBeing releasedDiscarding node-local and in-progress snapshotsFinalize to SUSPENDED
DELETINGBeing releasedCollecting the actor’s own snapshot prefixRecord removed

Allowed transitions

FromRPC or eventTo
(none)CreateActorSUSPENDED
SUSPENDEDResumeActorRESUMING → RUNNING
RUNNINGResumeActorunchanged; returns resumed=false
RUNNINGSuspendActorSUSPENDING → SUSPENDED
SUSPENDEDSuspendActorunchanged; returned as it is
RUNNING, CRASHED, or SUSPENDED with a node-local latest snapshotRevertActorREVERTING → SUSPENDED
anyDeleteActorDELETING → gone
RESUMING / SUSPENDINGan atelet failure the workflow cannot retryCRASHED
RUNNINGthe worker reports its sandbox diedCRASHED
RUNNING (or mid-operation)the worker pod is deleted, or its worker process restartsCRASHED

Anything else is rejected with FailedPrecondition, e.g. ResumeActor on a CRASHED actor, or RevertActor on a SUSPENDED actor whose snapshot is already durable-only (it is already where a revert would take it).

So what happens when a client retries in the middle of one of these transitions? Every workflow is re-entrant, and each step derives from the persisted record whether its work is already done, so a retry after a lost response or a failed ateapi replica fast-forwards to where the last attempt stopped. A RESUMING actor resumed again continues on the worker it was already assigned, and a SUSPENDING actor suspended again re-sends the same snapshot, which atelet answers without checkpointing twice.

The CRASHED state

CRASHED means the actor lost its sandbox, or an operation on it failed in a way that cannot simply be retried. ateapi records a human-readable reason and time in status.crash, releases the actor’s worker, and clears the assignment. The typical causes are listed below.

  • atelet’s Run, Restore, or Checkpoint returned a non-transient error, including a wakeupProbe that never answered 200. (Transport errors, Unavailable, and deadline errors are returned for a retry instead.)
  • The worker assigned to a RESUMING actor is gone, draining, no longer hosts it, or no longer matches its placement constraints.
  • The node holding the actor’s only copy of its latest snapshot no longer exists.
  • The sandbox exited on its own while RUNNING, which the worker reports through atelet (ReportActorCrashed).
  • The worker pod was deleted, or its worker process restarted and re-registered with a new epoch, while the actor was bound to it.

Substrate does not recover a crashed actor automatically. The actor keeps its snapshot entries (including a partial one from a failed suspend, so the objects it names can still be cleaned up), and there are two ways out of CRASHED. RevertActor terminates anything still running for it, detaches volumes, discards node-local and in-progress snapshots, and returns it to SUSPENDED at its last completed durable snapshot, so the next resume restores from there. DeleteActor removes it entirely.

Who writes each transition

flowchart LR
  subgraph WF_Resume["ResumeActor workflow"]
    R1[Load actor + template<br/>RUNNING? return]
    R2[Create pending volumes]
    R3[Assign worker<br/>→ RESUMING]
    R4[Attach volumes]
    R5[atelet Run or Restore]
    R6[Finalize → RUNNING]
    R1 --> R2 --> R3 --> R4 --> R5 --> R6
  end

  subgraph WF_Suspend["SuspendActor workflow"]
    S1[Load actor<br/>SUSPENDED? return]
    S2[Mark SUSPENDING<br/>new snapshot generation]
    S3[atelet Checkpoint]
    S4[Detach volumes]
    S5[Finalize → SUSPENDED<br/>release worker]
    S1 --> S2 --> S3 --> S4 --> S5
  end

  subgraph WF_Revert["RevertActor workflow"]
    V1[Mark REVERTING]
    V2[atelet Terminate,<br/>detach, release]
    V3[Discard pending snapshots]
    V4[Finalize → SUSPENDED]
    V1 --> V2 --> V3 --> V4
  end

  subgraph WF_Delete["DeleteActor workflow"]
    D1[Mark DELETING]
    D2[atelet Terminate,<br/>detach, release]
    D3[Delete volumes +<br/>actor's snapshot prefix]
    D4[Remove record]
    D1 --> D2 --> D3 --> D4
  end

As the diagram above shows, each RPC runs its own workflow inside ateapi. DeleteActor is best-effort across its cleanup steps, so if any of them fails, the record stays in DELETING and a retried delete picks up where it left off.

How are concurrent operations kept apart?

Every resume, suspend, revert, delete, and update (and every tag created from an actor) runs under a PostgreSQL lease named lease:actor:<atespace>:<name>. The lease has a 30-second TTL and is renewed in the background for as long as the workflow runs, and if renewal fails, the workflow’s context is canceled. A second operation on the same actor gets Aborted (“another operation is in progress for this actor”).

On top of the lease, every write carries a version precondition, so a write based on a stale read fails rather than overwriting. Together they prevent races like “resume + suspend at the same time” or “delete while a sandbox is still live.” The node side has its own guard too, since atelet and the worker processes serialize the operations they run for each actor UID with an in-process lock.

What can move an actor without a client RPC?

  • When the worker reports a dead sandbox, atelet forwards the worker’s ReportActorCrashed, and a RUNNING actor goes to CRASHED. An actor in the middle of another operation is left to that operation, which fails on its own.
  • When a worker pod enters Terminating, the worker-pool syncer marks the Worker DRAINING so nothing new is placed on it, and the worker propagates SIGTERM into its sandboxes. Once the pod is gone, ateapi deregisters the Worker. Actors already SUSPENDED are left alone, actors still bound to it go to CRASHED, and atelet is asked, best-effort, to reclaim their on-node state.
  • When atelet finishes a background upload or detach after an asynchronous suspend, it reports through ReportActorSnapshotState that the snapshot reached object storage or that it deleted the node-local copy. ateapi records the first as the snapshot’s durable copy becoming COMPLETED, and the second by clearing the snapshot’s node and the actor’s assignedNode. Neither changes the actor’s state.

Lifecycle events

Each committed state change emits an ate.actor.state_changed event (an OTLP log event, or a stdout record), and every crash an ate.actor.crashed event. The last record for an actor’s UID is the state it is in now. Actor identity is barred from metric labels, so these events are how a transition is attributed to one actor.

Where does state actually live?

All of it lives in ateapi’s PostgreSQL store, which holds the serialized Actor message. Every other component (atelet, atecontroller, and the atenet router) reads it through ateapi RPCs and never touches the database directly.