Durable mutation recovery

Durable mutation recovery

Mithai checkpoints approved mutations before dispatch and normally records their outcome before adapter result callbacks, final narration, or completed-turn persistence. If the settlement wait expires, it reports uncertainty while the admitted storage write finishes. The same session ledger serves model dispatch, guarded turn handlers/backends, handle, handle_with_results, API streams, and Slack. It is available even when a process stops before a conversation turn can be saved.

Execution contract

Tools with a resolved Human MCP policy are protected automatically, including when an auto-approval rule satisfies that policy. Hosts must also declare otherwise auto-executed mutations during startup:

engine.register_mutation_tools({"mcp__dashboards__create", "mcp__dashboards__patch"})

This declaration does not change approval policy. Synchronous API calls still cannot approve interactive tools. A replay is checked after normal approval; recorded success never authorizes another execution or a different operation. Read-only tools without a human policy or host declaration retain their normal repeatable behavior. When an MCP tool has neither protection nor a canonical owning skill’s explicit mcp_prerequisites read-only declaration, dispatch logs a warning naming the canonical tool and register_mutation_tools. The warning does not change execution or approval policy and never includes arguments. Read-sounding tool names are not evidence of read-only behavior.

Each mutation ID is SHA-256 of canonical JSON [actual_tool, approved_arguments] (sorted object keys, compact separators, no NaN). MCP aliases use the canonical mcp__<server>__<tool> target; native tools use their routed name. The public mithai.core.mutation_ledger.mutation_identity(tool, arguments) function computes this ID when given that canonical name. Arguments prepared after approval must still match the approved snapshot. A different patch has a different identity and passes through approval again. Equivalent dashboard titles do not authorize patching a dashboard, and Mithai does not normalize domain identities.

This is exact-argument replay protection, not semantic deduplication. After a crash, a model retry that rewords a title or changes another argument gets a new mutation ID, even in the same logical turn. It can execute again after normal approval (including auto-approval). Hosts must perform domain deduplication before dispatch: compare host-normalized scope_identity and normalized_title metadata from retained mutation state/artifacts with the proposed operation, and use authoritative resource readback when the outcome or metadata is missing. These metadata fields are stored for the host; Mithai does not compare them or use them to merge mutation identities. A matching title alone never authorizes a different patch.

Stored status	Meaning	Retry behavior
approved	Approval was obtained; execution intent was not committed, or dispatch proved non-execution.	A new attempt requires normal approval again.
execution_started	Intent was durably committed before entering the router.	Never replay automatically; public reads report outcome_unknown.
completed	The router returned a successful result and its outcome was committed.	Return mutation_completed, without executing again.
failed	The MCP server returned a structured CallToolResult with isError=true, or a native handler returned ToolExecutionResult(succeeded=False, failure_confirmed=True).	Return mutation_failed, without executing again.
outcome_unknown	A transport error or a native handler failure without failure_confirmed cannot establish whether an effect occurred.	Return mutation_outcome_unknown; reconcile externally.

Every recorded intent, settlement, reconciliation and metadata attachment also increments the logical turn’s revision and stores it on the entry. It is the replay cursor for mutation outcome events; it plays no part in replay protection.

A structured MCP failure is a known failed attempt, not proof that every possible side effect was rolled back. It does not authorize an automatic retry. Transport errors, timeouts, and protocol exceptions without a tool result remain uncertain.

After checkpoint persistence, dispatch rechecks cancellation and the deadline. If either prevents entry into the router, it records confirmed non-execution as approved; a later attempt still requires fresh approval.

Once routing begins, an exception, cancellation, process interruption, or failed outcome write leaves started intent blocking replay. A failed intent write returns mutation_checkpoint_unavailable and never enters the tool. These are safety receipts, not a cache of tool-result bodies: a duplicate returns a refusal with its bounded recorded metadata, not fabricated fresh evidence.

The additive tool_execution_started stream event announces local execution before these checkpoints. It is distinct from the ledger’s execution_started crash marker and proves no remote effect. Checkpoint waits are bounded; late storage work can finish durably without dispatching an abandoned write or emitting after stream termination. See the approved execution contract for release availability, storage containment, and consumer fixtures.

Bounded execution after approval (0.23.30)

Hosts can opt selected MCP mutations into a fresh execution allowance after an actual human approval. Register one immutable policy during startup:

from mithai.core.mcp_execution import ApprovedMutationBudget

engine.register_approved_mutation_budget(ApprovedMutationBudget(
    tools=frozenset({"mcp__inventory__create_item", "mcp__inventory__update_item"}),
    execution_seconds=120,
    request_ceiling_seconds=900,
))

Both durations must be positive finite numbers (booleans are rejected). The request ceiling must exceed the configured wrap-up allowance plus five seconds for MCP cancellation drain. Registration is idempotent for the same policy; a different second policy is rejected. Failed skill bind() rolls it back. Tool selection is exact, with aliases resolved to their canonical MCP target. There is no built-in domain-specific tool list. This policy does not alter Human MCP approval requirements or authorize a write by itself.

An eligible call must have a successful prepared MCP receipt and a fresh human approval of its exact tool, arguments and live surface. Auto-approval, denial, expired/cancelled approval, native handlers, unselected reads and fanout calls receive no new allowance. Native code calling MCPManager directly remains responsible for its own approval and deadline propagation; the automatic contract applies to MCP routes through the engine. Unselected tools keep their normal transport cap. Without a policy, existing session and transport timing defaults are unchanged.

The request ceiling is fixed before coalescing and includes human wait time. An earlier deadline_monotonic in the adapter’s captured interaction context wins. Hosts must set this ceiling within their external HTTP/gateway limit, leaving headroom for adapter delivery and persistence, and propagate request cancellation. The runtime reserves the configured wrap-up budget and five-second cancellation drain inside that ceiling. Approval pauses cannot move this fixed bound.

At successful approval, the call deadline is the earlier of now + execution allowance and the request ceiling minus those reserves. The owner stays alive to observe that call, but another approval wait cannot extend the call’s fixed deadline. An already exhausted effective budget is never revived. For example, with five seconds left on the session clock, a 120-second human wait still leaves five seconds under the default contract; an eligible approval now grants up to 120 execution seconds, limited by the fixed request ceiling.

The same call deadline reaches the router, reconnect/lock guards and MCP transport. MCPManager.call_tool_with_status(..., execution_timeout_seconds=120, deadline_monotonic=absolute_deadline, cancel_event=event) also exposes the finite transport cap directly; it bounds the SDK response wait and does not grant approval or replay permission. Omitting it preserves the 60-second cap. Connection and server failures can still end a call earlier than its allowance.

Cancellation retains the existing five-second drain and transport quarantine behavior. No detached mutation worker is introduced. Cancellation is not proof that the remote server stopped a dispatched effect. Unknown outcomes must be reconciled before retry, including retries with identical payloads. Confirmed non-execution permits a new attempt through fresh approval; this API never reuses an approval from a previous attempt.

Coalescing retains the original owner’s request lifetime; an attacher cannot start a duplicate transport call. Attachment timeout returns still_running. After response loss, reuse the original delivery/logical-turn identity and read the ledger. Existing retention and single-owner limitations still apply.

Turn handlers can read context.executor.deadline_monotonic for the live, approval-adjusted deadline and context.request_deadline_monotonic for the fixed ceiling. The original context.deadline_monotonic remains an initial snapshot; do not use it to cancel an approved call after its allowance was extended. Custom approval adapters should honor HumanRequest.remaining_timeout(default) and the optional deadline_monotonic; the runtime rejects late approved replies even if a custom adapter ignores expiry. Blocking host code must cooperate with deadlines and cancellation; a Python thread cannot safely be forcibly stopped.

Timeout outcomes

Outcome	Runtime evidence	Replay rule
Approval expired	approval_outcome: timed_out; no dispatch	Obtain fresh approval. No mutation outcome is created without intent.
Deadline rejected before dispatch	execution_status: not_executed, reason deadline (engine) or pre_dispatch_deadline (transport)	If intent exists, settle as approved; obtain fresh approval.
Timeout after SDK dispatch begins	reason execution_timeout, ledger/public status outcome_unknown	Never replay blindly; use authoritative reconciliation.

For guarded transport calls, MCPExecutionTimeout is a TimeoutError with dispatched and reason attributes. Treat dispatched=True conservatively: entering the SDK is not evidence that the server completed or rolled back. The ledger and mutation_outcome/v1 retain timeout reasons for replay.

Delivery and continuation identity

Logical-turn IDs are server-generated, scoped to session, agent, and user. They are allocated lazily on the first approved mutation and returned as TurnResult.logical_turn_id. A fresh delivery has a fresh logical turn. Hosts can supply IncomingMessage.delivery_id as a stable retry key; otherwise the transport’s message_id is used. Slack already supplies its message timestamp. Reuse a delivery ID only for redelivery of the same request. Explicit delivery IDs also distinguish simultaneous same-text requests; callers omitting them retain legacy live-request coalescing.

For an explicit continuation, authenticate the same session/user and select the existing identity before calling either handle method:

engine.resume_logical_turn(next_message, logical_turn_id)
result = engine.handle_with_results(next_message, adapter)

Unknown, foreign, erased, and expired logical IDs raise ValueError; they do not fall back to a fresh mutation. Do not populate continuation IDs from model tool arguments. Hosts remain responsible for deriving session/user scope from their authenticated caller, and for distinguishing continuation from unrelated work.

Authenticated /api/trigger callers can pass optional delivery_id and logical_turn_id strings, each 1–128 characters. Invalid field types/lengths return 400; an unavailable continuation returns 409. JSON responses, NDJSON done, and SSE run_completed include logical_turn_id when one exists. Retry an interrupted request with its original delivery ID, including after a runtime restart. /api/runs/{run_id}/events?after=N only replays the existing stream journal and never invokes the engine. That journal is not restart-durable; the mutation checkpoint is.

Host metadata and reconciliation API

engine.mutation_state(message) returns None when this delivery has no ledger, or a detached snapshot with this shape (mutation keys are their IDs):

{
  "id": "server-generated-logical-turn-id",
  "user_id": "authenticated-user",
  "delivery": "sha256-delivery-identity",
  "deliveries": ["sha256-delivery-identity"],
  "finished": false,
  "mutations": {
    "mutation-sha256": {
      "id": "mutation-sha256",
      "tool": "mcp__dashboards__create",
      "status": "completed",
      "metadata": {
        "id": "dashboard-id",
        "title": "Checkout errors",
        "url": "https://example.test/dashboards/dashboard-id",
        "normalized_title": "checkout errors",
        "scope_identity": "host-normalized-scope"
      }
    }
  }
}

delivery identifies the initial request; deliveries includes every admitted continuation delivery. Older records without deliveries retain their initial delivery as the sole known identity. finished only records that narration was persisted; it is not evidence about individual mutations. metadata is optional. Its only supported fields are the five string fields above, with a combined serialized UTF-8 limit of 4096 bytes. The host owns validation, normalization, and exclusion of credentials/raw data.

engine.attach_mutation_metadata(message, mutation_id, metadata)

This works during tool execution, before narration, and after restart. Native handlers and MCP host hooks can access the live message through get_tool_route_context(ctx).incoming_message. Attach resource metadata as soon as authoritative results are available; a missing metadata field is not proof that no resource was created. Completed-turn artifacts reserve turn_artifact.mutation_ledger with schema {"version": 1, "logical_turn_id": "...", "mutations": [...]}. Existing attach_turn_artifact calls preserve this section, and late metadata updates refresh artifacts for exactly this logical turn.

For an unknown outcome, stop or establish termination of the original dispatcher and obtain authoritative external readback before calling:

engine.reconcile_mutation(message, mutation_id, completed=True)

completed=True confirms the effect and blocks automatic replay while the logical-turn receipt is retained. completed=False asserts confirmed non-execution; it restores approved, requiring a new approval on the next attempt. Only unknown/started entries can be reconciled. Never use an error message, missing metadata, a model inference, or a timeout as evidence of non-execution. This is a trusted host API, not an LLM tool or an HTTP reconciliation endpoint.

Storage, bounds, and ownership

The ledger lives in the existing session record and shares its per-session lock, agent ownership checks, erasure generation/epoch, and inactivity-retention fence. Use one owning Engine/SessionManager per session; this contract does not add multi-process compare-and-swap or a distributed lease to arbitrary state backends. In-process duplicate dispatch claims serialize without holding locks over tool I/O.

Admission of a new mutation delivery durably refreshes the session inactivity clock before dispatch, including a new delivery continuing an existing logical turn. This prevents a nearly expired previous conversation from expiring the new receipt during execution. The session stores this clock as last_message_at, falling back to existing turn timestamps for older records. Ordinary turns advance it on completion; mutation narration and reads do not extend retention. Repeated deliveries do not refresh it, even when recovering interrupted narration.

Limits are 50 logical mutation-bearing turns per session, 32 mutations and 50 distinct admitted deliveries per logical turn. At the delivery limit, known redeliveries remain available but new continuation deliveries are refused; no delivery identities are discarded to make room. When admitting a fresh logical turn at the session limit, the oldest finished turn whose mutations are all settled is pruned to make room. Finished means narration was persisted; unfinished turns and turns with approved, started, or unknown mutations are retained. If none can be pruned, new mutation execution is refused. Ordinary read-only turns do not consume these slots.

Replay protection lasts only while the logical-turn receipt is retained. Hosts must reject stale delivery IDs outside this window: redelivery of a pruned ID would create a fresh logical turn and can execute again. Explicit continuation of a pruned logical-turn ID fails closed. Session erasure and configured inactivity expiry also retire ledger content alongside conversation history.

Production state backends must acknowledge durable writes and raise on failure. The filesystem backend writes a temporary file, fsyncs it, atomically replaces the record, and fsyncs the containing directory. MemoryStateBackend remains suitable only for tests or intentionally ephemeral deployments; it cannot provide restart recovery. Release preparation uses the normal master version-bump workflow; publishing occurs after merge and successful release CI.