Remote Dispatch Runbook¶

Remote dispatch lets one SASE controller discover, enroll, and operate agents on another machine. The supported setup path is explicit and credentialed:

Prepare the target machine and run its gateway on loopback.
Expose that loopback gateway through a private tailnet HTTPS endpoint.
Issue a target-local one-time bootstrap bundle.
Enroll the target from the controller with sase machine init.
Verify authenticated status, then launch and manage remote agents with %dispatch:<alias>.

Tailnet membership is not authorization. Enrollment uses a single-use bootstrap secret issued by the target gateway's credential store, and ordinary fleet calls use the credential written into the controller's protected SASE state.

%dispatch launch requests do not forward SASE_INSTRUCTIONS_FILE. The target renders instruction bundles with its own installed code; no instruction bytes cross the wire.

Install¶

Install SASE normally on each machine that will act as a target. The target needs the Python sase command and the packaged Rust entry points in the same installed tool environment:

uv tool install --reinstall sase
uv tool dir
"$(uv tool dir)/sase/bin/sase" version
"$(uv tool dir)/sase/bin/sase_gateway" --help
"$(uv tool dir)/sase/bin/sase_federation_worker" --help

On machines where noninteractive SSH does not load the uv-tool bin directory, use the absolute paths under $(uv tool dir)/sase/bin/. The dispatch resolver checks that installed-venv directory for sase_gateway and sase_federation_worker; the shell PATH does not need to expose those commands.

Restart the scheduler after updating a target or controller install:

sase scheduler restart

Run The Gateway¶

Run the gateway as the same OS user and with the same SASE home that will issue the bootstrap bundle. Keep it bound to loopback unless you are doing a short, deliberate LAN smoke test:

sase_gateway --bind 127.0.0.1:7629 --sase-home ~/.sase
curl -fsS http://127.0.0.1:7629/api/v1/health

The public health response must include a fleet.supported_protocol_versions list for automatic compatibility classification. Older gateways without that field can still be manually enrolled when the authenticated protocol negotiation accepts the controller, but discovery will report compatibility as unknown.

Supervision Via The Service Host¶

Supervise the target gateway with the sase service host instead of a hand-written unit. Register the host's native unit — a systemd user unit on Linux, a LaunchAgent on macOS — then enable the built-in gateway service proc (disabled by default, enabled per machine in the overlay) and start it:

sase service init --yes
sase service proc enable gateway
sase mobile gateway start
sase service proc show gateway
curl -fsS http://127.0.0.1:7629/api/v1/health

The important properties are loopback bind, the correct SASE home, restart-on-failure behavior, and an sase agent-bridge command the host can exec. When the service host supervises the gateway, it passes the resolved sase executable (the one running the service host) as the agent-bridge and helper-bridge command automatically, and mobile_gateway.agent_bridge_command is ignored. The setting applies only when you run the gateway yourself in the foreground with sase mobile gateway start; there, on machines where noninteractive SSH does not load the uv-tool bin directory, set it to the bare absolute sase path (not sase mobile agent-bridge — the gateway appends the subcommand itself). Without a usable bridge command, authenticated hello can succeed while remote launch fails with agent_bridge unavailable.

Expose Through Tailscale Serve¶

The gateway should stay on 127.0.0.1; Tailscale Serve terminates HTTPS inside the tailnet and proxies to the loopback port:

tailscale serve --bg --yes 7629
tailscale serve status
curl -fsS https://TARGET.tailnet-name.ts.net/api/v1/health

If the CLI prints a login.tailscale.com/f/serve?... URL, Tailscale Serve or HTTPS certificates are not enabled for the tailnet or node. Open the URL while signed in as a tailnet owner/admin and enable Serve before retrying. A successful setup shows a Serve config and the MagicDNS HTTPS health URL answers with the gateway health JSON.

Do not use Tailscale Funnel for remote dispatch. SASE's gateway is intended to remain inside a private tailnet or equivalent private access layer.

Issue A Bootstrap¶

Issue the bundle on the target, as the gateway user, against the gateway's SASE home:

umask 077
sase machine bootstrap --json > /tmp/sase-apollo-bootstrap.json

The file contains a live single-use secret. Do not pass it in argv, paste it into logs, or record it in a bead note. Move it to the controller over a protected channel and delete both temporary copies after enrollment. The default expiry is the gateway store's short TTL, so issue the bundle immediately before running init.

Enroll From The Controller¶

From the controller, initialize with the bundle file. Canonical init discovers, selects, enrolls, deploys, and verifies; sase machine discover remains available as an explicit inspection command:

sase machine init -B /path/to/bootstrap.json
sase machine list -j
sase machine status TARGET -j
sase doctor -D -C dispatch

sase machine init --check and sase init --check --json are offline checks; they do not discover peers or talk to gateways. Interactive bare sase init may offer the first explicit machine review before a local acknowledgment exists. Once a successful explicit sase machine init review has completed, bare sase init, sase init --all, and sase init --project use only local review state: they do not rescan discovery providers, prompt for newly appearing machines, or surface provider health diagnostics. Explicit sase machine init always performs discovery, preserves diagnostics from failed providers beside candidates from working ones, shows already enrolled machines beside new candidates, writes the machine record and credential, deploys the chezmoi-managed overlay through a tracked apply when configured, reloads config, and runs an authenticated hello before declaring success. Direct sase machine add and sase machine repair share that same activation path.

After a successful explicit init review, the controller records a machine-local acknowledgment in SASE state. The record may include candidates you skipped or no candidates at all; either way it marks the first review complete. It does not grant trust, sync to other machines, or update enrollment pins. Run sase machine discover whenever you want an explicit provider scan, sase machine init whenever you want to review or enroll discovered candidates, sase machine repair TARGET to rotate a mismatched or quarantined target, and sase machine status to check configured machines.

If enrollment partially succeeds after the target consumes the bootstrap, follow the command's recovery text. Retry a failed local apply when the credential is already stored; issue a fresh target bundle and run sase machine repair TARGET when the target credential needs to rotate. Repair keeps the still-applied credential until replacement activation succeeds. A submitted chezmoi apply whose outcome cannot be observed is reported in progress by proc identity and is never followed by an untracked second apply.

See the generated machine CLI reference for the command inventory and exact options.

sase machine add -S/--ssh-target records the SSH destination that terminal handoffs, such as remote sudo requests, use for that machine. It defaults to the alias itself; sase machine show ALIAS prints the effective target.

--bootstrap-file is available on init, add, and repair. The init workflow always requires an interactive stdin for candidate selection and alias prompts; it gets the bundle from --bootstrap-file or a hidden prompt, not from piped stdin. The lower level add and repair commands can instead read the bundle from a pipe when --bootstrap-file is omitted. Bootstrap secrets are never accepted as command-line values.

Launch And Operate¶

After status is healthy, launch remote agents by adding a dispatch selector to a normal launch prompt:

sase run "%dispatch:apollo summarize the current project state; do not change files"

The equivalent parenthesized form is %dispatch(apollo). Exactly one selector is allowed, and %dispatch:local is reserved — omit the directive for a local launch. V1 remote launch does not combine with %wait, %queue, %clan, or %hold. The controller strips only the dispatch selector, so other launch directives are processed on the target.

In sase's TUI, gD from prompt NORMAL mode or Ctrl+G D from INSERT mode opens the Launch Target picker. It lists local plus every enrolled alias. Local enrollment data labels each non-quarantined remote ok and enables it; quarantined rows remain visible but disabled. This is an eligibility label, not a gateway-health result—the picker performs no network probe. Choosing a remote inserts or replaces the single dispatch selector; choosing local removes it. The prompt's Target/Source context line appears once a remote is selected and makes the selected owner and portable source explicit before submission; it stays hidden for ordinary local launches.

Remote launch carries portable project evidence rather than the controller's local paths. A trusted launch integration can supply a Patch reference or explicit revision in the durable request payload. Merely mentioning a Patch or macro in the prompt does not supply that source-side evidence. Without payload evidence, including for an ordinary sase run invocation, the controller requires a clean Git checkout, a current branch with an upstream, and a HEAD that is already published. This check happens before the target processes the remaining directives. Local-only payloads — collected inputs, resolved launch units, attachments, files, and images — are rejected before submission. A submitted request is durable and idempotent; an acceptance-uncertain response can be retried without intentionally duplicating the launch.

sase's TUI consumes the same fleet records. sase tui --tmux prints the tmux target for the session. Open Admin Center with # and press 3 for the Machines tab: it reads the local controller and enrolled aliases without probing the network, and s runs a bounded authenticated hello only for the selected remote. Its connect, repair, rename, and remove actions show persistent CLI guidance rather than mutating immediately; Enter selects the machine's agent tab when the tab strip is visible, and otherwise returns to Agents with a machine:<alias> (machine:local for the local controller) filter; f always filters.

Once a machine is enrolled, the Agents tab shows local and remote rows in one list. Remote agent, session, and clan nodes carry a host-alias chip such as apollo or mac; local rows never carry a machine chip. Group with o until the header says by machine to render the local machine banner first, then remote aliases, each split into status subgroups. The Agents query spells it machine:local (machine:here still matches).

The list loads the bounded remote catalog across enrolled machines and keeps host failures visible as diagnostics rather than hiding healthy hosts.
Remote rows are grouped into the same session and clan nodes as local rows, using the membership the owning machine reports. Member turns nest under their remote parent and do not repeat its host chip.
The owning machine serves the same bounded recent window the local list shows (seven days, at most 200 completed rows). A completed plan-chain session has no separate root record, so its --plan turn (the one turn with no parent) stands in for the root and presents the session: every recent turn of it is served and the viewer nests them under one session row. Turns whose parent no record supplies, and artifact directories that carry no lifecycle marker at all, are never served. Workflow step rows are not agents and are not served, so a remote session's ×N can be lower than the owner's own, which also counts workflow steps.
A host whose feed is invalid or served from a stale cache says so loudly. The header names the alias and error (for example apollo: feed invalid: … (cached 5m ago), or 2 machines with feed errors), its by machine banner adds feed invalid or stale · cached 5m ago, its rows add a feed invalid or stale marker, and the detail panel shows a Feed error: line. A remote WAS RUNNING row also carries a last seen … ago label.
Remote rows show the owning machine's authored capacity and weight as cN / wN badges and Capacity: / Weight: detail lines, including c0 for a persisted compatibility zero-capacity record. Their load is never added to the load: gauge in sase's TUI Agents status row, which counts only the controller's own runners.
Remote stop, retry, fork, bounded content, machine status, launch-outcome checks, and pending question/gate actions appear only when the selected row advertises the matching capability.
Pending remote questions and gates are reconciled through the durable notification inbox rather than inferred from row-follow state. Dismissing one locally sticks for that revision, a new revision arrives as a fresh unread row, and a host outage does not dismiss them. Enter on a remote row with a pending request opens the same answer/approve flow; see Remote Attention.

Prompt submission first validates portable source proof off the TUI event loop. A preflight failure restores the prompt to the prompt bar with a source blocked: <reason> notice (when a modal is open it stashes the prompt and warns instead). After preflight passes, sase's TUI shows a provisional QUEUED owner row before the background launch settles. A structured accepted response keeps it QUEUED, while a settled response changes it to STARTING. When the launch finishes without a structured dispatch result—including the current rejection and failed-receipt paths—the row becomes outcome-unknown WAITING; run Agents: check dispatch launch outcome from the command palette. The provisional disappears when the target's authoritative fleet row arrives with the matching logical or exact locator.

Those operations are journaled through sase machine agent and sase machine attention. On a capable remote row, the normal Agents keys route to the remote operation: x stops, R retries, and F forks (it opens a Fork on <alias> prompt bar for the instruction). Bounded content, attention answers, and launch-outcome checks are in the command palette; their ace.keymaps.app fields (view_remote_agent_content, answer_remote_attention, check_dispatch_launch_outcome, plus a dedicated retry_remote_agent) are unbound by default.

From the shell, sase machine agent stop|retry ALIAS AGENT... and sase machine agent fork ALIAS AGENT [INSTRUCTION] look the agent up on the target including already-settled rows, matching AGENT exactly against the remote row's agent id, agent session id, or agent label. The command exits 0 only when every receipt is applied or already_settled. sase machine attention answer|approve ALIAS REQUEST acts on a pending remote question or gate; sase's TUI normally supplies the full request through a private request sidecar.

Stop, retry, and fork wait for a settled receipt inside the acceptance window: the command's -t/--timeout value, else dispatch.request_timeout_seconds, but never less than 30 seconds. A lost response is recovered with the same operation key. An outcome still unknown when the window ends is uncertain and must not be submitted again under a new key.

A fresh remote agent can be exact-stopped once the owner index has its record, including while the full fleet snapshot is still the previous cache.

A remote fleet stop does not dismiss the row. Killed and other served agent turns advertise retry and fork until they leave the recent-terminal presentation window. Owner dismissal still removes them.

To prove restart resilience, restart the target gateway service, then rerun:

sase machine status apollo
sase run "%dispatch:apollo report hostname and SASE version; do not change files"

The enrollment should survive a gateway process restart. Reissue a bootstrap only for a new or repaired enrollment, not for ordinary gateway restarts.

After upgrading sase or sase-core-rs on a target whose sase_gateway runs under a supervisor, restart that gateway process before trusting fleet status from another machine. Units with Restart=on-failure keep the old, still-healthy binary running after an install upgrade.

sase machine status TARGET appends version details to each healthy hello. Normally that is versions: sase-gateway X.Y.Z, …, fleet contract schema vN; a gateway too old to report its own version shows sase-gateway unknown, and a gateway too old to advertise a fleet-contract version shows fleet contract unknown. Capability-set schema (capabilities.schema_version) is independent of the fleet-data contract; matching sase-core-rs builds with capability v1 and fleet v4 are not skew. When the gateway's package version differs from the controller's installed sase-core-rs, or the advertised fleet-contract schema versions differ, the line instead reads version skew: …; restart target gateway. That warning usually means the supervised gateway is still serving the pre-upgrade binary. With -j/--json, each status row carries the same facts as gateway_version, service_versions, capability_schema_version, fleet_contract_schema_version, and a version_skew list.

Agent tabs (`%tab`) and dispatch¶

%tab:<name> travels with the prompt: the target stores the tab and the agent renders on that named tab on every machine. Lineage inheritance applies too — an agent running on a named tab that launches through %dispatch stamps %tab:<name> into the dispatched prompt, unless the prompt already has a %tab or is a session attach.

A %tab + %dispatch launch runs a version preflight with no network I/O: it reads the target's last-known fleet contract version from the federation cache-only host response. Agent tabs need fleet contract v7. When the cached version is older, the launch is refused with an upgrade hint; when it is unknown, the launch proceeds with a warning in the CLI result and a TUI toast. Upgrade controllers first, or the fleet in lockstep: an older controller reading a newer remote marks that host invalid, while a newer controller reading an older remote falls back to derived machine tabs and notes <alias>: tab data unavailable (upgrade sase).

Mobile gateway — loopback gateway and private remote access.
Machine initialization — enroll a target.
Integrations — other integration pages.

_[project]: A sase project is a named unit of work registered with SASE. _[workspace]: A sase workspace is a numbered clone of a project's primary repo, managed by the workspace store and tracked in that project's registry.json. _[agent]: A sase agent is an agent session or a single agent that does not belong to a session. _[prompt]: A prompt is the text SASE hands to a provider to start an agent turn. _[macro]: A macro is a reusable prompt triggered with #name in agent prompts. _[Patch]: A Patch is SASE's local unit of change. _[bead]: A bead is one tracked work item in SASE's git-native issue tracker. _[plan]: A plan is a persisted, approved artifact recording the intent behind agent work. _[TUI]: The TUI is SASE's terminal interface for running agents and reviewing results. _[epic]: An epic is an executable multi-phase plan run with sase bead work. _[gate]: A sase gate is a durable, command-backed request for one user decision, created when work must pause for the user. _[monitor]: A sase monitor is a monitor turn that runs one long command under a detached supervisor, so the command outlives the agent that started it. *[stitch]: A stitch is the lightweight ordered change record inside a Patch's STITCHES: section.