Remote Dispatch Runbook

Remote dispatch lets one SASE controller discover, enroll, and operate agents on another machine. The supported setup path is explicit and credentialed:

1. Prepare the target machine and run its gateway on loopback.
2. Expose that loopback gateway through a private tailnet HTTPS endpoint.
3. Issue a target-local one-time bootstrap bundle.
4. Enroll the target from the controller with sase machine init.
5. Verify authenticated status, then launch and manage remote agents with %dispatch:<alias>.

Tailnet membership is not authorization. Enrollment uses a single-use bootstrap secret issued by the target gateway's credential store, and ordinary fleet calls use the credential written into the controller's protected SASE state.

Install

Install SASE normally on each machine that will act as a target. The target needs the Python sase command and the packaged Rust entry points in the same installed tool environment:

uv tool install --reinstall sase
uv tool dir
"$(uv tool dir)/sase/bin/sase" version
"$(uv tool dir)/sase/bin/sase_gateway" --help
"$(uv tool dir)/sase/bin/sase_federation_worker" --help

On machines where noninteractive SSH does not load the uv-tool bin directory, use the absolute paths under $(uv tool dir)/sase/bin/. The dispatch resolver checks that installed-venv directory for sase_gateway and sase_federation_worker; the shell PATH does not need to expose those commands.

Restart AXE after updating a target or controller install:

sase axe stop
sase axe start

sase axe ensure is also acceptable when you want to heal a stopped orchestrator without forcing a stop/start cycle.

Run The Gateway

Run the gateway as the same OS user and with the same SASE home that will issue the bootstrap bundle. Keep it bound to loopback unless you are doing a short, deliberate LAN smoke test:

sase_gateway --bind 127.0.0.1:7629 --sase-home ~/.sase
curl -fsS http://127.0.0.1:7629/api/v1/health

The public health response must include a fleet.supported_protocol_versions list for automatic compatibility classification. Older gateways without that field can still be manually enrolled when the authenticated protocol negotiation accepts the controller, but discovery will report compatibility as unknown.

Linux Supervision

For a user-systemd target, run the installed gateway from the uv-tool environment:

TOOL_DIR="$(uv tool dir)"
systemd-run --user --unit=sase-gateway \
  --property=Restart=on-failure \
  --property=Environment="PATH=$HOME/.local/bin:$TOOL_DIR/sase/bin:/usr/bin" \
  "$TOOL_DIR/sase/bin/sase_gateway" \
  --bind 127.0.0.1:7629 \
  --sase-home "$HOME/.sase" \
  --agent-bridge-command "$HOME/.local/bin/sase"

systemctl --user status sase-gateway --no-pager
curl -fsS http://127.0.0.1:7629/api/v1/health

Use the host's normal unit naming if it already has a permanent service. The important properties are loopback bind, the correct SASE home, restart-on-failure behavior, and an sase agent-bridge command that systemd can exec. User units do not include ~/.local/bin on PATH by default, so pass --agent-bridge-command with the absolute installed sase path (or put that directory on the unit PATH). Without it, authenticated hello can succeed while remote launch fails with agent_bridge unavailable.

macOS Supervision

For macOS, prefer a LaunchAgent owned by the same user. Replace TOOL_DIR with the absolute uv tool dir result for that machine:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
  "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
  <key>Label</key>
  <string>sh.sase.gateway</string>
  <key>ProgramArguments</key>
  <array>
    <string>TOOL_DIR/sase/bin/sase_gateway</string>
    <string>--bind</string>
    <string>127.0.0.1:7629</string>
    <string>--sase-home</string>
    <string>/Users/YOU/.sase</string>
  </array>
  <key>KeepAlive</key>
  <true/>
  <key>RunAtLoad</key>
  <true/>
</dict>
</plist>

Load it with launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/sh.sase.gateway.plist and inspect it with launchctl print gui/$(id -u)/sh.sase.gateway.

Expose Through Tailscale Serve

The gateway should stay on 127.0.0.1; Tailscale Serve terminates HTTPS inside the tailnet and proxies to the loopback port:

tailscale serve --bg --yes 7629
tailscale serve status
curl -fsS https://TARGET.tailnet-name.ts.net/api/v1/health

If the CLI prints a login.tailscale.com/f/serve?... URL, Tailscale Serve or HTTPS certificates are not enabled for the tailnet or node. Open the URL while signed in as a tailnet owner/admin and enable Serve before retrying. A successful setup shows a Serve config and the MagicDNS HTTPS health URL answers with the gateway health JSON.

Do not use Tailscale Funnel for remote dispatch. SASE's gateway is intended to remain inside a private tailnet or equivalent private access layer.

Issue A Bootstrap

Issue the bundle on the target, as the gateway user, against the gateway's SASE home:

umask 077
sase machine bootstrap --json > /tmp/sase-apollo-bootstrap.json

The file contains a live single-use secret. Do not pass it in argv, paste it into logs, or record it in a bead note. Move it to the controller over a protected channel and delete both temporary copies after enrollment. The default expiry is the gateway store's short TTL, so issue the bundle immediately before running init.

Enroll From The Controller

From the controller, initialize with the bundle file. Canonical init discovers, selects, enrolls, deploys, and verifies; sase machine discover remains available as an explicit inspection command:

sase machine init -B /path/to/bootstrap.json
sase machine list -j
sase machine status TARGET -j
sase doctor -D -C dispatch

sase machine init --check and sase init --check --json are offline checks; they do not discover peers or talk to gateways. Interactive bare sase init may offer the first explicit machine review before a local acknowledgment exists. Once a successful explicit sase machine init review has completed, bare sase init, sase init --all, and sase init --project use only local review state: they do not rescan discovery providers, prompt for newly appearing machines, or surface provider health diagnostics. Explicit sase machine init always performs discovery, preserves diagnostics from failed providers beside candidates from working ones, shows already enrolled machines beside new candidates, writes the machine record and credential, deploys the chezmoi-managed overlay through a tracked apply when configured, reloads config, and runs an authenticated hello before declaring success. Direct sase machine add and sase machine repair share that same activation path.

After a successful explicit init review, the controller records a machine-local acknowledgment in SASE state. The record may include candidates you skipped or no candidates at all; either way it marks the first review complete. It does not grant trust, sync to other machines, or update enrollment pins. Run sase machine discover whenever you want an explicit provider scan, sase machine init whenever you want to review or enroll discovered candidates, sase machine repair TARGET to rotate a mismatched or quarantined target, and sase machine status to check configured machines.

If enrollment partially succeeds after the target consumes the bootstrap, follow the command's recovery text. Retry a failed local apply when the credential is already stored; issue a fresh target bundle and run sase machine repair TARGET when the target credential needs to rotate. Repair keeps the still-applied credential until replacement activation succeeds. A submitted chezmoi apply whose outcome cannot be observed is reported in progress by proc identity and is never followed by an untracked second apply.

The machine command group is deliberately split between offline inventory, explicit network work, and local mutations:

Command	Behavior
sase machine / list	Read configured aliases only; never contacts a provider or gateway.
sase machine discover	Query configured discovery providers explicitly; -p is repeatable.
sase machine bootstrap	Issue a target-local, single-use bundle; the secret is written only to stdout.
sase machine init	Interactively discover, select, enroll, reload config, and require an authenticated hello.
sase machine add	Enroll a named endpoint or discovered candidate, optionally recording an SSH handoff target.
sase machine show	Show one local machine record, including the effective SSH handoff target.
sase machine status	Run bounded authenticated hello checks for selected aliases, or every alias when none are given.
sase machine repair	Rotate a quarantined or mismatched enrollment with a fresh one-time bundle and activate it.
sase machine rename / remove	Change viewer-local alias state; removal also deletes the local credential reference.

sase machine add -S/--ssh-target records the SSH destination that terminal handoffs, such as remote sudo requests, use for that machine. It defaults to the alias itself; sase machine show ALIAS prints the effective target.

--bootstrap-file is available on init, add, and repair. The init workflow always requires an interactive stdin for candidate selection and alias prompts; it gets the bundle from --bootstrap-file or a hidden prompt, not from piped stdin. The lower level add and repair commands can instead read the bundle from a pipe when --bootstrap-file is omitted. Bootstrap secrets are never accepted as command-line values.

Launch And Operate

After status is healthy, launch remote agents by adding a dispatch selector to a normal launch prompt:

sase run "%dispatch:apollo summarize the current project state; do not change files"

The equivalent parenthesized form is %dispatch(apollo). Exactly one selector is allowed, and %dispatch:local is reserved — omit the directive for a local launch. V1 remote launch does not combine with %wait, %queue, or %clan. The controller strips only the dispatch selector, so other launch directives are processed on the target.

In sase's TUI, gD from prompt NORMAL mode or Ctrl+G D from INSERT mode opens the Launch Target picker. It lists here plus every enrolled alias. Local enrollment data labels each non-quarantined remote ok and enables it; quarantined rows remain visible but disabled. This is an eligibility label, not a gateway-health result—the picker performs no network probe. Choosing a remote inserts or replaces the single dispatch selector; choosing here removes it. The prompt's Target/Source context line makes the selected owner and portable source explicit before submission.

Remote launch carries portable project evidence rather than the controller's local paths. A trusted launch integration can supply a Patch reference or explicit revision in the durable request payload. Merely mentioning a Patch or xprompt in the prompt does not supply that source-side evidence. Without payload evidence, including for an ordinary sase run invocation, the controller requires a clean Git checkout, a current branch with an upstream, and a HEAD that is already published. This check happens before the target processes the remaining directives. Local-only payloads — collected inputs, resolved launch units, attachments, files, and images — are rejected before submission. A submitted request is durable and idempotent; an acceptance-uncertain response can be retried without intentionally duplicating the launch.

sase's TUI consumes the same fleet records. sase tui --tmux prints the tmux target for the session. Open Admin Center with # and press 3 for the Machines tab: it reads the local controller and enrolled aliases without probing the network, and s runs a bounded authenticated hello only for the selected remote. Its connect, repair, rename, and remove actions show persistent CLI guidance rather than mutating immediately; Enter returns to Agents with a machine:<alias> filter.

Once a machine is enrolled, the Agents tab shows local and remote rows in one list. Remote agent, family, and clan nodes carry a host-alias chip such as apollo or mac; local rows never carry a here chip. Group with o until the header says by machine to render the here machine banner first, then remote aliases, each split into status subgroups.

• The list loads the bounded remote catalog across enrolled machines and keeps host failures visible as diagnostics rather than hiding healthy hosts.
• Remote rows are grouped into the same family and clan nodes as local rows, using the membership the owning machine reports. Member shells nest under their remote parent and do not repeat its host chip.
• A host whose feed is invalid or served from a stale cache says so loudly. The header names the alias and error (for example apollo: feed invalid: … (cached 5m ago), or 2 machines with feed errors), its by machine banner adds feed invalid or stale · cached 5m ago, its rows add a feed invalid or stale marker, and the detail panel shows a Feed error: line. A remote WAS RUNNING row also carries a last seen … ago label.
• Remote rows show the owning machine's authored capacity and weight as cN / wN badges and Capacity: / Weight: detail lines, including c0 for a persisted legacy zero-capacity record. Their load is never added to the C/L capacity in sase's TUI header total, which counts only the controller's own runners.
• Remote stop, retry, fork, bounded content, machine status, launch-outcome checks, and pending question/gate actions appear only when the selected row advertises the matching capability.
• Pending remote questions and gates are reconciled through the durable notification inbox rather than inferred from row-follow state. They resurface while the request remains pending, and a host outage does not dismiss them; see Remote Attention.

Prompt submission first validates portable source proof off the TUI event loop. A preflight failure keeps and refocuses the draft. After preflight passes, sase's TUI shows a provisional QUEUED owner row before the background launch settles. A structured accepted response keeps it QUEUED, while a settled response changes it to STARTING. When the launch finishes without a structured dispatch result—including the current rejection and failed-receipt paths—the row becomes outcome-unknown WAITING; run Agents: check dispatch launch outcome from the command palette. The provisional disappears when the target's authoritative fleet row arrives with the matching logical or exact locator.

Those operations are journaled through sase machine agent and sase machine attention. Use sase's TUI command palette or configure the corresponding ace.keymaps.app fields for direct keys.

To prove restart resilience, restart the target gateway service, then rerun:

sase machine status apollo
sase run "%dispatch:apollo report hostname and SASE version; do not change files"

The enrollment should survive a gateway process restart. Reissue a bootstrap only for a new or repaired enrollment, not for ordinary gateway restarts.

After upgrading sase or sase-core-rs on a target whose sase_gateway runs under a supervisor, restart that gateway process before trusting fleet status from another machine. Units with Restart=on-failure keep the old, still-healthy binary running after an install upgrade.

sase machine status TARGET appends version details to each healthy hello. Normally that is versions: sase-gateway X.Y.Z, …, fleet contract schema vN; a gateway too old to report its own version shows sase-gateway unknown, and a gateway too old to advertise a fleet-contract version shows fleet contract unknown. Capability-set schema (capabilities.schema_version) is independent of the fleet-data contract; matching sase-core-rs builds with capability v1 and fleet v4 are not skew. When the gateway's package version differs from the controller's installed sase-core-rs, or the advertised fleet-contract schema versions differ, the line instead reads version skew: …; restart target gateway. That warning usually means the supervised gateway is still serving the pre-upgrade binary. With -j/--json, each status row carries the same facts as gateway_version, service_versions, capability_schema_version, fleet_contract_schema_version, and a version_skew list.