Lattice Fleet

Fleet is a separate control plane for many independent Lattice DDI clusters. Operators open one website and search, inbox, and proxy into each site’s console. Each site still runs its own Lattice cluster (IPAM, DHCP, DNS). Fleet does not hand out leases.

This is a different tarball from the cluster package (lattice-*-linux-amd64.tar.gz). The installer is fleet-install, not lattice-install. On a terminal with no --mode, that is the wizard (default: single-node, local Postgres). Non-interactive installs pass --mode.

If you administer Windows / AD and not Linux: windows-admin (SSH vs RDP, WinSCP, sudo). Those habits are the same; this page is the Fleet-specific path. Cluster DHCP/HA is deploy and ha — not this file. Rolling upgrade (Fleet first, then clusters; do not skip more than two cluster cuts under this Fleet): upgrade.

Just show me the steps — copy-paste at the bottom, without the lecture.

Vendor documentation (UIs, connection strings, and load-balancer flags change). Use these for the latest checkboxes; this page is what Lattice expects.

Two or more Fleet nodes behind a load balancer: fleet-ha.

1. What problem this solves

You already run Lattice at more than one site (or you will). Each site is its own cluster, its own Postgres, its own relays, its own entitlement. Opening every cluster URL to hunt a MAC is the pain.

Fleet sits in one place (your DC or a cloud VM). A small agent on each cluster opens an outbound WebSocket to Fleet. You do not punch an inbound hole at the branch. From Fleet you see health tiles, search across sites, and click through into that cluster’s console.

  operator browser  -->  Fleet (:8090, or HTTPS on your load balancer)
                              ^
                              | outbound WebSocket
                         cluster agent (on the Lattice VM)
                              |
                         local cluster API (127.0.0.1:8080)

What this is not

Community Fleet without a paid file is 2 clusters. Each of those clusters still needs its own Lattice DDI entitlement. Paid Fleet file: /etc/lattice/fleet.entitlement.json (max_clusters).

2. Postgres

Fleet is a manager of managers (HQ console and registry). A Fleet outage does not stop DHCP at the sites. Relays still point at each cluster.

sslmode=require (or verify-full) on a BYO DSN in production. Local single-node uses sslmode=disable on 127.0.0.1.

Single-node backups are the product. If this VM dies without a dump, you rebuild Fleet and re-enroll every site. DHCP still works; agent keys do not. At least daily:

sudo -u postgres pg_dump -Fc fleet -f /root/fleet.dump
# also copy /etc/lattice/fleet.env (admin token + encryption key)

Restore the dump and fleet.env onto a new VM so sites reattach with the same keys. Failover of a BYO writer is the database product’s job — Patroni or RDS. Lattice still assumes one writer.

Sizing for the Fleet VM itself: 2 CPU, 4 GB RAM, 20 GB disk for ≤50 clusters.

3. Install the Fleet server

Ubuntu Server 22.04 or 24.04 (derivatives on those LTS bases are best-effort). One virtio NIC, static IPv4, OpenSSH, NTP — same host habits as windows-admin. You do not need UDP 67 or trusted relays on this VM.

Download lattice-fleet-*-linux-amd64.tar.gz (or arm64) from Downloads. From a Windows PC, copy the .tar.gz to /tmp on the Ubuntu VM with WinSCP (or PowerShell scp), then in SSH:

sudo tar -C /tmp -xzf /tmp/lattice-fleet-*-linux-amd64.tar.gz
sudo /tmp/lattice-fleet-*-linux-amd64/scripts/fleet-install

No flags on a terminal: wizard. Default is single-node and create local Postgres. Type b to go back a screen. HA is a choice, not the default — that path asks for a DSN you already created, then you run the same unit on a second VM (fleet-ha).

Non-interactive single node (same as wizard default):

sudo /tmp/lattice-fleet-*-linux-amd64/scripts/fleet-install --mode server

Omit --pg-dsn and the installer installs PostgreSQL on this VM. Omit --admin-token and it generates one and prints it. BYO database: pass --pg-dsn 'postgres://fleet:SECRET@pg-host:5432/fleet?sslmode=require'.

--mode ha-node is the same unit as server but requires --pg-dsn, --admin-token, and --credential-key. It will not create Postgres.

Save the admin token (wizard / --admin-token / the generated line). That string is the bootstrap sign-in for the Fleet console. OIDC can replace it later.

The process listens on :8090 HTTP by default (--listen to change). Health: GET /readyz. TLS is typically on a load balancer, not baked in like community cluster Caddy on :443. The example HAProxy in the tarball (deploy/ha/fleet-haproxy.cfg) is plain HTTP :8090 with option httpchk GET /readyz.

Place a paid lattice.fleet.entitlement.json at /etc/lattice/fleet.entitlement.json or pass --entitlement.

If the host already has Fleet and you run the installer on a TTY with no --mode, the wizard offers Upgrade this node.

4. Enroll a cluster agent

The agent ships in the Fleet tarball, not the cluster tarball. Copy the same archive to the cluster Ubuntu VM (WinSCP), SSH in.

1. In Fleet: Clusters → register this site. Copy the one-time fleet_enroll_… token.
2. On the cluster console: Settings → Fleet management. Paste Fleet URL + enroll token (local API stays http://127.0.0.1:8080 unless you changed it). Save. You get a systemd snippet.
3. Install the agent from the Fleet tarball:

sudo tar -C /tmp -xzf /tmp/lattice-fleet-*-linux-amd64.tar.gz
sudo /tmp/lattice-fleet-*-linux-amd64/scripts/fleet-install \
  --mode agent \
  --fleet-url https://fleet.example.com \
  --enroll-token 'fleet_enroll_…' \
  --local-api http://127.0.0.1:8080

Install after the cluster is up, on one control-plane VM. Current --mode agent does not change permissions on that machine’s Lattice files. On Fleet 0.1.0-beta.18 and 0.1.0-beta.17 that chmod hid the paid entitlement, so HA on the site failed until you fixed the files — upgrade the agent from this archive. If that VM is down, DHCP at the site still works; enroll on another node when you want the Fleet console back.

Replace an agent. --mode agent --force with a new enroll token starts a new identity. A leftover /var/lib/lattice/fleet-agent.key is discarded so the new token is used.

4. After the first connect the agent writes fleet_agent_… to /var/lib/lattice/fleet-agent.key (cluster API token: /var/lib/lattice/fleet-agent.local-token). You can still set --agent-key + --force to persist a key you already have. Remove FLEET_ENROLL_TOKEN once the persistent key exists. Config: /etc/lattice/fleet-agent.env. Status: /var/lib/lattice/fleet-agent-status.json. Optional TLS: FLEET_CA_FILE (extra PEM roots) and FLEET_CA_HASH (SHA-256 pin of the Fleet server cert).

The agent talks outbound to Fleet. Open egress (and your corporate proxy, if any) from the cluster site; do not publish the cluster API to the internet for this path.

5. Open the Fleet console

Browser: http://<fleet-host>:8090/ (or https://fleet.example.com if the load balancer terminates TLS). Sign in with FLEET_ADMIN_TOKEN, or SSO when OIDC is configured under Settings.

Identity

IdP groups like fleet-cluster:acme (prefix configurable) or DB grants via the API. Break-glass mints a short-lived cluster operator token (audited).

SSO is operator login, not AD DNS. Vendor setup: Entra / Okta / AD FS (table at the top). Same distinction as first-login.

6. Upgrade (Fleet VM and agents)

Patches and new minors use the same command. Run the new tarball’s installer, not fleet-install already on PATH. From a Windows PC: copy the .tar.gz with WinSCP or PowerShell scp, then SSH in:

sudo tar -C /tmp -xzf /tmp/lattice-fleet-*-linux-amd64.tar.gz
sudo /tmp/lattice-fleet-*-linux-amd64/scripts/fleet-install --mode upgrade

Overview tiles show agent … — upgrade when the agent does not match this Fleet. Cluster software is a different archive — that cluster’s Help → Upgrade this node.

Order when you also upgrade clusters: this Fleet VM (HA: one node at a time), then agents rolling, then each cluster. Do not skip more than two cluster cuts under this Fleet. upgrade.

Do not downgrade. Rotate enroll tokens only when replacing agent identity.

7. Outbound HTTP proxy (agent)

Some sites require all internet egress through a corporate proxy. The agent only uses the proxy for traffic to Fleet (WebSocket and search-index upload). Calls to the local cluster API (FLEET_LOCAL_API) stay direct.

# unauthenticated
export FLEET_HTTP_PROXY=http://proxy.corp.example:8080

# credentials in the URL, or split (preferred for a secrets manager)
export FLEET_HTTP_PROXY=http://proxy.corp.example:8080
export FLEET_PROXY_USER=svc-fleet
export FLEET_PROXY_PASSWORD=...

# optional bypass
export FLEET_NO_PROXY=fleet.corp.example,*.corp.example

Put those in /etc/lattice/fleet-agent.env and restart the agent. Existing FLEET_AGENT_KEY is kept.

8. License

Separate file from the cluster entitlement. Missing Fleet file → Community (2 clusters). Each managed cluster still needs its own Lattice DDI entitlement.

Fleet unique-enforces one paid cluster license_id per active Fleet row (empty/community are unlimited). A second agent with a license already claimed is disconnected; Overview shows License conflicts. fleet_admin can Steal license seat on that cluster’s registry settings (audited), or remove the other cluster.

9. Connectivity modes

Webhooks: register the cluster in Fleet (response includes webhook.url and webhook.secret). On the cluster, create a webhook pointing at that URL with the same secret. Events land in Fleet Inbox. Set FLEET_PUBLIC_URL so generated URLs use your external hostname. HMAC is RFC 2104; cluster webhook how-to: webhooks.

Search index: agents push lease summaries (IP, MAC, hostname — not full payloads) to PUT /fleet/v1/agent/index. /search uses the index when it is fresher than 20 minutes. Disable: FLEET_INDEX_SYNC_INTERVAL=0 (default 5m). Occupancy History is always live fan-out of GET /api/v1/history (as-of needs the cluster lease table).

10. Security notes

• Connectivity settings (api_base_url, ssh_config, api_token) require a Fleet admin.
• Connect URLs must be HTTPS (HTTP allowed for loopback only). RFC1918 is allowed. Link-local and cloud metadata (169.254.169.254, fd00:ec2::254) are blocked.
• SSH tunnel requires a pinned host_key (base64-encoded SSH public key) in ssh_config.
• Agent → cluster API uses the cluster token sent once at enroll (/var/lib/lattice/fleet-agent.local-token) or FLEET_LOCAL_API_TOKEN. The agent key is persisted at /var/lib/lattice/fleet-agent.key.
• Stored secrets encrypt at rest when FLEET_CREDENTIAL_KEY is set (32-byte key, base64).
• Auth rate limits: 30 req/min/IP on SSO and bootstrap auth; 60 req/min/IP on agent connect.
• Set FLEET_ALLOWED_ORIGINS in production when Fleet is served from a known hostname.

11. Environment reference

Examples in the tarball: deploy/examples/fleet.env.example, fleet-agent.env.example.

12. API reference

Base: http://<fleet-host>:8090/fleet/v1
Auth: Authorization: Bearer <FLEET_ADMIN_TOKEN>

Agent WebSocket: ws://<fleet>/fleet/v1/agent/connect (or wss:// through TLS) with Authorization: Bearer fleet_enroll_… or fleet_agent_….

Related

• fleet-ha — two+ Fleet nodes, shared Postgres, load balancer
• deploy — install a cluster (DHCP/IPAM), not Fleet
• windows-admin — SSH, WinSCP, sudo
• REPORTING.md — support bundle; do not paste admin tokens

Just the steps

Copy lattice-fleet-*-linux-amd64.tar.gz to /tmp (WinSCP from Windows). Fleet is a control plane; DHCP stays on each cluster.

Single node (default — installer creates local Postgres):

sudo tar -C /tmp -xzf /tmp/lattice-fleet-*-linux-amd64.tar.gz
sudo /tmp/lattice-fleet-*-linux-amd64/scripts/fleet-install
# or: --mode server
curl -sf http://127.0.0.1:8090/readyz && echo OK

Save the printed admin token. Back up pg_dump + /etc/lattice/fleet.env.

HA (you already created Postgres): two VMs, same DSN, then fleet-ha.

sudo /tmp/lattice-fleet-*-linux-amd64/scripts/fleet-install --mode ha-node \
  --pg-dsn 'postgres://fleet:SECRET@writer:5432/fleet?sslmode=require' \
  --admin-token '<same-on-both>' \
  --credential-key '<same-on-both>' \
  --public-url https://fleet.example.com

Browser: http://<fleet-host>:8090/ — sign in with that admin token.

Agent on a cluster VM (same tarball). In Fleet: register the cluster, copy fleet_enroll_…. Cluster console: Settings → Fleet management. Then:

sudo tar -C /tmp -xzf /tmp/lattice-fleet-*-linux-amd64.tar.gz
sudo /tmp/lattice-fleet-*-linux-amd64/scripts/fleet-install \
  --mode agent \
  --fleet-url https://fleet.example.com \
  --enroll-token 'fleet_enroll_…' \
  --local-api http://127.0.0.1:8080

After connect, save fleet_agent_… in Settings (or --agent-key + --force).

Upgrade (Fleet node or agent host):

sudo tar -C /tmp -xzf /tmp/lattice-fleet-*-linux-amd64.tar.gz
sudo /tmp/lattice-fleet-*-linux-amd64/scripts/fleet-install --mode upgrade

HA Fleet: fleet-ha. DHCP stays on the cluster tarball (deploy). Fleet-first then rolling cluster: upgrade.