Fleet high availability

This page is Fleet availability: the control-plane website and registry. It is not cluster DHCP HA and it is not Microsoft DHCP failover.

• Cluster three-node Patroni / keepalived / UDP 67: ha.
• Single Fleet VM: fleet-deploy.
• Windows/SSH: windows-admin.

Just show me the steps — commands at the bottom.

Vendor documentation (use these for Patroni, RDS, and HAProxy flags; this page is how Lattice wires them).

1. What is highly available

lattice-fleet is stateless at the application layer. Two or more processes share one Fleet Postgres. DHCP and cluster IPAM never flow through Fleet — those stay on each site’s cluster. Fleet HA is console/API uptime and registry durability.

If you know Windows DHCP failover: two partners, a shared scope, MCLT. Fleet does none of that. There is no partner to click in the cluster console. Relays at a site still point at that cluster’s VIP or node (ha, relays).

If you know Lattice cluster HA: three Ubuntu VMs, etcd quorum, keepalived VIP, dhcpd on every node. Fleet HA is simpler and different:

--mode ha-node is an alias for server. Same systemd unit. The name is a reminder: same FLEET_PG_DSN, same bootstrap token (or OIDC), same --credential-key, load balancer in front. HA does not create Postgres. Bring a writer, then two Fleet VMs. The installer wizard defaults to single-node (local Postgres); pick HA only when the DSN already exists. A Fleet outage does not stop DHCP at the sites.

2. Target architecture

                    ┌─────────────┐
   Operators ──────►│ VIP / LB    │ :443 (TLS here)
                    └──────┬──────┘
                           │ HTTP :8090 to Fleet nodes
              ┌────────────┴────────────┐
              ▼                         ▼
      lattice-fleet A            lattice-fleet B
              │                         │
              └────────────┬────────────┘
                           ▼
              Patroni / RDS / your Postgres

The tarball’s deploy/ha/fleet-haproxy.cfg is plain HTTP on :8090 (option httpchk GET /readyz). Put TLS on the cloud LB or nginx in front if operators will use https://. LatticeDDI does not sell certificates.

Load balancer (active/standby)

The agent WebSocket lives in memory on one Fleet process. The tarball deploy/ha/fleet-haproxy.cfg sends all HTTP to the first healthy node and marks the second backup. Round-robin splits the socket from /clusters/{id}/proxy and 502s. Cloud LBs: one healthy target, or equivalent failover — not round-robin across live nodes.

OIDC SSO uses the fleet_session cookie. Any Fleet node can validate sessions (state is in Postgres).

Agent WebSocket

Agents reconnect to the LB hostname. All Fleet nodes share fleet_cluster agent keys in Postgres. Only one node holds the live WebSocket per cluster (in-memory hub). If that node dies, the agent reconnects to the remaining node. DHCP at the site does not notice.

3. Production install

1. Postgres (you bring this) — Patroni cluster or managed RDS/Aurora. One writer DSN for every Fleet node. The installer will not apt install postgresql on an HA node.
2. Fleet nodes — on each VM, from the Fleet tarball (fleet-deploy for WinSCP / extract):

sudo tar -C /tmp -xzf /tmp/lattice-fleet-*-linux-amd64.tar.gz
sudo /tmp/lattice-fleet-*-linux-amd64/scripts/fleet-install --mode ha-node \
  --pg-dsn 'postgres://fleet:SECRET@patroni-or-rds:5432/fleet?sslmode=require' \
  --admin-token '<same-on-all-nodes-or-use-oidc>' \
  --credential-key '<same-on-all-nodes>' \
  --public-url https://fleet.example.com \
  --allowed-origins https://fleet.example.com

3. Load balancer — point at both nodes’ :8090. Replace FLEET_NODE_A / FLEET_NODE_B in deploy/ha/fleet-haproxy.cfg.
4. Agents — --fleet-url https://fleet.example.com (the LB), never a single node’s IP.
5. Schema — fleet-install --mode upgrade on a node that already has fleet.env runs migrate. Manual once per cut: fleet-install --mode migrate --pg-dsn … --admin-token … on any one node.

Upgrade HA Fleet one node at a time. The other node keeps the console. DHCP is not on this path. fleet-deploy. Then agents, then each cluster (replica, replica, leader). Do not skip more than two cluster cuts under this Fleet: upgrade.

4. Sizing and backup

2 CPU / 4 GB RAM per Fleet node for ≤50 clusters. Size Postgres from Patroni or the managed-service guidance — not from cluster dhcpd rules.

• Fleet Postgres: registry, credentials, audit, alerts. Back it up like any control-plane database.
• Cluster DHCP/IPAM: still on each cluster’s Postgres. Fleet backup does not replace ha / cluster pg_dump.

Related

• fleet-deploy — what Fleet is, single-node install, agent enroll
• ha — paid cluster HA (DHCP/IPAM)
• deploy — cluster production install
• upgrade — Fleet first, then rolling cluster; two-cut window

Just the steps

Same FLEET_PG_DSN, admin token, and --credential-key on every Fleet VM. TLS on the load balancer.

sudo tar -C /tmp -xzf /tmp/lattice-fleet-*-linux-amd64.tar.gz
sudo /tmp/lattice-fleet-*-linux-amd64/scripts/fleet-install --mode ha-node \
  --pg-dsn 'postgres://fleet:SECRET@writer:5432/fleet?sslmode=require' \
  --admin-token '<same-everywhere>' \
  --credential-key '<same-everywhere>' \
  --public-url https://fleet.example.com \
  --allowed-origins https://fleet.example.com

Front with HAProxy (or a cloud LB): health-check GET /readyz. Agents use the LB hostname. Upgrade one Fleet node at a time (--mode upgrade). There is no Microsoft failover partner and no UDP VIP on this path. Full order (Fleet, agents, clusters): upgrade.