Runbooks

Add/drain dhcpd, Postgres failover, clock, pools, outbox. Shared DB, no pairwise failover.

There is no ISC failover protocol. Community is one node. Paid HA is three Ubuntu VMs in one site (same L2 for the VIP only) and a VIP (HA). Client VLANs do not share that L2. Extra site: --mode dhcp-only. Active/passive DC is restore onto a new cluster, then cut relays — not a stretched trio. HA — Active / passive DC.

Add a dhcpd node (paid)

lattice-install --mode join-token
lattice-install --mode dhcp-only --peer <primary-ip> --token <join-token> --ca-hash <hash>

Relays at that site point at this VM. Confirm a node row with recent last_seen_at. Join a control-plane member with ha-join, not dhcp-only.

Drain / upgrade (paid HA)

lattice-install --mode upgrade on a replica, then on the leader (switchover runs automatically). --force if switchover stalls. Do not delete leases.

Postgres failover (paid HA)

Patroni holds the writer. Processes use the VIP DSN. /readyz is 503 until ping works. Check lattice_outbox_lag. Pending rows retry.

Diag: lattice diag -o /tmp/lattice.tgz. Kubernetes examples (not the production SKU): k8s.

More: Clock · Pool exhaustion · Outbox · Backup.