HA is not Microsoft DHCP failover

Community is single-node (lattice-install --mode community). Same binaries. HA is a paid entitlement, not a second download. Purchase is closed.

Before any node: install-prerequisites (VM size, Ubuntu, static IP vs VIP, NTP, entitlement).

Just show me the steps — installer commands at the bottom.

What you deploy

Same L2 is only the three control-plane VMs, not your client networks.

Three Ubuntu Server 22.04 or 24.04 VMs in the same site (low RTT, same L2 for VRRP). Ubuntu derivatives on those LTS bases are best-effort. Each of those three nodes runs containers:

Health checks are HTTP /readyz, never UDP/67.

Postgres is the only source of truth for prefixes, pools, reservations, and leases. dhcpd on every node writes SQL to the Patroni VIP (HAProxy → current primary). Discover on A and Request on B is normal. Extra dhcpd processes put DHCP next to clients and let either node Ack; they share one writer, so first-grant is not N times the node count. Plan DHCP load from lease time: DHCP lease time.

If you know Microsoft DHCP failover

Windows DHCP failover is two servers, a shared scope, a sync protocol, MCLT, and “hot standby” or “load balance %.” Microsoft’s current description is DHCP failover. Lattice does none of that.

• There is no partner server to configure in the console.
• There is no split scope to import. Throw the relationship away. migrate.
• Three Ubuntu VMs share one Postgres. Every dhcpd is identical. Relays point at one VIP.
• “Restart DHCP on the primary” is the wrong instinct. Look at Cluster last grant and /readyz. troubleshooting.
• Two VMs is not HA (etcd needs three). Stretching 2+1 across data centers is not HA. windows-admin if the Ubuntu part is unfamiliar.

Two nodes cannot form a safe etcd/Patroni quorum. The installer refuses --peers that is not exactly three IPs. Fail closed: synchronous_mode on, use_pg_rewind off. If two nodes die, the survivor does not become a writable primary.

The third node is a full member (witness and dhcpd/api/dns). Extra load or remote segments use dhcp-only / dns-only nodes, not a fourth etcd member. Do not put one of the three Patroni/etcd members in another DC: 2+1 across sites cannot quorum if the majority site is gone.

lattice-install
# or
lattice-install --mode ha-primary --vip 10.0.0.10 --peers 10.0.0.11,10.0.0.12,10.0.0.13
lattice-install --mode ha-join --peer 10.0.0.11 --token <printed-on-primary> --ca-hash <printed-on-primary>
# token is spent; on the primary:
lattice-install --mode join-token
lattice-install --mode ha-join --peer 10.0.0.11 --token <new> --ca-hash <same>

One package. The installer installs Docker if needed, builds images from the tarball binaries, and uses named volumes (etcd-data, pg-data, backup-data). Host packages are not touched besides Docker.

DHCP-only / DNS-only

lattice-install --mode dhcp-only --peer 10.0.0.11 --token JOIN
# or
lattice-install --mode dhcp-only --pg-dsn 'postgres://lattice:…@10.0.0.10:5432/lattice?sslmode=require'

Installs only dhcpd (or lattice-dns) + maint. No Patroni, etcd, or HAProxy. Relays at that site point at this node, not the control-plane VIP. Lease writes are SQL to the cluster VIP. Clients on that site do not need to share an L2 with the control plane.

Latency: control-plane RTT should stay under ~5 ms (etcd elections). DHCP-only nodes can be farther; keep Postgres RTT under ~50 ms so DORA still completes. Above ~100 ms, stand up a second cluster in that site (see below) — do not stretch etcd.

Active / passive DC

Lattice does not stretch the three-node control plane across data centers. That is etcd + synchronous Postgres + VRRP, not DHCP. Duplicate leases are worse than a short outage.

Pick one of these. Mixing two live writers on the same pools is not supported.

One cluster, local DHCP at the other site (metro / nearby DC)

DC-A holds the three control-plane VMs and the VIP. DC-B (or a campus) runs --mode dhcp-only. Relays in DC-B point at that node, not the DC-A VIP. Lease writes are SQL to DC-A.

Use this when Postgres RTT from the leaf is ≲ ~50 ms. Client VLANs in DC-B can be any subnet.

If DC-A is unreachable, DC-B dhcpd /readyz goes 503 (no writer). Existing leases keep working until they expire. New DORA fails until DC-A (or a restored cluster) is back. That is local serving while the WAN to Postgres is up — not a DR failover.

Disaster recovery (the usual enterprise active/passive)

DC-A is the live cluster. Off-box backups (s3:// or sftp://) land where DC-B can read them. Optionally pre-stage three Ubuntu VMs and an air-gap image bundle in DC-B — do not start a second live writer on the same addressing plan.

On disaster:

1. Confirm DC-A is fenced (powered off or isolated). Two writers on the same pools duplicate IPs.
2. On a new cluster in DC-B: lattice backup restore --from s3://… (or sftp://). Never restore onto a live writer dhcpd is still using.
3. Cut relays (and the DNS name for the console) to the DC-B VIP. A relay “helper 2” is fine after cutover; it must not answer the same pools while DC-A is live.

RPO is the last good backup (use PITR/WAL if you need seconds). RTO is restore + the relay cut. Leases already in client RAM survive until expiry even while the server is down.

Planned move of the whole cluster (vMotion / live migrate)

Yes, for a planned window — VMware vMotion, Hyper-V Live Migration, Proxmox, and the like — if the destination still presents the same L2 and the same three IPs + VIP. Relays do not change. Treat it like a rolling upgrade:

1. Move one VM at a time. Replicas first, then the Patroni leader (same order as lattice-install --mode upgrade).
2. Keep etcd RTT ≲ ~5 ms the entire time. A long stun on one node looks like a reboot; the other two hold quorum.
3. Pin all three in the same site / host group when you are done. DRS must not leave 2 in DC-A and 1 in DC-B.

vMotion does not help when DC-A is already dead — you cannot migrate a VM that is down. That is restore (above) or a hypervisor DR product that restarts the three VMs together on replicated storage in DC-B (same IPs, same L2). If the stretch can partition, fence one site so keepalived does not run VRRP on both sides.

A stretched vSphere cluster that freely places VMs across DCs is the 2+1 layout we reject. Keep the trio together.

Both DCs must grant new leases if the other is down

That is two clusters, not a stretched HA pair. Give each DC non-overlapping pools (split scope) or different subnets. Relays in each DC point at the local VIP. Fleet (separate archive) is the pane of glass. Do not run the same pool on two writers.

Do not

• Put 2 Patroni nodes in DC-A and 1 in DC-B (or the reverse). Majority dies with one site. Do not let DRS do it for you.
• Stretch keepalived across a routed WAN. VRRP needs L2. A stretched VLAN that partitions can run the VIP on both sides — fence one site.
• Point DC-B relays at the DC-A VIP across a high-RTT WAN and expect DORA to feel local. Use a dhcp-only leaf, or a second cluster.
• Invent BGP anycast inside Lattice. You may put your own anycast in front; withdraw when /readyz fails.

Bring your own Postgres

lattice-install --mode byo-postgres --pg-dsn 'postgres://…' --vip 10.0.0.10 --peers a,b,c

Skips bundled Patroni/etcd. You own failover. Lattice still assumes one writer.

Air-gap

On a connected host with images already built:

lattice-install --mode save-bundle --out lattice-images.tar

On the dark site:

lattice-install --mode ha-primary --bundle lattice-images.tar --vip … --peers …

Operations

Sidecar (lattice-maint): VACUUM / ANALYZE / REINDEX on a cron (30 3 * * * by default), container self-heal (never restarts a running Patroni primary on a flaky check), disk/VIP//readyz alerts. New/resolved alerts POST to webhook, Alertmanager, PagerDuty, and/or SMTP (PATCH /cluster notify).

Console: Cluster — dhcpd heartbeats for everyone. Paid HA (control plane, VIP, backups, alerts) is below when licensed. Community shows an upgrade note there, not an installer.

TLS: cluster CA + server cert at /etc/lattice/tls/ (SAN: VIP, peers, localhost). HAProxy :443, Postgres sslmode=require through the VIP. Replace haproxy.pem with a real certificate when you have one. Join uses HTTPS (curl -k until the joiner has ca.crt).

Prometheus (api /metrics): lattice_cluster_member_ready, lattice_cluster_disk_used_ratio, lattice_cluster_replication_lag_bytes, lattice_cluster_vip_up.

What Community vs paid includes

Split-brain

• etcd quorum 2 of 3.
• HAProxy postgres backend checks Patroni GET /primary (HTTP 200 only on the writer).
• synchronous_mode: true, use_pg_rewind: false.
• Uncertainty → no writer. DHCP /readyz fails; VIP moves off a dead node; clients retry. Duplicate leases are worse than a short outage.

Just the steps

Three Ubuntu VMs, same site, same L2. VIP is an unused IPv4 on that subnet — not in netplan. Relays and browsers use the VIP.

lattice-install --mode ha-primary --vip 10.0.0.10 --peers 10.0.0.11,10.0.0.12,10.0.0.13
lattice-install --mode ha-join --peer 10.0.0.11 --token <printed-on-primary> --ca-hash <printed-on-primary>
# token is spent; on the primary:
lattice-install --mode join-token
lattice-install --mode ha-join --peer 10.0.0.11 --token <new> --ca-hash <same>

Extra site: lattice-install --mode dhcp-only --peer 10.0.0.11 --token JOIN (relays there point at that node). There is no Microsoft failover partner. Health-check HTTP /readyz.

Upgrade: replica, replica, then the current leader. If you also run Fleet: Fleet first. upgrade.