Administrator manual

Who this is for: the person who installs Lattice, designs the addressing plan, points relays at it, chooses a DNS mode, mints API tokens, and is on the hook when DHCP is quiet. Lease time (1-day default, RFC T1): DHCP lease time.

Companion: Console user.

1. What Lattice is

Lattice is software-only DDI:

1. IPAM — the addressing plan (prefixes, subnets, pools, reservations, excludes). Source of truth.
2. DHCP — identical lattice-dhcpd processes that allocate from Postgres. No primary DHCP server. No split pools for HA.
3. DNS — pluggable adapters that write A/PTR into AD, BIND, or PowerDNS, or an embedded authoritative server (lattice-dns) for Lattice-managed internal zones.

One Postgres database is the system of record for intent and leases. DHCP nodes do not keep authoritative leases in memory. The UI talks only to lattice-api. dhcpd talks SQL on the hot path, not HTTP.

Public site: latticeddi.com. That hostname is the product, not the customer’s API hostname. Customer deployments keep their own URL.

Fleet is a separate control plane if you run more than one cluster. It is not this install. Skip it unless you operate many sites.

Design rules you will feel every day

• Correctness over cleverness. A duplicate lease is a product-killing bug. Allocation is a SQL transaction with FOR UPDATE SKIP LOCKED.
• Intent vs reality. A prefix is a plan. A reservation is intent DHCP must honour. A lease is what dhcpd actually handed out. There is no generic “used” flag.
• Ack never waits on DNS. Grant commits the lease, then the worker claims an outbox row. Missing A/PTR after Ack is an outbox problem, not a DHCP rollback.
• Lease times are wall-clock. NTP/chrony on every node is a hard dependency. UTC everywhere.
• API first. If you cannot do it with Authorization: Bearer lattice_live_… against /api/v1, it is not a product feature.

What Lattice is not

• Not Infoblox DNS security, RPZ, or threat intel.
• Not DCIM (racks, cables). NetBox import creates missing prefixes only.
• Not a recursive resolver, workstation cache, or public/external DNS.
• Not the AD forest (_msdcs, site SRV). Leave that on the DCs.
• Not DNSSEC (internal zones stay unsigned on purpose).
• Not ISC/Microsoft pairwise DHCP failover. Shared Postgres is HA for leases.
• Not a BGP anycast speaker. You may put a VIP or anycast in front; health-check HTTP /readyz, never UDP/67.

2. Pieces you install

Same binaries for community and paid. The signed entitlement is the SKU.

Process config vs network intent

Process settings are environment variables (optional YAML LATTICE_CONFIG). Network intent lives only in Postgres.

Do not run dhcpd and api on the same LATTICE_LISTEN_HTTP port on one host.

3. Editions and license

Lattice enforces a signed entitlement file, not a serial typed into a box. Ed25519, public key compiled into the binaries. No phone-home.

Managed IPs (the meter) = distinct addresses that are leased + reserved + quarantined + assigned. Prefix size, free inventory, offered, and excluded do not count. A /16 with 200 live addresses is 200, not 65,536.

Community includes DHCPv4, IPAM, API, BIND TSIG, and AD GSS-TSIG. Cluster UI, join, bundled Patroni, backups UI, and dhcp-only extra nodes need a paid entitlement. The cap is how many addresses you may catalog and lease, not a packets-per-second rating — plan DHCP load from lease time (DHCP lease time). Tiers match pricing.

Enforcement (this is the contract)

1. Missing or invalid file (including a hand-edited max_managed_ips) → community, 1,000. The unsigned cap is never trusted.
2. Over the cap: DHCP does not crash. Existing leases still renew. New grants (counted at Commit, not when an Offer is sketched), new reservations, and new assignments return 402 license_limit.
3. After expires_at + grace_days (default 14): same as over-limit (renew only).
4. A reservation that already counts still gets DHCP when over the cap.
5. No per-node license. Any number of dhcpd processes.
6. max_managed_ips: 0 in a valid signed file is a revoke (renew only). Deleting the file falls back to community.

Admins replace the file in the console (License) or PUT /api/v1/license. The body is the JSON file. It is stored in Postgres so every node sees it. Operator and readonly can look; they cannot replace.

Download from the customer portal. Detail: License. A missing or invalid file is Community. Hand-editing the cap fails verify. DHCP keeps renewing existing leases when over cap.

4. Topology

Pick one. You can rebuild later with the same binaries and the same entitlement.

Single node (community, or paid that can take a window)

• One Ubuntu 22.04/24.04 VM
• Relays and browsers point at that VM (HTTPS :443, DHCP :67, DNS :53). API :8080 is loopback-only.
• One Postgres. The community SKU is one node.

Three-node HA (paid) — HA.

• Three Ubuntu VMs in the same site (low RTT, same L2). That L2 is only for Patroni, etcd, and the keepalived VIP — not a limit on client VLANs.
• One unused IPv4 on that subnet — the VIP. Not in netplan. keepalived owns it.
• Relays and browsers point at the VIP (HTTPS :443, DHCP :67, DNS :53). Relays may sit on any VLAN; they unicast to the VIP.
• Each of those three nodes runs api, worker, dhcpd, lattice-dns, maint, Patroni, etcd, HAProxy, keepalived
• etcd quorum is 2 of 3. Two nodes cannot form a safe cluster. If two die, the survivor does not become a writable primary (fail closed)
• Extra dhcp-only / dns-only nodes after the control plane exists: another subnet or DC. Relays at that site point at the leaf, not the VIP. Postgres RTT ≲ ~50 ms. Do not put a fourth etcd member, or stretch 2+1 Patroni across DCs.
• Two data centers: live cluster in the primary; dhcp-only at the other site if RTT allows; or restore a backup onto a new cluster in the DR DC and cut relays. Planned vMotion of all three is fine if DC-B keeps the same L2 and IPs (one at a time, replicas first). Not a stretched 2+1 pair. HA — Active / passive DC.

Bring your own Postgres (paid)

• lattice-install --mode byo-postgres --pg-dsn '…'
• You own failover. Lattice still assumes one writer.

Public cloud: keepalived needs real L2 and a free address the platform will accept. Many VPCs do not. Prefer single-node, or BYO Postgres plus a cloud load balancer, unless you already know how to run VRRP there.

Host sizing, netplan, NTP, ports, firewall: Install prerequisites. HA internals: Paid HA.

5. Install

1. Ubuntu Server 22.04 or 24.04. Derivatives on those LTS bases are best-effort. Fresh VM. Do not share it with another DHCP server, BIND on :53, or a second Postgres on :5432.
2. Stable IPv4 in netplan before the installer. Lattice detects the address; it does not rewrite the NIC (that can drop SSH).
3. Chrony synchronized. Offset well under 500 ms. UTC.
4. Paid: entitlement on disk or ready to paste. Community: skip; missing file is 1,000 IPs.
5. Extract the tarball, then:

sudo ./scripts/lattice-install

No arguments = wizard. It asks license, host specs, topology, node IP vs VIP, firewall, then a summary. b goes back. Nothing is installed until you confirm. The installer installs Docker if needed.

Non-interactive examples (full flags in lattice-install --help and Paid HA):

lattice-install --mode community
lattice-install --mode ha-primary --vip 10.0.0.10 --peers 10.0.0.11,10.0.0.12,10.0.0.13
lattice-install --mode join-token
lattice-install --mode ha-join --peer 10.0.0.11 --token <join-token> --ca-hash <printed-on-primary>
lattice-install --mode dhcp-only --peer 10.0.0.11 --token <join-token> --ca-hash …
lattice-install --mode upgrade
lattice-install --mode save-bundle --out lattice-images.tar

Join tokens are one-shot. The string printed on the primary is spent on the first successful join. Mint the next with lattice-install --mode join-token. CA hash is printed on the primary. Neither is shown in the Cluster console.

Air-gap: save-bundle on a connected host, copy the tar with the release, --bundle on the dark site.

6. First login and tokens

Open https://<this-vm> (or the VIP). The browser warns on the self-signed cert until you replace it. lattice-install already minted the first admin and printed it; it also wrote /etc/lattice/bootstrap.token. Paste that lattice_live_… string. Copy once. Closing the tab forgets it; paste the file again. Do not open http://127.0.0.1:8080 from your laptop.

Until the first token exists, the API is loopback-only. After the first token, Bearer is required. Missing/invalid Bearer is 401. Wrong role is 403.

Hiding a button is not security. The API enforces RBAC.

The secret lives in memory + sessionStorage (this tab), not localStorage. A new tab does not inherit it. Closing the tab drops it. XSS on the Lattice origin can still steal it — treat the UI as a client, not a vault. Serve UI and API from the same origin. Do not set Access-Control-Allow-Origin: *. Do not put the token in query strings or logs.

Mint more under Tokens. Rate limit: 10 token creates / min / IP.

Automation: Authorization: Bearer lattice_live_…. Idempotency: header Idempotency-Key on creates. Same key + same body → original 201. Same key + different body → 409.

7. Addressing plan (the mental model)

Build top-down. DHCP does nothing until a pool exists.

tenant (default is seeded)
  └── prefix          addressing plan block (may never be a DHCP scope)
        └── subnet    L3 network Lattice may serve
              ├── pool(s)         dynamic range
              ├── reservation(s)  this client always gets this IP
              ├── exclude(s)      never offer (gateway, VIP, already in use)
              └── options         extra DHCP options

Children of a prefix must sit strictly inside the parent. Pool ranges must sit inside the subnet and must not overlap. IPv4 inventory expands at most 65,536 addresses per pool.

How to mark an address

A reservation is a row dhcpd cannot violate. Dynamic clients never receive that IP. That MAC/DUID always gets it (in that subnet). At least one of MAC, client-id, DUID is required.

Deleting a subnet with active leases is 409 unless you check Force (?force=true), which drops those leases.

Tenants: prefixes and classes hang off a tenant. Omit tenant_id to use default. v1 is a slug, not hard isolation. Operators cannot list tenants.

First subnet

1. Addresses (/plan) → add a prefix (e.g. 10.20.0.0/16). This is the plan, not the DHCP scope.
2. Add a subnet inside it (e.g. 10.20.20.0/24) with gateway / VLAN / site if you have them. Lease time defaults to 1 day (86400 s) — leave it for campus VLANs (DHCP lease time).
3. Exclude the gateway (and any VIP). Single IP: start = end.
4. Add a pool (e.g. .10–.200). DHCP will not Offer until this exists.
5. Point relays at this node (paid HA: the VIP, or a dhcp-only node at that site — clients do not need to share the HA L2), UDP :67. Direct clients need an interface binding; most sites use relays. Community dhcpd is Dockerized: a same-VLAN broadcast client often never arrives — prove with a relay or lattice-dhcptest, not a laptop on the DHCP server VLAN.
6. Prove it on the node: lattice-dhcptest -server 127.0.0.1:67 -giaddr 10.20.20.1 -n 5. -giaddr must sit in the subnet CIDR.

8. How DHCP actually works

Each lattice-dhcpd is interchangeable. Discover may land on node A and Request on node B. That is normal. Request re-validates against Postgres. Never assume “I offered this.”

Packet path (v4)

1. Receive UDP (direct or relayed).
2. Identify subnet: option 82 link-selection (RFC 3527) if present; else giaddr; else ingress interface.
3. Identify client: client-identifier if present, else chaddr (MAC).
4. Load reservation. If this client is reserved in this subnet, that IP is the only candidate.
5. Discover — Offer an existing valid lease, else peek a free candidate and Offer. Ordinary Discover does not write Postgres. Reservations still take a short offered row so the reserved IP is held. Does not commit a long lease.
6. Request — if the IP is still this client’s (reservation, existing lease, or offered-to-this-client): commit lease, enqueue DNS upsert, Ack. Else Nak.
7. Release / Decline — transactional state change + DNS delete event. Decline quarantines the IP for LATTICE_QUARANTINE_TTL.
8. Renew / Rebind — extend expiry if the lease still belongs to this client.

Never allocate: reserved, excluded, leased, quarantined.

Lease states: offered · leased · released · expired · declined · quarantined. A client has at most one leased address per subnet. Unique IP among offered/leased/quarantined.

Server identifier (option 54) is not PXE. PXE is next_server (siaddr / option 66) and boot_file (option 67) on the subnet. Mismatch of server-id on Request is ignored on purpose so a VIP does not NAK a valid Request.

Relays and trust

Point DHCP relays at the VIP (HA) or the node (single). Health-check HTTP /readyz, never UDP/67. /readyz on dhcpd is a Postgres ping plus a subnet-count query.

LATTICE_TRUSTED_RELAYS is a comma-separated allow list on the dhcpd process. Empty drops relayed packets; direct clients on the same L2 still get leases. List your relay IPs/CIDRs. Option 82 is accepted and logged; subnet selection still uses giaddr / link-selection. Trusted sources are env, not an IPAM screen. Changing this list does not open firewall ports.

Built-in options vs JSON options

Mask, router, and lease-time come from the subnet row. Set Gateway so IPAM and DHCP agree. NTP servers on the subnet becomes DHCPv4 option 42 (this is for clients; host clocks are chrony on the Lattice VMs — two different problems).

Extra options: code + JSON payload + optional vendor class.

{"addresses":["8.8.8.8","1.1.1.1"]}
{"ip":"10.20.20.1"}
{"string":"office.example.com"}
{"uint32":86400}
{"hex":"0102"}

Priority (later wins): global, then subnet, then pool, then class, then reservation. A stored option 42 wins over the subnet NTP field if both exist.

There is no dhcpd.conf snippet to edit on each node. Config is Postgres, refreshed via LISTEN/NOTIFY plus a short poll. Hot-reload does not drop UDP sockets.

Client classes

Match vendor_class, user_class, or mac_prefix. Attach the class to a pool to steer allocation. Class-scoped options override subnet options. Admin writes; operator reads.

IPv6

Prefixes and subnets are first-class. Lattice does not explode a /64 into inventory rows. Pools still cap expansion at 65,536 where inventory exists.

• IA_NA — DUID identity. Leave PD len blank on the pool.
• IA_PD — set PD len (e.g. 64). The lease stores prefix_len. That pool is not an IA_NA/v4 host pool.

Reservations may use DUID. If the VLAN is SLAAC-only, do not create a DHCPv6 pool.

PXE

On the subnet: PXE next-server and PXE boot file. Do not encode these as JSON DHCP options.

9. DNS — pick one mode per zone

Ack never waits. The worker claims the outbox (lease.granted / renewed / released / expired / declined). Failures retry with backoff; after 8 attempts the row is failed (dead-letter). Duplicate upserts are expected (at-least-once). Adapters must be idempotent. Do not roll back the lease.

Do not point the same zone at embedded and an external adapter. That is split brain.

Secrets are refs (env:LATTICE_TSIG_SECRET, file:/etc/lattice/lattice.keytab), never shown again after create. Store the actual secret on the worker host, mode 0600.

BIND TSIG

RFC 2136 + HMAC-SHA256. allow-update { key lattice.; }; on the zones Lattice owns. Worker → BIND 53. dhcpd does not need that path. Docs: BIND TSIG.

Active Directory (GSS-TSIG)

Does not talk LDAP. Subprocess nsupdate -g (in-process GSS is not implemented). Dedicated AD user, keytab 0600, clock skew vs DCs under 5 minutes. AD GSS-TSIG is on every tier. Docs: AD DDNS.

PowerDNS HTTP

Worker PATCHes rrsets on the Authoritative HTTP API. Create the zone in PowerDNS first. Docs: PowerDNS.

lattice-dns (embedded)

Authoritative only (RecursionAvailable=0). Serves SOA/NS, A/AAAA/PTR from leased rows plus static records. Off-zone → REFUSED. Name in zone, wrong type → NODATA. Missing name → NXDOMAIN. Recursion, RPZ, public DNS, AD-forest hosting, DNSSEC: out of scope.

AXFR is TCP. List the slave IPs in allow_transfer — empty means any client can zone-transfer. Optional transfer TSIG. Serial bump sends NOTIFY to also_notify. IXFR is served when the slave serial matches the previous compiled snapshot; otherwise AXFR. Process restart forgets history, so the next IXFR is AXFR.

Listen is :53.

Create a dns_server with adapter embedded, then bind the subnet’s DDNS adapter to that name (or embedded). Static records live on DNS (/dns): A, AAAA, CNAME, MX, SRV, CAA.

If workstations need a cache, run Unbound/BIND in front and conditional-forward our zones; leave AD on the DCs.

Hostname on the lease: option 12 is sanitized to one DNS-safe label. FQDN is zone plus that label.

10. Console map (admin)

11. Webhooks, audit, NetBox

Audit is append-only IPAM mutation (create/update/delete, token create/revoke, force-release, NetBox import, …). Readonly can look. Occupancy (who held an IP, where a MAC has been) is History / GET /history, not Audit. History.

Webhooks POST outbox JSON after the DNS adapter runs (or is skipped). Same retry as DNS. Never rolls back the lease or the IPAM write. URL must be https (plain http only for localhost). Header X-Lattice-Signature: sha256=… is HMAC-SHA256 of the raw body. Topics: lease.granted, lease.renewed, lease.released, lease.expired, lease.declined, ipam.audit, or *. Operator cannot read webhook secret_ref.

NetBox: Lattice is master for DHCP subnets, pools, reservations, and leases. Import only creates missing prefixes. It never deletes and never writes pools. There is no live pull with a stored NetBox token — dump from NetBox, POST the JSON here.

12. High availability and backups (paid)

Community Cluster shows dhcpd heartbeats and an upgrade note, not the HA installer. After a paid file is on License, deploy with lattice-install (wizard) on three Ubuntu VMs in one site (same L2 for the VIP). Client VLANs stay on relays. Other DC: dhcp-only, or restore + cutover — HA — Active / passive DC. Community stays one node without that file.

Health is HTTP /readyz, not UDP/67. UDP 67 and 53 on the VIP are keepalived IPVS (open-source HAProxy has no generic mode udp). DNS TCP/53 stays on HAProxy. UI/API is HTTPS :443.

Fail closed: synchronous_mode on, use_pg_rewind off. Uncertainty → no writer. Duplicate leases are worse than a short outage.

Backups (Cluster UI, admin): off-box dump to s3://, sftp://, or a local path. That is also the DR DC path: restore is one-shot onto a new cluster in the passive site, then cut relays and the VIP. Do not stretch the three HA VMs across DCs (HA — Active / passive DC):

lattice backup restore --from s3://bucket/file.dump

Do not restore a dump onto a live writer that dhcpd is still using. PITR (WAL archive) is the real RPO; logical dump is a second copy. See Postgres.

Alert destinations (PATCH /cluster notify): webhook URL, Alertmanager base URL (we POST /api/v2/alerts), PagerDuty Events API v2 routing key, SMTP. Optional.

Upgrade: lattice-install --mode upgrade on a replica first, then the leader (switchover runs). --force if switchover stalls.

Diag: lattice diag -o bundle.tgz — logs, redacted configs, docker ps, Patroni/etcd, /readyz.

etcd is Patroni’s DCS only. Lattice binaries never speak etcd; they speak SQL.

13. Observability

Lattice does not speak syslog, SNMP, or NTP itself.

• Logs: JSON slog on stdout. DHCP lines include node, xid, mac, ip, subnet_id, msg_type. Do not parse logs for lease truth — Postgres is the store.
• Metrics: Prometheus /metrics on api, dhcpd HTTP, worker. API and worker are open on loopback. Scrapes from another host need Authorization: Bearer lattice_live_… (readonly is enough). dhcpd /metrics is loopback-only.
• Webhooks: SIEM/CMDB subscribe to ipam.audit and/or lease topics.
• Overview: derived alerts (pool free, stale dhcpd, outbox lag, license) from Postgres every 8s.

Alert on lattice_outbox_lag, lattice_pool_free, lattice_dns_updates_total{result="failed"}, process up. Clock offset > 500 ms on hosts. GSS-TSIG dies opaquely beyond ~5 minutes vs DCs.

Ship journald → Vector/rsyslog. Examples: Observability.

14. Day-2 operations

Runbooks: Runbooks.

Postgres

Single writer. dhcpd /readyz is 503 while the writer is gone; the VIP should take the node out. Application code does not retry a write against a replica.

DSN is in /etc/lattice/cluster.env (sslmode=require). Do not share one database user across api, dhcpd, and worker.

15. Kubernetes

Kubernetes examples keep Postgres external. They are not a hosted Lattice cluster and not an operator. Prefer lattice-install on Ubuntu unless you already run k8s as a platform. See k8s.

16. Checklist: a competent admin’s afternoon

• Host(s) meet the spec table for licensed IPs; Ubuntu 22.04/24.04; NTP; static IPv4
• Entitlement in place (or accept community 1,000)
• lattice-install completed; /readyz is 200
• First admin token copied once; extra operator/readonly tokens minted
• Prefix → subnet → exclude gateway → pool
• Relays point at VIP/node; LATTICE_TRUSTED_RELAYS set in production
• One DNS mode per zone; worker can reach BIND/AD/PowerDNS or lattice-dns
• A test client (or lattice-dhcptest) gets a unique lease; Overview shows it
• Release/expire removes DNS (eventually)
• Metrics scraped; webhook or SIEM on ipam.audit if required
• Paid HA: three nodes, VIP unused in netplan, Cluster page shows members, backup destination set

If any of that fails, Console user for console clicks. Day-2 symptoms: Runbooks.