A disaster recovery cluster that reports healthy is worse than one that reports broken. Broken gets a ticket. Healthy gets ignored until the day you need it.
This one had been silently out of sync for days. Primary looked fine. Secondary said it was connecting. Nothing in either log was marked fatal. The Merkle tree told a different story, and so did the last successful WAL index, which had not moved since the network team completed a change window.
The setup
Six nodes: a three-node cluster in the primary data centre, three more at the DR site. Between them, NAT. The DR site addressed the primary through translated public addresses; the primary's own nodes knew each other by private addresses on the internal subnet.
Replication had been configured and verified during commissioning. It worked. Then it stopped, without anyone changing a line of Vault configuration.
What actually happens
Vault's replication does not rely solely on the address you hand it at activation time. The primary cluster advertises its own cluster addresses to the secondary over the heartbeat, and the secondary stores what it receives in known_primary_cluster_addrs. That list is how it reconnects after any interruption.
Which is the whole problem. The addresses the primary advertises are the ones its nodes know about themselves — the private addresses. The secondary dutifully overwrites the working translated address with a list of private addresses that are not routable from the DR site.
So the first connection succeeds, using the address a human typed. Every reconnection after that fails, using addresses a machine supplied. And because the secondary keeps retrying, its status reads as connecting rather than failed.
Check known_primary_cluster_addrs on the secondary, not just replication status. If it holds addresses the DR site cannot route to, the status field is telling you about intent, not connectivity.
What does not fix it
Re-issuing the secondary token and re-activating. It works, briefly, for exactly as long as it takes the next heartbeat to overwrite the address list again. If your fix survives a test but not a weekend, this is usually why.
Opening the firewall wider also does nothing. The addresses are not blocked, they are unroutable — a different failure with a similar shape from the application's point of view.
The fix
Stop advertising addresses. Advertise names, and let each site resolve those names to something it can actually reach.
Configure the cluster address as a hostname on every node, then arrange split-horizon resolution so the name resolves to the private address inside the primary data centre and to the translated address from DR. Whatever the primary advertises is now a name, and the name means the right thing on both sides of the NAT boundary.
cluster_addr = "https://vault-node-1.dc.internal:8201"
api_addr = "https://vault.dc.internal:8200"Two details worth getting right:
- Certificates. Once the nodes address each other by name, those names have to appear in the SAN of the cluster certificates, or the TLS handshake fails and you have swapped one silent failure for a louder one.
- Every node. All six. A node missed during the change is a node that reintroduces the private address into the advertisement the moment it becomes active.
Hosts-file entries will prove the theory in an afternoon. They should not be the final state. Anything that lives only in /etc/hosts is one server rebuild away from being gone, and nobody rebuilding that server in eighteen months will know why the entry was there. Get it into DNS, with the reasoning attached to the change record.
What I would do differently
Two things, both cheap.
First, monitor the WAL index on the secondary rather than the replication status string. Status describes what the cluster is trying to do. The index describes what it has done. Alert on the gap growing, not on the state changing.
Second, treat DR failover as an exercise with a date on it. A cluster that has never been failed over is a cluster whose DR posture is a document, not a capability. The gap between those two is discovered at the worst possible moment.
Working through something similar? Send me a note.