A disaster recovery cluster that reports healthy is harder to trust than one that reports broken. Broken gets a ticket. Healthy gets ignored until the day it is needed.
One way a Vault replication setup can reach that state is address advertisement across a NAT boundary. It is worth understanding before designing a cross-site topology, because the symptom is subtle and the fix belongs in the design rather than in an operational workaround.
The topology in question
Consider a replication setup where the primary cluster sits in one data centre and the secondary in another, with network address translation between them. The secondary reaches the primary through a translated address. The primary's own nodes know each other by their private addresses on an internal subnet.
Replication is configured with the translated address, activation succeeds, and the two clusters synchronise. Everything looks correct.
What the heartbeat does
Vault replication does not rely solely on the address supplied at activation time. The primary advertises its own cluster addresses to the secondary, and the secondary records what it receives in known_primary_cluster_addrs. That stored list is what the secondary uses to reconnect after any interruption.
This is the part worth thinking through at design time. The addresses a primary advertises are the ones its nodes know about themselves, which in this topology are private and not routable from the secondary's site. The working translated address supplied by an operator can therefore be replaced by a list of addresses the secondary cannot reach.
The first connection succeeds because a human supplied a reachable address. Later reconnections can fail because a machine supplied unreachable ones. And because the secondary keeps retrying, the state it reports is connecting rather than failed.
Check known_primary_cluster_addrs on the secondary, not only the replication status field. If it holds addresses the secondary's site cannot route to, the status is describing intent rather than connectivity.
Why the obvious remedies do not hold
Re-issuing the secondary token and re-activating restores replication, but only until the next advertisement overwrites the address list again. A fix that survives a test and then fails days later often has this shape.
Widening firewall rules does not help either. The addresses are not being blocked; they are unroutable from that site. Both look similar from the application's perspective, which is part of why the diagnosis takes longer than it should.
Designing around it
The durable approach is to advertise names rather than addresses, and let each site resolve those names to something it can reach.
Configure the cluster address as a hostname on every node, then arrange split-horizon DNS so the name resolves to the private address within the primary data centre and to the translated address from the secondary site. Whatever the primary advertises is then a name, and the name resolves correctly on both sides of the NAT boundary.
cluster_addr = "https://vault-node-1.dc.internal:8201"
api_addr = "https://vault.dc.internal:8200"Two details matter when implementing this:
- Certificates. Once nodes address each other by name, those names must appear in the SAN of the cluster certificates. Otherwise the TLS handshake fails, and one silent problem is exchanged for a louder one.
- Every node. A node missed during the change will reintroduce a private address into the advertisement as soon as it becomes active, so the configuration has to be applied consistently across the cluster.
Host file entries will demonstrate the behaviour quickly, but they should not be the end state. Anything that exists only in /etc/hosts disappears at the next server rebuild, and whoever rebuilds it will have no record of why the entry was there. The resolution belongs in DNS, with the reasoning captured in the change record.
Two design habits worth adopting
Monitor the WAL index on the secondary rather than the replication status string. Status describes what a cluster is attempting; the index describes what it has achieved. Alerting on a growing gap catches this class of problem, while alerting on a state change does not.
Treat DR failover as a scheduled exercise. A cluster that has never been failed over has a documented DR posture rather than a demonstrated one, and the difference between those tends to surface at the worst possible moment.
Questions or corrections? Feel free to send me a note.