Deciding who leads: failure detection without split-brain

Lesson 2.2 said "the failover controller promotes the standby" and moved on. Here's the buried problem: in a distributed system you cannot distinguish a dead node from an unreachable one. Missed heartbeats mean the primary is down — or that the network between it and you is. Promote the standby while the isolated-but-alive primary keeps publishing, and you have two exchanges emitting conflicting fills: split-brain, the worst state a trading venue can enter. Minutes of downtime embarrass; double-published fills destroy.

The solution has two interlocking halves:

Half 1 — quorum elections. Leadership requires majority consent from a fixed group (say, 3 or 5 nodes across zones). Two disjoint majorities of the same group cannot exist, so two simultaneous leaders cannot be elected — even under any network partition. A candidate that can't reach a majority simply cannot win; a partitioned minority (possibly containing the old leader) can never out-vote the rest.

stateDiagram-v2
    [*] --> Follower
    Follower --> Candidate: leader lease expired
    Candidate --> Leader: majority votes, epoch = old + 1
    Candidate --> Follower: lost — another candidate won
    Leader --> Follower: sees a higher epoch anywhere
    note right of Leader
        Every message the leader emits
        carries its epoch. Downstream
        rejects any epoch lower than
        the highest ever seen.
    end note
    

Half 2 — epoch fencing. Election prevents two elected leaders, but the deposed old primary may not know it was deposed — it might still be publishing. So every output carries the leader's epoch, and every downstream system — journals, gateways, market data relays — remembers the highest epoch seen and drops anything older. The instant the epoch-5 leader's first message lands, epoch-4's messages become noise, everywhere, without the old primary's cooperation. It isn't asked to stop; it's made irrelevant.

Tuning the tripwire: leader leases of ~1–2 s with heartbeats every few hundred ms give failover well inside the four-nines budget. Aggressive timeouts buy speed but cause spurious elections during GC pauses or network hiccups — each a brief availability blip. The Part 3 refinement will be failing over on quality degradation, not just silence (3.5); for now, tune conservatively and measure.