Bug#121063 Members of one group assign different GNOs to the same transaction

[matias-sanchez]

Bug#121063: https://bugs.mysql.com/bug.php?id=121063

What happens

Two members of the same group assign different GNOs to the same transaction, in multi-primary mode, when a member leaves and rejoins during a rolling restart under write load. The group splits into internally consistent sets that disagree on the GTID of the same transaction. It can surface on the group_replication_applier channel as Error_code 1032 or 1062, or stay silent with every member ONLINE and the disagreement present only in the binary logs.

Reproduced on 8.4.8, 8.4.11 and 9.7.2. A self contained reproducer is attached to the bug.

Where it starts

A reserved synode is only ours while we still hold the node index it was reserved under.

local_synode_allocator stamps the member's current index into the synode, synode.node = my_nodeno. Still inside reserve_synode_number, the task yields in the while (too_far(*msgno)) loop at TIMED_TASK_WAIT. During that yield, site_install_action reassigns site->nodeno. The reservation still carries the old index, so proposer_task brands and proposes into a slot that now belongs to another node.

The header comment of xcom_base.cc states the rule at line 107: only node N may propose a value for synode {X N}. With two proposers on the same slot at cnt=0, acceptor.promise is never raised, since it is assigned in exactly one place, inside handle_simple_prepare, which is the phase 1 decision. Both proposals are accepted, both are learned, and handle_learn keeps whichever LEARN arrived first because of the /* Avoid re-learn */ guard. Members that heard different values first deliver different payloads.

A bpftrace trace of one such slot followed from ACCEPT through LEARN to DELIVER is in the bug, posted 28 Aug 2026.

The change

Before proposing, verify the reservation still carries the member's own node index. If it does not, drop it through retry_new and take a new one.

This is the same check incr_msgno (xcom_base.cc:668) already makes whenever it advances, with the comment In case site and node number has changed. The client transaction is not lost, it goes out in a slot that does belong to the member.

Testing

In the lab the anomaly goes to zero: 0 stale proposals out of 15,523,048, against 993 predicted from the unpatched rate, and 0 divergences out of 5,667,504 transactions, against 17.1 expected. No run carrying the patch has reproduced the divergence, on two hosts.

Performance: 12 runs per arm across two hosts, load only, no restarts, no emulated latency, comparing the patched plugin against the same source built without the patch and against the stock image. Ratio of median successful operations per 10 s window, bootstrap over runs: patched over unpatched [0.9998, 1.0004], patched over stock [0.9994, 1.0003]. A positive control with a known throttle was detected at [0.9944, 0.9953], so the method resolves differences of about 0.5%. The workload is rate limited, so this measures behaviour under a production-like load and not peak capacity.

I do not have an MTR test for this. The reproduction needs seven members, rolling restarts and sustained write load, which does not fit the standard framework. The reproducer attached to the bug builds that environment and decides the verdict by decoding every member's binary log and comparing all 21 pairs.

[jujose-1]

This check covers a view change while reserve_synode_number() is suspended, which matches the reported trace.

However, wait_for_cache() below is another TASK_CALL and may suspend at TIMED_TASK_WAIT. If the member is renumbered during that wait, the reservation can become stale after this check.

Could we revalidate the local reservation immediately after wait_for_cache() succeeds—after the null check and before locking or modifying the Paxos machine?

Keeping both checks would avoid waiting on an already stale reservation while also establishing ownership after the final possible yield.

[matias-sanchez]

Hi @jujose-1 huge thanks on your review.
Good catch, thanks. Added the revalidation right after wait_for_cache(), after the null check, and kept the first check as you suggested.
I tested this over a 3 hours runs with rolling restarts under load and I didn't see any meaningful impact on throughput, with the GNO mismatch not reproducing. Feels safe.

[tiagoportelajorge]

Hi!

Just to add a note here... the loop that follows is the main loop of the proposer, where each of the 8 proposers are waiting for the message to be accepted.

In short, I don't think that this check makes sense here, and maybe the one inside the loop is enough, because even with view changes, we keep the proposals active. They will end up being accepted in another configuration, but it will avoid the said problem.

[jujose-1]

Hi Matias,
Thank you for the contribution and for working on this fix. I’ve left a few review comments for your consideration. Please take a look when you have a chance.

Regards,
Justin Jose

[matias-sanchez]

Hi @jujose-1, thanks for taking the time to review this and for the comments. Added the requested changes.

[jujose-1]

Hi @matias-sanchez ,

Thank you for updating the patch and addressing the earlier feedback. The production change looks good to me. I’ve left one additional comment regarding the unit-test setup.

I’ll also wait for @tiagoportelajorge 's review and any additional feedback he may have.

Thank you again for working on this.

Regards,
Justin Jose

[kamil-holubicki]

My proposal is to add two debug asserts:

1. Proposer side (offending node) in prepare_push_2p() after the ballot setup: assert(p->proposer.bal.node == p->synode.node). The node can propose only for its synode. Detect the case when the current node index mismatches the index at the synode reservation time.

2. Acceptor side in handle_simple_accept(): assert(m->proposal.cnt != 0 || m->proposal.node == m->synode.node);. In fast path (cnt==0) the node can propose only for its synode.

[matias-sanchez]

Thanks @kamil-holubicki . However, I'm not sure on how to safely add these asserts, because the issue covered is on multi primary and I'm not sure how to add that assert so that it does not fire when paxos_single_leader is ON, as in that mode the synode carries the leader's node index, not the proposer's. @jujose-1 would adding site->max_active_leaders != active_leaders_all be the proper way to validate this? Also, do you recommend adding this on this PR or would you prefer a follow up one?

[jujose-1]

Thanks, @matias-sanchez , for updating the unit test. The revised test now uses valid memberships and verifies the same reservation across the view change. This looks good to me.

Thanks also, @kamil-holubicki, for the suggestions.

@jujose-1 would adding site->max_active_leaders != active_leaders_all be the proper way to validate this? Also, do you recommend adding this on this PR or would you prefer a follow up one?

My understanding is that the proposed assertions would cover the scenario reported in this bug, but they may not hold for remotely allocated synodes, where the synode can legitimately carry the allocating leader’s node index rather than the proposer’s.

Given that the reported issue involves a locally allocated reservation, and the current fix scopes the ownership validation to that path, I suggest keeping this PR focused on that change. The acceptor side does not currently have the allocation provenance needed to apply the same invariant safely, so I would prefer not to add the proposed assertions here.

[tiagoportelajorge]

I need to think a bit more about this, because I don't know exactly how find_site_def behaves for synodes that were still in the proposal phase. It works well for synodes that are already in the stream and accepted by the group.

That synode might still happen, but now necessarily with the data you are proposing. That is why we have checks like if (match_my_msg(ep->p->learner.msg, ep->client_msg->p))

[tiagoportelajorge]

HI @matias-sanchez , thank you for working on this!

I've reopened a conversation because I think that the code can be simplified. I've also dropped some food for thought, even for me to double-check the behaviour of find_site_def

[matias-sanchez]

Thanks @jujose-1 and @tiagoportelajorge for the review. @tiagoportelajorge I will look deeper into the code on the points you raised and update soon, as at first glance I am still not sure how it could be simplified, so let me know any further thoughts or guidance you may have.

[tiagoportelajorge]

Thanks @jujose-1 and @tiagoportelajorge for the review. @tiagoportelajorge I will look deeper into the code on the points you raised and update soon, as at first glance I am still not sure how it could be simplified, so let me know any further thoughts or guidance you may have.

Hi @matias-sanchez,

Please check my comment in the reopened thread:

Hi!
Just to add a note here... the loop that follows is the main loop of the proposer, where each of the 8 proposers are waiting for the message to be accepted.
In short, I don't think that this check makes sense here, and maybe the one inside the loop is enough, because even with view changes, we keep the proposals active. They will end up being accepted in another configuration, but it will avoid the said problem.

Regards,

Tiago