Repository navigation
[Mocha] Fix RabbitMQ consumers not being re-attached after the broker takes them away - #10543
Draft
tobias-tengler wants to merge 4 commits into
Draft
tobias-tengler wants to merge 4 commits into
tobias-tengler wants to merge 4 commits into
Conversation
Contributor
Patch coverage87.2% of changed lines covered (383/439)
Uncovered changed lines (JSON){
"sha": "8dba6144372bd856b5d2e3f8ed7c4d88bf5919d8",
"files": [
{ "path": "src/Mocha/src/Mocha.Transport.RabbitMQ/Connection/RabbitMQDispatcher.cs", "ranges": [[139, 139], [146, 147], [149, 149]] },
{ "path": "src/Mocha/src/Mocha.Transport.RabbitMQ/Connection/RabbitMQConsumerManager.cs", "ranges": [[98, 98], [100, 100], [106, 106], [204, 206], [366, 371], [487, 488], [490, 490], [523, 524], [550, 551], [568, 570], [575, 576], [578, 578], [663, 664], [666, 666], [711, 711], [713, 714], [762, 765]] },
{ "path": "src/Mocha/src/Mocha.Transport.RabbitMQ/Connection/RabbitMQConnectionManagerBase.cs", "ranges": [[215, 216], [240, 241], [243, 243], [251, 253], [285, 285], [321, 321], [355, 355], [386, 387], [392, 392], [473, 474], [476, 476]] }
]
}Project coverage: 59.1% (312726/528892 lines) |
Contributor
|
✅ Nitro CLI E2E snapshots: no flow recordings changed vs |
…ys up When the broker closes a consumer's channel or cancels the consumer (for example on a consumer timeout) while the connection stays open, the consumer is never re-attached and the queue stalls.
…hem away Re-attach a consumer with a fresh channel and backoff when its channel is closed or its consumer is cancelled by anything other than Mocha itself, cancel handlers still running on the lost channel, check consumer health in the periodic health monitor, and log lost consumers, re-attachment and skipped acks.
tobias-tengler
force-pushed
the
tte/rabbitmq-consumer-reattach
branch
from
October 19, 2026 07:36
33654db to
aee642f
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When the broker closes a consumer's channel or cancels the consumer while the connection stays up, the RabbitMQ transport never consumes from that queue again until the process restarts, and it logs nothing. A consumer timeout causes exactly this: before RabbitMQ 4.3 it closes the channel with 406 PRECONDITION_FAILED, and from 4.3 it sends
basic.canceland keeps the channel open. The client library never recovers channel-level losses, and the consumer manager only re-attached consumers after a connection event. Handlers still running on the lost channel also kept their token and kept retrying, and their acks were skipped silently.This PR makes
RabbitMQConsumerManagertreat any channel shutdown or server-sent cancel that Mocha did not initiate itself as a lost consumer. It re-attaches the consumer on a fresh channel with backoff for as long as the registration is alive, and keeps retrying with an Error log while the queue is missing. Each channel now gets its own handler cancellation token, which is cancelled when that channel's consumer is lost. The periodic health check now also re-attaches registered consumers that are not attached. Lost consumers, re-attachment, and acks or nacks skipped because the channel is closed are now logged. Integration tests against a real broker cover both a closed channel and a cancelled consumer, the returned message being delivered again first, and disposal not causing a re-attach.