Skip to content

[Mocha] Fix RabbitMQ consumers not being re-attached after the broker takes them away - #10543

Draft
tobias-tengler wants to merge 4 commits into
mainfrom
tte/rabbitmq-consumer-reattach
Draft

tobias-tengler wants to merge 4 commits into
mainfrom
tte/rabbitmq-consumer-reattach

Conversation

@tobias-tengler

Copy link
Copy Markdown
Member

When the broker closes a consumer's channel or cancels the consumer while the connection stays up, the RabbitMQ transport never consumes from that queue again until the process restarts, and it logs nothing. A consumer timeout causes exactly this: before RabbitMQ 4.3 it closes the channel with 406 PRECONDITION_FAILED, and from 4.3 it sends basic.cancel and keeps the channel open. The client library never recovers channel-level losses, and the consumer manager only re-attached consumers after a connection event. Handlers still running on the lost channel also kept their token and kept retrying, and their acks were skipped silently.

This PR makes RabbitMQConsumerManager treat any channel shutdown or server-sent cancel that Mocha did not initiate itself as a lost consumer. It re-attaches the consumer on a fresh channel with backoff for as long as the registration is alive, and keeps retrying with an Error log while the queue is missing. Each channel now gets its own handler cancellation token, which is cancelled when that channel's consumer is lost. The periodic health check now also re-attaches registered consumers that are not attached. Lost consumers, re-attachment, and acks or nacks skipped because the channel is closed are now logged. Integration tests against a real broker cover both a closed channel and a cancelled consumer, the returned message being delivered again first, and disposal not causing a re-attach.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Patch coverage

87.2% of changed lines covered (383/439)

File Covered Changed Patch %
…/Mocha.Transport.RabbitMQ/Connection/RabbitMQDispatcher.cs 10 14 71.4% 🔴
…/Connection/RabbitMQConsumerManager.cs 224 259 86.5% 🟡
…/Connection/RabbitMQConnectionManagerBase.cs 113 130 86.9% 🟡
…/Middlewares/Receive/RabbitMQAcknowledgementMiddleware.cs 10 10 100.0% 🟢
…/src/Mocha.Transport.RabbitMQ/RabbitMQDispatchEndpoint.cs 4 4 100.0% 🟢
…/src/Mocha.Transport.RabbitMQ/RabbitMQMessagingTransport.cs 2 2 100.0% 🟢
…/src/Mocha.Transport.RabbitMQ/RabbitMQReceiveEndpoint.cs 20 20 100.0% 🟢
Uncovered changed lines (JSON)
{
  "sha": "8dba6144372bd856b5d2e3f8ed7c4d88bf5919d8",
  "files": [
    { "path": "src/Mocha/src/Mocha.Transport.RabbitMQ/Connection/RabbitMQDispatcher.cs", "ranges": [[139, 139], [146, 147], [149, 149]] },
    { "path": "src/Mocha/src/Mocha.Transport.RabbitMQ/Connection/RabbitMQConsumerManager.cs", "ranges": [[98, 98], [100, 100], [106, 106], [204, 206], [366, 371], [487, 488], [490, 490], [523, 524], [550, 551], [568, 570], [575, 576], [578, 578], [663, 664], [666, 666], [711, 711], [713, 714], [762, 765]] },
    { "path": "src/Mocha/src/Mocha.Transport.RabbitMQ/Connection/RabbitMQConnectionManagerBase.cs", "ranges": [[215, 216], [240, 241], [243, 243], [251, 253], [285, 285], [321, 321], [355, 355], [386, 387], [392, 392], [473, 474], [476, 476]] }
  ]
}

Project coverage: 59.1% (312726/528892 lines)

@github-actions

Copy link
Copy Markdown
Contributor

✅ Nitro CLI E2E snapshots: no flow recordings changed vs main.

…ys up

When the broker closes a consumer's channel or cancels the consumer
(for example on a consumer timeout) while the connection stays open,
the consumer is never re-attached and the queue stalls.
…hem away

Re-attach a consumer with a fresh channel and backoff when its channel is
closed or its consumer is cancelled by anything other than Mocha itself,
cancel handlers still running on the lost channel, check consumer health
in the periodic health monitor, and log lost consumers, re-attachment and
skipped acks.
@tobias-tengler
tobias-tengler force-pushed the tte/rabbitmq-consumer-reattach branch from 33654db to aee642f Compare October 19, 2026 07:36

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant