Delayed exchanges just started piling up messages without forwarding to queues after moving data from one server to another

[KristjanTammekivi]

Hi,

we moved rabbitmq from one server another by shutting down, copying the data directory, starting up. After this messages that go to delayed exchanges with a delay (I tested publishing without a x-delay header and those messages went through) just got stuck in the exchange and we observed the messages delayed number increasing but never decreasing. Later we noticed that same thing happened to every delayed exchange.

We tried restarting rabbitmq but that didn't help. In the end we disabled/enabled the delayed exchange plugin and after that new messages started appearing in the queues.

Is there anything we could have done, some magical rabbitmqctl eval command or something to immediately forward all delayed messages before toggling the plugin, anything to not lose the messages?

Versions:
Rabbitmq 3.8.16
[E*] rabbitmq_delayed_message_exchange 3.8.9.0199d11c

[lukebakken]

RabbitMQ 3.8.6 is completely out of support.

[KristjanTammekivi]

I'll request an upgrade, but any other tips You could give me, or is it completely out of the question because of the old version?

[lukebakken]

is it completely out of the question because of the old version

Partially. If you provide a very, very easy means to reproduce the issue (docker compose project, shell scripts), I can take the time to investigate. Team RabbitMQ is small, so the less work we have to do to reproduce a problem is time we can spend fixing it 😄 Thanks

[KristjanTammekivi]

I think it was a freak accident of sorts, we've restarted/moved rabbitmq before without the issue. Anyway, thanks for your time, hope it doesn't happen after the upgrade 🤞