-
Notifications
You must be signed in to change notification settings - Fork 6
Description
Context
During message processing in RabbitMQ using the @reprocessing decorator, when an exception occurs inside the try, the catch block executes the reprocessing logic but does not acknowledge (ack) the message, nor does it properly handle the channel closure. As a result, the message remains stuck in the queue, unacknowledged, which leads to:
- Accumulation of unconfirmed messages
- Increased memory usage (memory leak)
- After about 30 minutes, the container/pod is automatically restarted due to excessive memory usage or a process failure
- Errors such as:
IllegalOperationError: Channel closed
PRECONDITION_FAILED - delivery acknowledge on channel 1 timed out
Likely Cause
The exception inside the try is caught, but the RabbitMQ channel is either automatically closed (or already closed when attempting to resend), causing the Channel closed error. Additionally, the reprocess method is called without any asynchronous handling, and the message is never explicitly acknowledged, either on success or failure.
Problematic Code Snippet
catch (error) {
if (!REPROCESSING.ENABLED || options.normalizeOnly) return;
mqSendReprocessing.reprocess({
middleware: target.constructor.name,
tries: state?.reprocessing?.tries,
progress: {
step: error?.step,
total: error?.total
},
data: { state, payload }
});
// Missing: message acknowledgment
}Impact
This bug can cause unavailability of services that rely on MQ queues, performance degradation, constant pod restarts, and loss of traceability for reprocessing attempts.