Message is moved to DLQ without clear cause

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Message is moved to DLQ without clear cause

gijsbert802
I'm using camel to consume messages from an ActiveMQ queue. We don't want automatic redeliveries, so I've set MaximumRedeliveries on the activemq connection to 0 (by setting a RedeliveryPolicy on ActiveMQConnectionFactory).

This works except that occasionally a message is now moved to the DLQ without a clear cause. There is nothing in the camel or ActiveMQ logs. The dlqDeliveryFailureCause property is set to "java.lang.Throwable: dispatch to ... exceeds redelivery policy limit...", but I can't see the underlying cause for redelivery.

How can I see what's happening here?

Thanks.
Gijsbert
Reply | Threaded
Open this post in threaded view
|

Re: Message is moved to DLQ without clear cause

Tim Bain
You might get more insight by turning on debug logging in the broker.  See
http://activemq.apache.org/how-do-i-enable-debug-logging.html.
On Jan 14, 2015 3:10 AM, "gijsbert802" <[hidden email]> wrote:

> I'm using camel to consume messages from an ActiveMQ queue. We don't want
> automatic redeliveries, so I've set MaximumRedeliveries on the activemq
> connection to 0 (by setting a RedeliveryPolicy on
> ActiveMQConnectionFactory).
>
> This works except that occasionally a message is now moved to the DLQ
> without a clear cause. There is nothing in the camel or ActiveMQ logs. The
> dlqDeliveryFailureCause property is set to "java.lang.Throwable: dispatch
> to
> ... exceeds redelivery policy limit...", but I can't see the underlying
> cause for redelivery.
>
> How can I see what's happening here?
>
> Thanks.
> Gijsbert
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Message-is-moved-to-DLQ-without-clear-cause-tp4689889.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Message is moved to DLQ without clear cause

gijsbert802
I tried that, but I only see that a message is being sent to my "incoming" queue and then some time later it is sent to the DLQ. There are no other references to that message-id in the logs.

Something like this:

2015-01-14 14:18:39,473 DEBUG o.a.activemq.broker.region.Queue - localhost Message ID:ZD68105205-2.local-54923-1421245093739-1:1:10:1:1 sent to queue://incoming
...
2015-01-14 14:19:10,824 DEBUG o.a.activemq.broker.region.Queue - localhost Message ID:ZD68105205-2.local-54923-1421245093739-1:1:10:1:1 sent to queue://ActiveMQ.DLQ
Reply | Threaded
Open this post in threaded view
|

Re: Message is moved to DLQ without clear cause

Tim Bain
Can you reproduce this in a standalone debugging environment (not
production)?  Maybe not right away, but reasonably?  Because you could
start the broker with the debugging port opened and attach Eclipse (or your
favorite debugger) to the broker process, then set a breakpoint on the line
in Queue.java that outputs the log line you've copied above (you'll want a
conditional breakpoint with a condition that the destination is
queue://ActiveMQ.DLQ so you don't stop there on every message).  Then when
you hit the breakpoint, you can hopefully walk up the stack trace to figure
out what caused the issue.  (In Eclipse, the Drop To Frame button will let
you restart whatever method you've currently selected in the stack trace,
so you can walk through the code again and see how you got to the log
line.)  I expect you'll find that the broker is being told by the ActiveMQ
code in the client that the message failed delivery, so then you'll
probably want to do the same kind of debugging exercise on the client...

Also, have you enabled debug logging for both ActiveMQ and Camel in the
client?  If not, I'd do that before I went to the debugger.

On Wed, Jan 14, 2015 at 7:24 AM, gijsbert802 <[hidden email]>
wrote:

> I tried that, but I only see that a message is being sent to my "incoming"
> queue and then some time later it is sent to the DLQ. There are no other
> references to that message-id in the logs.
>
> Something like this:
>
> 2015-01-14 14:18:39,473 DEBUG o.a.activemq.broker.region.Queue - localhost
> Message ID:ZD68105205-2.local-54923-1421245093739-1:1:10:1:1 sent to
> queue://incoming
> ...
> 2015-01-14 14:19:10,824 DEBUG o.a.activemq.broker.region.Queue - localhost
> Message ID:ZD68105205-2.local-54923-1421245093739-1:1:10:1:1 sent to
> queue://ActiveMQ.DLQ
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Message-is-moved-to-DLQ-without-clear-cause-tp4689889p4689899.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Message is moved to DLQ without clear cause

artnaseef
Can you try using a logging error handler in camel, as per the following page?

http://camel.apache.org/error-handler.html

A likely cause is an error in the camel route that leads to the failure.  However, it's also possible that the connection was dropped (check the ActiveMQ log files for signs of lost connections at the time of the message going to the DLQ).
Reply | Threaded
Open this post in threaded view
|

Re: Message is moved to DLQ without clear cause

gijsbert802
I added logging error handlers to my routes and also enabled debug logging.

When I search the logs for messages that have been moved to ActiveMQ.DLQ (by grepping on the messageId or breadcrumbId) there are no errors or warnings. I do see these two debug messages, but they also show up for messages that are sent successfully, so I'm not sure if this means anything..

2015-01-16 16:51:56,811 DEBUG o.apache.activemq.ActiveMQConnection - org.apache.activemq.ActiveMQConnection$3@159c9f59 no dispatcher for MessageDispatch …
2015-01-16 16:51:56,820 DEBUG o.a.activemq.ActiveMQMessageConsumer - ID:ZD68105205-2.local-54944-1421423463691-1:1:2:30 received with excessive redelivered: MessageDispatch

ActiveMQ also shows no errors or warnings in the logs.
Reply | Threaded
Open this post in threaded view
|

Re: Message is moved to DLQ without clear cause

gijsbert802
In reply to this post by Tim Bain
Thanks for the tip. Though I'm not sure if I can reproduce this in my development environment. I'll let you know if I find anything.