Redelivery sticks to and blocks a consumer thread

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Redelivery sticks to and blocks a consumer thread

Jason Rosenberg
All,

I have been experimenting with using exponentialBackOff as a redelivery policy.  I have multiple parallel consumers.  I'm using AMQ 4.1.1, using Spring's DefaultMessageListener, within Tomcat 6.

I've noticed that when a message consumer has a failure, and throws an exception, and the message gets scheduled for retry, the particular consumer thread gets tied to the redelivery, and it doesn't process any more messages until it's gone through the full redelivery schedule for the error prone message.  So, with the exponentialBackOff mode, it's easy to set it to wait 10 minutes before redelivering a failed message, and in that time that consumer will be blocked.

If the consumer has a pre-fetch greater than 1, this means other messages which might be perfectly valid for processing are blocked until the troubled message goes through it's full redelivery cycle (which could be infinite!)....

I'm wondering if there's an alternate way to configure things, so that a message marked for redelivery goes back on the queue to be scheduled by the next available consumer (and not until it's next scheduled redelivery time has arrived!)....

If I have a queue with multiple consumers, and perhaps even a small percentage of the messages are corrupt and force multiple retries, it could eventually happen that all consumers are blocked waiting idly to retry the message at the top of it's list, even though other messages are readily available.  This would seem to be the case even when the prefetch limit is 1.

Thoughts?

Jason
Reply | Threaded
Open this post in threaded view
|

Re: Redelivery sticks to and blocks a consumer thread

Demian Mrakovich
Hi, bumping this one... Has this (I'd have to say major) issue been addressed in newer versions of ActiveMQ?

Jason Rosenberg wrote
All,

I have been experimenting with using exponentialBackOff as a redelivery policy.  I have multiple parallel consumers.  I'm using AMQ 4.1.1, using Spring's DefaultMessageListener, within Tomcat 6.

I've noticed that when a message consumer has a failure, and throws an exception, and the message gets scheduled for retry, the particular consumer thread gets tied to the redelivery, and it doesn't process any more messages until it's gone through the full redelivery schedule for the error prone message.  So, with the exponentialBackOff mode, it's easy to set it to wait 10 minutes before redelivering a failed message, and in that time that consumer will be blocked.

If the consumer has a pre-fetch greater than 1, this means other messages which might be perfectly valid for processing are blocked until the troubled message goes through it's full redelivery cycle (which could be infinite!)....

I'm wondering if there's an alternate way to configure things, so that a message marked for redelivery goes back on the queue to be scheduled by the next available consumer (and not until it's next scheduled redelivery time has arrived!)....

If I have a queue with multiple consumers, and perhaps even a small percentage of the messages are corrupt and force multiple retries, it could eventually happen that all consumers are blocked waiting idly to retry the message at the top of it's list, even though other messages are readily available.  This would seem to be the case even when the prefetch limit is 1.

Thoughts?

Jason
Reply | Threaded
Open this post in threaded view
|

Re: Redelivery sticks to and blocks a consumer thread

rajdavies
I'd certainly try 5.1

cheers,

Rob
On 6 Jun 2008, at 14:05, Demian Mrakovich wrote:

>
> Hi, bumping this one... Has this (I'd have to say major) issue been  
> addressed
> in newer versions of ActiveMQ?
>
>
> Jason Rosenberg wrote:
>>
>> All,
>>
>> I have been experimenting with using exponentialBackOff as a  
>> redelivery
>> policy.  I have multiple parallel consumers.  I'm using AMQ 4.1.1,  
>> using
>> Spring's DefaultMessageListener, within Tomcat 6.
>>
>> I've noticed that when a message consumer has a failure, and throws  
>> an
>> exception, and the message gets scheduled for retry, the particular
>> consumer thread gets tied to the redelivery, and it doesn't process  
>> any
>> more messages until it's gone through the full redelivery schedule  
>> for the
>> error prone message.  So, with the exponentialBackOff mode, it's  
>> easy to
>> set it to wait 10 minutes before redelivering a failed message, and  
>> in
>> that time that consumer will be blocked.
>>
>> If the consumer has a pre-fetch greater than 1, this means other  
>> messages
>> which might be perfectly valid for processing are blocked until the
>> troubled message goes through it's full redelivery cycle (which  
>> could be
>> infinite!)....
>>
>> I'm wondering if there's an alternate way to configure things, so  
>> that a
>> message marked for redelivery goes back on the queue to be  
>> scheduled by
>> the next available consumer (and not until it's next scheduled  
>> redelivery
>> time has arrived!)....
>>
>> If I have a queue with multiple consumers, and perhaps even a small
>> percentage of the messages are corrupt and force multiple retries, it
>> could eventually happen that all consumers are blocked waiting idly  
>> to
>> retry the message at the top of it's list, even though other  
>> messages are
>> readily available.  This would seem to be the case even when the  
>> prefetch
>> limit is 1.
>>
>> Thoughts?
>>
>> Jason
>>
>
> --
> View this message in context: http://www.nabble.com/Redelivery-sticks-to-and-blocks-a-consumer-thread-tp13261850s2354p17691674.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Reply | Threaded
Open this post in threaded view
|

Re: Redelivery sticks to and blocks a consumer thread

Demian Mrakovich
Thanks,

So you can confirm that threads are no longer blocked during retries in 5.1? Can I find any docs/info on the retry mechanism in 5.1 anywhere?
 
rajdavies wrote
I'd certainly try 5.1

cheers,

Rob
On 6 Jun 2008, at 14:05, Demian Mrakovich wrote:

>
> Hi, bumping this one... Has this (I'd have to say major) issue been  
> addressed
> in newer versions of ActiveMQ?
>
>
> Jason Rosenberg wrote:
>>
>> All,
>>
>> I have been experimenting with using exponentialBackOff as a  
>> redelivery
>> policy.  I have multiple parallel consumers.  I'm using AMQ 4.1.1,  
>> using
>> Spring's DefaultMessageListener, within Tomcat 6.
>>
>> I've noticed that when a message consumer has a failure, and throws  
>> an
>> exception, and the message gets scheduled for retry, the particular
>> consumer thread gets tied to the redelivery, and it doesn't process  
>> any
>> more messages until it's gone through the full redelivery schedule  
>> for the
>> error prone message.  So, with the exponentialBackOff mode, it's  
>> easy to
>> set it to wait 10 minutes before redelivering a failed message, and  
>> in
>> that time that consumer will be blocked.
>>
>> If the consumer has a pre-fetch greater than 1, this means other  
>> messages
>> which might be perfectly valid for processing are blocked until the
>> troubled message goes through it's full redelivery cycle (which  
>> could be
>> infinite!)....
>>
>> I'm wondering if there's an alternate way to configure things, so  
>> that a
>> message marked for redelivery goes back on the queue to be  
>> scheduled by
>> the next available consumer (and not until it's next scheduled  
>> redelivery
>> time has arrived!)....
>>
>> If I have a queue with multiple consumers, and perhaps even a small
>> percentage of the messages are corrupt and force multiple retries, it
>> could eventually happen that all consumers are blocked waiting idly  
>> to
>> retry the message at the top of it's list, even though other  
>> messages are
>> readily available.  This would seem to be the case even when the  
>> prefetch
>> limit is 1.
>>
>> Thoughts?
>>
>> Jason
>>
>
> --
> View this message in context: http://www.nabble.com/Redelivery-sticks-to-and-blocks-a-consumer-thread-tp13261850s2354p17691674.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Redelivery sticks to and blocks a consumer thread

Demian Mrakovich
In reply to this post by rajdavies
OK, I found some time to test this on 5.1 myself. Unfortunately the issue is NOT resolved as far as I can tell.

I don't see how anyone can live with this...
Consider a system which processes 100 messages/sec, with MaximumRedeliveries to 8 for a total of let's say an hour (in case of network failure against remote services etc.) before going to DLQ. Then we get about 20 corrupt messages in a row going into redelivery for a full 8 retries. The remaining incoming messages are fine, but what is the condition of our system now? With thousands of messages on queue, which could have been processed in the mean time, the system is effectively down for a full hour!

I understand this blocking mechanism is by design to ensure correct order of messages, but I think this should be optional, because having the system down for a full hour is much worse than a few messages out of order (at least for some systems) . So, is there a configuration option I have missed somewhere or is this still a firm design decision?  

Hoping for a reply,
Demian

rajdavies wrote
I'd certainly try 5.1

cheers,

Rob
On 6 Jun 2008, at 14:05, Demian Mrakovich wrote:

>
> Hi, bumping this one... Has this (I'd have to say major) issue been  
> addressed
> in newer versions of ActiveMQ?
>
>
> Jason Rosenberg wrote:
>>
>> All,
>>
>> I have been experimenting with using exponentialBackOff as a  
>> redelivery
>> policy.  I have multiple parallel consumers.  I'm using AMQ 4.1.1,  
>> using
>> Spring's DefaultMessageListener, within Tomcat 6.
>>
>> I've noticed that when a message consumer has a failure, and throws  
>> an
>> exception, and the message gets scheduled for retry, the particular
>> consumer thread gets tied to the redelivery, and it doesn't process  
>> any
>> more messages until it's gone through the full redelivery schedule  
>> for the
>> error prone message.  So, with the exponentialBackOff mode, it's  
>> easy to
>> set it to wait 10 minutes before redelivering a failed message, and  
>> in
>> that time that consumer will be blocked.
>>
>> If the consumer has a pre-fetch greater than 1, this means other  
>> messages
>> which might be perfectly valid for processing are blocked until the
>> troubled message goes through it's full redelivery cycle (which  
>> could be
>> infinite!)....
>>
>> I'm wondering if there's an alternate way to configure things, so  
>> that a
>> message marked for redelivery goes back on the queue to be  
>> scheduled by
>> the next available consumer (and not until it's next scheduled  
>> redelivery
>> time has arrived!)....
>>
>> If I have a queue with multiple consumers, and perhaps even a small
>> percentage of the messages are corrupt and force multiple retries, it
>> could eventually happen that all consumers are blocked waiting idly  
>> to
>> retry the message at the top of it's list, even though other  
>> messages are
>> readily available.  This would seem to be the case even when the  
>> prefetch
>> limit is 1.
>>
>> Thoughts?
>>
>> Jason
>>
>
> --
> View this message in context: http://www.nabble.com/Redelivery-sticks-to-and-blocks-a-consumer-thread-tp13261850s2354p17691674.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Redelivery sticks to and blocks a consumer thread

Ishitori
This post was updated on .
In reply to this post by Jason Rosenberg
Is there any progress conserning this issue? I am using the latest version of ActiveMQ 5.5.0 and the situation is the same: consumer threads doesn't receive any messages until full redelivery cycle is completed.

Can this be somehow resolved? Expected behaviour is: failed to process message is sent back to queue and after some time gots redelivered back.

UPD: Is this problem is due to using exponential backoff? If I switch it off will it resolve the problem?
UPD2: I don't care about message order. Is there any way to get what I want even if message order will be different?

UPD3: I found https://issues.apache.org/jira/browse/AMQ-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel#issue-tabs which says, that the resolution to that issue is:

1. Create a separate DLQ for a queue
2. Set redeliverypolicy to 0
3. Create camel route to move messages from DLQ back to queue after some period of time.

Okay, thats workaround I could use, but I now I am aware of a situation that message will be tried to be redelivered infinite amount of time. Can I set that this individual DLQ should be used only limited number of times? I am okay that retry will occur without exponential backoff, but I want number of retries to be limited.
Reply | Threaded
Open this post in threaded view
|

Re: Redelivery sticks to and blocks a consumer thread

gtully
try a current 5.6-SNAPSHOT
see: https://issues.apache.org/jira/browse/AMQ-1853

On 3 November 2011 11:54, Ishitori <[hidden email]> wrote:

> Is there any progress conserning this issue? I am using the latest version of
> ActiveMQ 5.5.0 and the situation is the same: consumer threads doesn't
> receive any messages until full redelivery cycle is completed.
>
> Can this be somehow resolved? Expected behaviour is: failed to process
> message is sent back to queue and after some time gots redelivered back
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Redelivery-sticks-to-and-blocks-a-consumer-thread-tp2358923p3985367.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



--
http://fusesource.com
http://blog.garytully.com
Reply | Threaded
Open this post in threaded view
|

Re: Redelivery sticks to and blocks a consumer thread

Ishitori
Hi, Gary.

Thanks for the answer. I forgot to mention that I am using NMS. Does it make the difference? I mean, that you gave the link to ActiveMQ client code, right? Is there something like that for NMS?
Reply | Threaded
Open this post in threaded view
|

Re: Redelivery sticks to and blocks a consumer thread

gtully
sorry, not sure about NMS. Keeping message order is the default so I
think NMS would need a similar enhancement to support out of order
delivery.

On 3 November 2011 14:18, Ishitori <[hidden email]> wrote:

> Hi, Gary.
>
> Thanks for the answer. I forgot to mention that I am using NMS. Does it make
> the difference? I mean, that you gave the link to ActiveMQ client code,
> right? Is there something like that for NMS?
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Redelivery-sticks-to-and-blocks-a-consumer-thread-tp2358923p3986026.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



--
http://fusesource.com
http://blog.garytully.com
Reply | Threaded
Open this post in threaded view
|

Re: Redelivery sticks to and blocks a consumer thread

Ishitori
Okay, it seems I have 2 options:

1) Merge all changes from Java into NMS
2) Try to use camel case routes to define a rule. The scheme will be the following:

* Consumer receives a message and it can't process it
* Consumer move message to its own DLQ, which is actually works as a temporary storage
* There is a camel route defined, like you have pointed in https://issues.apache.org/jira/browse/AMQ-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#issue-tabs that executes the following logic:

if (message.headers.RedeliveryCount < 5)
{
    Move message back to original queue
}
else
{
   move message to real DLQ
}

Can this being done using the page https://issues.apache.org/jira/browse/AMQ-2710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#issue-tabs and conditional logic from http://activemq.apache.org/sample-camel-routes.html ?
Reply | Threaded
Open this post in threaded view
|

Re: Redelivery sticks to and blocks a consumer thread

Ishitori
Hm, it seems that I don't fully understand how consumers are working.

I've merged fix from the JMS and it seems working nice if there is a small amount of messages to resend. But if I try to put a big amount of "bad" messages in queue, I will still get the same behavior: all my consumers got stuck.

Why does it happen? I have 20 consumers and 500 bad messages in a queue. Does it mean, that every consumer got stuck with a bad message and try to honor redelivery policy, though every consumer does release all prefetched messages? Can I change this behaviour or it is a way how ActiveMQ works (honoring of redelivery policy is up to Consumer and it is not the task of the broker...)

Maybe I should create a separate queue where I will put every message that I couldn't process at the first time? And some consumers will watch this queue and redelivery policy will be honored by them? What if a big amount of bad messages will be put there? Will I experience the same problem?