Broker not delivering persistent messages to consumer on queue

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Broker not delivering persistent messages to consumer on queue

xabhi
Hi,
I had a weird issue where broker was not delivering persistent messages to consumer on some queues. I tried following things on one of the affected queues:

1. I added a new consumer on that queue but it didn't receive any messages.
2. I restarted the original consumer but it didn't receive any messages after the restart.
3. I purged the messages that were held on the queue but then messages started accumulating again and broker didn't deliver any of the new messages. When I purged the expiry count didn't increase neither the dequeue and dispatch counters.
4. I sent 100 non-persistent messages on that queue, surprisingly it received those messages.
5. I tried sending 100 persistent messages on that queue, it didn't deliver anyone of them, all the messages were held by broker.

Only broker restart fixed this issue.

Another weird thing is that this kind of thing (where broker doesn't deliver messages to queue consumer for persistent messages) happened multiple times but it got auto resolved after 30 mins or so. But at one point it remained for about an hour and I had to restart the broker as it is affecting production setups.

I am attaching thread dumps of activemq process from the time issue was happening. Can someone please help.

jstack.jstack
jstack3.jstack3

Thanks,
Abhi
Reply | Threaded
Open this post in threaded view
|

Re: Broker not delivering persistent messages to consumer on queue

Tim Bain
Were there other messages in the queue at the time that would not have been
consumable by your new consumer due to selectors, message groups, etc.?
On Feb 9, 2015 8:22 AM, "xabhi" <[hidden email]> wrote:

> Hi,
> I had a weird issue where broker was not delivering persistent messages to
> consumer on some queues. I tried following things on one of the affected
> queues:
>
> 1. I added a new consumer on that queue but it didn't receive any messages.
> 2. I restarted the original consumer but it didn't receive any messages
> after the restart.
> 3. I purged the messages that were held on the queue but then messages
> started accumulating again and broker didn't deliver any of the new
> messages. When I purged the expiry count didn't increase neither the
> dequeue
> and dispatch counters.
> 4. I sent 100 non-persistent messages on that queue, surprisingly it
> received those messages.
> 5. I tried sending 100 persistent messages on that queue, it didn't deliver
> anyone of them, all the messages were held by broker.
>
> Only broker restart fixed this issue.
>
> Another weird thing is that this kind of thing (where broker doesn't
> deliver
> messages to queue consumer for persistent messages) happened multiple times
> but it got auto resolved after 30 mins or so. But at one point it remained
> for about an hour and I had to restart the broker as it is affecting
> production setups.
>
> I am attaching thread dumps of activemq process from the time issue was
> happening. Can someone please help.
>
> jstack.jstack
> <http://activemq.2283324.n4.nabble.com/file/n4691245/jstack.jstack>
> jstack3.jstack3
> <http://activemq.2283324.n4.nabble.com/file/n4691245/jstack3.jstack3>
>
> Thanks,
> Abhi
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Broker-not-delivering-persistent-messages-to-consumer-on-queue-tp4691245.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Broker not delivering persistent messages to consumer on queue

xabhi
Thanks for your prompt response.

As I mentioned before in my observations, I purged the queue and the new messages that came after purging were also not delivered to the consumer. And My consumer does not use any selectors or message groups.

Moreover, I believe purging the queue would rule out the case where there are unconsumed messages due to selectors or message groups.

Thanks,
Abhi
Reply | Threaded
Open this post in threaded view
|

Re: Broker not delivering persistent messages to consumer on queue

xabhi
Can someone from activemq dev take a look at this and help in debugging the root cause. As it affects the production setup every now and then and only way to resolve this is broker restart. I have also attached jstacks of activemq at the time the issue was happening.

Thanks,
Abhi
Reply | Threaded
Open this post in threaded view
|

Re: Broker not delivering persistent messages to consumer on queue

xabhi
Can anyone suggest what configuration would be causing the broker to go in this state where it is delivering non-persistent messages but not persistent messages on queues?

I tried sending persistent messages from STOMP producers but those were also not delivered.

This issue is not happening for topics. I am using ActiveMQ 5.10.0

This issue happens every weekend around same time.

Please suggest some direction/configuration I should look into.

Thanks,
Abhi