Messages in queue but not consumed

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Messages in queue but not consumed

bedag2013
We've noticed a problem in our production ActiveMQ 5.8 instances where messages remain in a queue even when there are active consumers on that queue. Most messages are consumed by the consumers, but a few do not and remain in the queue. They just seem to be stuck there.  Here's our setup:

-       2 instances of ActiveMQ 5.8 in a failover (master/slave) setup on linux
-       JDBC (MSSQL) for the message store

I've been able to reproduce this situation manually by doing the following:

1.      Create 2 consumers on a queue.
2.      Run 2 Producers (from 2 separate processes) simultaneously to send 1000 messages each to the queue.
3.      Use JMX to check the QueueSize of the queue.

Not everytime, but every second or third run of the 2 simultaneous producers, not all of the 2000 messages will be consumed.  When this happens, the consumers are still running and idle, but the QueueSize is non-zero (can be anything from 1 - 5, usually) and I can see the messages in the database. The unconsumed messages do not stop later messages from being consumed, however. A broker restart causes the 'stuck' messages to be immediately sent to the consumers.

It seems to have something to do with the multiple simultaneous producers as running just 1 producer at a time works as normal with all messages consumed. I have tried to create a self-contained unit test (i.e. vm broker) for this, but was not able to reproduce the situation. However, a unit test connecting to a remote broker did show the same situation (with many more stuck messages). That would lead me to guess that the issue is perhaps in the JDBC store code, but that's just a guess.

Is this a known issue?  Is there any workaround for it? Is it corrected in a version later than 5.8?

Thanks for the help.
Nick


Reply | Threaded
Open this post in threaded view
|

Re: Messages in queue but not consumed

RuralHunter
Maybe this one?
https://issues.apache.org/jira/browse/AMQ-2009

于 2014/7/3 17:26, Rahn Nicholas, Bedag 写道:

> We've noticed a problem in our production ActiveMQ 5.8 instances where messages remain in a queue even when there are active consumers on that queue. Most messages are consumed by the consumers, but a few do not and remain in the queue. They just seem to be stuck there.  Here's our setup:
>
> -       2 instances of ActiveMQ 5.8 in a failover (master/slave) setup on linux
> -       JDBC (MSSQL) for the message store
>
> I've been able to reproduce this situation manually by doing the following:
>
> 1.      Create 2 consumers on a queue.
> 2.      Run 2 Producers (from 2 separate processes) simultaneously to send 1000 messages each to the queue.
> 3.      Use JMX to check the QueueSize of the queue.
>
> Not everytime, but every second or third run of the 2 simultaneous producers, not all of the 2000 messages will be consumed.  When this happens, the consumers are still running and idle, but the QueueSize is non-zero (can be anything from 1 - 5, usually) and I can see the messages in the database. The unconsumed messages do not stop later messages from being consumed, however. A broker restart causes the 'stuck' messages to be immediately sent to the consumers.
>
> It seems to have something to do with the multiple simultaneous producers as running just 1 producer at a time works as normal with all messages consumed. I have tried to create a self-contained unit test (i.e. vm broker) for this, but was not able to reproduce the situation. However, a unit test connecting to a remote broker did show the same situation (with many more stuck messages). That would lead me to guess that the issue is perhaps in the JDBC store code, but that's just a guess.
>
> Is this a known issue?  Is there any workaround for it? Is it corrected in a version later than 5.8?
>
> Thanks for the help.
> Nick
>
>
>

Reply | Threaded
Open this post in threaded view
|

RE: Messages in queue but not consumed

Donnell Alwyn
In reply to this post by bedag2013
I had a similar issue in ActiveMQ 5.7.

Messages stuck on the pending queue would never come off, new messages would process ok. A broker restart was required to get messages off the pending queue.

Upgrading to 5.9 solved my problem.

Regards
Alwyn

------------------------------------------------------------------------
Alwyn Donnell

ISD Middleware and Architecture
Mizuho International
Bracken House
1 Friday Street
London EC4M 9JA

email: [hidden email]
Tel.: +44 (0)20 7090 6569


-----Original Message-----
From: Rahn Nicholas, Bedag [mailto:[hidden email]]
Sent: 03 July 2014 10:27
To: '[hidden email]'
Subject: Messages in queue but not consumed

We've noticed a problem in our production ActiveMQ 5.8 instances where messages remain in a queue even when there are active consumers on that queue. Most messages are consumed by the consumers, but a few do not and remain in the queue. They just seem to be stuck there.  Here's our setup:

-       2 instances of ActiveMQ 5.8 in a failover (master/slave) setup on linux
-       JDBC (MSSQL) for the message store

I've been able to reproduce this situation manually by doing the following:

1.      Create 2 consumers on a queue.
2.      Run 2 Producers (from 2 separate processes) simultaneously to send 1000 messages each to the queue.
3.      Use JMX to check the QueueSize of the queue.

Not everytime, but every second or third run of the 2 simultaneous producers, not all of the 2000 messages will be consumed.  When this happens, the consumers are still running and idle, but the QueueSize is non-zero (can be anything from 1 - 5, usually) and I can see the messages in the database. The unconsumed messages do not stop later messages from being consumed, however. A broker restart causes the 'stuck' messages to be immediately sent to the consumers.

It seems to have something to do with the multiple simultaneous producers as running just 1 producer at a time works as normal with all messages consumed. I have tried to create a self-contained unit test (i.e. vm broker) for this, but was not able to reproduce the situation. However, a unit test connecting to a remote broker did show the same situation (with many more stuck messages). That would lead me to guess that the issue is perhaps in the JDBC store code, but that's just a guess.

Is this a known issue?  Is there any workaround for it? Is it corrected in a version later than 5.8?

Thanks for the help.
Nick



This message and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this message in error please delete it and any files transmitted with it, after notifying [hidden email].
 
Any opinions expressed in this message may be those of the author and not necessarily those of the company. The company accepts no responsibility for the accuracy or completeness of any information contained herein. This message is not intended to create legal relations between the company and the recipient.

Recipients should please note that messages sent via the Internet may be intercepted and that caution should therefore be exercised before dispatching to the company any confidential or sensitive information.
Mizuho International plc Bracken House, One Friday Street, London EC4M 9JA. TEL. 020 72361090. Wholly owned subsidiary of Mizuho Securities Co., Ltd. Member of Mizuho Financial Group. Authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority. Member of the London Stock Exchange.

Registered in England No. 1203696. Registered office as above.
Reply | Threaded
Open this post in threaded view
|

RE: Messages in queue but not consumed

bedag2013
Thanks for the tip.

Does anyone know which change/bug-fix/Jira in 5.9 could have fixed this? Any clues/keywords to locating the fix...?

Thanks,
Nick


-----Original Message-----
From: Donnell Alwyn [mailto:[hidden email]]
Sent: Donnerstag, 3. Juli 2014 11:53
To: '[hidden email]'
Subject: RE: Messages in queue but not consumed

I had a similar issue in ActiveMQ 5.7.

Messages stuck on the pending queue would never come off, new messages would process ok. A broker restart was required to get messages off the pending queue.

Upgrading to 5.9 solved my problem.

Regards
Alwyn

------------------------------------------------------------------------
Alwyn Donnell

ISD Middleware and Architecture
Mizuho International
Bracken House
1 Friday Street
London EC4M 9JA

email: [hidden email]
Tel.: +44 (0)20 7090 6569


-----Original Message-----
From: Rahn Nicholas, Bedag [mailto:[hidden email]]
Sent: 03 July 2014 10:27
To: '[hidden email]'
Subject: Messages in queue but not consumed

We've noticed a problem in our production ActiveMQ 5.8 instances where messages remain in a queue even when there are active consumers on that queue. Most messages are consumed by the consumers, but a few do not and remain in the queue. They just seem to be stuck there.  Here's our setup:

-       2 instances of ActiveMQ 5.8 in a failover (master/slave) setup on linux
-       JDBC (MSSQL) for the message store

I've been able to reproduce this situation manually by doing the following:

1.      Create 2 consumers on a queue.
2.      Run 2 Producers (from 2 separate processes) simultaneously to send 1000 messages each to the queue.
3.      Use JMX to check the QueueSize of the queue.

Not everytime, but every second or third run of the 2 simultaneous producers, not all of the 2000 messages will be consumed.  When this happens, the consumers are still running and idle, but the QueueSize is non-zero (can be anything from 1 - 5, usually) and I can see the messages in the database. The unconsumed messages do not stop later messages from being consumed, however. A broker restart causes the 'stuck' messages to be immediately sent to the consumers.

It seems to have something to do with the multiple simultaneous producers as running just 1 producer at a time works as normal with all messages consumed. I have tried to create a self-contained unit test (i.e. vm broker) for this, but was not able to reproduce the situation. However, a unit test connecting to a remote broker did show the same situation (with many more stuck messages). That would lead me to guess that the issue is perhaps in the JDBC store code, but that's just a guess.

Is this a known issue?  Is there any workaround for it? Is it corrected in a version later than 5.8?

Thanks for the help.
Nick



This message and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this message in error please delete it and any files transmitted with it, after notifying [hidden email].
 
Any opinions expressed in this message may be those of the author and not necessarily those of the company. The company accepts no responsibility for the accuracy or completeness of any information contained herein. This message is not intended to create legal relations between the company and the recipient.

Recipients should please note that messages sent via the Internet may be intercepted and that caution should therefore be exercised before dispatching to the company any confidential or sensitive information.
Mizuho International plc Bracken House, One Friday Street, London EC4M 9JA. TEL. 020 72361090. Wholly owned subsidiary of Mizuho Securities Co., Ltd. Member of Mizuho Financial Group. Authorised by the Prudential Regulation Authority and regulated by the Financial Conduct Authority and the Prudential Regulation Authority. Member of the London Stock Exchange.

Registered in England No. 1203696. Registered office as above.