Messages stuck in queue which are not expired

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Messages stuck in queue which are not expired

Martin Carpella
Hi,

We've ran into some problems since we updated to Activemq 5.12.1. Our
most busy queue has stuck messages which also do NOT expire.
The queue has around 200 producers (each producer has it's own message
group, making sure messages of a producer do not overtake each other)
which send non-persistent messages with a timeout of 40 seconds. They
produce around 20-30 msgs / second. 5 cached consumers exist.

Our problem is that all 5 consumers are consuming messages but some of
those messages are apparently not delivered. They get stuck in the
queue and stay there. They do not expire.
The only solution to "clear" the queue is to use a QueueBrowser and
inspect it. Once I connect with the QueueBrowser, all messages are
apparently moved to expiration. After that the processing works for a
couple of minutes until the messages start clogging up again.

The consumers do not use any form of selector other than the JMS
message group. The operation on the server side is very lightweight
and the load on the server is low so i do not think that it's the
fault of the server for not processing the messages fast enough (and
they should at least time out after their expiration deadline is
reached).
The problem scales apparently with the amount of the producers /
produced messages. Systems with ~100 producers have much fewer stuck
messages.

All our other queues use message groups as well but work as intended.
A maybe noticable difference is that the messages that get stuck are
non-persistent and have a TTL. We have some high-throughput queues
with non-expiring, non-persistent messages, which do not show those
symptoms.

Good ideas on what could be the issues are very welcome! Thanks in advance!

Best regards,
Martin
Reply | Threaded
Open this post in threaded view
|

Re: Messages stuck in queue which are not expired

Tim Bain
Martin, did you ever resolve this issue?

If not, I'd recommend looking at the messages that expire to see if there
is a pattern to them.

Also, do you have a single broker, or a network of brokers?  If the latter,
what is your networkTTL set to?

Tim
On Dec 2, 2015 9:50 AM, "Martin Carpella" <[hidden email]> wrote:

> Hi,
>
> We've ran into some problems since we updated to Activemq 5.12.1. Our
> most busy queue has stuck messages which also do NOT expire.
> The queue has around 200 producers (each producer has it's own message
> group, making sure messages of a producer do not overtake each other)
> which send non-persistent messages with a timeout of 40 seconds. They
> produce around 20-30 msgs / second. 5 cached consumers exist.
>
> Our problem is that all 5 consumers are consuming messages but some of
> those messages are apparently not delivered. They get stuck in the
> queue and stay there. They do not expire.
> The only solution to "clear" the queue is to use a QueueBrowser and
> inspect it. Once I connect with the QueueBrowser, all messages are
> apparently moved to expiration. After that the processing works for a
> couple of minutes until the messages start clogging up again.
>
> The consumers do not use any form of selector other than the JMS
> message group. The operation on the server side is very lightweight
> and the load on the server is low so i do not think that it's the
> fault of the server for not processing the messages fast enough (and
> they should at least time out after their expiration deadline is
> reached).
> The problem scales apparently with the amount of the producers /
> produced messages. Systems with ~100 producers have much fewer stuck
> messages.
>
> All our other queues use message groups as well but work as intended.
> A maybe noticable difference is that the messages that get stuck are
> non-persistent and have a TTL. We have some high-throughput queues
> with non-expiring, non-persistent messages, which do not show those
> symptoms.
>
> Good ideas on what could be the issues are very welcome! Thanks in advance!
>
> Best regards,
> Martin
>
Reply | Threaded
Open this post in threaded view
|

Re: Messages stuck in queue which are not expired

Martin C.
Hello,

sorry for the late response, missed the message. We are currently very
confident that the issue was triggered because one of the client
applications was still using a very old 5.7 library to connect to the
5.12 broker. Once we updated the client library in that particular
application, things started to improve and we are currently testing if
upgrading everything across all clients and also the broker to 5.13
fixes this for good.

Once done, I'll post our test results.

Best regards,
Martin


On Sun, Dec 20, 2015 at 4:50 PM, Tim Bain <[hidden email]> wrote:

> Martin, did you ever resolve this issue?
>
> If not, I'd recommend looking at the messages that expire to see if there
> is a pattern to them.
>
> Also, do you have a single broker, or a network of brokers?  If the latter,
> what is your networkTTL set to?
>
> Tim
> On Dec 2, 2015 9:50 AM, "Martin Carpella" <[hidden email]> wrote:
>
>> Hi,
>>
>> We've ran into some problems since we updated to Activemq 5.12.1. Our
>> most busy queue has stuck messages which also do NOT expire.
>> The queue has around 200 producers (each producer has it's own message
>> group, making sure messages of a producer do not overtake each other)
>> which send non-persistent messages with a timeout of 40 seconds. They
>> produce around 20-30 msgs / second. 5 cached consumers exist.
>>
>> Our problem is that all 5 consumers are consuming messages but some of
>> those messages are apparently not delivered. They get stuck in the
>> queue and stay there. They do not expire.
>> The only solution to "clear" the queue is to use a QueueBrowser and
>> inspect it. Once I connect with the QueueBrowser, all messages are
>> apparently moved to expiration. After that the processing works for a
>> couple of minutes until the messages start clogging up again.
>>
>> The consumers do not use any form of selector other than the JMS
>> message group. The operation on the server side is very lightweight
>> and the load on the server is low so i do not think that it's the
>> fault of the server for not processing the messages fast enough (and
>> they should at least time out after their expiration deadline is
>> reached).
>> The problem scales apparently with the amount of the producers /
>> produced messages. Systems with ~100 producers have much fewer stuck
>> messages.
>>
>> All our other queues use message groups as well but work as intended.
>> A maybe noticable difference is that the messages that get stuck are
>> non-persistent and have a TTL. We have some high-throughput queues
>> with non-expiring, non-persistent messages, which do not show those
>> symptoms.
>>
>> Good ideas on what could be the issues are very welcome! Thanks in advance!
>>
>> Best regards,
>> Martin
>>
Reply | Threaded
Open this post in threaded view
|

Re: Messages stuck in queue which are not expired

Tim Bain
Great, thanks for reporting back.

Tim

On Fri, Jan 8, 2016 at 1:04 PM, Martin C. <[hidden email]> wrote:

> Hello,
>
> sorry for the late response, missed the message. We are currently very
> confident that the issue was triggered because one of the client
> applications was still using a very old 5.7 library to connect to the
> 5.12 broker. Once we updated the client library in that particular
> application, things started to improve and we are currently testing if
> upgrading everything across all clients and also the broker to 5.13
> fixes this for good.
>
> Once done, I'll post our test results.
>
> Best regards,
> Martin
>
>
> On Sun, Dec 20, 2015 at 4:50 PM, Tim Bain <[hidden email]> wrote:
> > Martin, did you ever resolve this issue?
> >
> > If not, I'd recommend looking at the messages that expire to see if there
> > is a pattern to them.
> >
> > Also, do you have a single broker, or a network of brokers?  If the
> latter,
> > what is your networkTTL set to?
> >
> > Tim
> > On Dec 2, 2015 9:50 AM, "Martin Carpella" <[hidden email]>
> wrote:
> >
> >> Hi,
> >>
> >> We've ran into some problems since we updated to Activemq 5.12.1. Our
> >> most busy queue has stuck messages which also do NOT expire.
> >> The queue has around 200 producers (each producer has it's own message
> >> group, making sure messages of a producer do not overtake each other)
> >> which send non-persistent messages with a timeout of 40 seconds. They
> >> produce around 20-30 msgs / second. 5 cached consumers exist.
> >>
> >> Our problem is that all 5 consumers are consuming messages but some of
> >> those messages are apparently not delivered. They get stuck in the
> >> queue and stay there. They do not expire.
> >> The only solution to "clear" the queue is to use a QueueBrowser and
> >> inspect it. Once I connect with the QueueBrowser, all messages are
> >> apparently moved to expiration. After that the processing works for a
> >> couple of minutes until the messages start clogging up again.
> >>
> >> The consumers do not use any form of selector other than the JMS
> >> message group. The operation on the server side is very lightweight
> >> and the load on the server is low so i do not think that it's the
> >> fault of the server for not processing the messages fast enough (and
> >> they should at least time out after their expiration deadline is
> >> reached).
> >> The problem scales apparently with the amount of the producers /
> >> produced messages. Systems with ~100 producers have much fewer stuck
> >> messages.
> >>
> >> All our other queues use message groups as well but work as intended.
> >> A maybe noticable difference is that the messages that get stuck are
> >> non-persistent and have a TTL. We have some high-throughput queues
> >> with non-expiring, non-persistent messages, which do not show those
> >> symptoms.
> >>
> >> Good ideas on what could be the issues are very welcome! Thanks in
> advance!
> >>
> >> Best regards,
> >> Martin
> >>
>
Reply | Threaded
Open this post in threaded view
|

Re: Messages stuck in queue which are not expired

Martin C.
Hi,

It's been quite some time now and I just wanted to follow up for
archiving purposes. Since we updated everything to 5.13, we don't see
the issue any more. I am convinced it had been caused by the 5.7
client library.

Best regards,
Martin

On Sat, Jan 9, 2016 at 6:24 AM, Tim Bain <[hidden email]> wrote:

> Great, thanks for reporting back.
>
> Tim
>
> On Fri, Jan 8, 2016 at 1:04 PM, Martin C. <[hidden email]> wrote:
>
>> Hello,
>>
>> sorry for the late response, missed the message. We are currently very
>> confident that the issue was triggered because one of the client
>> applications was still using a very old 5.7 library to connect to the
>> 5.12 broker. Once we updated the client library in that particular
>> application, things started to improve and we are currently testing if
>> upgrading everything across all clients and also the broker to 5.13
>> fixes this for good.
>>
>> Once done, I'll post our test results.
>>
>> Best regards,
>> Martin
>>
>>
>> On Sun, Dec 20, 2015 at 4:50 PM, Tim Bain <[hidden email]> wrote:
>> > Martin, did you ever resolve this issue?
>> >
>> > If not, I'd recommend looking at the messages that expire to see if there
>> > is a pattern to them.
>> >
>> > Also, do you have a single broker, or a network of brokers?  If the
>> latter,
>> > what is your networkTTL set to?
>> >
>> > Tim
>> > On Dec 2, 2015 9:50 AM, "Martin Carpella" <[hidden email]>
>> wrote:
>> >
>> >> Hi,
>> >>
>> >> We've ran into some problems since we updated to Activemq 5.12.1. Our
>> >> most busy queue has stuck messages which also do NOT expire.
>> >> The queue has around 200 producers (each producer has it's own message
>> >> group, making sure messages of a producer do not overtake each other)
>> >> which send non-persistent messages with a timeout of 40 seconds. They
>> >> produce around 20-30 msgs / second. 5 cached consumers exist.
>> >>
>> >> Our problem is that all 5 consumers are consuming messages but some of
>> >> those messages are apparently not delivered. They get stuck in the
>> >> queue and stay there. They do not expire.
>> >> The only solution to "clear" the queue is to use a QueueBrowser and
>> >> inspect it. Once I connect with the QueueBrowser, all messages are
>> >> apparently moved to expiration. After that the processing works for a
>> >> couple of minutes until the messages start clogging up again.
>> >>
>> >> The consumers do not use any form of selector other than the JMS
>> >> message group. The operation on the server side is very lightweight
>> >> and the load on the server is low so i do not think that it's the
>> >> fault of the server for not processing the messages fast enough (and
>> >> they should at least time out after their expiration deadline is
>> >> reached).
>> >> The problem scales apparently with the amount of the producers /
>> >> produced messages. Systems with ~100 producers have much fewer stuck
>> >> messages.
>> >>
>> >> All our other queues use message groups as well but work as intended.
>> >> A maybe noticable difference is that the messages that get stuck are
>> >> non-persistent and have a TTL. We have some high-throughput queues
>> >> with non-expiring, non-persistent messages, which do not show those
>> >> symptoms.
>> >>
>> >> Good ideas on what could be the issues are very welcome! Thanks in
>> advance!
>> >>
>> >> Best regards,
>> >> Martin
>> >>
>>