Artemis 2.11.0 - Cluster node OOM - Queue build up on internal $.artemis.internal.sf... queue

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Artemis 2.11.0 - Cluster node OOM - Queue build up on internal $.artemis.internal.sf... queue

Graham Stewart
Hi,

Firstly thank you and the team for Artemis. We have been using the fruits
of your labour from way back in HornetQ days. Thank you.

We have encountered a similar issue to the one described in:
https://issues.apache.org/jira/browse/ARTEMIS-2373

I do not have a reproducible test case but I will try to describe what we
are seeing.

The cluster is symmetric across 24 nodes.The environment is under continual
steady load.
Sporadically - say once in 10 days we see a node flip into a state where
one of the
$.artemis.internal.sf.... queues starts to back up. Typically we see pretty
much 0 queue depth on these internal queues.
The queue continues to fill pushing heap usage to its limit and
eventually after an hour OOM.

We are running using an Embedded server.
We see warnings that the $.artemis.internal.sf... queues do not have Dead
Letter or Expiry Addresses configured. Should they be?

I realise this is big ask, but what other details can I provide that might
help diagnose the issue.

Many thanks

Graham Stewart
Reply | Threaded
Open this post in threaded view
|

Re: Artemis 2.11.0 - Cluster node OOM - Queue build up on internal $.artemis.internal.sf... queue

brusdev
Hi Graham,

can you enable the dumping of the java heap when a
java.lang.OutOfMemoryError exception is thrown and send a screenshot or
summary?

To enable the heap dump, you could uncomment the following line in
`artemis.profile` file:
# JAVA_ARGS="$JAVA_ARGS -XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=${ARTEMIS_OOME_DUMP}"

Thanks,
Domenico



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Artemis 2.11.0 - Cluster node OOM - Queue build up on internal $.artemis.internal.sf... queue

Graham Stewart
Thanks Domenico,

We have a heap dump from when this happened last week - it's large - 8GB.

We can see the internal queue that blew up was

$.artemis.internal.sf.cluster-prodemea-r186951.39.7adc9da9-6c30-11ea-b3c0-0050568f93e5


From the heap dump that QueueImpl has

messagesReplaced 0

messagesKilled 0

messagesExpired 0

ackAttempts 17215914
messagesAcknowleged 17215914
messagesAdded 18437336

queueMemorySize 108598463

Can you let me know what other  details would be useful? Can run some OQL
against the dump to pull things out if that would help.
I tried to attach a thread dump - but it exceeds the mailing list limit.
Would it be useful?

Thanks

Graham


On Mon, 30 Mar 2020 at 21:51, brusdev <[hidden email]> wrote:

> Hi Graham,
>
> can you enable the dumping of the java heap when a
> java.lang.OutOfMemoryError exception is thrown and send a screenshot or
> summary?
>
> To enable the heap dump, you could uncomment the following line in
> `artemis.profile` file:
> # JAVA_ARGS="$JAVA_ARGS -XX:+HeapDumpOnOutOfMemoryError
> -XX:HeapDumpPath=${ARTEMIS_OOME_DUMP}"
>
> Thanks,
> Domenico
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Artemis 2.11.0 - Cluster node OOM - Queue build up on internal $.artemis.internal.sf... queue

Graham Stewart
Hi,

Have done some more investigations into the heap dump.
All 1.2m messages on the internal queue were addressed to a specific
jms.topic.
There were many other producers publishing to the same topic at the time
and there was no apparent issue with the topic consumer.

A question:
Should we assign a Dead Letter address for the internal
$.artemis.internal.sf.... queues? I am confused on whether we should or
should not do that.

Really appreciate any help you can give me on this.

Many thanks

Graham



On Mon, 30 Mar 2020 at 23:08, Graham Stewart <[hidden email]>
wrote:

> Thanks Domenico,
>
> We have a heap dump from when this happened last week - it's large - 8GB.
>
> We can see the internal queue that blew up was
>
>
> $.artemis.internal.sf.cluster-prodemea-r186951.39.7adc9da9-6c30-11ea-b3c0-0050568f93e5
>
>
> From the heap dump that QueueImpl has
>
> messagesReplaced 0
>
> messagesKilled 0
>
> messagesExpired 0
>
> ackAttempts 17215914
> messagesAcknowleged 17215914
> messagesAdded 18437336
>
> queueMemorySize 108598463
>
> Can you let me know what other  details would be useful? Can run some OQL
> against the dump to pull things out if that would help.
> I tried to attach a thread dump - but it exceeds the mailing list limit.
> Would it be useful?
>
> Thanks
>
> Graham
>
>
> On Mon, 30 Mar 2020 at 21:51, brusdev <[hidden email]> wrote:
>
>> Hi Graham,
>>
>> can you enable the dumping of the java heap when a
>> java.lang.OutOfMemoryError exception is thrown and send a screenshot or
>> summary?
>>
>> To enable the heap dump, you could uncomment the following line in
>> `artemis.profile` file:
>> # JAVA_ARGS="$JAVA_ARGS -XX:+HeapDumpOnOutOfMemoryError
>> -XX:HeapDumpPath=${ARTEMIS_OOME_DUMP}"
>>
>> Thanks,
>> Domenico
>>
>>
>>
>> --
>> Sent from:
>> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Artemis 2.11.0 - Cluster node OOM - Queue build up on internal $.artemis.internal.sf... queue

brusdev
Hi Graham,

the warnings `No Expiry Address configured for queue...` or `No Expiry
Address configured for queue...` are often due to the lack of the default
address-setting (address-setting match="#") or to the mismatch from
`wildcard-addresses` node and the `address-settings` nodes (the `any-words`
char is different from the char used in the address-setting matches).

Regards,
Domenico



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html