OutOfMemoryError in activeMQ and applcation gets stopped

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

OutOfMemoryError in activeMQ and applcation gets stopped

sainath-2
We have LiveTransfer application that use ActiveMQ to transfer data from
Production database to operational Database

Application is getting issue for every 50+ days i.e. OutOfMemoryError in
activeMQ and LT gets stopped. After restarting the AMQ services, it will
resume the transfer.

----
INFO  | jvm 1    | 2019/09/06 00:56:34 | java.lang.OutOfMemoryError: Java
heap space
INFO  | jvm 1    | 2019/09/06 00:56:34 | Dumping heap to
D:\RockwellSoftware\heap_log\java_pid5216.hprof ...
INFO  | jvm 1    | 2019/09/06 00:56:58 | Heap dump file created [4259842078
bytes in 23.626 secs]
----

Due to this issue we found data missing between Production database to
operational Database.

Currently activemq memory set to 4 GB. We continuously monitor AMQ memory
using JConsole and found memory is consuming more than i.e. 2.8 GB

This system is only dedicated for AMQ and Live transfer

AactiveMQ version-5.14.1

JDK version : 1.8 Update 60

Could you please help to provide the root cause and fix on the issue.




--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: OutOfMemoryError in activeMQ and applcation gets stopped

christopher.l.shannon
Have you configured limits on memory usage in the broker settings? You need
to look at a heap dump and see exactly where the memory is being used (if
it's messages on a destination, in prefetch etc).

Some general info about OOM and how to prevent it:
https://activemq.apache.org/javalangoutofmemory.html

On Mon, Nov 25, 2019 at 10:38 AM sainath <[hidden email]>
wrote:

> We have LiveTransfer application that use ActiveMQ to transfer data from
> Production database to operational Database
>
> Application is getting issue for every 50+ days i.e. OutOfMemoryError in
> activeMQ and LT gets stopped. After restarting the AMQ services, it will
> resume the transfer.
>
> ----
> INFO  | jvm 1    | 2019/09/06 00:56:34 | java.lang.OutOfMemoryError: Java
> heap space
> INFO  | jvm 1    | 2019/09/06 00:56:34 | Dumping heap to
> D:\RockwellSoftware\heap_log\java_pid5216.hprof ...
> INFO  | jvm 1    | 2019/09/06 00:56:58 | Heap dump file created [4259842078
> bytes in 23.626 secs]
> ----
>
> Due to this issue we found data missing between Production database to
> operational Database.
>
> Currently activemq memory set to 4 GB. We continuously monitor AMQ memory
> using JConsole and found memory is consuming more than i.e. 2.8 GB
>
> This system is only dedicated for AMQ and Live transfer
>
> AactiveMQ version-5.14.1
>
> JDK version : 1.8 Update 60
>
> Could you please help to provide the root cause and fix on the issue.
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Reply | Threaded
Open this post in threaded view
|

Re: OutOfMemoryError in activeMQ and applcation gets stopped

Tim Bain
Are your consumers keeping up with your producers? One easy way to run out
of memory is to produce messages faster than you consume them.

Tim

On Mon, Nov 25, 2019, 11:16 AM Christopher Shannon <
[hidden email]> wrote:

> Have you configured limits on memory usage in the broker settings? You need
> to look at a heap dump and see exactly where the memory is being used (if
> it's messages on a destination, in prefetch etc).
>
> Some general info about OOM and how to prevent it:
> https://activemq.apache.org/javalangoutofmemory.html
>
> On Mon, Nov 25, 2019 at 10:38 AM sainath <[hidden email]>
> wrote:
>
> > We have LiveTransfer application that use ActiveMQ to transfer data from
> > Production database to operational Database
> >
> > Application is getting issue for every 50+ days i.e. OutOfMemoryError in
> > activeMQ and LT gets stopped. After restarting the AMQ services, it will
> > resume the transfer.
> >
> > ----
> > INFO  | jvm 1    | 2019/09/06 00:56:34 | java.lang.OutOfMemoryError: Java
> > heap space
> > INFO  | jvm 1    | 2019/09/06 00:56:34 | Dumping heap to
> > D:\RockwellSoftware\heap_log\java_pid5216.hprof ...
> > INFO  | jvm 1    | 2019/09/06 00:56:58 | Heap dump file created
> [4259842078
> > bytes in 23.626 secs]
> > ----
> >
> > Due to this issue we found data missing between Production database to
> > operational Database.
> >
> > Currently activemq memory set to 4 GB. We continuously monitor AMQ memory
> > using JConsole and found memory is consuming more than i.e. 2.8 GB
> >
> > This system is only dedicated for AMQ and Live transfer
> >
> > AactiveMQ version-5.14.1
> >
> > JDK version : 1.8 Update 60
> >
> > Could you please help to provide the root cause and fix on the issue.
> >
> >
> >
> >
> > --
> > Sent from:
> > http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: OutOfMemoryError in activeMQ and applcation gets stopped

sainath-2
Hi All,

Yes, we have configured the limits on memory usage in the broker settings.

We also checked the heap dump and found that still 500 MB free space
available. So what could you be the reason for  OutOfMemoryError in activeMQ

Also we are monitoring the AMQ queues and there is no pending message.







--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: OutOfMemoryError in activeMQ and applcation gets stopped

Tim Bain
Are you by any chance using CMS as your GC strategy? It's got a nasty
shortcoming where because it doesn't compact Old Gen, it's possible to have
heap fragmentation that means it's not possible to allocate contiguous
space for a large object even though there's still space free. If so, you
should consider moving to G1, though if you were going to switch to it I'd
recommend you upgrade to a recent Java 8 JRE. There were some G1 bugs
addressed over the course of development of Java 8, and I'm not sure that
Update 60 would be recent enough to have all of them.

Ultimately, though, any shortcomings of the garbage collector as space gets
tight might just be hastening a problem you'd have hit anyway after a
little more time. So Chris's questions about exactly what the memory was
being used for remain relevant.

Tim

On Tue, Nov 26, 2019 at 8:49 PM sainath <[hidden email]>
wrote:

> Hi All,
>
> Yes, we have configured the limits on memory usage in the broker settings.
>
> We also checked the heap dump and found that still 500 MB free space
> available. So what could you be the reason for  OutOfMemoryError in
> activeMQ
>
> Also we are monitoring the AMQ queues and there is no pending message.
>
>
>
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Reply | Threaded
Open this post in threaded view
|

Re: OutOfMemoryError in activeMQ and applcation gets stopped

sainath-2
We have tried with java Update 181 and it still ran out of memory. Could you
please help us to provide some suggestion on this.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: OutOfMemoryError in activeMQ and applcation gets stopped

sainath-2
In reply to this post by Tim Bain
Hi All,

We disabled JMX and advisories, this seems to fix the memory issue.

In the ActiveMQ.xml file we

Added these flags useJmx="false"  and advisorySupport="false"

eg:

    <broker xmlns="http://activemq.apache.org/schema/core"
brokerName="localhost" dataDirectory="${activemq.data}" useJmx="false"
advisorySupport="false">

For any clients we added jms.watchTopicAdvisories=false

eg:
loaderBrokerURL=failover\:(tcp\://localhost\:61616)?jms.watchTopicAdvisories=false

Has anyone had to do this or understand why this would help?  Is there a bug
with these components that’s causing a memory leak?




--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: OutOfMemoryError in activeMQ and applcation gets stopped

Tim Bain
Does disabling one but not both of those things also fix the problem? Or is
the problem only eliminated if both are disabled?

Tim

On Mon, Jun 1, 2020, 7:44 AM sainath <[hidden email]>
wrote:

> Hi All,
>
> We disabled JMX and advisories, this seems to fix the memory issue.
>
> In the ActiveMQ.xml file we
>
> Added these flags useJmx="false"  and advisorySupport="false"
>
> eg:
>
>     <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="localhost" dataDirectory="${activemq.data}" useJmx="false"
> advisorySupport="false">
>
> For any clients we added jms.watchTopicAdvisories=false
>
> eg:
>
> loaderBrokerURL=failover\:(tcp\://localhost\:61616)?jms.watchTopicAdvisories=false
>
> Has anyone had to do this or understand why this would help?  Is there a
> bug
> with these components that’s causing a memory leak?
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Reply | Threaded
Open this post in threaded view
|

Re: OutOfMemoryError in activeMQ and applcation gets stopped

sainath-2
We tried just JMX which made a big difference but it seems that  advisories
still had some effect and decided to be safe and disabled both.  If someone
is able to read heap dumps we would be more than happy to collect them to
verify what if any objects are being retained.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: OutOfMemoryError in activeMQ and applcation gets stopped

jbonofre
Hi,

What ActiveMQ did you test ?

Regards
JB

> Le 2 juin 2020 à 17:26, sainath <[hidden email]> a écrit :
>
> We tried just JMX which made a big difference but it seems that  advisories
> still had some effect and decided to be safe and disabled both.  If someone
> is able to read heap dumps we would be more than happy to collect them to
> verify what if any objects are being retained.
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html