OutOfMemoryError in activeMQ and applcation gets stopped

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

OutOfMemoryError in activeMQ and applcation gets stopped

sainath-2
We have LiveTransfer application that use ActiveMQ to transfer data from
Production database to operational Database

Application is getting issue for every 50+ days i.e. OutOfMemoryError in
activeMQ and LT gets stopped. After restarting the AMQ services, it will
resume the transfer.

----
INFO  | jvm 1    | 2019/09/06 00:56:34 | java.lang.OutOfMemoryError: Java
heap space
INFO  | jvm 1    | 2019/09/06 00:56:34 | Dumping heap to
D:\RockwellSoftware\heap_log\java_pid5216.hprof ...
INFO  | jvm 1    | 2019/09/06 00:56:58 | Heap dump file created [4259842078
bytes in 23.626 secs]
----

Due to this issue we found data missing between Production database to
operational Database.

Currently activemq memory set to 4 GB. We continuously monitor AMQ memory
using JConsole and found memory is consuming more than i.e. 2.8 GB

This system is only dedicated for AMQ and Live transfer

AactiveMQ version-5.14.1

JDK version : 1.8 Update 60

Could you please help to provide the root cause and fix on the issue.




--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: OutOfMemoryError in activeMQ and applcation gets stopped

christopher.l.shannon
Have you configured limits on memory usage in the broker settings? You need
to look at a heap dump and see exactly where the memory is being used (if
it's messages on a destination, in prefetch etc).

Some general info about OOM and how to prevent it:
https://activemq.apache.org/javalangoutofmemory.html

On Mon, Nov 25, 2019 at 10:38 AM sainath <[hidden email]>
wrote:

> We have LiveTransfer application that use ActiveMQ to transfer data from
> Production database to operational Database
>
> Application is getting issue for every 50+ days i.e. OutOfMemoryError in
> activeMQ and LT gets stopped. After restarting the AMQ services, it will
> resume the transfer.
>
> ----
> INFO  | jvm 1    | 2019/09/06 00:56:34 | java.lang.OutOfMemoryError: Java
> heap space
> INFO  | jvm 1    | 2019/09/06 00:56:34 | Dumping heap to
> D:\RockwellSoftware\heap_log\java_pid5216.hprof ...
> INFO  | jvm 1    | 2019/09/06 00:56:58 | Heap dump file created [4259842078
> bytes in 23.626 secs]
> ----
>
> Due to this issue we found data missing between Production database to
> operational Database.
>
> Currently activemq memory set to 4 GB. We continuously monitor AMQ memory
> using JConsole and found memory is consuming more than i.e. 2.8 GB
>
> This system is only dedicated for AMQ and Live transfer
>
> AactiveMQ version-5.14.1
>
> JDK version : 1.8 Update 60
>
> Could you please help to provide the root cause and fix on the issue.
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Reply | Threaded
Open this post in threaded view
|

Re: OutOfMemoryError in activeMQ and applcation gets stopped

Tim Bain
Are your consumers keeping up with your producers? One easy way to run out
of memory is to produce messages faster than you consume them.

Tim

On Mon, Nov 25, 2019, 11:16 AM Christopher Shannon <
[hidden email]> wrote:

> Have you configured limits on memory usage in the broker settings? You need
> to look at a heap dump and see exactly where the memory is being used (if
> it's messages on a destination, in prefetch etc).
>
> Some general info about OOM and how to prevent it:
> https://activemq.apache.org/javalangoutofmemory.html
>
> On Mon, Nov 25, 2019 at 10:38 AM sainath <[hidden email]>
> wrote:
>
> > We have LiveTransfer application that use ActiveMQ to transfer data from
> > Production database to operational Database
> >
> > Application is getting issue for every 50+ days i.e. OutOfMemoryError in
> > activeMQ and LT gets stopped. After restarting the AMQ services, it will
> > resume the transfer.
> >
> > ----
> > INFO  | jvm 1    | 2019/09/06 00:56:34 | java.lang.OutOfMemoryError: Java
> > heap space
> > INFO  | jvm 1    | 2019/09/06 00:56:34 | Dumping heap to
> > D:\RockwellSoftware\heap_log\java_pid5216.hprof ...
> > INFO  | jvm 1    | 2019/09/06 00:56:58 | Heap dump file created
> [4259842078
> > bytes in 23.626 secs]
> > ----
> >
> > Due to this issue we found data missing between Production database to
> > operational Database.
> >
> > Currently activemq memory set to 4 GB. We continuously monitor AMQ memory
> > using JConsole and found memory is consuming more than i.e. 2.8 GB
> >
> > This system is only dedicated for AMQ and Live transfer
> >
> > AactiveMQ version-5.14.1
> >
> > JDK version : 1.8 Update 60
> >
> > Could you please help to provide the root cause and fix on the issue.
> >
> >
> >
> >
> > --
> > Sent from:
> > http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: OutOfMemoryError in activeMQ and applcation gets stopped

sainath-2
Hi All,

Yes, we have configured the limits on memory usage in the broker settings.

We also checked the heap dump and found that still 500 MB free space
available. So what could you be the reason for  OutOfMemoryError in activeMQ

Also we are monitoring the AMQ queues and there is no pending message.







--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: OutOfMemoryError in activeMQ and applcation gets stopped

Tim Bain
Are you by any chance using CMS as your GC strategy? It's got a nasty
shortcoming where because it doesn't compact Old Gen, it's possible to have
heap fragmentation that means it's not possible to allocate contiguous
space for a large object even though there's still space free. If so, you
should consider moving to G1, though if you were going to switch to it I'd
recommend you upgrade to a recent Java 8 JRE. There were some G1 bugs
addressed over the course of development of Java 8, and I'm not sure that
Update 60 would be recent enough to have all of them.

Ultimately, though, any shortcomings of the garbage collector as space gets
tight might just be hastening a problem you'd have hit anyway after a
little more time. So Chris's questions about exactly what the memory was
being used for remain relevant.

Tim

On Tue, Nov 26, 2019 at 8:49 PM sainath <[hidden email]>
wrote:

> Hi All,
>
> Yes, we have configured the limits on memory usage in the broker settings.
>
> We also checked the heap dump and found that still 500 MB free space
> available. So what could you be the reason for  OutOfMemoryError in
> activeMQ
>
> Also we are monitoring the AMQ queues and there is no pending message.
>
>
>
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>