Active MQ Automatically Getting Restarted

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Active MQ Automatically Getting Restarted

tejas13

Hello Team,

Last one year we are using Active MQ in production system.

We are using version 5.14.5

In early days we were facing max connection exceeded problem, we resolved
that by identifying unnecessary open connections.

Now we are facing one more problem, after each 10 to 15 days, active mq is
getting restarted automatically. Before restarting it goes into hang mode.

Few findings from log analysis.
1]  JVM Hanged/Crashed

2] INFO   | jvm 3    | java.lang.OutOfMemoryError: GC overhead limit
exceeded
INFO   | jvm 3    | at
java.util.concurrent.CopyOnWriteArrayList.remove(Unknown Source)[:1.7.0_80]
INFO   | jvm 3    |  | at
java.util.concurrent.CopyOnWriteArraySet.remove(Unknown Source)[:1.7.0_80]

We have allocated 8 GB memory only for Active MQ.

Please help us to resolve this issue.





--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Active MQ Automatically Getting Restarted

Tim Bain
What garbage collection algorithm are you using? If it's CMS, it's possible
to get an OOM even though there is heap space still available.

What limits do you have configured on the memory store, and what percentage
is used shortly before the OOM occurs?

Tim

On Jan 22, 2018 3:51 AM, "tejas13" <[hidden email]> wrote:

>
> Hello Team,
>
> Last one year we are using Active MQ in production system.
>
> We are using version 5.14.5
>
> In early days we were facing max connection exceeded problem, we resolved
> that by identifying unnecessary open connections.
>
> Now we are facing one more problem, after each 10 to 15 days, active mq is
> getting restarted automatically. Before restarting it goes into hang mode.
>
> Few findings from log analysis.
> 1]  JVM Hanged/Crashed
>
> 2] INFO   | jvm 3    | java.lang.OutOfMemoryError: GC overhead limit
> exceeded
> INFO   | jvm 3    |     at
> java.util.concurrent.CopyOnWriteArrayList.remove(Unknown
> Source)[:1.7.0_80]
> INFO   | jvm 3    |  |  at
> java.util.concurrent.CopyOnWriteArraySet.remove(Unknown Source)[:1.7.0_80]
>
> We have allocated 8 GB memory only for Active MQ.
>
> Please help us to resolve this issue.
>
>
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Active MQ Automatically Getting Restarted

tejas13
Hello Team,

Thank you for reply.

We are not using any external GC mechanism.

We have allocated 8 GB memory. At the time of OOM greater than 90% memory
store is used.

Please help us.





--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Active MQ Automatically Getting Restarted

Tim Bain
The JVM uses a GC strategy even if you don't explicitly pick one; that's
why you don't have to explicitly free your heap variables in Java like you
do in C++. It's probably Parallel if you haven't customized it, but you
should check that. If I recall correctly, JConsole and/or JVisualVM might
tell you the GC strategy in use; if not, you can find the default in your
JVM's documentation.

Either way, if you're at 90+%, the GC is probably using most of the CPU
trying unsuccessfully to free up memory, which will make the broker
considerably less responsive.

So what Producer Flow Control limits have you put in place on the broker to
limit the amount of memory used by messages in the memory store? If you
haven't configured those, you probably need to do so. (Details of how to do
it and what options are available are on the wiki.)

Tim

On Jan 31, 2018 12:54 AM, "tejas13" <[hidden email]> wrote:

> Hello Team,
>
> Thank you for reply.
>
> We are not using any external GC mechanism.
>
> We have allocated 8 GB memory. At the time of OOM greater than 90% memory
> store is used.
>
> Please help us.
>
>
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805
> .html
>