ActiveMQ 5.10 CPU spikes to 90-100% and stays there after a period of time

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

ActiveMQ 5.10 CPU spikes to 90-100% and stays there after a period of time

ryan segura
For reasons that I cannot figure out the ActiveMQ broker will spike its CPU to 90-100% and stay there.  I can kill the broker and restart the process (consumers will then reconnect) and it works fine again with low CPU utilization (0-1%).

How can I troubleshoot this?  There are hundreds of consumers and thousands of topics.  I wish there was some easy way for me to know what consumer or topic or queue is hogging all of the CPU.  How do people troubleshoot these problems?  As it is right now I feel very blind to this process.

Any guidance or assistance is appreciated.  Thank you.
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ 5.10 CPU spikes to 90-100% and stays there after a period of time

Tim Bain
You can start by using jstack to dump all threads' current stack traces to
give you an idea of what code is being run, though you won't know what data
it's being run against.  If you identify a suspicious block, you could then
start the broker with the debugging port enabled (either uncomment the line
in the start script or pass that variable to the script as a shell
variable), attach a debugger, and set a breakpoint to walk through the code.

A word of warning: breakpoints can cause clients (which can be other
brokers in a network of brokers) to disconnect because they don't receive a
response before the timeout.  So work quickly.  Or if messing up consumers
on a production broker isn't an acceptable risk, then use conditional
breakpoints to insert your own logging into the code.  Most people don't
know it, but in Eclipse the conditional expression block can be used to
execute arbitrary code every time you hit a breakpoint, as long as your
last line returns a boolean.  And the debugger won't stop if you return
false, so you can do:   LOG.info("Value of foo is: " + foo); return
false;   and you'll see the value of foo each time the breakpoint is hit.
(If there isn't a Log4J logger available where you set your breakpoint you
can use System.out.println(), but Log4J is preferred if it's available.)
On Dec 30, 2014 3:54 AM, "ryan segura" <[hidden email]> wrote:

> For reasons that I cannot figure out the ActiveMQ broker will spike its CPU
> to 90-100% and stay there.  I can kill the broker and restart the process
> (consumers will then reconnect) and it works fine again with low CPU
> utilization (0-1%).
>
> How can I troubleshoot this?  There are hundreds of consumers and thousands
> of topics.  I wish there was some easy way for me to know what consumer or
> topic or queue is hogging all of the CPU.  How do people troubleshoot these
> problems?  As it is right now I feel very blind to this process.
>
> Any guidance or assistance is appreciated.  Thank you.
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-5-10-CPU-spikes-to-90-100-and-stays-there-after-a-period-of-time-tp4689332.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ 5.10 CPU spikes to 90-100% and stays there after a period of time

Kevin Burton
In reply to this post by ryan segura
I suspect this is full GC.  Enable GC logging. You might already have a
gc.log and I’d take a look at that.  If you tail -F the file and you’re
seeing constant “Full GC” back to back at like 1 second apart you’re in a
GC storm and you need to increase your memory.

I *suspect* there’s a memory leak somewhere.  If I stop all my clients
ActiveMQ never recovers.  Either that or all the clients are using up
resources and the broker can’t release them since it spends all its time
doing full GCs.

I’m not sure this is the case for you though but I suspect it is…

On Tue, Dec 30, 2014 at 12:52 AM, ryan segura <[hidden email]>
wrote:

> For reasons that I cannot figure out the ActiveMQ broker will spike its CPU
> to 90-100% and stay there.  I can kill the broker and restart the process
> (consumers will then reconnect) and it works fine again with low CPU
> utilization (0-1%).
>
> How can I troubleshoot this?  There are hundreds of consumers and thousands
> of topics.  I wish there was some easy way for me to know what consumer or
> topic or queue is hogging all of the CPU.  How do people troubleshoot these
> problems?  As it is right now I feel very blind to this process.
>
> Any guidance or assistance is appreciated.  Thank you.
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-5-10-CPU-spikes-to-90-100-and-stays-there-after-a-period-of-time-tp4689332.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



--

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ 5.10 CPU spikes to 90-100% and stays there after a period of time

Tim Bain
If you have JMX enabled, just attach JConsole and look at the Memory tab;
it's easier than enabling GC logging for this basic use-case, and doesn't
require a broker restart.  If you do find lots of full GCs, then it might
be worth turning on GC logging to figure out why, but it's overkill for
figuring out if it's happening in the first place.
On Dec 30, 2014 2:26 PM, "Kevin Burton" <[hidden email]> wrote:

> I suspect this is full GC.  Enable GC logging. You might already have a
> gc.log and I’d take a look at that.  If you tail -F the file and you’re
> seeing constant “Full GC” back to back at like 1 second apart you’re in a
> GC storm and you need to increase your memory.
>
> I *suspect* there’s a memory leak somewhere.  If I stop all my clients
> ActiveMQ never recovers.  Either that or all the clients are using up
> resources and the broker can’t release them since it spends all its time
> doing full GCs.
>
> I’m not sure this is the case for you though but I suspect it is…
>
> On Tue, Dec 30, 2014 at 12:52 AM, ryan segura <[hidden email]>
> wrote:
>
> > For reasons that I cannot figure out the ActiveMQ broker will spike its
> CPU
> > to 90-100% and stay there.  I can kill the broker and restart the process
> > (consumers will then reconnect) and it works fine again with low CPU
> > utilization (0-1%).
> >
> > How can I troubleshoot this?  There are hundreds of consumers and
> thousands
> > of topics.  I wish there was some easy way for me to know what consumer
> or
> > topic or queue is hogging all of the CPU.  How do people troubleshoot
> these
> > problems?  As it is right now I feel very blind to this process.
> >
> > Any guidance or assistance is appreciated.  Thank you.
> >
> >
> >
> > --
> > View this message in context:
> >
> http://activemq.2283324.n4.nabble.com/ActiveMQ-5-10-CPU-spikes-to-90-100-and-stays-there-after-a-period-of-time-tp4689332.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>