Quantcast

ActiveMQ threads clogging CPU

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

ActiveMQ threads clogging CPU

reindheer
Our application runs on arm 32 bit linux, centos, java 1.8, activemq 5.13.3. The application communicates with backoffice through activemq connected through mobile sim card. application has embedded activemq and broker is at back office.

Whenever there is low or no signal, the connection is lost and reconnects to the broker using the
static:(failover:(tcp://${Broker1}:66617,tcp://${Broker2}:66617)?maxReconnectAttempts=0&randomize=true) in our embedded application.

Now whenerver we run the pplication initially teh CPU utilization for user is at 20%, after 2-4 hours the CPU utilization for user shot up to 100and sometimes upto 150%.

We collected jvm threaddump when at 20% and at max CPU utilization and found that major difference is due to the threads created by ActiveMQ. Most of them are "ActiveMQ Connector:" & "ActiveMQ Task-" threads. Could some one please explain the reason for these threads creation, is ActiveMQ not cleaning up threads, or some configuration to be done on our side.

Did any one face this issue earlier.

Thanks You,
Dheeraj
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ActiveMQ threads clogging CPU

Tim Bain
Could you post the full stack trace of each of the two categories of
threads you referenced?

If I understand your description, you have an intermittent network
connection, but the CPU utilization eventually grows to 100-150% *even
after* you re-establish a strong network connection over the mobile
network?  Or are you just saying that you're seeing high CPU during a
network drop, but the CPU drops back to the 20% range once connectivity is
re-established?  Also, since ActiveMQ clients are unable to continue
running when they're unable to publish messages and are essentially unable
to do work when they can't receive new messages to consume, I would expect
there to be a period of high CPU utilization immediately after a reconnect,
as the client completes any work that was blocked waiting for the
connection to be re-established, so I'd only be concerned about CPU
utilization that stayed high even after any built-up backlog of work was
completed.

Tim

On Wed, Apr 12, 2017 at 8:10 AM, reindheer <[hidden email]> wrote:

> Our application runs on arm 32 bit linux, centos, java 1.8, activemq
> 5.13.3.
> The application communicates with backoffice through activemq connected
> through mobile sim card. application has embedded activemq and broker is at
> back office.
>
> Whenever there is low or no signal, the connection is lost and reconnects
> to
> the broker using the
> static:(failover:(tcp://${Broker1}:66617,tcp://${Broker2}:66617)?
> maxReconnectAttempts=0&amp;randomize=true)
> in our embedded application.
>
> Now whenerver we run the pplication initially teh CPU utilization for user
> is at 20%, after 2-4 hours the CPU utilization for user shot up to 100and
> sometimes upto 150%.
>
> We collected jvm threaddump when at 20% and at max CPU utilization and
> found
> that major difference is due to the threads created by ActiveMQ. Most of
> them are "ActiveMQ Connector:" & "ActiveMQ Task-" threads. Could some one
> please explain the reason for these threads creation, is ActiveMQ not
> cleaning up threads, or some configuration to be done on our side.
>
> Did any one face this issue earlier.
>
> Thanks You,
> Dheeraj
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/ActiveMQ-threads-clogging-CPU-tp4724902.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: ActiveMQ threads clogging CPU

reindheer
This post was updated on .
Hi Tim,

Thank you for replying.

Yes we have intermittent network connection, as our app runs on a vehicle which is travelling. We observed this issue when our app runs on a vehicle which looses connection. On one particular instance it looses connection for 6 hours continously.

Regardless of whether it is connected or disconnected we are observing this issue after few hours. For one vehicle we see its occurs after 3 hours and on some after 6 hours. We see 20% CPU utilization when app is started and when it is 100 or more we see 228 threads for ActiveMQ Task out of total 460 threads in jvm. And thread name that start with ActiveMQ are 272. Also attaching activemq.txt below for ActiveMQ Task and Transport thread stack trace. We see ActiveMQ Task mostly in waiting state.

Please let me know if i am not clear.

Thank You,
Dheeraj

activemq.txt
Loading...