Connecting to broker slows down

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Connecting to broker slows down

Priit
Hey


I have a problem with the broker itself slowing down over a few days of uptime. Cpu and memory utilization are both ok. But establishing a connection to the broker takes longer and longer. I'm using tcp+ssl connector. Same issue with nio and regular tcp.

I have a setup of two brokers, both on version 5.7.0 (were previously on 5.5.0 same problem).


Any ideas?


Regards


Reply | Threaded
Open this post in threaded view
|

Re: Connecting to broker slows down

gtully
maybe post a thread dump of the broker. possibly if connections are
not pooled and are never released you could see this but that would be
reflected in fds and threads in use (so would appear as a memory
leak).

On 14 May 2013 13:41, Priit <[hidden email]> wrote:

> Hey
>
>
> I have a problem with the broker itself slowing down over a few days of
> uptime. Cpu and memory utilization are both ok. But establishing a
> connection to the broker takes longer and longer. I'm using tcp+ssl
> connector. Same issue with nio and regular tcp.
>
> I have a setup of two brokers, both on version 5.7.0 (were previously on
> 5.5.0 same problem).
>
>
> Any ideas?
>
>
> Regards
>
>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Connecting-to-broker-slows-down-tp4667017.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.



--
http://redhat.com
http://blog.garytully.com
Reply | Threaded
Open this post in threaded view
|

Re: Connecting to broker slows down

Priit
Reply | Threaded
Open this post in threaded view
|

Re: Connecting to broker slows down

ceposta
Are the brokers sluggish at the moment of those thread dumps?
You have 165 connections on broker A, 51 on broker B.
Those numbers don't seem too bad. What does the JVM memory look like?
What does the network look like between producers/consumers and broker?


On Wed, May 15, 2013 at 12:04 AM, Priit <[hidden email]> wrote:

> threaddump-2013-05-08---08-27-09_brokerB.txt
> <
> http://activemq.2283324.n4.nabble.com/file/n4667042/threaddump-2013-05-08---08-27-09_brokerB.txt
> >
>
> threaddump-2013-05-08---08-27-15_brokerA.txt
> <
> http://activemq.2283324.n4.nabble.com/file/n4667042/threaddump-2013-05-08---08-27-15_brokerA.txt
> >
>
>
> Threaddumps from both brokers
>
>
> Regards
>
>
> Priit
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Connecting-to-broker-slows-down-tp4667017p4667042.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Reply | Threaded
Open this post in threaded view
|

Re: Connecting to broker slows down

Priit
Brokers were slow at the time. Jvm max heap is set at 2048MB, starting the brokers clean memory usage usually starts from 200MB and slowly starts rising as more threads are being used. Usual thread usage will go up to about 300-400 on brokers together. Right now broker b is running 246 threads and using 400-500MB and broker a 132 threads and 600-900MB. Gradually memory usage increases with both brokers over time (could be a memory leak?). When the slowdown becomes noticeable jvm is not even pushing close to its heap limit. The producer consumer topology is difficult to explain. We're using an application that itself uses activemq as a backbone to relay messages to both internal components and external client application parts.
Reply | Threaded
Open this post in threaded view
|

Re: Connecting to broker slows down

Priit
Seems to be a cluster problem - if i take one node down, slowdown doesn't happen. What i also noticed that sometimes cpu usage would rise out of the normal range (on both brokers) and would only go down
when i stopped the network bridge (or restarted one of the brokers).

Network connector settings are:
<networkConnector
                uri="static://(ssl://brokerb:20561)?group=${activemq.group_id}"
                dynamicOnly="true"
                networkTTL="3"
                prefetchSize="1"
                decreaseNetworkConsumerPriority="true"
/>

I'm using static connector as multicast is not allowed between the networks. Maybe someone has ideas how to better configure network of brokers.