TCP connections leak

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

TCP connections leak

sssig23
Hello,

We are using ActiveMQ to deliver JMS messages in a distributed peer environment and there seem to be TCP connections leak issue happening on the server machine

Here are some details, but sorry if I am missing posting something here

netstat output on server machine -

tcp        0      0 ::ffff:10.5.69.95:8501      ::ffff:10.40.205.111:59726  ESTABLISHED
tcp        0      0 ::ffff:10.5.69.95:8501      ::ffff:10.40.205.111:59035  ESTABLISHED
tcp        0      0 ::ffff:10.5.69.95:8501      ::ffff:10.40.205.111:58414  ESTABLISHED

netstat output on client machine -

tcp        0      0  ::ffff:10.40.205.111:59726 ::ffff:10.5.69.95:8501    ESTABLISHED
tcp        0      0 ::ffff:10.40.205.111:59035  ::ffff:10.5.69.95:8501      ESTABLISHED
tcp        0      0 ::ffff:10.40.205.111:58414  ::ffff:10.5.69.95:8501      ESTABLISHED

After some time, tcp connections on client machine are cleared

tcp        0      0  ::ffff:10.40.205.111:59726 ::ffff:10.5.69.95:8501    ESTABLISHED

But tcp connnections on server machine remain

tcp        0      0 ::ffff:10.5.69.95:8501      ::ffff:10.40.205.111:59726  ESTABLISHED
tcp        0      0 ::ffff:10.5.69.95:8501      ::ffff:10.40.205.111:59035  ESTABLISHED
tcp        0      0 ::ffff:10.5.69.95:8501      ::ffff:10.40.205.111:58414  ESTABLISHED

And this behavior continues and on server machine keep TCP connections keep growing whereas the client machine do not

After about 1 hour on server machine -

tcp        0      0 ::ffff:10.5.69.95:8501      ::ffff:10.40.205.111:59726  ESTABLISHED
tcp        0      0 ::ffff:10.5.69.95:8501      ::ffff:10.40.205.111:59035  ESTABLISHED
tcp        0      0 ::ffff:10.5.69.95:8501      ::ffff:10.40.205.111:58414  ESTABLISHED
tcp        0      0 ::ffff:10.5.69.95:8501      ::ffff:10.40.205.111:61280  ESTABLISHED
tcp        0      0 ::ffff:10.5.69.95:8501      ::ffff:10.40.205.111:64875  ESTABLISHED

Rate of this tcp growth is variable and takes few days to reach hundreds of TCP connections and a week or two it gets into thousands before it reaches ulimit on file system and requires a restart of the application

Using transport.soTimeout=60000 does not seem to be very useful, could this be a firewall related issue or any TCP tuning parameters that can resolve the issue?

Thank you for any feedback
Reply | Threaded
Open this post in threaded view
|

Re: TCP connections leak

Tim Bain
What does "tcp connections on client machine are cleared" mean?  TCP
connections need to be closed properly to avoid leaving the other side in a
bad state, so if it means "I kill -9'ed the client processes" or something
else that would result in an improper closing of the connection, then fix
that and see if your leak goes away.

Also, there are options to use ActiveMQ's built-in keep-alive features to
detect broken TCP connections; see
http://activemq.apache.org/tcp-transport-reference.html.  But first try to
avoid breaking the connections in the first place.
On Jun 8, 2015 2:03 PM, "sssig23" <[hidden email]> wrote:

> Hello,
>
> We are using ActiveMQ to deliver JMS messages in a distributed peer
> environment and there seem to be TCP connections leak issue happening on
> the
> server machine
>
> Here are some details, but sorry if I am missing posting something here
>
> netstat output on server machine -
>
> tcp        0      0 ::ffff:10.5.69.95:8501      ::ffff:10.40.205.111:59726
> ESTABLISHED
> tcp        0      0 ::ffff:10.5.69.95:8501      ::ffff:10.40.205.111:59035
> ESTABLISHED
> tcp        0      0 ::ffff:10.5.69.95:8501      ::ffff:10.40.205.111:58414
> ESTABLISHED
>
> netstat output on client machine -
>
> tcp        0      0  ::ffff:10.40.205.111:59726 ::ffff:10.5.69.95:8501
> ESTABLISHED
> tcp        0      0 ::ffff:10.40.205.111:59035  ::ffff:10.5.69.95:8501
> ESTABLISHED
> tcp        0      0 ::ffff:10.40.205.111:58414  ::ffff:10.5.69.95:8501
> ESTABLISHED
>
> After some time, tcp connections on client machine are cleared
>
> tcp        0      0  ::ffff:10.40.205.111:59726 ::ffff:10.5.69.95:8501
> ESTABLISHED
>
> But tcp connnections on server machine remain
>
> tcp        0      0 ::ffff:10.5.69.95:8501      ::ffff:10.40.205.111:59726
> ESTABLISHED
> tcp        0      0 ::ffff:10.5.69.95:8501      ::ffff:10.40.205.111:59035
> ESTABLISHED
> tcp        0      0 ::ffff:10.5.69.95:8501      ::ffff:10.40.205.111:58414
> ESTABLISHED
>
> And this behavior continues and on server machine keep TCP connections keep
> growing whereas the client machine do not
>
> After about 1 hour on server machine -
>
> tcp        0      0 ::ffff:10.5.69.95:8501      ::ffff:10.40.205.111:59726
> ESTABLISHED
> tcp        0      0 ::ffff:10.5.69.95:8501      ::ffff:10.40.205.111:59035
> ESTABLISHED
> tcp        0      0 ::ffff:10.5.69.95:8501      ::ffff:10.40.205.111:58414
> ESTABLISHED
> tcp        0      0 ::ffff:10.5.69.95:8501      ::ffff:10.40.205.111:61280
> ESTABLISHED
> tcp        0      0 ::ffff:10.5.69.95:8501      ::ffff:10.40.205.111:64875
> ESTABLISHED
>
> Rate of this tcp growth is variable and takes few days to reach hundreds of
> TCP connections and a week or two it gets into thousands before it reaches
> ulimit on file system and requires a restart of the application
>
> Using transport.soTimeout=60000 does not seem to be very useful, could this
> be a firewall related issue or any TCP tuning parameters that can resolve
> the issue?
>
> Thank you for any feedback
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/TCP-connections-leak-tp4697435.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: TCP connections leak

sssig23
Thanks Tim for your reply.

Yes, the issue seems to be TCP connections seem to be in bad state - we are not sure what is causing this behavior but guessing it could be related to firewall idle session timeout (or possibly some other firewall rule?) which is dropping the TCP connection on the client machine but not the server machine. We believe it cannot be kill -9 as the client process is still running and continues to establish new TCP connections to the server

We don't have complete control or access to the firewall as we don't own them. Can you please elaborate on parameters for ActiveMQ keep-alive features to detect the broken TCP connections?

Thanks for your feedback!




Reply | Threaded
Open this post in threaded view
|

Re: TCP connections leak

Tim Bain
In theory, the useKeepAlive=true TcpTransport option (enabled by default,
as described in the link I referenced earlier) should detect dead
connections, but earlier this year another user saw a situation where one
side didn't detect a network cable pull even though the useKeepAlive option
was enabled.  So you can try the (unrelated) keepAlive=true option in
addition to see if it makes any difference for you and your firewall.  See
that page for details (including a link to a detailed description of TCP
keep-alive).

Tim
On Jun 9, 2015 10:05 AM, "sssig23" <[hidden email]> wrote:

> Thanks Tim for your reply.
>
> Yes, the issue seems to be TCP connections seem to be in bad state - we are
> not sure what is causing this behavior but guessing it could be related to
> firewall idle session timeout (or possibly some other firewall rule?) which
> is dropping the TCP connection on the client machine but not the server
> machine. We believe it cannot be kill -9 as the client process is still
> running and continues to establish new TCP connections to the server
>
> We don't have complete control or access to the firewall as we don't own
> them. Can you please elaborate on parameters for ActiveMQ keep-alive
> features to detect the broken TCP connections?
>
> Thanks for your feedback!
>
>
>
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/TCP-connections-leak-tp4697435p4697508.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>