ACTIVEMQ 5.14.4 Connection Reset error (Broker and Client)

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

ACTIVEMQ 5.14.4 Connection Reset error (Broker and Client)

yogu13
Hello,

 I am facing a weird connection reset errors being logged at debug level
randomly. I am using ACTIVEMQ 5.14.4 with JDK 8-141 deployed on a Windows
2012 R2 Machine. The ActiveMQ Broker has 4 GB heap defined with almost 14000
connections (Yeah we havent used any pooling yet it uses a different
connection for every queue/topic per thread) for queues and topics across
various java processes.

The message seen is logs are :-

07:09:55:740|1209-00246:DEBUG
[org.apache.activemq.broker.TransportConnection.Transport] - Transport
Connection to: tcp://172.31.26.125:50898 failed: java.net.SocketException:
Connection reset Thread {ActiveMQ Transport:
tcp:///172.31.26.125:50898@61616}
java.net.SocketException: Connection reset
                at
java.net.SocketInputStream.read(SocketInputStream.java:210)
                at
java.net.SocketInputStream.read(SocketInputStream.java:141)
                at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
                at
org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:634)
                at
org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:59)
                at
org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:619)
                at java.io.DataInputStream.readInt(DataInputStream.java:387)
                at
org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268)
                at
org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240)
                at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232)
                at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)
                at java.lang.Thread.run(Thread.java:748)
07:09:55:740|0324-00246:DEBUG
[org.apache.activemq.broker.jmx.ManagementContext] - Unregistering MBean
org.apache.activemq:type=Broker,brokerName=AMQ,connector=clientConnectors,connectorName=tcp,connectionViewType=clientId,connectionName=ID_WIN-31THCOHACHB-49421-1509705222509-4_1
Thread {ActiveMQ Transport: tcp:///172.31.26.125:50898@61616}
07:09:55:740|0312-00246:DEBUG
[org.apache.activemq.broker.jmx.ManagementContext] - Unregistering MBean
org.apache.activemq:type=Broker,brokerName=AMQ,connector=clientConnectors,connectorName=tcp,connectionViewType=remoteAddress,connectionName=tcp_//172.31.26.125_50898
Thread {ActiveMQ Transport: tcp:///172.31.26.125:50898@61616}
07:09:55:740|0155-00675:DEBUG
[org.apache.activemq.broker.TransportConnection] - Stopping connection:
tcp://172.31.26.125:50898 Thread {ActiveMQ BrokerService[AMQ] Task-103}
07:09:55:740|0160-00675:DEBUG
[org.apache.activemq.transport.tcp.TcpTransport] - Stopping transport
tcp:///172.31.26.125:50898@61616 Thread {ActiveMQ BrokerService[AMQ]
Task-103}
07:09:55:740|0306-00675:DEBUG [org.apache.activemq.thread.TaskRunnerFactory]
- Initialized TaskRunnerFactory[ActiveMQ Task] using ExecutorService:
java.util.concurrent.ThreadPoolExecutor@30b3bb2e[Running, pool size = 0,
active threads = 0, queued tasks = 0, completed tasks = 0] Thread {ActiveMQ
BrokerService[AMQ] Task-103}
07:09:55:741|0156-00714:DEBUG
[org.apache.activemq.transport.tcp.TcpTransport] - Closed socket
Socket[addr=/172.31.26.125,port=50898,localport=61616] Thread {ActiveMQ
Task-1}
07:09:55:741|0271-00675:DEBUG [org.apache.activemq.util.ThreadPoolUtils] -
Forcing shutdown of ExecutorService:
java.util.concurrent.ThreadPoolExecutor@30b3bb2e[Running, pool size = 1,
active threads = 0, queued tasks = 0, completed tasks = 1] Thread {ActiveMQ
BrokerService[AMQ] Task-103}
07:09:55:741|0153-00675:DEBUG
[org.apache.activemq.broker.TransportConnection] - Stopped transport:
tcp://172.31.26.125:50898 Thread {ActiveMQ BrokerService[AMQ] Task-103}
07:09:55:741|0168-00675:DEBUG
[org.apache.activemq.broker.TransportConnection] - Cleaning up connection
resources: tcp://172.31.26.125:50898 Thread {ActiveMQ BrokerService[AMQ]
Task-103}
07:09:55:741|0173-00675:DEBUG
[org.apache.activemq.broker.TransportConnection] - remove connection id:
ID:WIN-31THCOHACHB-49421-1509705222509-5:1 Thread {ActiveMQ
BrokerService[AMQ] Task-103}

Things that i have tried
   1. Setting TCP level Keep Alive to true
   2. Disabling InactivityMonitorThread by setting Max Inactivity Duration
to 0
   3. using connection pool for creating connections which brought the
entire connection count to 300-350 connections in all, thus reducing load on
OS

But none of the above changes have worked for me to get rid of the problem.

my entire infrastructure is hosted on AWS and CPU usage not going beyond
15-20%. Also I am not using failover protocol

I would like to know 1) are these exception traces considered to be harmful
? 2) If yes what could be causing this connection reset ?

Let me know if any more info is needed.
Regards,
-Yogesh



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html
Reply | Threaded
Open this post in threaded view
|

Re: ACTIVEMQ 5.14.4 Connection Reset error (Broker and Client)

yogu13
Figured out what the problem was...

For people wondering why this happened, this was an OS issue Windows 2008
R2, looks like TCP Keep Alive is not set by default. On moving to Windows
2012 R2 the problem never surfaced.

Regards,
-Yogesh



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html