Quantcast

Occasionally EOF on connection

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Occasionally EOF on connection

lucious_77
Hello all,

in our framework we use AMQ 5.14.1 and CMS version 3.9.4. From time to time we observe connection drops on client's side like the one below:

20170418 10:00:41.902 CRITICAL: void CMSConnectionManager::sltConnectionError(CMSExceptionListener::errorType, QString) "DataInputStream::readLong - Reached EOF" "tcp://127.0.0.1:61616?connection.useAsyncSend=true&soKeepAlive=true&wireFormat.MaxInactivityDuration=240000"

in the same time on Broker side this is reported in log file:

2017-04-18 10:00:29,358 | WARN | Transport Connection to: tcp://127.0.0.1:48612 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker 8

questions:

1. is there any way to log more detailed information when EOF occurs ? I mean we don;t want to flood log file with general DEBUG messages but rather TransportConnection only to figure out what;s going on when EOF appears ?
2. perhaps we do have something wrong with broker/client settings ?

broker (part):
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" dataDirectory="${activemq.data}" useJmx="true" advisorySupport="true" persistent="true" schedulePeriodForDestinationPurge="10000" networkConnectorStartAsync="true" allowTempAutoCreationOnSend="true" deleteAllMessagesOnStartup="false" >

<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="false" >
<pendingSubscriberPolicy>
<vmCursor/>
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="false" >
<pendingQueuePolicy>
<vmQueueCursor/>
</pendingQueuePolicy>
</policyEntry>

<policyEntry tempTopic="true" producerFlowControl="false" >
<pendingSubscriberPolicy>
<vmCursor/>
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry tempQueue="true" producerFlowControl="false" gcInactiveDestinations="true" inactiveTimoutBeforeGC="60000">
<pendingQueuePolicy>
<vmQueueCursor/>
</pendingQueuePolicy>
</policyEntry>

</policyEntries>
</policyMap>
</destinationPolicy>

<persistenceAdapter>
<levelDB directory="${activemq.data}/leveldb" logSize="107374182"/>
</persistenceAdapter>


<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="80" />
</memoryUsage>
<storeUsage>
<storeUsage limit="20 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="10 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>

<transportConnectors>
<transportConnector name="openwire" uri="nio://0.0.0.0:61616?transport.maximumConnections=1000&amp;wireFormat.maxFrameSize=314572800&amp;transport.keepAlive=true&amp;wireFormat.maxInactivityDuration=240000"/>
</transportConnectors>

client:
[activemq]
URL="tcp://127.0.0.1:61616?connection.useAsyncSend=true&soKeepAlive=true&wireFormat.MaxInactivityDuration=240000"

Thx
Marcin
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Occasionally EOF on connection

Tim Bain
Nothing in your configuration looks obviously wrong to me.

Is the connection between your broker and the clients that are having
problems unstable during the period of time that these problems are
occurring? Simultaneous EOFs on both ends of a socket connection makes me
think of packet loss or network disconnection; could that be what's going
on here?

If you're allowed to use it in your environment, Wireshark could help you
figure out whether the problem is a failure of packets to be sent (e.g. due
to network problems) or if one side or the other is choosing to close the
connection.

Tim

On Apr 19, 2017 8:37 AM, "Marcin Giedz" <[hidden email]> wrote:

> Hello all,
>
> in our framework we use AMQ 5.14.1 and CMS version 3.9.4. From time to
> time we observe connection drops on client's side like the one below:
>
> 20170418 10:00:41.902 CRITICAL: void CMSConnectionManager::
> sltConnectionError(CMSExceptionListener::errorType, QString)
> "DataInputStream::readLong - Reached EOF" "tcp://127.0.0.1:61616?
> connection.useAsyncSend=true&soKeepAlive=true&wireFormat.
> MaxInactivityDuration=240000"
>
> in the same time on Broker side this is reported in log file:
>
> 2017-04-18 10:00:29,358 | WARN | Transport Connection to: tcp://
> 127.0.0.1:48612 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport
> | ActiveMQ NIO Worker 8
>
> questions:
>
> 1. is there any way to log more detailed information when EOF occurs ? I
> mean we don;t want to flood log file with general DEBUG messages but rather
> TransportConnection only to figure out what;s going on when EOF appears ?
> 2. perhaps we do have something wrong with broker/client settings ?
>
> broker (part):
> <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="localhost" dataDirectory="${activemq.data}" useJmx="true"
> advisorySupport="true" persistent="true" schedulePeriodForDestinationPurge="10000"
> networkConnectorStartAsync="true" allowTempAutoCreationOnSend="true"
> deleteAllMessagesOnStartup="false" >
>
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">" producerFlowControl="false" >
> <pendingSubscriberPolicy>
> <vmCursor/>
> </pendingSubscriberPolicy>
> </policyEntry>
> <policyEntry queue=">" producerFlowControl="false" >
> <pendingQueuePolicy>
> <vmQueueCursor/>
> </pendingQueuePolicy>
> </policyEntry>
>
> <policyEntry tempTopic="true" producerFlowControl="false" >
> <pendingSubscriberPolicy>
> <vmCursor/>
> </pendingSubscriberPolicy>
> </policyEntry>
> <policyEntry tempQueue="true" producerFlowControl="false"
> gcInactiveDestinations="true" inactiveTimoutBeforeGC="60000">
> <pendingQueuePolicy>
> <vmQueueCursor/>
> </pendingQueuePolicy>
> </policyEntry>
>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
>
> <persistenceAdapter>
> <levelDB directory="${activemq.data}/leveldb" logSize="107374182"/>
> </persistenceAdapter>
>
>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage percentOfJvmHeap="80" />
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="20 gb"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="10 gb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
>
> <transportConnectors>
> <transportConnector name="openwire" uri="nio://0.0.0.0:61616?
> transport.maximumConnections=1000&amp;wireFormat.
> maxFrameSize=314572800&amp;transport.keepAlive=true&amp;wireFormat.
> maxInactivityDuration=240000"/>
> </transportConnectors>
>
> client:
> [activemq]
> URL="tcp://127.0.0.1:61616?connection.useAsyncSend=true&
> soKeepAlive=true&wireFormat.MaxInactivityDuration=240000"
>
> Thx
> Marcin
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Occasionally EOF on connection

lucious_77
Hi Tim,

the main issue in this case is that this is over localhost connection - URL contains 127.0.0.1 ;) we've got plenty of AMQ clients sitting on the same host AMQ broker is running.

wireshark is not a problem at all but I wonder how to "narrow" its filtering. Perhaps I can also change log4j in AMQ broker to log more details on transport connection level but I can't find how to do this without global DEBUG/TRACE option ?

Thx
Marcin


Od: "Tim Bain" <[hidden email]>
Do: "users" <[hidden email]>
Wysłane: czwartek, 20 kwiecień 2017 7:04:07
Temat: Re: Occasionally EOF on connection

Nothing in your configuration looks obviously wrong to me.

Is the connection between your broker and the clients that are having
problems unstable during the period of time that these problems are
occurring? Simultaneous EOFs on both ends of a socket connection makes me
think of packet loss or network disconnection; could that be what's going
on here?

If you're allowed to use it in your environment, Wireshark could help you
figure out whether the problem is a failure of packets to be sent (e.g. due
to network problems) or if one side or the other is choosing to close the
connection.

Tim

On Apr 19, 2017 8:37 AM, "Marcin Giedz" <[hidden email]> wrote:

> Hello all,
>
> in our framework we use AMQ 5.14.1 and CMS version 3.9.4. From time to
> time we observe connection drops on client's side like the one below:
>
> 20170418 10:00:41.902 CRITICAL: void CMSConnectionManager::
> sltConnectionError(CMSExceptionListener::errorType, QString)
> "DataInputStream::readLong - Reached EOF" "tcp://127.0.0.1:61616?
> connection.useAsyncSend=true&soKeepAlive=true&wireFormat.
> MaxInactivityDuration=240000"
>
> in the same time on Broker side this is reported in log file:
>
> 2017-04-18 10:00:29,358 | WARN | Transport Connection to: tcp://
> 127.0.0.1:48612 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport
> | ActiveMQ NIO Worker 8
>
> questions:
>
> 1. is there any way to log more detailed information when EOF occurs ? I
> mean we don;t want to flood log file with general DEBUG messages but rather
> TransportConnection only to figure out what;s going on when EOF appears ?
> 2. perhaps we do have something wrong with broker/client settings ?
>
> broker (part):
> <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="localhost" dataDirectory="${activemq.data}" useJmx="true"
> advisorySupport="true" persistent="true" schedulePeriodForDestinationPurge="10000"
> networkConnectorStartAsync="true" allowTempAutoCreationOnSend="true"
> deleteAllMessagesOnStartup="false" >
>
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">" producerFlowControl="false" >
> <pendingSubscriberPolicy>
> <vmCursor/>
> </pendingSubscriberPolicy>
> </policyEntry>
> <policyEntry queue=">" producerFlowControl="false" >
> <pendingQueuePolicy>
> <vmQueueCursor/>
> </pendingQueuePolicy>
> </policyEntry>
>
> <policyEntry tempTopic="true" producerFlowControl="false" >
> <pendingSubscriberPolicy>
> <vmCursor/>
> </pendingSubscriberPolicy>
> </policyEntry>
> <policyEntry tempQueue="true" producerFlowControl="false"
> gcInactiveDestinations="true" inactiveTimoutBeforeGC="60000">
> <pendingQueuePolicy>
> <vmQueueCursor/>
> </pendingQueuePolicy>
> </policyEntry>
>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
>
> <persistenceAdapter>
> <levelDB directory="${activemq.data}/leveldb" logSize="107374182"/>
> </persistenceAdapter>
>
>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage percentOfJvmHeap="80" />
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="20 gb"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="10 gb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
>
> <transportConnectors>
> <transportConnector name="openwire" uri="nio://0.0.0.0:61616?
> transport.maximumConnections=1000&amp;wireFormat.
> maxFrameSize=314572800&amp;transport.keepAlive=true&amp;wireFormat.
> maxInactivityDuration=240000"/>
> </transportConnectors>
>
> client:
> [activemq]
> URL="tcp://127.0.0.1:61616?connection.useAsyncSend=true&
> soKeepAlive=true&wireFormat.MaxInactivityDuration=240000"
>
> Thx
> Marcin
>

-
Loading...