Quantcast

Occasionally EOF on connection

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Occasionally EOF on connection

lucious_77
Hello all,

in our framework we use AMQ 5.14.1 and CMS version 3.9.4. From time to time we observe connection drops on client's side like the one below:

20170418 10:00:41.902 CRITICAL: void CMSConnectionManager::sltConnectionError(CMSExceptionListener::errorType, QString) "DataInputStream::readLong - Reached EOF" "tcp://127.0.0.1:61616?connection.useAsyncSend=true&soKeepAlive=true&wireFormat.MaxInactivityDuration=240000"

in the same time on Broker side this is reported in log file:

2017-04-18 10:00:29,358 | WARN | Transport Connection to: tcp://127.0.0.1:48612 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ NIO Worker 8

questions:

1. is there any way to log more detailed information when EOF occurs ? I mean we don;t want to flood log file with general DEBUG messages but rather TransportConnection only to figure out what;s going on when EOF appears ?
2. perhaps we do have something wrong with broker/client settings ?

broker (part):
<broker xmlns="http://activemq.apache.org/schema/core" brokerName="localhost" dataDirectory="${activemq.data}" useJmx="true" advisorySupport="true" persistent="true" schedulePeriodForDestinationPurge="10000" networkConnectorStartAsync="true" allowTempAutoCreationOnSend="true" deleteAllMessagesOnStartup="false" >

<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry topic=">" producerFlowControl="false" >
<pendingSubscriberPolicy>
<vmCursor/>
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry queue=">" producerFlowControl="false" >
<pendingQueuePolicy>
<vmQueueCursor/>
</pendingQueuePolicy>
</policyEntry>

<policyEntry tempTopic="true" producerFlowControl="false" >
<pendingSubscriberPolicy>
<vmCursor/>
</pendingSubscriberPolicy>
</policyEntry>
<policyEntry tempQueue="true" producerFlowControl="false" gcInactiveDestinations="true" inactiveTimoutBeforeGC="60000">
<pendingQueuePolicy>
<vmQueueCursor/>
</pendingQueuePolicy>
</policyEntry>

</policyEntries>
</policyMap>
</destinationPolicy>

<persistenceAdapter>
<levelDB directory="${activemq.data}/leveldb" logSize="107374182"/>
</persistenceAdapter>


<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage percentOfJvmHeap="80" />
</memoryUsage>
<storeUsage>
<storeUsage limit="20 gb"/>
</storeUsage>
<tempUsage>
<tempUsage limit="10 gb"/>
</tempUsage>
</systemUsage>
</systemUsage>

<transportConnectors>
<transportConnector name="openwire" uri="nio://0.0.0.0:61616?transport.maximumConnections=1000&amp;wireFormat.maxFrameSize=314572800&amp;transport.keepAlive=true&amp;wireFormat.maxInactivityDuration=240000"/>
</transportConnectors>

client:
[activemq]
URL="tcp://127.0.0.1:61616?connection.useAsyncSend=true&soKeepAlive=true&wireFormat.MaxInactivityDuration=240000"

Thx
Marcin
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Occasionally EOF on connection

Tim Bain
Nothing in your configuration looks obviously wrong to me.

Is the connection between your broker and the clients that are having
problems unstable during the period of time that these problems are
occurring? Simultaneous EOFs on both ends of a socket connection makes me
think of packet loss or network disconnection; could that be what's going
on here?

If you're allowed to use it in your environment, Wireshark could help you
figure out whether the problem is a failure of packets to be sent (e.g. due
to network problems) or if one side or the other is choosing to close the
connection.

Tim

On Apr 19, 2017 8:37 AM, "Marcin Giedz" <[hidden email]> wrote:

> Hello all,
>
> in our framework we use AMQ 5.14.1 and CMS version 3.9.4. From time to
> time we observe connection drops on client's side like the one below:
>
> 20170418 10:00:41.902 CRITICAL: void CMSConnectionManager::
> sltConnectionError(CMSExceptionListener::errorType, QString)
> "DataInputStream::readLong - Reached EOF" "tcp://127.0.0.1:61616?
> connection.useAsyncSend=true&soKeepAlive=true&wireFormat.
> MaxInactivityDuration=240000"
>
> in the same time on Broker side this is reported in log file:
>
> 2017-04-18 10:00:29,358 | WARN | Transport Connection to: tcp://
> 127.0.0.1:48612 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport
> | ActiveMQ NIO Worker 8
>
> questions:
>
> 1. is there any way to log more detailed information when EOF occurs ? I
> mean we don;t want to flood log file with general DEBUG messages but rather
> TransportConnection only to figure out what;s going on when EOF appears ?
> 2. perhaps we do have something wrong with broker/client settings ?
>
> broker (part):
> <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="localhost" dataDirectory="${activemq.data}" useJmx="true"
> advisorySupport="true" persistent="true" schedulePeriodForDestinationPurge="10000"
> networkConnectorStartAsync="true" allowTempAutoCreationOnSend="true"
> deleteAllMessagesOnStartup="false" >
>
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">" producerFlowControl="false" >
> <pendingSubscriberPolicy>
> <vmCursor/>
> </pendingSubscriberPolicy>
> </policyEntry>
> <policyEntry queue=">" producerFlowControl="false" >
> <pendingQueuePolicy>
> <vmQueueCursor/>
> </pendingQueuePolicy>
> </policyEntry>
>
> <policyEntry tempTopic="true" producerFlowControl="false" >
> <pendingSubscriberPolicy>
> <vmCursor/>
> </pendingSubscriberPolicy>
> </policyEntry>
> <policyEntry tempQueue="true" producerFlowControl="false"
> gcInactiveDestinations="true" inactiveTimoutBeforeGC="60000">
> <pendingQueuePolicy>
> <vmQueueCursor/>
> </pendingQueuePolicy>
> </policyEntry>
>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
>
> <persistenceAdapter>
> <levelDB directory="${activemq.data}/leveldb" logSize="107374182"/>
> </persistenceAdapter>
>
>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage percentOfJvmHeap="80" />
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="20 gb"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="10 gb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
>
> <transportConnectors>
> <transportConnector name="openwire" uri="nio://0.0.0.0:61616?
> transport.maximumConnections=1000&amp;wireFormat.
> maxFrameSize=314572800&amp;transport.keepAlive=true&amp;wireFormat.
> maxInactivityDuration=240000"/>
> </transportConnectors>
>
> client:
> [activemq]
> URL="tcp://127.0.0.1:61616?connection.useAsyncSend=true&
> soKeepAlive=true&wireFormat.MaxInactivityDuration=240000"
>
> Thx
> Marcin
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Occasionally EOF on connection

lucious_77
Hi Tim,

the main issue in this case is that this is over localhost connection - URL contains 127.0.0.1 ;) we've got plenty of AMQ clients sitting on the same host AMQ broker is running.

wireshark is not a problem at all but I wonder how to "narrow" its filtering. Perhaps I can also change log4j in AMQ broker to log more details on transport connection level but I can't find how to do this without global DEBUG/TRACE option ?

Thx
Marcin


Od: "Tim Bain" <[hidden email]>
Do: "users" <[hidden email]>
Wysłane: czwartek, 20 kwiecień 2017 7:04:07
Temat: Re: Occasionally EOF on connection

Nothing in your configuration looks obviously wrong to me.

Is the connection between your broker and the clients that are having
problems unstable during the period of time that these problems are
occurring? Simultaneous EOFs on both ends of a socket connection makes me
think of packet loss or network disconnection; could that be what's going
on here?

If you're allowed to use it in your environment, Wireshark could help you
figure out whether the problem is a failure of packets to be sent (e.g. due
to network problems) or if one side or the other is choosing to close the
connection.

Tim

On Apr 19, 2017 8:37 AM, "Marcin Giedz" <[hidden email]> wrote:

> Hello all,
>
> in our framework we use AMQ 5.14.1 and CMS version 3.9.4. From time to
> time we observe connection drops on client's side like the one below:
>
> 20170418 10:00:41.902 CRITICAL: void CMSConnectionManager::
> sltConnectionError(CMSExceptionListener::errorType, QString)
> "DataInputStream::readLong - Reached EOF" "tcp://127.0.0.1:61616?
> connection.useAsyncSend=true&soKeepAlive=true&wireFormat.
> MaxInactivityDuration=240000"
>
> in the same time on Broker side this is reported in log file:
>
> 2017-04-18 10:00:29,358 | WARN | Transport Connection to: tcp://
> 127.0.0.1:48612 failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport
> | ActiveMQ NIO Worker 8
>
> questions:
>
> 1. is there any way to log more detailed information when EOF occurs ? I
> mean we don;t want to flood log file with general DEBUG messages but rather
> TransportConnection only to figure out what;s going on when EOF appears ?
> 2. perhaps we do have something wrong with broker/client settings ?
>
> broker (part):
> <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="localhost" dataDirectory="${activemq.data}" useJmx="true"
> advisorySupport="true" persistent="true" schedulePeriodForDestinationPurge="10000"
> networkConnectorStartAsync="true" allowTempAutoCreationOnSend="true"
> deleteAllMessagesOnStartup="false" >
>
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <policyEntry topic=">" producerFlowControl="false" >
> <pendingSubscriberPolicy>
> <vmCursor/>
> </pendingSubscriberPolicy>
> </policyEntry>
> <policyEntry queue=">" producerFlowControl="false" >
> <pendingQueuePolicy>
> <vmQueueCursor/>
> </pendingQueuePolicy>
> </policyEntry>
>
> <policyEntry tempTopic="true" producerFlowControl="false" >
> <pendingSubscriberPolicy>
> <vmCursor/>
> </pendingSubscriberPolicy>
> </policyEntry>
> <policyEntry tempQueue="true" producerFlowControl="false"
> gcInactiveDestinations="true" inactiveTimoutBeforeGC="60000">
> <pendingQueuePolicy>
> <vmQueueCursor/>
> </pendingQueuePolicy>
> </policyEntry>
>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
>
> <persistenceAdapter>
> <levelDB directory="${activemq.data}/leveldb" logSize="107374182"/>
> </persistenceAdapter>
>
>
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage percentOfJvmHeap="80" />
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="20 gb"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="10 gb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
>
> <transportConnectors>
> <transportConnector name="openwire" uri="nio://0.0.0.0:61616?
> transport.maximumConnections=1000&amp;wireFormat.
> maxFrameSize=314572800&amp;transport.keepAlive=true&amp;wireFormat.
> maxInactivityDuration=240000"/>
> </transportConnectors>
>
> client:
> [activemq]
> URL="tcp://127.0.0.1:61616?connection.useAsyncSend=true&
> soKeepAlive=true&wireFormat.MaxInactivityDuration=240000"
>
> Thx
> Marcin
>

-
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Occasionally EOF on connection

Tim Bain
Your right, a connection to localhost should be rock-solid. Sorry I missed
that.

Is there any possibility that a firewall (either OS or application) might
be interfering with your connections? I feel like I'm grasping at straws a
little bit, but I can't explain EOFs on both sides of the connection.

The other variable here is that you're using the CMS client library (which
I have no experience with and which isn't the subject of most posts on this
list), and maybe the EOF it's showing doesn't mean the same thing as an
EOFException would in the Java client library. @Tim Bish, do you see
anything in the described behavior that would be specific to the CMS client
and that might explain what's going on?

Tim

On Apr 20, 2017 8:58 PM, "Marcin Giedz" <[hidden email]> wrote:

> Hi Tim,
>
> the main issue in this case is that this is over localhost connection -
> URL contains 127.0.0.1 ;) we've got plenty of AMQ clients sitting on the
> same host AMQ broker is running.
>
> wireshark is not a problem at all but I wonder how to "narrow" its
> filtering. Perhaps I can also change log4j in AMQ broker to log more
> details on transport connection level but I can't find how to do this
> without global DEBUG/TRACE option ?
>
> Thx
> Marcin
>
>
> Od: "Tim Bain" <[hidden email]>
> Do: "users" <[hidden email]>
> Wysłane: czwartek, 20 kwiecień 2017 7:04:07
> Temat: Re: Occasionally EOF on connection
>
> Nothing in your configuration looks obviously wrong to me.
>
> Is the connection between your broker and the clients that are having
> problems unstable during the period of time that these problems are
> occurring? Simultaneous EOFs on both ends of a socket connection makes me
> think of packet loss or network disconnection; could that be what's going
> on here?
>
> If you're allowed to use it in your environment, Wireshark could help you
> figure out whether the problem is a failure of packets to be sent (e.g. due
> to network problems) or if one side or the other is choosing to close the
> connection.
>
> Tim
>
> On Apr 19, 2017 8:37 AM, "Marcin Giedz" <[hidden email]> wrote:
>
> > Hello all,
> >
> > in our framework we use AMQ 5.14.1 and CMS version 3.9.4. From time to
> > time we observe connection drops on client's side like the one below:
> >
> > 20170418 10:00:41.902 CRITICAL: void CMSConnectionManager::
> > sltConnectionError(CMSExceptionListener::errorType, QString)
> > "DataInputStream::readLong - Reached EOF" "tcp://127.0.0.1:61616?
> > connection.useAsyncSend=true&soKeepAlive=true&wireFormat.
> > MaxInactivityDuration=240000"
> >
> > in the same time on Broker side this is reported in log file:
> >
> > 2017-04-18 10:00:29,358 | WARN | Transport Connection to: tcp://
> > 127.0.0.1:48612 failed: java.io.EOFException |
> org.apache.activemq.broker.TransportConnection.Transport
> > | ActiveMQ NIO Worker 8
> >
> > questions:
> >
> > 1. is there any way to log more detailed information when EOF occurs ? I
> > mean we don;t want to flood log file with general DEBUG messages but
> rather
> > TransportConnection only to figure out what;s going on when EOF appears ?
> > 2. perhaps we do have something wrong with broker/client settings ?
> >
> > broker (part):
> > <broker xmlns="http://activemq.apache.org/schema/core"
> > brokerName="localhost" dataDirectory="${activemq.data}" useJmx="true"
> > advisorySupport="true" persistent="true" schedulePeriodForDestinationPu
> rge="10000"
> > networkConnectorStartAsync="true" allowTempAutoCreationOnSend="true"
> > deleteAllMessagesOnStartup="false" >
> >
> > <destinationPolicy>
> > <policyMap>
> > <policyEntries>
> > <policyEntry topic=">" producerFlowControl="false" >
> > <pendingSubscriberPolicy>
> > <vmCursor/>
> > </pendingSubscriberPolicy>
> > </policyEntry>
> > <policyEntry queue=">" producerFlowControl="false" >
> > <pendingQueuePolicy>
> > <vmQueueCursor/>
> > </pendingQueuePolicy>
> > </policyEntry>
> >
> > <policyEntry tempTopic="true" producerFlowControl="false" >
> > <pendingSubscriberPolicy>
> > <vmCursor/>
> > </pendingSubscriberPolicy>
> > </policyEntry>
> > <policyEntry tempQueue="true" producerFlowControl="false"
> > gcInactiveDestinations="true" inactiveTimoutBeforeGC="60000">
> > <pendingQueuePolicy>
> > <vmQueueCursor/>
> > </pendingQueuePolicy>
> > </policyEntry>
> >
> > </policyEntries>
> > </policyMap>
> > </destinationPolicy>
> >
> > <persistenceAdapter>
> > <levelDB directory="${activemq.data}/leveldb" logSize="107374182"/>
> > </persistenceAdapter>
> >
> >
> > <systemUsage>
> > <systemUsage>
> > <memoryUsage>
> > <memoryUsage percentOfJvmHeap="80" />
> > </memoryUsage>
> > <storeUsage>
> > <storeUsage limit="20 gb"/>
> > </storeUsage>
> > <tempUsage>
> > <tempUsage limit="10 gb"/>
> > </tempUsage>
> > </systemUsage>
> > </systemUsage>
> >
> > <transportConnectors>
> > <transportConnector name="openwire" uri="nio://0.0.0.0:61616?
> > transport.maximumConnections=1000&amp;wireFormat.
> > maxFrameSize=314572800&amp;transport.keepAlive=true&amp;wireFormat.
> > maxInactivityDuration=240000"/>
> > </transportConnectors>
> >
> > client:
> > [activemq]
> > URL="tcp://127.0.0.1:61616?connection.useAsyncSend=true&
> > soKeepAlive=true&wireFormat.MaxInactivityDuration=240000"
> >
> > Thx
> > Marcin
> >
>
> -
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Occasionally EOF on connection

tabish121@gmail.com
On 04/26/2017 04:00 PM, Tim Bain wrote:

> Your right, a connection to localhost should be rock-solid. Sorry I missed
> that.
>
> Is there any possibility that a firewall (either OS or application) might
> be interfering with your connections? I feel like I'm grasping at straws a
> little bit, but I can't explain EOFs on both sides of the connection.
>
> The other variable here is that you're using the CMS client library (which
> I have no experience with and which isn't the subject of most posts on this
> list), and maybe the EOF it's showing doesn't mean the same thing as an
> EOFException would in the Java client library. @Tim Bish, do you see
> anything in the described behavior that would be specific to the CMS client
> and that might explain what's going on?

With the little information provided all I can say is that it appears
the connection was forcibly broken on both ends and the client and
broker reacted accordingly by issuing an EOFException. Without more
debugging there's really no why to know for sure what is going on.

>
> Tim
>
> On Apr 20, 2017 8:58 PM, "Marcin Giedz" <[hidden email]> wrote:
>
>> Hi Tim,
>>
>> the main issue in this case is that this is over localhost connection -
>> URL contains 127.0.0.1 ;) we've got plenty of AMQ clients sitting on the
>> same host AMQ broker is running.
>>
>> wireshark is not a problem at all but I wonder how to "narrow" its
>> filtering. Perhaps I can also change log4j in AMQ broker to log more
>> details on transport connection level but I can't find how to do this
>> without global DEBUG/TRACE option ?
>>
>> Thx
>> Marcin
>>
>>
>> Od: "Tim Bain" <[hidden email]>
>> Do: "users" <[hidden email]>
>> Wysłane: czwartek, 20 kwiecień 2017 7:04:07
>> Temat: Re: Occasionally EOF on connection
>>
>> Nothing in your configuration looks obviously wrong to me.
>>
>> Is the connection between your broker and the clients that are having
>> problems unstable during the period of time that these problems are
>> occurring? Simultaneous EOFs on both ends of a socket connection makes me
>> think of packet loss or network disconnection; could that be what's going
>> on here?
>>
>> If you're allowed to use it in your environment, Wireshark could help you
>> figure out whether the problem is a failure of packets to be sent (e.g. due
>> to network problems) or if one side or the other is choosing to close the
>> connection.
>>
>> Tim
>>
>> On Apr 19, 2017 8:37 AM, "Marcin Giedz" <[hidden email]> wrote:
>>
>>> Hello all,
>>>
>>> in our framework we use AMQ 5.14.1 and CMS version 3.9.4. From time to
>>> time we observe connection drops on client's side like the one below:
>>>
>>> 20170418 10:00:41.902 CRITICAL: void CMSConnectionManager::
>>> sltConnectionError(CMSExceptionListener::errorType, QString)
>>> "DataInputStream::readLong - Reached EOF" "tcp://127.0.0.1:61616?
>>> connection.useAsyncSend=true&soKeepAlive=true&wireFormat.
>>> MaxInactivityDuration=240000"
>>>
>>> in the same time on Broker side this is reported in log file:
>>>
>>> 2017-04-18 10:00:29,358 | WARN | Transport Connection to: tcp://
>>> 127.0.0.1:48612 failed: java.io.EOFException |
>> org.apache.activemq.broker.TransportConnection.Transport
>>> | ActiveMQ NIO Worker 8
>>>
>>> questions:
>>>
>>> 1. is there any way to log more detailed information when EOF occurs ? I
>>> mean we don;t want to flood log file with general DEBUG messages but
>> rather
>>> TransportConnection only to figure out what;s going on when EOF appears ?
>>> 2. perhaps we do have something wrong with broker/client settings ?
>>>
>>> broker (part):
>>> <broker xmlns="http://activemq.apache.org/schema/core"
>>> brokerName="localhost" dataDirectory="${activemq.data}" useJmx="true"
>>> advisorySupport="true" persistent="true" schedulePeriodForDestinationPu
>> rge="10000"
>>> networkConnectorStartAsync="true" allowTempAutoCreationOnSend="true"
>>> deleteAllMessagesOnStartup="false" >
>>>
>>> <destinationPolicy>
>>> <policyMap>
>>> <policyEntries>
>>> <policyEntry topic=">" producerFlowControl="false" >
>>> <pendingSubscriberPolicy>
>>> <vmCursor/>
>>> </pendingSubscriberPolicy>
>>> </policyEntry>
>>> <policyEntry queue=">" producerFlowControl="false" >
>>> <pendingQueuePolicy>
>>> <vmQueueCursor/>
>>> </pendingQueuePolicy>
>>> </policyEntry>
>>>
>>> <policyEntry tempTopic="true" producerFlowControl="false" >
>>> <pendingSubscriberPolicy>
>>> <vmCursor/>
>>> </pendingSubscriberPolicy>
>>> </policyEntry>
>>> <policyEntry tempQueue="true" producerFlowControl="false"
>>> gcInactiveDestinations="true" inactiveTimoutBeforeGC="60000">
>>> <pendingQueuePolicy>
>>> <vmQueueCursor/>
>>> </pendingQueuePolicy>
>>> </policyEntry>
>>>
>>> </policyEntries>
>>> </policyMap>
>>> </destinationPolicy>
>>>
>>> <persistenceAdapter>
>>> <levelDB directory="${activemq.data}/leveldb" logSize="107374182"/>
>>> </persistenceAdapter>
>>>
>>>
>>> <systemUsage>
>>> <systemUsage>
>>> <memoryUsage>
>>> <memoryUsage percentOfJvmHeap="80" />
>>> </memoryUsage>
>>> <storeUsage>
>>> <storeUsage limit="20 gb"/>
>>> </storeUsage>
>>> <tempUsage>
>>> <tempUsage limit="10 gb"/>
>>> </tempUsage>
>>> </systemUsage>
>>> </systemUsage>
>>>
>>> <transportConnectors>
>>> <transportConnector name="openwire" uri="nio://0.0.0.0:61616?
>>> transport.maximumConnections=1000&amp;wireFormat.
>>> maxFrameSize=314572800&amp;transport.keepAlive=true&amp;wireFormat.
>>> maxInactivityDuration=240000"/>
>>> </transportConnectors>
>>>
>>> client:
>>> [activemq]
>>> URL="tcp://127.0.0.1:61616?connection.useAsyncSend=true&
>>> soKeepAlive=true&wireFormat.MaxInactivityDuration=240000"
>>>
>>> Thx
>>> Marcin
>>>
>> -
>>


--
Tim Bish
twitter: @tabish121
blog: http://timbish.blogspot.com/

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Occasionally EOF on connection

lucious_77
Tim how can I provide with more detailed logs? I can try and adjust broker side to have more debugs but client is unlikely to extend its messages

Thx
Marcin



Od: "Timothy Bish" <[hidden email]>
Do: "users" <[hidden email]>
Wysłane: środa, 26 kwiecień 2017 22:16:34
Temat: Re: Occasionally EOF on connection

On 04/26/2017 04:00 PM, Tim Bain wrote:

> Your right, a connection to localhost should be rock-solid. Sorry I missed
> that.
>
> Is there any possibility that a firewall (either OS or application) might
> be interfering with your connections? I feel like I'm grasping at straws a
> little bit, but I can't explain EOFs on both sides of the connection.
>
> The other variable here is that you're using the CMS client library (which
> I have no experience with and which isn't the subject of most posts on this
> list), and maybe the EOF it's showing doesn't mean the same thing as an
> EOFException would in the Java client library. @Tim Bish, do you see
> anything in the described behavior that would be specific to the CMS client
> and that might explain what's going on?

With the little information provided all I can say is that it appears
the connection was forcibly broken on both ends and the client and
broker reacted accordingly by issuing an EOFException. Without more
debugging there's really no why to know for sure what is going on.

>
> Tim
>
> On Apr 20, 2017 8:58 PM, "Marcin Giedz" <[hidden email]> wrote:
>
>> Hi Tim,
>>
>> the main issue in this case is that this is over localhost connection -
>> URL contains 127.0.0.1 ;) we've got plenty of AMQ clients sitting on the
>> same host AMQ broker is running.
>>
>> wireshark is not a problem at all but I wonder how to "narrow" its
>> filtering. Perhaps I can also change log4j in AMQ broker to log more
>> details on transport connection level but I can't find how to do this
>> without global DEBUG/TRACE option ?
>>
>> Thx
>> Marcin
>>
>>
>> Od: "Tim Bain" <[hidden email]>
>> Do: "users" <[hidden email]>
>> Wysłane: czwartek, 20 kwiecień 2017 7:04:07
>> Temat: Re: Occasionally EOF on connection
>>
>> Nothing in your configuration looks obviously wrong to me.
>>
>> Is the connection between your broker and the clients that are having
>> problems unstable during the period of time that these problems are
>> occurring? Simultaneous EOFs on both ends of a socket connection makes me
>> think of packet loss or network disconnection; could that be what's going
>> on here?
>>
>> If you're allowed to use it in your environment, Wireshark could help you
>> figure out whether the problem is a failure of packets to be sent (e.g. due
>> to network problems) or if one side or the other is choosing to close the
>> connection.
>>
>> Tim
>>
>> On Apr 19, 2017 8:37 AM, "Marcin Giedz" <[hidden email]> wrote:
>>
>>> Hello all,
>>>
>>> in our framework we use AMQ 5.14.1 and CMS version 3.9.4. From time to
>>> time we observe connection drops on client's side like the one below:
>>>
>>> 20170418 10:00:41.902 CRITICAL: void CMSConnectionManager::
>>> sltConnectionError(CMSExceptionListener::errorType, QString)
>>> "DataInputStream::readLong - Reached EOF" "tcp://127.0.0.1:61616?
>>> connection.useAsyncSend=true&soKeepAlive=true&wireFormat.
>>> MaxInactivityDuration=240000"
>>>
>>> in the same time on Broker side this is reported in log file:
>>>
>>> 2017-04-18 10:00:29,358 | WARN | Transport Connection to: tcp://
>>> 127.0.0.1:48612 failed: java.io.EOFException |
>> org.apache.activemq.broker.TransportConnection.Transport
>>> | ActiveMQ NIO Worker 8
>>>
>>> questions:
>>>
>>> 1. is there any way to log more detailed information when EOF occurs ? I
>>> mean we don;t want to flood log file with general DEBUG messages but
>> rather
>>> TransportConnection only to figure out what;s going on when EOF appears ?
>>> 2. perhaps we do have something wrong with broker/client settings ?
>>>
>>> broker (part):
>>> <broker xmlns="http://activemq.apache.org/schema/core"
>>> brokerName="localhost" dataDirectory="${activemq.data}" useJmx="true"
>>> advisorySupport="true" persistent="true" schedulePeriodForDestinationPu
>> rge="10000"
>>> networkConnectorStartAsync="true" allowTempAutoCreationOnSend="true"
>>> deleteAllMessagesOnStartup="false" >
>>>
>>> <destinationPolicy>
>>> <policyMap>
>>> <policyEntries>
>>> <policyEntry topic=">" producerFlowControl="false" >
>>> <pendingSubscriberPolicy>
>>> <vmCursor/>
>>> </pendingSubscriberPolicy>
>>> </policyEntry>
>>> <policyEntry queue=">" producerFlowControl="false" >
>>> <pendingQueuePolicy>
>>> <vmQueueCursor/>
>>> </pendingQueuePolicy>
>>> </policyEntry>
>>>
>>> <policyEntry tempTopic="true" producerFlowControl="false" >
>>> <pendingSubscriberPolicy>
>>> <vmCursor/>
>>> </pendingSubscriberPolicy>
>>> </policyEntry>
>>> <policyEntry tempQueue="true" producerFlowControl="false"
>>> gcInactiveDestinations="true" inactiveTimoutBeforeGC="60000">
>>> <pendingQueuePolicy>
>>> <vmQueueCursor/>
>>> </pendingQueuePolicy>
>>> </policyEntry>
>>>
>>> </policyEntries>
>>> </policyMap>
>>> </destinationPolicy>
>>>
>>> <persistenceAdapter>
>>> <levelDB directory="${activemq.data}/leveldb" logSize="107374182"/>
>>> </persistenceAdapter>
>>>
>>>
>>> <systemUsage>
>>> <systemUsage>
>>> <memoryUsage>
>>> <memoryUsage percentOfJvmHeap="80" />
>>> </memoryUsage>
>>> <storeUsage>
>>> <storeUsage limit="20 gb"/>
>>> </storeUsage>
>>> <tempUsage>
>>> <tempUsage limit="10 gb"/>
>>> </tempUsage>
>>> </systemUsage>
>>> </systemUsage>
>>>
>>> <transportConnectors>
>>> <transportConnector name="openwire" uri="nio://0.0.0.0:61616?
>>> transport.maximumConnections=1000&amp;wireFormat.
>>> maxFrameSize=314572800&amp;transport.keepAlive=true&amp;wireFormat.
>>> maxInactivityDuration=240000"/>
>>> </transportConnectors>
>>>
>>> client:
>>> [activemq]
>>> URL="tcp://127.0.0.1:61616?connection.useAsyncSend=true&
>>> soKeepAlive=true&wireFormat.MaxInactivityDuration=240000"
>>>
>>> Thx
>>> Marcin
>>>
>> -
>>

Loading...