Persistent messages disappearing

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Persistent messages disappearing

jamesH
Hi

I'm having an issue with persistent messages getting lost with network connector forwarding messages. I've searched forums, tried fixing with inactivitymonitor but nothing seems to work.
We have two external amq:s which puts messages on amq 5.9 locally through stomp.
After some time (random) an java.io.EOFEXception is thrown and it succesffully reconnects 0.1s later. But that is when the problem starts. Messages on the external dequeues seems to be dequeued (enqueued and dequeued correctly) but the messages are never received on the receiving amq (5.9). Rarely any messages gets through but it seems to be that some messages goes through and are received. The receiver amq enqueued/dequeued does not match th external amqs. It is alot lower like if it never received the messages. A restart to the external amqs fixes the problem and it starts to send all messages correctly. But why does the external amqs dequeue messages when they are not received by the receiving amq? The messages are all marked with persistence and it is correct according to the webconsole.

The external activemq:s are configured with:  
          <transportConnectors>
           
            <transportConnector name="openwire" uri="tcp://0.0.0.0:61616?transport.useInactivityMonitor=false&amp;maximumConnections=1000&amp;wireformat.maxFrameSize=104857600"/>
           
            <transportConnector name="openwireSSL" uri="ssl://0.0.0.0:61617?transport.useInactivityMonitor=false&amp;maximumConnections=1000&amp;wireformat.maxFrameSize=104857600"/>
            <transportConnector name="stomp+ssl" uri="stomp+ssl://0.0.0.0:61618?maximumConnections=1000&amp;wireformat.maxFrameSize=104857600&amp;transport.closeAsync=false"/>
            <transportConnector name="stomp" uri="stomp://0.0.0.0:61619?maximumConnections=1000&amp;wireformat.maxFrameSize=104857600&amp;transport.closeAsync=false"/>
          </transportConnectors>

Which seems to not be a problem and works fine.
These messages are forwarded to another amq which servers as the receiver of all these messages. The external amq:s forwards the messages with

      <broker xmlns="http://activemq.apache.org/schema/core" brokerName="ext" dataDirectory="${activemq.data}" populateJMSXUserID="true" persistent="true">

<networkConnectors>
          <networkConnector name="extToReceiver" uri="static:(failover:(ssl://rewritten.com:61617))?maxReconnectDelay=5000&amp;useExponentialBackOff=false" userName="extusername" password="extpassword" staticBridge="true" networkTTL="3" decreaseNetworkConsumerPriority="true">
            <staticallyIncludedDestinations>
                    <queue physicalName="messages.in"/>
                </staticallyIncludedDestinations>
          </networkConnector>
        </networkConnectors>



The broker dmz

      <broker xmlns="http://activemq.apache.org/schema/core" brokerName="dmz-broker" dataDirectory="${activemq.data}" populateJMSXUserID="true" persistent="true">

uses the simpleauthenticationplugin and the transportconnector

     <transportConnector name="openwireSSL" uri="ssl://0.0.0.0:61617?maximumConnections=1000&amp;wireformat.maxFrameSize=104857600&amp;transport.useInactivityMonitor=false"/>



The external ativemqs shows this error in the log
2014-05-20 11:24:35,417 | WARN  | Transport (tcp://uritodmz-broker:61617) failed, reason:  java.io.EOFException, attempting to automatically reconnect | org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ Transport: ssl:///uritodmz-broker:61617
2014-05-20 11:24:51,219 | INFO  | Successfully reconnected to ssl://uritodmz-broker:61617 | org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ Task-3

All help is appreciated!

Thank you
James
Reply | Threaded
Open this post in threaded view
|

Re: Persistent messages disappearing

gtully
don't use failover with a networkConnector. it may be ok with
staticBridge, but it is not something that is tested. The network
connector keeps track of bridge creation and will recreate a bridge on
a failure. So hiding any failure with failover is duplication of
effort and may also be impeding bridge recreation.
If you need failover to choose from a master slave pair use
maxReconnects=0 or masterSlave: see:
https://issues.apache.org/jira/browse/AMQ-3564

On 23 May 2014 11:00, jamesH <[hidden email]> wrote:

> Hi
>
> I'm having an issue with persistent messages getting lost with network
> connector forwarding messages. I've searched forums, tried fixing with
> inactivitymonitor but nothing seems to work.
> We have two external amq:s which puts messages on amq 5.9 locally through
> stomp.
> After some time (random) an java.io.EOFEXception is thrown and it
> succesffully reconnects 0.1s later. But that is when the problem starts.
> Messages on the external dequeues seems to be dequeued (enqueued and
> dequeued correctly) but the messages are never received on the receiving amq
> (5.9). Rarely any messages gets through but it seems to be that some
> messages goes through and are received. The receiver amq enqueued/dequeued
> does not match th external amqs. It is alot lower like if it never received
> the messages. A restart to the external amqs fixes the problem and it starts
> to send all messages correctly. But why does the external amqs dequeue
> messages when they are not received by the receiving amq? The messages are
> all marked with persistence and it is correct according to the webconsole.
>
> The external activemq:s are configured with:
>           <transportConnectors>
>
>             <transportConnector name="openwire"
> uri="tcp://0.0.0.0:61616?transport.useInactivityMonitor=false&amp;maximumConnections=1000&amp;wireformat.maxFrameSize=104857600"/>
>
>             <transportConnector name="openwireSSL"
> uri="ssl://0.0.0.0:61617?transport.useInactivityMonitor=false&amp;maximumConnections=1000&amp;wireformat.maxFrameSize=104857600"/>
>             <transportConnector name="stomp+ssl"
> uri="stomp+ssl://0.0.0.0:61618?maximumConnections=1000&amp;wireformat.maxFrameSize=104857600&amp;transport.closeAsync=false"/>
>             <transportConnector name="stomp"
> uri="stomp://0.0.0.0:61619?maximumConnections=1000&amp;wireformat.maxFrameSize=104857600&amp;transport.closeAsync=false"/>
>           </transportConnectors>
>
> Which seems to not be a problem and works fine.
> These messages are forwarded to another amq which servers as the receiver of
> all these messages. The external amq:s forwards the messages with
>
>       <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="ext" dataDirectory="${activemq.data}" populateJMSXUserID="true"
> persistent="true">
>
> <networkConnectors>
>           <networkConnector name="extToReceiver"
> uri="static:(failover:(ssl://rewritten.com:61617))?maxReconnectDelay=5000&amp;useExponentialBackOff=false"
> userName="extusername" password="extpassword" staticBridge="true"
> networkTTL="3" decreaseNetworkConsumerPriority="true">
>             <staticallyIncludedDestinations>
>                     <queue physicalName="messages.in"/>
>                 </staticallyIncludedDestinations>
>           </networkConnector>
>         </networkConnectors>
>
>
>
> The broker dmz
>
>       <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="dmz-broker" dataDirectory="${activemq.data}"
> populateJMSXUserID="true" persistent="true">
>
> uses the simpleauthenticationplugin and the transportconnector
>
>      <transportConnector name="openwireSSL"
> uri="ssl://0.0.0.0:61617?maximumConnections=1000&amp;wireformat.maxFrameSize=104857600&amp;transport.useInactivityMonitor=false"/>
>
>
>
> The external ativemqs shows this error in the log
> 2014-05-20 11:24:35,417 | WARN  | Transport (tcp://uritodmz-broker:61617)
> failed, reason:  java.io.EOFException, attempting to automatically reconnect
> | org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ
> Transport: ssl:///uritodmz-broker:61617
> 2014-05-20 11:24:51,219 | INFO  | Successfully reconnected to
> ssl://uritodmz-broker:61617 |
> org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ Task-3
>
> All help is appreciated!
>
> Thank you
> James
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Persistent-messages-disappearing-tp4681353.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.



--
http://redhat.com
http://blog.garytully.com
Reply | Threaded
Open this post in threaded view
|

Re: Persistent messages disappearing

jamesH
Thanks, I'll give it a try.