Messages stuck in broker

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Messages stuck in broker

Shobhana
Hi,

I have a network of brokers with just 2 brokers in the network. Relevant
configurations are shown in attached files :

On Node-1 :
activemq-PrimaryBorker.xml
<http://activemq.2283324.n4.nabble.com/file/t377415/activemq-PrimaryBorker.xml>  

On Node-2 :
activemq-SecondaryBorker.xml
<http://activemq.2283324.n4.nabble.com/file/t377415/activemq-SecondaryBorker.xml>  

When a topic subscriber is connected to Node-1 and publisher is connected to
Node-2, sometimes I see that messages are stuck in the broker and not given
to subscriber. Statistics on Node-1 show that consumer is connected :
{"timestamp":1510559053,"status":200,"request":{"mbean":"org.apache.activemq:brokerName=PrimaryBroker,destinationName=topicname,destinationType=Topic,type=Broker","attribute":"ConsumerCount","type":"read"},"value":1}

I can also see some messages are enqueued :
{"timestamp":1510559073,"status":200,"request":{"mbean":"org.apache.activemq:brokerName=PrimaryBroker,destinationName=topicname,destinationType=Topic,type=Broker","attribute":"EnqueueCount","type":"read"},"value":12}

However, dequeue count is 0 :
{"timestamp":1510559078,"status":200,"request":{"mbean":"org.apache.activemq:brokerName=PrimaryBroker,destinationName=topicname,destinationType=Topic,type=Broker","attribute":"DequeueCount","type":"read"},"value":0}

I had used same configuration in my test environment before rolling into
production and in my test env, my subscriber would always receive the
messages sent to any of the brokers. The main difference between my test env
and production env is that the production env has a high no of connections
(several tens of thousands) and topics (again several tens of thousands)!

Any hints on why messages are not delivered? How can I analyse further?

Thanks & Regards,
Shobhana



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Messages stuck in broker

Shobhana
I see a bunch of these messages in the logs of Node-2 :

2017-11-13 15:30:06,506 | INFO  | Establishing network connection from
vm://SecondaryBroker?create=false&async=false to tcp://x.x.x.x:61616 |
org.apache.activemq.network.DiscoveryNetworkConnector | ActiveMQ Task-4
2017-11-13 15:37:49,129 | INFO  | ActiveMQ.Advisory.MasterBroker Inactive
for longer than 1800000 ms - removing ... |
org.apache.activemq.broker.region.Topic | ActiveMQ Broker[SecondaryBroker]
Scheduler
2017-11-13 15:38:37,095 | WARN  | Network connection between
vm://SecondaryBroker#16 and tcp://x.x.x.x/x.x.x.x:61616@46009 shutdown due
to a remote error: java.util.concurrent.TimeoutException |
org.apache.activemq.network.DemandForwardingBridgeSupport |
triggerStartAsyncNetworkBridgeCreation:
remoteBroker=tcp://x.x.x.x/x.x.x.x:61616@46009, localBroker=
vm://SecondaryBroker#16
2017-11-13 15:38:37,101 | INFO  | Connector vm://SecondaryBroker stopped |
org.apache.activemq.broker.TransportConnector | ActiveMQ
BrokerService[SecondaryBroker] Task-21
2017-11-13 15:38:37,102 | INFO  | SecondaryBroker bridge to Unknown stopped
| org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ
BrokerService[SecondaryBroker] Task-21
2017-11-13 15:38:38,101 | INFO  | Establishing network connection from
vm://SecondaryBroker?create=false&async=false to tcp://x.x.x.x:61616 |
org.apache.activemq.network.DiscoveryNetworkConnector | ActiveMQ Task-5
2017-11-13 15:38:38,102 | INFO  | Connector vm://SecondaryBroker started |
org.apache.activemq.broker.TransportConnector | ActiveMQ Task-5
2017-11-13 15:39:08,169 | WARN  | Network connection between
vm://SecondaryBroker#20 and tcp://x.x.x.x/x.x.x.x:61616@46010 shutdown due
to a remote error: java.util.concurrent.TimeoutException |
org.apache.activemq.network.DemandForwardingBridgeSupport |
triggerStartAsyncNetworkBridgeCreation:
remoteBroker=tcp://x.x.x.x/x.x.x.x:61616@46010, localBroker=
vm://SecondaryBroker#20
2017-11-13 15:39:08,171 | INFO  | Connector vm://SecondaryBroker stopped |
org.apache.activemq.broker.TransportConnector | ActiveMQ
BrokerService[SecondaryBroker] Task-25
2017-11-13 15:39:08,171 | INFO  | SecondaryBroker bridge to Unknown stopped
| org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ
BrokerService[SecondaryBroker] Task-25

I could only understand second statement in the logs above. How do I
indicate to the broker that network connector should not be timed out? Can I
increase the maxInactivityDuration of the transport connector to a large
value in both brokers?

What is the reason for other logs? I couldn't see anything relevant in the
logs of Node-1 during this time.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Messages stuck in broker

Tim Bain
In reply to this post by Shobhana
I believe that no messages are being passed because you haven't enabled any
destinations to be sent across the network bridge. I see you're excluding
some, but you're not including any so the exclusion has no effect.

Tim

On Nov 13, 2017 2:59 AM, "Shobhana" <[hidden email]> wrote:

> Hi,
>
> I have a network of brokers with just 2 brokers in the network. Relevant
> configurations are shown in attached files :
>
> On Node-1 :
> activemq-PrimaryBorker.xml
> <http://activemq.2283324.n4.nabble.com/file/t377415/
> activemq-PrimaryBorker.xml>
>
> On Node-2 :
> activemq-SecondaryBorker.xml
> <http://activemq.2283324.n4.nabble.com/file/t377415/
> activemq-SecondaryBorker.xml>
>
> When a topic subscriber is connected to Node-1 and publisher is connected
> to
> Node-2, sometimes I see that messages are stuck in the broker and not given
> to subscriber. Statistics on Node-1 show that consumer is connected :
> {"timestamp":1510559053,"status":200,"request":{"mbean"
> :"org.apache.activemq:brokerName=PrimaryBroker,destinationName=topicname,
> destinationType=Topic,type=Broker","attribute":"
> ConsumerCount","type":"read"},"value":1}
>
> I can also see some messages are enqueued :
> {"timestamp":1510559073,"status":200,"request":{"mbean"
> :"org.apache.activemq:brokerName=PrimaryBroker,destinationName=topicname,
> destinationType=Topic,type=Broker","attribute":"
> EnqueueCount","type":"read"},"value":12}
>
> However, dequeue count is 0 :
> {"timestamp":1510559078,"status":200,"request":{"mbean"
> :"org.apache.activemq:brokerName=PrimaryBroker,destinationName=topicname,
> destinationType=Topic,type=Broker","attribute":"
> DequeueCount","type":"read"},"value":0}
>
> I had used same configuration in my test environment before rolling into
> production and in my test env, my subscriber would always receive the
> messages sent to any of the brokers. The main difference between my test
> env
> and production env is that the production env has a high no of connections
> (several tens of thousands) and topics (again several tens of thousands)!
>
> Any hints on why messages are not delivered? How can I analyse further?
>
> Thanks & Regards,
> Shobhana
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Messages stuck in broker

Tim Bain
In reply to this post by Shobhana
The inactivity monitor is supposed to keep the connection from timing out,
but clearly that's not happening. You haven't done anything to
intentionally disable its keepalive feature, have you? I don't see anything
in your config files, but I want to make sure.

Tim

On Nov 13, 2017 3:54 AM, "Shobhana" <[hidden email]> wrote:

> I see a bunch of these messages in the logs of Node-2 :
>
> 2017-11-13 15:30:06,506 | INFO  | Establishing network connection from
> vm://SecondaryBroker?create=false&async=false to tcp://x.x.x.x:61616 |
> org.apache.activemq.network.DiscoveryNetworkConnector | ActiveMQ Task-4
> 2017-11-13 15:37:49,129 | INFO  | ActiveMQ.Advisory.MasterBroker Inactive
> for longer than 1800000 ms - removing ... |
> org.apache.activemq.broker.region.Topic | ActiveMQ Broker[SecondaryBroker]
> Scheduler
> 2017-11-13 15:38:37,095 | WARN  | Network connection between
> vm://SecondaryBroker#16 and tcp://x.x.x.x/x.x.x.x:61616@46009 shutdown due
> to a remote error: java.util.concurrent.TimeoutException |
> org.apache.activemq.network.DemandForwardingBridgeSupport |
> triggerStartAsyncNetworkBridgeCreation:
> remoteBroker=tcp://x.x.x.x/x.x.x.x:61616@46009, localBroker=
> vm://SecondaryBroker#16
> 2017-11-13 15:38:37,101 | INFO  | Connector vm://SecondaryBroker stopped |
> org.apache.activemq.broker.TransportConnector | ActiveMQ
> BrokerService[SecondaryBroker] Task-21
> 2017-11-13 15:38:37,102 | INFO  | SecondaryBroker bridge to Unknown stopped
> | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ
> BrokerService[SecondaryBroker] Task-21
> 2017-11-13 15:38:38,101 | INFO  | Establishing network connection from
> vm://SecondaryBroker?create=false&async=false to tcp://x.x.x.x:61616 |
> org.apache.activemq.network.DiscoveryNetworkConnector | ActiveMQ Task-5
> 2017-11-13 15:38:38,102 | INFO  | Connector vm://SecondaryBroker started |
> org.apache.activemq.broker.TransportConnector | ActiveMQ Task-5
> 2017-11-13 15:39:08,169 | WARN  | Network connection between
> vm://SecondaryBroker#20 and tcp://x.x.x.x/x.x.x.x:61616@46010 shutdown due
> to a remote error: java.util.concurrent.TimeoutException |
> org.apache.activemq.network.DemandForwardingBridgeSupport |
> triggerStartAsyncNetworkBridgeCreation:
> remoteBroker=tcp://x.x.x.x/x.x.x.x:61616@46010, localBroker=
> vm://SecondaryBroker#20
> 2017-11-13 15:39:08,171 | INFO  | Connector vm://SecondaryBroker stopped |
> org.apache.activemq.broker.TransportConnector | ActiveMQ
> BrokerService[SecondaryBroker] Task-25
> 2017-11-13 15:39:08,171 | INFO  | SecondaryBroker bridge to Unknown stopped
> | org.apache.activemq.network.DemandForwardingBridgeSupport | ActiveMQ
> BrokerService[SecondaryBroker] Task-25
>
> I could only understand second statement in the logs above. How do I
> indicate to the broker that network connector should not be timed out? Can
> I
> increase the maxInactivityDuration of the transport connector to a large
> value in both brokers?
>
> What is the reason for other logs? I couldn't see anything relevant in the
> logs of Node-1 during this time.
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>