durable subscribers causing message flow to halt

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

durable subscribers causing message flow to halt

michael.hart
We are running ActiveMQ 5.10.0 with a network of brokers. For one topic called 'results' we have one subscriber running on a single broker 'X' and many (9 in staging, 24 in production and growing) brokers (let's call them sensors) connected in with full duplex connections, each broker with a producer connected and sending messages to that topic. Consumers and producers are all connected using stomp. brokers are connected using openwire.

There are other queues and topics on the brokers, but let's focus on just this one.

Until yesterday, we were using non-durable consumers, and non-persistent messaging. Yesterday we enabled durable consumers by setting the client-id on the CONNECT header, and setting activemq.subscriptionName on the SUBSCRIBE header. At first things seem to work, but very quickly the brokers on the sensors stopped forwarding messages and went into producer flow control.

Under normal runtime, when I run "/opt/activemq/bin/activemq-admin query -QTopic=sensor-results" I should see one network connector section per sensor broker. However once we enabled durable subscribers, only a few of the sensor brokers would show up, even though they were actually connected (tcp connection was active). No amount of restarts of anything fixed it. The number of sensor brokers connected/subscribed was transient, between 2 and 8 would be connected, usually around 4.

Then it got worse. We rolled back our code change and went back to non-durable subscribers, but things did not get better. We had offlineDurableSubscriberTimeout and offlineDurableSubscriberTaskSchedule set to one minute, but the durable subscritions did not clear out. We eventually shutdown the entire network of brokers, deleted the kahadb directory on every broker, and restarted everything.

It appears to us that the subscription from single broker 'X' did not make it to the sensor brokers, or only did sometimes, but then refused to clear out. We don't know how to further debug this, and would appreciate any help and suggestions.

thanks
mike
Reply | Threaded
Open this post in threaded view
|

Re: durable subscribers causing message flow to halt

Tim Bain
Is there any consistency in which sensor brokers succeed and which fail, or
is it random every time?

If you're able to download the source code for your version of ActiveMQ and
attach a debugger to one of your sensor brokers, you could set breakpoints
and watch the subscription request from broker X be processed.
Topic.addSubscription() would be a good place for your first breakpoint
when you're failing to add the subscription, and TopicRegion.doCleanup()
would be a good place for a breakpoint when you're failing to close offline
durable subscribers.

Tim

On Thu, Sep 18, 2014 at 8:42 AM, michael.hart <[hidden email]>
wrote:

> We are running ActiveMQ 5.10.0 with a network of brokers. For one topic
> called 'results' we have one subscriber running on a single broker 'X' and
> many (9 in staging, 24 in production and growing) brokers (let's call them
> sensors) connected in with full duplex connections, each broker with a
> producer connected and sending messages to that topic. Consumers and
> producers are all connected using stomp. brokers are connected using
> openwire.
>
> There are other queues and topics on the brokers, but let's focus on just
> this one.
>
> Until yesterday, we were using non-durable consumers, and non-persistent
> messaging. Yesterday we enabled durable consumers by setting the client-id
> on the CONNECT header, and setting activemq.subscriptionName on the
> SUBSCRIBE header. At first things seem to work, but very quickly the
> brokers
> on the sensors stopped forwarding messages and went into producer flow
> control.
>
> Under normal runtime, when I run "/opt/activemq/bin/activemq-admin query
> -QTopic=sensor-results" I should see one network connector section per
> sensor broker. However once we enabled durable subscribers, only a few of
> the sensor brokers would show up, even though they were actually connected
> (tcp connection was active). No amount of restarts of anything fixed it.
> The
> number of sensor brokers connected/subscribed was transient, between 2 and
> 8
> would be connected, usually around 4.
>
> Then it got worse. We rolled back our code change and went back to
> non-durable subscribers, but things did not get better. We had
> offlineDurableSubscriberTimeout and offlineDurableSubscriberTaskSchedule
> set
> to one minute, but the durable subscritions did not clear out. We
> eventually
> shutdown the entire network of brokers, deleted the kahadb directory on
> every broker, and restarted everything.
>
> It appears to us that the subscription from single broker 'X' did not make
> it to the sensor brokers, or only did sometimes, but then refused to clear
> out. We don't know how to further debug this, and would appreciate any help
> and suggestions.
>
> thanks
> mike
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/durable-subscribers-causing-message-flow-to-halt-tp4685694.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: durable subscribers causing message flow to halt

michael.hart
Thanks Tim. There is no consistency, it appears to be somewhat random.

I can't attach a debugger unfortunately, I'll have to setup a test network of sorts to try to reproduce it. :(

mike