ActiveMQ.NMS consumer hangs after sometime

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

ActiveMQ.NMS consumer hangs after sometime

Ishitori
I am using 2 ActiveMQ brokers 5.5.0 with failover on Windows. I have a C# windows service which connects to the broker queue via NMS. Windows service creates 100 threads and each thread has its own consumer. I use AutoAcknowledge mode. I run 4 instances of the windows service on separate machines. PrefetchSize is set to 1 and I use synchronous receive method with small (10 ms) wait time.

I also have the following setup on brokers for queues:

<policyEntry queue=">" producerFlowControl="false" memoryLimit="1mb">

After working for some period of time (usually a day or two) all consumers stop receiving any messages, but I can see messages in brokers. Usually all windows service instances stop receiving messages simultaneously, but not every time. If I restart services then they start processing messages.

I've created a dump file and I found a lot of threads are waiting for a mutex to be released. Here is the stack trace:

System.Threading.Monitor.Wait(System.Object, System.TimeSpan)
Apache.NMS.ActiveMQ.Util.SimplePriorityMessageDispatchChannel.Dequeue(System.TimeSpan)
Apache.NMS.ActiveMQ.MessageConsumer.Dequeue(System.TimeSpan)
Apache.NMS.ActiveMQ.MessageConsumer.Receive(System.TimeSpan)

I also have a lot of DedicatedTaskRunners waiting for something:

System.Threading.Monitor.Wait(System.Object)
Apache.NMS.ActiveMQ.Threads.DedicatedTaskRunner.Run()
System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
System.Threading.ThreadHelper.ThreadStart()

I've found a few issues with the same problem, but I can't find my case:

1) http://activemq.2283324.n4.nabble.com/Client-threads-hang-if-ActiveMQ-dies-td2363080.html - I don't use transacted messages anymore, neither prefetchSize=0. I don't use ReceiveNoWait as well (though I could move waiting logic to my application), because it seems to cause some problems.

2) http://activemq.2283324.n4.nabble.com/Consumer-Receive-hangs-td2364683.html - I set producerFlowControl to false, and afair the broker could use as much memory as the server has.

I also found an issue which describes the same problem and is marked as resolved - https://issues.apache.org/jira/browse/AMQ-2446

What else could cause such a problem?
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ.NMS consumer hangs after sometime

tabish121@gmail.com
Your best bet is to try and create a test case that can reproduce the
issue.  The stack traces look fine, seems they are waiting on more
messages.  

Make sure you are using the latest NMS ActiveMQ version to ensure you
have all the latest fixes.

On Tue, 2012-02-07 at 05:17 -0800, Ishitori wrote:

> I am using 2 ActiveMQ brokers 5.5.0 with failover on Windows. I have a C#
> windows service which connects to the broker queue via NMS. Windows service
> creates 100 threads and each thread has its own consumer. I use
> AutoAcknowledge mo
> de. I run 4 instances of the windows service on separate
> machines. PrefetchSize is set to 1 and I use synchronous receive method with
> small (10 ms) wait time.
>
> I also have the following setup on brokers for queues:
>
> <policyEntry queue=">" producerFlowControl="false" memoryLimit="1mb">
>
> After working for some period of time (usually a day or two) all consumers
> stop receiving any messages, but I can see messages in brokers. Usually all
> windows service instances stop receiving messages simultaneously, but not
> every time. If I restart services then they start processing messages.
>
> I've created a dump file and I found a lot of threads are waiting for a
> mutex to be released. Here is the stack trace:
>
> System.Threading.Monitor.Wait(System.Object, System.TimeSpan)
> Apache.NMS.ActiveMQ.Util.SimplePriorityMessageDispatchChannel.Dequeue(System.TimeSpan)
> Apache.NMS.ActiveMQ.MessageConsumer.Dequeue(System.TimeSpan)
> Apache.NMS.ActiveMQ.MessageConsumer.Receive(System.TimeSpan)
>
> I also have a lot of DedicatedTaskRunners waiting for something:
>
> System.Threading.Monitor.Wait(System.Object)
> Apache.NMS.ActiveMQ.Threads.DedicatedTaskRunner.Run()
> System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext,
> System.Threading.ContextCallback, System.Object, Boolean)
> System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext,
> System.Threading.ContextCallback, System.Object)
> System.Threading.ThreadHelper.ThreadStart()
>
> I've found a few issues with the same problem, but I can't find my case:
>
> 1)
> http://activemq.2283324.n4.nabble.com/Client-threads-hang-if-ActiveMQ-dies-td2363080.html
> - I don't use transacted messages anymore, neither prefetchSize=0. I don't
> use ReceiveNoWait as well (though I could move waiting logic to my
> application), because it seems to cause some problems.
>
> 2)
> http://activemq.2283324.n4.nabble.com/Consumer-Receive-hangs-td2364683.html
> - I set producerFlowControl to false, and afair the broker could use as much
> memory as the server has.
>
> I also found an issue which describes the same problem and is marked as
> resolved - https://issues.apache.org/jira/browse/AMQ-2446
>
> What else could cause such a problem?
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-NMS-consumer-hangs-after-sometime-tp4364689p4364689.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.

--
Tim Bish
Sr Software Engineer | FuseSource Corp
[hidden email] | www.fusesource.com
skype: tabish121 | twitter: @tabish121
blog: http://timbish.blogspot.com/

Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ.NMS consumer hangs after sometime

Ishitori
It is hard to reproduce this problem in a test case.
But I do use latest version of ActiveMQ.NMS. I am also think about upgrading 5.5.0 ActiveMQ to 5.6.0 to resolve the problem with Scheduler (I use it for redelivery).

Can you recommend me anything to get closer to the root of the problem?