Consumer hangs on creation

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Consumer hangs on creation

Ishitori
There were a situation in my production environment when an application couldn't create a consumer. It looks like a thread freeze for infinite. There are no exceptions or errors happen, but the thread got blocked.

I am using AMQ 5.6.0 with the latest NMS client library.

I created a dump file and here is the stack trace:

System.Threading.WaitHandle.WaitOneNative(System.Runtime.InteropServices.SafeHandle, UInt32, Boolean, Boolean)
System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle, Int64, Boolean, Boolean)
Apache.NMS.ActiveMQ.Transport.FutureResponse.get_Response()
Apache.NMS.ActiveMQ.Transport.ResponseCorrelator.Request(Apache.NMS.ActiveMQ.Commands.Command, System.TimeSpan)
Apache.NMS.ActiveMQ.Connection.SyncRequest(Apache.NMS.ActiveMQ.Commands.Command, System.TimeSpan)
Apache.NMS.ActiveMQ.Session.CreateConsumer(Apache.NMS.IDestination, System.String, Boolean)
...

So it is seems that the broker didn't respond to the consumer who was waiting for the response forever.

Why the broker didn't respond anything? What could cause that? And why consumer didn't throw an exception after the timeout is reached? If there is a timeout at all?
Reply | Threaded
Open this post in threaded view
|

Re: Consumer hangs on creation

tabish121@gmail.com
On Wed, 2012-10-17 at 04:44 -0700, Ishitori wrote:

> There were a situation in my production environment when an application
> couldn't create a consumer. It looks like a thread freeze for infinite.
> There are no exceptions or errors happen, but the thread got blocked.
>
> I am using AMQ 5.6.0 with the latest NMS client library.
>
> I created a dump file and here is the stack trace:
>
> System.Threading.WaitHandle.WaitOneNative(System.Runtime.InteropServices.SafeHandle,
> UInt32, Boolean, Boolean)
> System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle,
> Int64, Boolean, Boolean)
> Apache.NMS.ActiveMQ.Transport.FutureResponse.get_Response()
> Apache.NMS.ActiveMQ.Transport.ResponseCorrelator.Request(Apache.NMS.ActiveMQ.Commands.Command,
> System.TimeSpan)
> Apache.NMS.ActiveMQ.Connection.SyncRequest(Apache.NMS.ActiveMQ.Commands.Command,
> System.TimeSpan)
> Apache.NMS.ActiveMQ.Session.CreateConsumer(Apache.NMS.IDestination,
> System.String, Boolean)
> ...
>

Is your client using failover?
If so was there a broker running?
What do the broker logs say?

> So it is seems that the broker didn't respond to the consumer who was
> waiting for the response forever.
>
> Why the broker didn't respond anything? What could cause that? And why
> consumer didn't throw an exception after the timeout is reached? If there is
> a timeout at all?
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Consumer-hangs-on-creation-tp4657882.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.

--
Tim Bish
Sr Software Engineer | RedHat Inc.
[hidden email] | www.fusesource.com | www.redhat.com
skype: tabish121 | twitter: @tabish121
blog: http://timbish.blogspot.com/

Reply | Threaded
Open this post in threaded view
|

Re: Consumer hangs on creation

Ishitori
I managed to resolve the issue by restarting 2nd broker. But I want to understand why it is happened.

> Is your client using failover?

Yes, it does. Moreover I am using 2 duplex connectors from one broker to another with conduitSubscriptions = "false" on both connectors.

If so was there a broker running?

Yes they were running, but I think we expirienced the problem with the 2nd broker at that moment.

What do the broker logs say?

I can see

WARN | Transport Connection to: tcp://192.168.0.18:58614 failed: java.io.EOFException
WARN | Transport Connection to: tcp://192.168.0.18:59866 failed: java.io.EOFException
WARN | Transport Connection to: tcp://192.168.0.18:59920 failed: java.io.EOFException

where 192.168.0.18 it is an address of the 2nd broker. So I can guess that connections between brokers went down for some reason, right?

And a strange information message:

INFO: TopicSubscription: consumer=ID:<Broker1>-65074-1350392991110-8:2:1:1, destinations=15, dispatched=1000, delivered=6333, matched=39550, discarded=0: Pending message cursor [org.apache.activemq.broker.region.cursors.VMPendingMessageCursor@ec2384] is full, temp usage (15%) or memory usage (100%) limit reached, blocking message add() pending the release of resources.

I don't use topics in ActiveMQ at all, only named queues. Why it tells me something about a TopicSubscription where I shouldn't have any?

Any ideas?
Reply | Threaded
Open this post in threaded view
|

Re: Consumer hangs on creation

tabish121@gmail.com
On Thu, 2012-10-18 at 02:30 -0700, Ishitori wrote:

> I managed to resolve the issue by restarting 2nd broker. But I want to
> understand why it is happened.
>
> > Is your client using failover?
>
> Yes, it does. Moreover I am using 2 duplex connectors from one broker to
> another with conduitSubscriptions = "false" on both connectors.
>
> If so was there a broker running?
>
> Yes they were running, but I think we expirienced the problem with the 2nd
> broker at that moment.

With failover the calls can hang if the connection is down.  You can
configure timeouts so that exceptions are thrown.  Once the connection
is re-established the call should have completed.  You can get events
from the connection interrupted/resumed events to know when things are
down.

>
> What do the broker logs say?
>
> I can see
>
> WARN | Transport Connection to: tcp://192.168.0.18:58614 failed:
> java.io.EOFException
> WARN | Transport Connection to: tcp://192.168.0.18:59866 failed:
> java.io.EOFException
> WARN | Transport Connection to: tcp://192.168.0.18:59920 failed:
> java.io.EOFException
>
> where 192.168.0.18 it is an address of the 2nd broker. So I can guess that
> connections between brokers went down for some reason, right?
>
> And a strange information message:
>
> INFO: TopicSubscription: consumer=ID:<Broker1>-65074-1350392991110-8:2:1:1,
> destinations=15, dispatched=1000, delivered=6333, matched=39550,
> discarded=0: Pending message cursor
> [org.apache.activemq.broker.region.cursors.VMPendingMessageCursor@ec2384] is
> full, temp usage (15%) or memory usage (100%) limit reached, blocking
> message add() pending the release of resources.
>
> I don't use topics in ActiveMQ at all, only named queues. Why it tells me
> something about a TopicSubscription where I shouldn't have any?
>
> Any ideas?
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Consumer-hangs-on-creation-tp4657882p4657921.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.

--
Tim Bish
Sr Software Engineer | RedHat Inc.
[hidden email] | www.fusesource.com | www.redhat.com
skype: tabish121 | twitter: @tabish121
blog: http://timbish.blogspot.com/