potential "deadlock" between BrokerFilter and Producer creation with VM connection

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

potential "deadlock" between BrokerFilter and Producer creation with VM connection

zyc1984
Hi,

We have identified a potential "deadlock" between BrokerFilter and Producer
creation with VM connection
This block is random, it happened on one given machine but not the others.

ActiveMQ version 5.9.0

Thread #310 : notified by BrokerFilter.addConsumer() and it creates a
Producer with local VM connection
Thread #308 : notifed by BrokerFilter.messageConsumed() and it tries to get
Consumer info in our application with read lock
Thread 380 : it is treating the Producer creation command but it's blocked
by Thread #308 a lock inside held by ActiveMQ BrokerFilter stack.


As this issue is not always happening, i guess it is related to the
Threading model in ActiveMQ.
And I would like to know in which case VM connection handling thread will be
blocked by BrokerFilter and in which case it is not. Is it related to jvm
arguments ? because the configuration of ActiveMQ is hard coded in our case.

Thanks,
zyc1984




"ActiveMQ Transport: tcp:///22.0.195.149:64315@64078" #310 daemon prio=5
os_prio=64 tid=0x000000010274e000 nid=0x14d waiting on condition
[0xfffffffe76ffd000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xfffffffe8335fb60> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
        at
org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
        at
org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:87)
        at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1394)
        at
org.apache.activemq.ActiveMQSession.syncSendPacket(ActiveMQSession.java:1925)
        at
org.apache.activemq.ActiveMQMessageProducer.<init>(ActiveMQMessageProducer.java:125)
        at
org.apache.activemq.ActiveMQSession.createProducer(ActiveMQSession.java:969)
        at EventFeeder.start(GenericEventFeeder.java:63)
        at DefaultConsumerManager.startFeed(DefaultConsumerManager.java:192)
        at
DefaultConsumerAdministration.consumerConnected(DefaultConsumerAdministration.java:249)
        at ConsumerListener.onConsumerConnect(ConsumerListener.java:31)
        at ActiveMQInterceptor.addConsumer(ActiveMQInterceptor.java:80)
        at
org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:102)
        at
org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:587)
        at org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:347)
        at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
        at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:149)
        at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
        at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
        at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
        at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
        at java.lang.Thread.run(Thread.java:745)



"ActiveMQ Transport: tcp:///22.0.195.149:64313@64078" #308 daemon prio=5
os_prio=64 tid=0x00000001028d5000 nid=0x14b waiting on condition
[0xfffffffe773fd000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0xfffffffe81003250> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
        at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
        at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
        at
DefaultConsumerAdministration.consumed(DefaultConsumerAdministration.java:367)
        at ConsumerListener.onMessageConsumed(ConsumerListener.java:45)
        at ActiveMQInterceptor.messageConsumed(ActiveMQInterceptor.java:123)
        at
org.apache.activemq.broker.MutableBrokerFilter.messageConsumed(MutableBrokerFilter.java:350)
        at
org.apache.activemq.broker.region.BaseDestination.messageConsumed(BaseDestination.java:465)
        at org.apache.activemq.broker.region.Queue.acknowledge(Queue.java:917)
        at org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1703)
        at
org.apache.activemq.broker.region.QueueSubscription.acknowledge(QueueSubscription.java:63)
        at
org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:235)
        - locked <0xfffffffe833683f0> (a java.lang.Object)
        at
org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:412)
        at
org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:458)
        at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:82)
        at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:82)
        at
org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:277)
        at
org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:82)
        at
org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:92)
        at
org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:476)
        at org.apache.activemq.command.MessageAck.visit(MessageAck.java:236)
        at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
        at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:149)
        at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
        at
org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
        at
org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)
        at
org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
        at
org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:214)
        at
org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None



"ActiveMQ VMTransport: vm://BOS_JMS_BROKER#1-3" #380 daemon prio=5
os_prio=64 tid=0x0000000102bad000 nid=0x18a waiting for monitor entry
[0xfffffffe763fd000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at
org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:654)
        - waiting to lock <0xfffffffe833683f0> (a java.lang.Object)
        - locked <0xfffffffe83368400> (a java.lang.Object)
        at
org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:161)
        at
org.apache.activemq.broker.region.Queue.doActualDispatch(Queue.java:1998)
        at org.apache.activemq.broker.region.Queue.doDispatch(Queue.java:1946)
        at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2086)
        at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1581)
        - locked <0xfffffffe83368658> (a java.lang.Object)
        at org.apache.activemq.broker.region.Queue.wakeup(Queue.java:1803)
        at org.apache.activemq.broker.region.Queue.messageSent(Queue.java:1797)
        at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:866)
        at org.apache.activemq.broker.region.Queue.send(Queue.java:700)
        at
org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:394)
        at
org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:442)
        at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:147)
        at
org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
        at
org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:307)
        at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:147)
        at
org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:152)
        at
org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:467)
        at
org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:751)
        at
org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:292)
        at
org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:149)
        at
org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116)
        at
org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
        at
org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:247)
        at
org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:129)
        at
org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:47)
        at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - <0xfffffffe833687b8> (a java.util.concurrent.ThreadPoolExecutor$Worker)
        - <0xfffffffe83407310> (a
java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)





--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html