We were able to reproduce this issue with just ActiveMQ (minus our application). It seems that when the consumers throw a exceptions frequently, something goes wrong trying to rollback the transaction and the LevelDB. It seems that a LevelDB thread is locking it up.
Attached is a test which simulates the behavior. For us it failed after 50000 messages but the number may vary on the machine.
We suspect the following thread to be the culprit as it continuously seems to be blocked.
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
- java.util.concurrent.locks.LockSupport.park(java.lang.Object) @bci=20, line=186 (Compiled frame [deoptimized])
- java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt() @bci=7, line=834 (Compiled frame [deoptimized])
- java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(int) @bci=89, line=994 (Compiled frame [deoptimized])
- java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(int) @bci=47, line=1303 (Interpreted frame)
- java.util.concurrent.CountDownLatch.await() @bci=11, line=236 (Interpreted frame)
- org.apache.activemq.leveldb.LevelDBStore.rollback(org.apache.activemq.command.TransactionId) @bci=152, line=498 (Interpreted frame)
- org.apache.activemq.transaction.LocalTransaction.rollback() @bci=111, line=94 (Interpreted frame)
- org.apache.activemq.broker.TransactionBroker.rollbackTransaction(org.apache.activemq.broker.ConnectionContext, org.apache.activemq.command.TransactionId) @bci=15, line=258 (Interpreted frame)
- org.apache.activemq.broker.BrokerFilter.rollbackTransaction(org.apache.activemq.broker.ConnectionContext, org.apache.activemq.command.TransactionId) @bci=12, line=142 (Interpreted frame)
- org.apache.activemq.broker.MutableBrokerFilter.rollbackTransaction(org.apache.activemq.broker.ConnectionContext, org.apache.activemq.command.TransactionId) @bci=12, line=147 (Interpreted frame)
- org.apache.activemq.broker.TransportConnection.processRollbackTransaction(org.apache.activemq.command.TransactionInfo) @bci=44, line=442 (Interpreted frame)
- org.apache.activemq.command.TransactionInfo.visit(org.apache.activemq.state.CommandVisitor) @bci=146, line=104 (Compiled frame [deoptimized])
- org.apache.activemq.broker.TransportConnection.service(org.apache.activemq.command.Command) @bci=52, line=292 (Compiled frame [deoptimized])
- org.apache.activemq.broker.TransportConnection$1.onCommand(java.lang.Object) @bci=95, line=149 (Compiled frame [deoptimized])
- org.apache.activemq.transport.MutexTransport.onCommand(java.lang.Object) @bci=58, line=50 (Compiled frame [deoptimized])
- org.apache.activemq.transport.WireFormatNegotiator.onCommand(java.lang.Object) @bci=35, line=113 (Compiled frame [deoptimized])
- org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(java.lang.Object) @bci=162, line=270 (Compiled frame [deoptimized])
- org.apache.activemq.transport.TransportSupport.doConsume(java.lang.Object) @bci=27, line=83 (Compiled frame [deoptimized])
- org.apache.activemq.transport.tcp.TcpTransport.doRun() @bci=13, line=214 (Compiled frame [deoptimized])
- org.apache.activemq.transport.tcp.TcpTransport.run() @bci=70, line=196 (Compiled frame [deoptimized])
- java.lang.Thread.run() @bci=17, line=724 (Interpreted frame)
Thanks.