Consumer performance problem with Tx

[James Black]

Hi,

I have a problem that when using transactions all my consumers for a particular queue are getting blocking waiting for the commit.  We are using the Spring DMLC and are caching at the consumer level.  This is ActiveMQ 5.5.1

All threads end up at a WAITING state with the same stack trace as below:

"service--51" - Thread t@96
   java.lang.Thread.State: WAITING
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <12021c84> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
        at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:374)
        at org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
        at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:87)
        at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1284)
        at org.apache.activemq.TransactionContext.syncSendPacketWithInterruptionHandling(TransactionContext.java:707)
        at org.apache.activemq.TransactionContext.commit(TransactionContext.java:299)
        at org.apache.activemq.ActiveMQSession.commit(ActiveMQSession.java:560)
        at org.springframework.jms.support.JmsUtils.commitIfNecessary(JmsUtils.java:217)
        at org.springframework.jms.listener.AbstractMessageListenerContainer.commitIfNecessary(AbstractMessageListenerContainer.java:577)
        at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:482)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325)
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:243)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1103)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1095)
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:992)
        at java.lang.Thread.run(Thread.java:724)
   Locked ownable synchronizers:
        - None

I know that the transaction speed is limited by the disk performance but we do not see the disk IO jump beyond what we would expect it can handle.

Anyone got any ideas?

Thanks,

Justin

[Matt Pavlovich-2]

Hi James-

ActiveMQ 5.5.1 is quite aged at this point. Any reason you aren’t looking at a newer version?

Transactions and connection pooling is tricky— are you doing XA or JMS Local?  I suggest trying to get it all working, then tune.. CACHE_NONE first.

-Matt

On Sep 2, 2014, at 7:05 AM, James Black <[hidden email]> wrote:

> Hi,
>
> I have a problem that when using transactions all my consumers for a
> particular queue are getting blocking waiting for the commit.  We are using
> the Spring DMLC and are caching at the consumer level.  This is ActiveMQ
> 5.5.1
>
> All threads end up at a WAITING state with the same stack trace as below:
>
> "service--51" - Thread t@96
>   java.lang.Thread.State: WAITING
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <12021c84> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
> at
> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:374)
> at
> org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
> at
> org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:87)
> at
> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1284)
> at
> org.apache.activemq.TransactionContext.syncSendPacketWithInterruptionHandling(TransactionContext.java:707)
> at
> org.apache.activemq.TransactionContext.commit(TransactionContext.java:299)
> at org.apache.activemq.ActiveMQSession.commit(ActiveMQSession.java:560)
> at
> org.springframework.jms.support.JmsUtils.commitIfNecessary(JmsUtils.java:217)
> at
> org.springframework.jms.listener.AbstractMessageListenerContainer.commitIfNecessary(AbstractMessageListenerContainer.java:577)
> at
> org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:482)
> at
> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325)
> at
> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:243)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1103)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1095)
> at
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:992)
> at java.lang.Thread.run(Thread.java:724)
>   Locked ownable synchronizers:
> - None
>
> I know that the transaction speed is limited by the disk performance but we
> do not see the disk IO jump beyond what we would expect it can handle.
>
> Anyone got any ideas?
>
> Thanks,
>
> Justin
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Consumer-performance-problem-with-Tx-tp4685226.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.

[James Black]

Hi Matt,

thanks for the response.

The reason that we have not migrated to a later version of ActiveMq is that we are looking to move away from using ActiveMq due to the problems we have had.  Therefore, we wanted to avoid going through the whole testing process for a later version of ActiveMq.  We use shared nothing master/slave replicated which has been removed in later ActiveMq versions.

We are doing JMS local (originally we tried XA but had massive problems with that) and we have tested with all caching settings for the DMLC to no avail.  Our connection is not provided from a pool but directly from standard ActiveMq connection factory (org.apache.activemq.ActiveMQConnectionFactory), due to the fact that it is cached in the DMLC.

Thanks,

Justin

[Matt Pavlovich-2]

Justin-

Sorry to hear that you’ve had problems with ActiveMQ in the past. I’ve had a lot of successful deployments in large-scale environments (3,000+ brokers, hub and spoke with 1 broker serving up to 1,000 clients, 4,000 queues and 3,000 total connections) with transactions, and the full boat of features with newer releases of ActiveMQ. If you don’t have luck with alternatives, I suggest you give newer versions a second look.

The replicated Master-Slave was definitely a lightly-maintained feature and has a lot of operational problems (how to resync after an outage, etc).

Back to your original issue.. I've seen a lot of issues with Spring JMS Template on the consumer side (esp with transactions). Recommend trying with straight JMS code to isolate if you have a pooling/caching issue or a broker bug and go from there.

-Matt

On Sep 4, 2014, at 3:54 AM, James Black <[hidden email]> wrote:

> Hi Matt,
>
> thanks for the response.
>
> The reason that we have not migrated to a later version of ActiveMq is that
> we are looking to move away from using ActiveMq due to the problems we have
> had.  Therefore, we wanted to avoid going through the whole testing process
> for a later version of ActiveMq.  We use shared nothing master/slave
> replicated which has been removed in later ActiveMq versions.
>
> We are doing JMS local (originally we tried XA but had massive problems with
> that) and we have tested with all caching settings for the DMLC to no avail.
> Our connection is not provided from a pool but directly from standard
> ActiveMq connection factory (org.apache.activemq.ActiveMQConnectionFactory),
> due to the fact that it is cached in the DMLC.
>
> Thanks,
>
> Justin
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Consumer-performance-problem-with-Tx-tp4685226p4685279.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.

[James Black]

Hi Matt,

we don't use JmsTemplate in this particular bit of code.

I have done some testing this morning and the problem seems to be contention with the shared connection.  The Spring DMLC is configured to cache at Connection, Session and Consumer level.  If caching is enabled at all (connection+) then it uses a shared single connection.  I can verify this by monitoring ActiveMq through Jmx.

So if I have a DMLC with 20 consumer threads, all sharing the same connection it is slow, but at low throughput you would not notice.

If I create 20 separate DMLCs all with a single thread and all with different connections, then the performance is great in comparison!

Not sure why this would be as I believed that it is normal to share the connection for JMS unlike JDBC and I would of assumed that ActiveMQ is optimised for that case?

Thanks,

Justin

[Matt Pavlovich-2]

Justin-

JMS connections act and behave very different from JDBC connections.  Just like a long query can block other apps trying to access the connection on a shared JDBC connection, a slow consumer can impact others trying to access it as well. You can do multiple sessions on top of a single connection, but it can get hairy and I don’t get the sense that the intention was to support connection pooling, but rather allow for a single program to do all its producing and consuming over a single connection— not several programs sharing a single connection.

Separate connections per consumer is always a safe starting point. Pre-tuning for shared JMS connections, usually results in de-tuning. I think you’ll find single connection per consumer will meet your performance needs.  

If you absolutely feel like you must share sessions on top of a single connection, reduce prefetch=“1”, ensure your memoryLimit is well above the default “1 mb”, use NIO on the transportConnector, optimizedDispatch=“true” and dedicatedTaskRunner=“false”.

-Matt

On Sep 8, 2014, at 6:25 AM, James Black <[hidden email]> wrote:

> Hi Matt,
>
> we don't use JmsTemplate in this particular bit of code.
>
> I have done some testing this morning and the problem seems to be contention
> with the shared connection.  The Spring DMLC is configured to cache at
> Connection, Session and Consumer level.  If caching is enabled at all
> (connection+) then it uses a shared single connection.  I can verify this by
> monitoring ActiveMq through Jmx.
>
> So if I have a DMLC with 20 consumer threads, all sharing the same
> connection it is slow, but at low throughput you would not notice.
>
> If I create 20 separate DMLCs all with a single thread and all with
> different connections, then the performance is great in comparison!
>
> Not sure why this would be as I believed that it is normal to share the
> connection for JMS unlike JDBC and I would of assumed that ActiveMQ is
> optimised for that case?
>
> Thanks,
>
> Justin
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Consumer-performance-problem-with-Tx-tp4685226p4685365.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.