Poor performances with large number of consumers

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Poor performances with large number of consumers

ghuillaume
Hello,

I have a network of 2 non-persistent brokers which is dedicated to dispatch
messages to 10000 different consumers using a single topic.
Each consumer use a topic selector to filter messages.
The messages are produced by hundreds of producers : multiple
Java/SpringBoot instances with a CachedConnexionFactory.

I have other queues and topics but it's not the point here.

My problem is that the more I have consumers on my topics, the more I have
latency to deliver messages. I've designed a JMeter test suite to produce
messages and measure performances easily, below the results :
- With no consumer on the topic : up to 10k msg/second can be delivered with
good performances (max/average time in ms reported by Jmeter : 400/0,8).
JMeter is not able to produce more messages, I can't measure performances
with higher throughput.
- With 10K consumers : from 1k msg/second, poor performances (max/average
time in ms reported by Jmeter : 30000/31). I have good performances with
lower throughput

All my clients (producers and consumers) are connected on the 2 brokers
using failover transport reference, so I don't know on which broker the
messages will be produced to and I don't know on which broker a message need
to be forwarded to be consumed.


You will find my config in attachment, and below some details about it :
- conduitSubscriptions=false on networkConnector : because I use topic
selectors
- advisorySupport=false : because if true it doubles the number of consumers
on my topic in a 2 nodes network scenario (i.e. : if I have 10K consumers, I
will have 10K on each instance instead of 5K per instance), which impact
negatively the performances
- All my topics and queues are "staticallyIncludedDestinations" : because
dynamic require advisorySupport
- MaxPageSize=20000 on the topic policyEntry : it seems to be efficient. I
think I need a large number to be able to evaluate a large amout of topic
selector at the same time
- constantPendingMessageLimitStrategy=20 on the topic because I have a lot
of consumers (20 x 10000 = 2000000 pending messages)
- The two brokers declare the same transport connector so it results in a
duplex bridge between them.

I'm using ActiveMQ 5.15.3

activemq.xml
<http://activemq.2283324.n4.nabble.com/file/t379617/activemq.xml>  



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Poor performances with large number of consumers

Tim Bain
The advice I've always heard about selectors in ActiveMQ is that applying
selectors across many consumers is expensive, and that using them with large
numbers of consumers is likely to result in performance problems. The fact
that you're applying selectors to 10,000 consumers on your topic makes me
suspect that that might be what's causing your performance problems here.

If you wanted to test that theory, you could attach JVisualVM or a similar
tool to the broker and run CPU sampling (not profiling!) to determine where
the broker is spending its time. If you find that you're spending all of the
time evaluating selectors, that would confirm the theory.

Tim



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html