Consume rate drops dracstically with message process latency, even with many consumers per queue.
TLDR: When we introduce some latency on the consumers, the maximum consume
rate drops drastically, even though we have enough consumer threads to
handle the rate.
We are currently evaluating ActiveMQ, focusing on produce/consume rate:
The setup is the following:
- Virtual Topics: 100
- Consumer Queues: 3 per virtual topic amounting to 300 queues.
- Produce rate: 10 messages/second on each Virtual topic, amounting to
-- PooledConnectionFactory with 16 connections
- Consumers: 4 different consumers on each queue, amounting to 1200
consumers (i.e. sessions + consumer)
-- another PooledConnectionFactory with 16 connections
- Broker configuration: default settings, producerFlowControl is enabled
- Everything runs on a single machine, i7 2.5Ghz, 16GB memory
When the consumers process messages very fast we can achieve a consume rate
of 3000 messages/sec which is the expected one for the 1000messages/sec that
we produce (3 queues per topic).
When we introduce some latency on the consumers, the maximum consume rate
drops drastically, even though we have enough consumers (== threads) to
handle the rate:
- Latency 10 msec, Consume Rate drops to 1300 messages/sec
- Latency 20 msec, Consume Rate drops to 650 messages/sec
- Latency 100 msec, Consume Rate drops to 130 messages/sec
In all cases produce rate also drops to approximately 1/3 of consume rate,
due to flow control.
Any advice on why the consume rate drops?
Below are code samples, with the connections and producers/consumers
Re: Consume rate drops dracstically with message process latency, even with many consumers per queue.
My guess is that this is because messages are sitting in the prefetch
buffer of consumers while they are being slow to consume their current
message, making those messages unavailable for other consumers who might be
If you change your consumers' prefetch size to 0, do you get the expected