How can i recover from exhausted memory limits?

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

How can i recover from exhausted memory limits?

Laures
(my broker is running with deactivated producer flow control at the moment)

when a activemq broker gets flooded with messages or the consumer fails it will stop accepting messages once certain (configurable) limits are reached. In Broker Networks this effect can take down the whole cluster.

I'm currently using the default configuration for systemUsage and experience the following behavior:

    1. consumer fails or becomes very slow (happens rarely but usaly during holydays or on vacation...)
    2. broker A (the one the consumer connects to) gets filled and stops accepting messages
    3. all other brokers get filled up and stop to accept messages
    4. the cluster is basicly down

if the consumer comes back online now it will try to reconnect to one of the cluster nodes but the nodes will not accept the connection becaus this would create advisory messages that can't be handled because the broker is already full.

How do i have to configure the systemUsage-limits or producer flow control (or whatever) so that my productive destinations are limited and blocked but the broker will still be able to accept advisories so my consumer can revover?

Greetings,
Alexander
Reply | Threaded
Open this post in threaded view
|

Re: How can i recover from exhausted memory limits?

Laures
after reading through the documentation i found:

storeUsageHighWaterMark (a destination policy)

my idea is to configure a watermark of 95% for ">" and reconfigure it to 100% for "ActiveMQ.Advisory.>".
This way there should always be space for consumer and connection advisories even when no normal messages are accepted.

will try the configuration tomorrow.