Configuring ActiveMQ for reliability

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Configuring ActiveMQ for reliability

Robin Kåveland Hansen-2
Hi,

Vi have been running ActiveMQ-5.6.0 in production since january last year and
we’ve been looking to upgrade to a newer version for a while. Our main
requirement is reliability and we run the broker with jdbc persistence against
Oracle.

However for the last few version upgrades we have attempted we have been
able to find message loss scenarios that hit versions newer than 5.6.0.

At this point we’re running out of ideas, it seems that we can not come up
with a configuration for 5.7.0, 5.9.0/1 brokers that does not have message
loss in the following scenario:
- Start batchjob that produces a few thousand messages with client A
- Messages are consumed by client B who posts them to C
- Let C consume a few hundred

kill -9 C
kill -9 activemq

B and C have audit.

We’ve also been able to cause message loss in ‘nicer’ scenarios (TERM
instead of KILL) in other ways. Clearly we seem to be missing something
here, because it works fine with 5.6.0 (unless the client runs > 5.7.0). We
sometimes observe duplicate messages with 5.6.0, but this is fine.

On the broker, the following is set for all queues:
producerFlowControl=“false"
queuePrefetch=“1"
optimizedDispatch=“true”

We are using transacted clients (these are configured with camel). We have
tried various permutations of the following options on ActiveMQConnectionFactory:
optimizeAcknowledge
alwaysSyncSend
dispatchAsync
sendAcksAsync
We also ensured that we’re sending with the appropriate delivery mode (2).

Are we missing something? Does anyone have tips for us?
--  
Kind regards,
Robin Kåveland Hansen

Reply | Threaded
Open this post in threaded view
|

Re: Configuring ActiveMQ for reliability

ceposta
Can you put together a test that shows message loss? I can help you
debug it and find the fix if there is indeed message loss as seen by
the broker.



On Mon, May 5, 2014 at 1:59 AM, Robin Kåveland Hansen
<[hidden email]> wrote:

> Hi,
>
> Vi have been running ActiveMQ-5.6.0 in production since january last year and
> we’ve been looking to upgrade to a newer version for a while. Our main
> requirement is reliability and we run the broker with jdbc persistence against
> Oracle.
>
> However for the last few version upgrades we have attempted we have been
> able to find message loss scenarios that hit versions newer than 5.6.0.
>
> At this point we’re running out of ideas, it seems that we can not come up
> with a configuration for 5.7.0, 5.9.0/1 brokers that does not have message
> loss in the following scenario:
> - Start batchjob that produces a few thousand messages with client A
> - Messages are consumed by client B who posts them to C
> - Let C consume a few hundred
>
> kill -9 C
> kill -9 activemq
>
> B and C have audit.
>
> We’ve also been able to cause message loss in ‘nicer’ scenarios (TERM
> instead of KILL) in other ways. Clearly we seem to be missing something
> here, because it works fine with 5.6.0 (unless the client runs > 5.7.0). We
> sometimes observe duplicate messages with 5.6.0, but this is fine.
>
> On the broker, the following is set for all queues:
> producerFlowControl=“false"
> queuePrefetch=“1"
> optimizedDispatch=“true”
>
> We are using transacted clients (these are configured with camel). We have
> tried various permutations of the following options on ActiveMQConnectionFactory:
> optimizeAcknowledge
> alwaysSyncSend
> dispatchAsync
> sendAcksAsync
> We also ensured that we’re sending with the appropriate delivery mode (2).
>
> Are we missing something? Does anyone have tips for us?
> --
> Kind regards,
> Robin Kåveland Hansen
>



--
Christian Posta
http://www.christianposta.com/blog
twitter: @christianposta
Reply | Threaded
Open this post in threaded view
|

Re: Configuring ActiveMQ for reliability

Robin Kåveland Hansen
Hi,

I am trying to do this, but it is difficult to figure out what is the minimal testcase here.

It’ll take a while before this is ready and I’m afraid I can’t make a junit-testcase here. :-/


On 6 May 2014 19:36, Christian Posta <[hidden email] (mailto:[hidden email])> wrote:

> Can you put together a test that shows message loss? I can help you
> debug it and find the fix if there is indeed message loss as seen by
> the broker.
>  
>  
>  
> On Mon, May 5, 2014 at 1:59 AM, Robin Kåveland Hansen
> <[hidden email] (mailto:[hidden email])> wrote:
> > Hi,
> >
> > Vi have been running ActiveMQ-5.6.0 in production since january last year and
> > we’ve been looking to upgrade to a newer version for a while. Our main
> > requirement is reliability and we run the broker with jdbc persistence against
> > Oracle.
> >
> > However for the last few version upgrades we have attempted we have been
> > able to find message loss scenarios that hit versions newer than 5.6.0.
> >
> > At this point we’re running out of ideas, it seems that we can not come up
> > with a configuration for 5.7.0, 5.9.0/1 brokers that does not have message
> > loss in the following scenario:
> > - Start batchjob that produces a few thousand messages with client A
> > - Messages are consumed by client B who posts them to C
> > - Let C consume a few hundred
> >
> > kill -9 C
> > kill -9 activemq
> >
> > B and C have audit.
> >
> > We’ve also been able to cause message loss in ‘nicer’ scenarios (TERM
> > instead of KILL) in other ways. Clearly we seem to be missing something
> > here, because it works fine with 5.6.0 (unless the client runs > 5.7.0). We
> > sometimes observe duplicate messages with 5.6.0, but this is fine.
> >
> > On the broker, the following is set for all queues:
> > producerFlowControl=“false"
> > queuePrefetch=“1"
> > optimizedDispatch=“true”
> >
> > We are using transacted clients (these are configured with camel). We have
> > tried various permutations of the following options on ActiveMQConnectionFactory:
> > optimizeAcknowledge
> > alwaysSyncSend
> > dispatchAsync
> > sendAcksAsync
> > We also ensured that we’re sending with the appropriate delivery mode (2).
> >
> > Are we missing something? Does anyone have tips for us?
> > --
> > Kind regards,
> > Robin Kåveland Hansen
> >
>  
>  
>  
> --
> Christian Posta
> http://www.christianposta.com/blog
> twitter: @christianposta



--  
Vennlig hilsen,
Robin Kåveland Hansen