ActiveMQ out-of-order redelivery - resource adapter

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

ActiveMQ out-of-order redelivery - resource adapter

smil2k
Dear List!

We experienced the following problem using JBoss 7 and ActiveMQ-RAR:

In case a message cannot be processed by an MDB then out-of-order delivery
might happen.
We are using message grouping, still the order of the messages changes.

Does anybody has experience on enforcing strict message ordering with
ActiveMQ and JBoss 7?

Cheers,

Tamas
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ out-of-order redelivery - resource adapter

ceposta
Can you elaborate? "cannot be processed by MDB" then out of delivery might
happen.

So you mean if the MDB fails and the message goes to DLQ?


On Tue, Jun 3, 2014 at 2:58 AM, Cserveny Tamas <[hidden email]>
wrote:

> Dear List!
>
> We experienced the following problem using JBoss 7 and ActiveMQ-RAR:
>
> In case a message cannot be processed by an MDB then out-of-order delivery
> might happen.
> We are using message grouping, still the order of the messages changes.
>
> Does anybody has experience on enforcing strict message ordering with
> ActiveMQ and JBoss 7?
>
> Cheers,
>
> Tamas
>



--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ out-of-order redelivery - resource adapter

smil2k
Hi,

In case the MDB throws an exception, then a mixup might happen.

Example: Let's assume 3 messages.

In case 1 throws an exception, my expectation would be this:

1 1retry 2 3

But in reality the following happens in a minority case:

1 2 1retry 3

So message 2 is processed before the 1retry.

Cheers,

Tamas


Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ out-of-order redelivery - resource adapter

ceposta
You'll have to check the settings on the MDB. You may have multiple
concurrent sessions? Or multiple sessions? ActiveMQ by default will retry
at the consumer level so you shouldn't end up with that.


On Thu, Jun 5, 2014 at 6:41 AM, smil2k <[hidden email]> wrote:

> Hi,
>
> In case the MDB throws an exception, then a mixup might happen.
>
> Example: Let's assume 3 messages.
>
> In case 1 throws an exception, my expectation would be this:
>
> 1 1retry 2 3
>
> But in reality the following happens in a minority case:
>
> 1 2 1retry 3
>
> So message 2 is processed before the 1retry.
>
> Cheers,
>
> Tamas
>
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-out-of-order-redelivery-resource-adapter-tp4681601p4681719.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ out-of-order redelivery - resource adapter

smil2k
Hello,

This is what I thought also, but it seems that in the RAR ActiveMQ does not use the normal consumer.
I've set the appropriate settings on the MDB maxSessions is 1 and works fine when there is no exception.

The problem only occurs with rollback and only in minority of the cases.

Cheers,

Tamas
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ out-of-order redelivery - resource adapter

gtully
maybe try the 5.10 RAR, there were a bunch of fixes relating to
transaction timeout and eager acking for 5.10

On 13 June 2014 02:40, smil2k <[hidden email]> wrote:

> Hello,
>
> This is what I thought also, but it seems that in the RAR ActiveMQ does not
> use the normal consumer.
> I've set the appropriate settings on the MDB maxSessions is 1 and works fine
> when there is no exception.
>
> The problem only occurs with rollback and only in minority of the cases.
>
> Cheers,
>
> Tamas
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-out-of-order-redelivery-resource-adapter-tp4681601p4682095.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.



--
http://redhat.com
http://blog.garytully.com
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ out-of-order redelivery - resource adapter

smil2k
Hi,  

I've checked it and it seems that the RA does not block delivery of the messages until the redelivery policy takes effect. The consumer does that, but application servers are using the Session directly to delivery the messages somehow.

I've opened a case for that: AMQ-5400, I'll try to provide a patch :-)

Cheers,

Tamas
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ out-of-order redelivery - resource adapter

smil2k
Hi,

I've created a patch which blocks the session until the redelivery timeout by stopping the session executor's queue.

I have an automated test which can check the order of the messages, I'll leave it running for the night.

The tool creates messages and dispatches them to a queue with exclusive consumer on. They put it into a queue with 40 consumers with message grouping. The consumers will forward it back to the tool again. 3% of the messages will throw an exception during processing for the first time.
Results are promising after 50k messages. (no out of order messages).

Btw,
I also tried master-slave fail-over and application server fail-over while the tool was running. Is it normal to get duplicated messages in case I kill AMQ or the APP server with -9?

Cheers,

Tamas
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ out-of-order redelivery - resource adapter

smil2k
Result of the night:
After 1 million delivered messages with 27k generated errors using two application servers the order was always kept and the system was still delivering messages just fine.

Cheers,

Tamas