ActiveMQ redelivering acknowledged messages

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

ActiveMQ redelivering acknowledged messages

sbarlabanov
Hi,

what would be the reason for ActiveMQ (5.10) to redeliver already acknowledged messages?

We have an issue in a production system, that a consumer with a non transacted session in a CLIENT_ACKNOWLEDGE mode running in an app server (Glassfish) using ActiveMQ RAR, receives already consumed and acknowledged messages again.
This happens after we restart the application and it reconnects to a running standalone ActiveMQ instance. The application was up the last three days and was restarted (because of some maintenance). After the restart the application received all persistent, consumed messages since the last three days. All redelivered messages have JmsRedelivered=true.
No errors/warnings in activemq.log, no exceptions on the consumer side. Nothing.
The storage is KahaDB with discSyncs enabled and async index writes.
The redelivered messages have been reprocessed and acknowledged again. They seem to disappear - at least I do not see anything in the ActiveMQ JMX and KahaDB has only one data file from today (otherwise KahaDB would have several data files containing those messages).
Any ideas?

Best regards,
Sergiy
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ redelivering acknowledged messages

Kevin Burton
My main suggestion is to make CERTAIN all your settings are correct. JMS is
a difficult API and it’s possible to easily make mistakes and shoot
yourself in the foot.

Every time I think something is wrong it’s because I was accidentally using
the API wrong.

On Mon, Nov 3, 2014 at 8:11 AM, sbarlabanov <[hidden email]>
wrote:

> Hi,
>
> what would be the reason for ActiveMQ (5.10) to redeliver already
> acknowledged messages?
>
> We have an issue in a production system, that a consumer with a non
> transacted session in a CLIENT_ACKNOWLEDGE mode running in an app server
> (Glassfish) using ActiveMQ RAR, receives already consumed and acknowledged
> messages again.
> This happens after we restart the application and it reconnects to a
> running
> standalone ActiveMQ instance. The application was up the last three days
> and
> was restarted (because of some maintenance). After the restart the
> application received all persistent, consumed messages since the last three
> days. All redelivered messages have JmsRedelivered=true.
> No errors/warnings in activemq.log, no exceptions on the consumer side.
> Nothing.
> The storage is KahaDB with discSyncs enabled and async index writes.
> The redelivered messages have been reprocessed and acknowledged again. They
> seem to disappear - at least I do not see anything in the ActiveMQ JMX and
> KahaDB has only one data file from today (otherwise KahaDB would have
> several data files containing those messages).
> Any ideas?
>
> Best regards,
> Sergiy
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-redelivering-acknowledged-messages-tp4686898.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



--

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ redelivering acknowledged messages

Tim Bain
I second Kevin's suggestion; redelivered messages with CLIENT_ACKNOWLEDGE
could certainly be a bug somewhere in ActiveMQ, but it could very easily be
a case where your application logic is failing to acknowledge those
messages properly, such that the broker believes it needs to re-deliver
them once you disconnect and re-connect.  I'd make double-sure (by stepping
through with a debugger) that your acknowledgments are being done properly
when everything works as expected, and also scour your code to make
sure that there aren't any paths (e.g. exception handling) by which you
could bypass the acknowledgment if something unusual happened.

On Mon, Nov 3, 2014 at 11:17 AM, Kevin Burton <[hidden email]> wrote:

> My main suggestion is to make CERTAIN all your settings are correct. JMS is
> a difficult API and it’s possible to easily make mistakes and shoot
> yourself in the foot.
>
> Every time I think something is wrong it’s because I was accidentally using
> the API wrong.
>
> On Mon, Nov 3, 2014 at 8:11 AM, sbarlabanov <[hidden email]>
> wrote:
>
> > Hi,
> >
> > what would be the reason for ActiveMQ (5.10) to redeliver already
> > acknowledged messages?
> >
> > We have an issue in a production system, that a consumer with a non
> > transacted session in a CLIENT_ACKNOWLEDGE mode running in an app server
> > (Glassfish) using ActiveMQ RAR, receives already consumed and
> acknowledged
> > messages again.
> > This happens after we restart the application and it reconnects to a
> > running
> > standalone ActiveMQ instance. The application was up the last three days
> > and
> > was restarted (because of some maintenance). After the restart the
> > application received all persistent, consumed messages since the last
> three
> > days. All redelivered messages have JmsRedelivered=true.
> > No errors/warnings in activemq.log, no exceptions on the consumer side.
> > Nothing.
> > The storage is KahaDB with discSyncs enabled and async index writes.
> > The redelivered messages have been reprocessed and acknowledged again.
> They
> > seem to disappear - at least I do not see anything in the ActiveMQ JMX
> and
> > KahaDB has only one data file from today (otherwise KahaDB would have
> > several data files containing those messages).
> > Any ideas?
> >
> > Best regards,
> > Sergiy
> >
> >
> >
> >
> > --
> > View this message in context:
> >
> http://activemq.2283324.n4.nabble.com/ActiveMQ-redelivering-acknowledged-messages-tp4686898.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ redelivering acknowledged messages

sbarlabanov
I double checked everything and did a little bit debugging.
According to the current state of ActiveMQ code (5.10) , it may indeed loose acknowledgements.

1. Since 5.10 CLIENT_ACKNOWLEDGE is ignored in the application server - see the commit adb49f56 by Gary from 15.03.2014. This was quite a strange commit, since this was a very important change breaking the previous behavior even if it had not been really spec conform - it did work since years and suddenly it was changed without any corresponding Jira and any public information.
Now AUTO_ACKNOWLEDGE is used instead.

2. Despite http://activemq.apache.org/async-sends.html stating that persistent messages are sent synchronously outside a transaction, the ACKs are sent ASYnchronously outside a transaction in an application server by default (and I did not find any property which would change this behavior).
Since the ACKs are sent asynchronously, they may be lost obviously. And the corresponding messages will be redelivered. This is quite a big pain since everybody would expect an acknowledge of a persistent message to be synchronous regardless whether it is AUTO_ACK or CLIENT_ACK.

So the workaround in our case is to use JTA transactions instead of CLIENT_ACKNOWLEDGE and Message#acknowledge. This would be at least more spec conform.

But nevertheless it would be nice to see any comments from ActiveMQ developers about points 1 and 2 I described above.
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ redelivering acknowledged messages

Tim Bain
Gary Tully, does the first change Sergiy referenced (
http://mail-archives.apache.org/mod_mbox/activemq-commits/201403.mbox/%3C58f67097f342495e921e1a25e951aec6@...%3E)
really disable CLIENT_ACKNOWLEDGE from 5.10 onward, or is he
misunderstanding the change you made?  If he's got it right, can you please
explain why that option is no longer supported?

On Tue, Nov 4, 2014 at 7:37 AM, sbarlabanov <[hidden email]>
wrote:

> I double checked everything and did a little bit debugging.
> According to the current state of ActiveMQ code (5.10) , it may indeed
> loose
> acknowledgements.
>
> 1. Since 5.10 CLIENT_ACKNOWLEDGE is ignored in the application server - see
> the commit adb49f56 by Gary from 15.03.2014. This was quite a strange
> commit, since this was a very important change breaking the previous
> behavior even if it had not been really spec conform - it did work since
> years and suddenly it was changed without any corresponding Jira and any
> public information.
> Now AUTO_ACKNOWLEDGE is used instead.
>
> 2. Despite http://activemq.apache.org/async-sends.html stating that
> persistent messages are sent synchronously outside a transaction, the ACKs
> are sent ASYnchronously outside a transaction in an application server by
> default (and I did not find any property which would change this behavior).
> Since the ACKs are sent asynchronously, they may be lost obviously. And the
> corresponding messages will be redelivered. This is quite a big pain since
> everybody would expect an acknowledge of a persistent message to be
> synchronous regardless whether it is AUTO_ACK or CLIENT_ACK.
>
> So the workaround in our case is to use JTA transactions instead of
> CLIENT_ACKNOWLEDGE and Message#acknowledge. This would be at least more
> spec
> conform.
>
> But nevertheless it would be nice to see any comments from ActiveMQ
> developers about points 1 and 2 I described above.
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-redelivering-acknowledged-messages-tp4686898p4686933.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ redelivering acknowledged messages

gtully
It looks like my bad w.r.t CLIENT_ACK, my focus was on the tck and
transaction semantics.
For folks who use the rar managed connection factory in plain jms mode
there is a need to control the ack mode.
I added  connection info param useSessionArgs  to allow that.
  - CLIENT_ACKNOWLEDGE mode support should be back in 5.11
https://issues.apache.org/jira/browse/AMQ-5264
https://git-wip-us.apache.org/repos/asf?p=activemq.git;a=commit;h=9e139017

On 25 November 2014 at 15:47, Tim Bain <[hidden email]> wrote:

> Gary Tully, does the first change Sergiy referenced (
>
> http://mail-archives.apache.org/mod_mbox/activemq-commits/201403.mbox/%3C58f67097f342495e921e1a25e951aec6@...%3E
> )
> really disable CLIENT_ACKNOWLEDGE from 5.10 onward, or is he
> misunderstanding the change you made?  If he's got it right, can you please
> explain why that option is no longer supported?
>
> On Tue, Nov 4, 2014 at 7:37 AM, sbarlabanov <[hidden email]>
> wrote:
>
> > I double checked everything and did a little bit debugging.
> > According to the current state of ActiveMQ code (5.10) , it may indeed
> > loose
> > acknowledgements.
> >
> > 1. Since 5.10 CLIENT_ACKNOWLEDGE is ignored in the application server -
> see
> > the commit adb49f56 by Gary from 15.03.2014. This was quite a strange
> > commit, since this was a very important change breaking the previous
> > behavior even if it had not been really spec conform - it did work since
> > years and suddenly it was changed without any corresponding Jira and any
> > public information.
> > Now AUTO_ACKNOWLEDGE is used instead.
> >
> > 2. Despite http://activemq.apache.org/async-sends.html stating that
> > persistent messages are sent synchronously outside a transaction, the
> ACKs
> > are sent ASYnchronously outside a transaction in an application server by
> > default (and I did not find any property which would change this
> behavior).
> > Since the ACKs are sent asynchronously, they may be lost obviously. And
> the
> > corresponding messages will be redelivered. This is quite a big pain
> since
> > everybody would expect an acknowledge of a persistent message to be
> > synchronous regardless whether it is AUTO_ACK or CLIENT_ACK.
> >
> > So the workaround in our case is to use JTA transactions instead of
> > CLIENT_ACKNOWLEDGE and Message#acknowledge. This would be at least more
> > spec
> > conform.
> >
> > But nevertheless it would be nice to see any comments from ActiveMQ
> > developers about points 1 and 2 I described above.
> >
> >
> >
> > --
> > View this message in context:
> >
> http://activemq.2283324.n4.nabble.com/ActiveMQ-redelivering-acknowledged-messages-tp4686898p4686933.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ redelivering acknowledged messages

Tim Bain
Cool, thanks for taking another look at it.

On Fri, Dec 5, 2014 at 8:22 AM, Gary Tully <[hidden email]> wrote:

> It looks like my bad w.r.t CLIENT_ACK, my focus was on the tck and
> transaction semantics.
> For folks who use the rar managed connection factory in plain jms mode
> there is a need to control the ack mode.
> I added  connection info param useSessionArgs  to allow that.
>   - CLIENT_ACKNOWLEDGE mode support should be back in 5.11
> https://issues.apache.org/jira/browse/AMQ-5264
> https://git-wip-us.apache.org/repos/asf?p=activemq.git;a=commit;h=9e139017
>
> On 25 November 2014 at 15:47, Tim Bain <[hidden email]> wrote:
>
> > Gary Tully, does the first change Sergiy referenced (
> >
> >
> http://mail-archives.apache.org/mod_mbox/activemq-commits/201403.mbox/%3C58f67097f342495e921e1a25e951aec6@...%3E
> > )
> > really disable CLIENT_ACKNOWLEDGE from 5.10 onward, or is he
> > misunderstanding the change you made?  If he's got it right, can you
> please
> > explain why that option is no longer supported?
> >
> > On Tue, Nov 4, 2014 at 7:37 AM, sbarlabanov <
> [hidden email]>
> > wrote:
> >
> > > I double checked everything and did a little bit debugging.
> > > According to the current state of ActiveMQ code (5.10) , it may indeed
> > > loose
> > > acknowledgements.
> > >
> > > 1. Since 5.10 CLIENT_ACKNOWLEDGE is ignored in the application server -
> > see
> > > the commit adb49f56 by Gary from 15.03.2014. This was quite a strange
> > > commit, since this was a very important change breaking the previous
> > > behavior even if it had not been really spec conform - it did work
> since
> > > years and suddenly it was changed without any corresponding Jira and
> any
> > > public information.
> > > Now AUTO_ACKNOWLEDGE is used instead.
> > >
> > > 2. Despite http://activemq.apache.org/async-sends.html stating that
> > > persistent messages are sent synchronously outside a transaction, the
> > ACKs
> > > are sent ASYnchronously outside a transaction in an application server
> by
> > > default (and I did not find any property which would change this
> > behavior).
> > > Since the ACKs are sent asynchronously, they may be lost obviously. And
> > the
> > > corresponding messages will be redelivered. This is quite a big pain
> > since
> > > everybody would expect an acknowledge of a persistent message to be
> > > synchronous regardless whether it is AUTO_ACK or CLIENT_ACK.
> > >
> > > So the workaround in our case is to use JTA transactions instead of
> > > CLIENT_ACKNOWLEDGE and Message#acknowledge. This would be at least more
> > > spec
> > > conform.
> > >
> > > But nevertheless it would be nice to see any comments from ActiveMQ
> > > developers about points 1 and 2 I described above.
> > >
> > >
> > >
> > > --
> > > View this message in context:
> > >
> >
> http://activemq.2283324.n4.nabble.com/ActiveMQ-redelivering-acknowledged-messages-tp4686898p4686933.html
> > > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> > >
> >
>