Dropping slow AMQP consumer behaviour

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Dropping slow AMQP consumer behaviour

Marcel Meulemans
I have a question about what the expected behaviour of dropping a slow AMQP
consumer is/should be. I am using 5.11-SNAPSHOT. The behaviour I am
observing is as follows:

- I have an AMQP (proton-c) client that is consuming messages from ActiveMQ
from a specific queue via a specific AMQP link.
- The client is on a different machine from ActiveMQ
- While device is correctly consuming messages turn off the wifi on the
AMQP machine (there are still messages in this queue).
- Obviously communication stops.
- Turn back on the wifi.
- ActiveMQ will have dropped the "slow" consumer: 2014-12-22 11:57:07,414 |
INFO  | aborting slow consumer: ID:ubuntu-hp-54642-1418997390497-2:16:0:0
for destination:queue://somequeue |
org.apache.activemq.broker.region.policy.AbortSlowConsumerStrategy |
ActiveMQ Broker[localhost] Scheduler
- Both computers still consider the TCP connection to be active (the
connection is still listed also by ActiveMQ
- On the client nothing happens: no messages are consumed anymore
(obviously), AMQP link is open on remote and local side.

Currently I seem to be unable to recover from this situation because the
client does not know ActiveMQ has drop it as a consumer. I expected
ActiveMQ to close the AMQP link, or maybe even the session, after which
recovering would be fairly trivial.

Any ideas?

--
Marcel
Reply | Threaded
Open this post in threaded view
|

Re: Dropping slow AMQP consumer behaviour

Tim Bain
Are you using a non-zero wireFormat.maxInactivityDuration?  This type of
situation is exactly what it's intended to detect.
On Dec 23, 2014 2:51 AM, "Marcel Meulemans" <[hidden email]>
wrote:

> I have a question about what the expected behaviour of dropping a slow AMQP
> consumer is/should be. I am using 5.11-SNAPSHOT. The behaviour I am
> observing is as follows:
>
> - I have an AMQP (proton-c) client that is consuming messages from ActiveMQ
> from a specific queue via a specific AMQP link.
> - The client is on a different machine from ActiveMQ
> - While device is correctly consuming messages turn off the wifi on the
> AMQP machine (there are still messages in this queue).
> - Obviously communication stops.
> - Turn back on the wifi.
> - ActiveMQ will have dropped the "slow" consumer: 2014-12-22 11:57:07,414 |
> INFO  | aborting slow consumer: ID:ubuntu-hp-54642-1418997390497-2:16:0:0
> for destination:queue://somequeue |
> org.apache.activemq.broker.region.policy.AbortSlowConsumerStrategy |
> ActiveMQ Broker[localhost] Scheduler
> - Both computers still consider the TCP connection to be active (the
> connection is still listed also by ActiveMQ
> - On the client nothing happens: no messages are consumed anymore
> (obviously), AMQP link is open on remote and local side.
>
> Currently I seem to be unable to recover from this situation because the
> client does not know ActiveMQ has drop it as a consumer. I expected
> ActiveMQ to close the AMQP link, or maybe even the session, after which
> recovering would be fairly trivial.
>
> Any ideas?
>
> --
> Marcel
>
Reply | Threaded
Open this post in threaded view
|

Re: Dropping slow AMQP consumer behaviour

Marcel Meulemans
> Are you using a non-zero wireFormat.maxInactivityDuration?  This type of
> situation is exactly what it's intended to detect.

I don't think the AMQP protocol converter implements an
InactivityMonitor a.t.m. and I won't help I think. The problem is not
that there is no communication between the two parties (the
communication is only interrupted briefly), the problem is that the
client side does not know that ActiveMQ has dropped him as a consumer
of a particular queue.

--
Marcel
Reply | Threaded
Open this post in threaded view
|

Re: Dropping slow AMQP consumer behaviour

Tim Bain
I'd have thought that if the broker no longer considered the consumer to be
active, it would have stopped sending keep-alive packets, which would then
cause the consumer to abort, but I haven't looked at that code in any depth.

Have you tried setting abortConnection=true on the
AbortSlowConsumerStrategy, to have the broker close the TCP connection when
it aborts the consumer, rather than just sending a request to the consumer
to ask it nicely to abort?  I'm not sure if it will change the behavior,
but it's worth a shot.

On Tue, Dec 23, 2014 at 8:27 AM, Marcel Meulemans <
[hidden email]> wrote:

> > Are you using a non-zero wireFormat.maxInactivityDuration?  This type of
> > situation is exactly what it's intended to detect.
>
> I don't think the AMQP protocol converter implements an
> InactivityMonitor a.t.m. and I won't help I think. The problem is not
> that there is no communication between the two parties (the
> communication is only interrupted briefly), the problem is that the
> client side does not know that ActiveMQ has dropped him as a consumer
> of a particular queue.
>
> --
> Marcel
>
Reply | Threaded
Open this post in threaded view
|

Re: Dropping slow AMQP consumer behaviour

Marcel Meulemans
On Tue, Dec 23, 2014 at 5:36 PM, Tim Bain <[hidden email]> wrote:
> Have you tried setting abortConnection=true on the
> AbortSlowConsumerStrategy, to have the broker close the TCP connection when
> it aborts the consumer, rather than just sending a request to the consumer
> to ask it nicely to abort?  I'm not sure if it will change the behavior,
> but it's worth a shot.

Thanks for the tip. This doesn't change the behaviour much but does
shed some light on the problem. Now ActiveMQ closes the TCP connection
but still the client is unaware so this means the TCP connection is
really "broken" in some way. My conclusion is that the problem is not
solvable without some kind of keep-alive mechinisme (TCP or AMQP).

--
Marcel