Force Producer to fail when blocked by slow consumer?

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Force Producer to fail when blocked by slow consumer?

Paulon
Hey,

we have a setup whereby we are sending persistent messages to an MQ in SessionTransacted batches.

The problem we are facing is that if the MQ fills up, our producer blocks indefinitely.

We dont have control over the server side queue, so we need to be able to abort sending (rather than wait forever)

We have set the sendTimeout on the connectionFactory however, while this throws the expected exception, any attempt to do a session.close() blocks waiting for a Response object.

Is there a way to get the blocked send call to just fail/timeout to enable us to close the session/connection object and terminate the thread/process?

Otherwise we are at the mercy of a slow/poorly configured consumer that can block our system.

Cheers!
Reply | Threaded
Open this post in threaded view
|

Re: Force Producer to fail when blocked by slow consumer?

christopher.l.shannon
There are a couple options you can configure.  Take a look at this page
http://activemq.apache.org/producer-flow-control.html

Look at the sendFailIfNoSpace and sendFailIfNoSpaceAfterTimeout options.
These will allow the producer to fail immediately or after a set amount o
time instead of waiting indefinitely.

On Mon, Apr 4, 2016 at 11:18 AM, Paulon <[hidden email]> wrote:

> Hey,
>
> we have a setup whereby we are sending persistent messages to an MQ in
> SessionTransacted batches.
>
> The problem we are facing is that if the MQ fills up, our producer blocks
> indefinitely.
>
> We dont have control over the server side queue, so we need to be able to
> abort sending (rather than wait forever)
>
> We have set the sendTimeout on the connectionFactory however, while this
> throws the expected exception, any attempt to do a session.close() blocks
> waiting for a Response object.
>
> Is there a way to get the blocked send call to just fail/timeout to enable
> us to close the session/connection object and terminate the thread/process?
>
> Otherwise we are at the mercy of a slow/poorly configured consumer that can
> block our system.
>
> Cheers!
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Force-Producer-to-fail-when-blocked-by-slow-consumer-tp4710264.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Force Producer to fail when blocked by slow consumer?

Paulon
Hey,

are they not all configured on the MQ server side though?

I don't have access to that as its on the Customer side (we are delivering to them)
Is there anything that can be configured on the Producer side?

Cheers!
Paul
Reply | Threaded
Open this post in threaded view
|

Re: Force Producer to fail when blocked by slow consumer?

christopher.l.shannon
Yes those settings are configured on the server side because it's the
server that decides when to block.  The broker is only going to throw an
exception to the client if those settings have been enabled otherwise the
client is just going to wait for space.

As you pointed out I think the only other thing you can do on the client
side is that timeout setting, but you said are seeing it hang on close.  It
does seem weird to me that it would hang on that close request and not be
able to process that so there could be a bug there.

If you could submit a small unit test that shows the close hanging when the
producer is blocked that would be great and someone could look into it to
see if there is some improvement that could be done to fix that.
Otherwise, when I get some time I can try and reproduce the issue and
create the test case.


On Mon, Apr 4, 2016 at 12:24 PM, Paulon <[hidden email]> wrote:

> Hey,
>
> are they not all configured on the MQ server side though?
>
> I don't have access to that as its on the Customer side (we are delivering
> to them)
> Is there anything that can be configured on the Producer side?
>
> Cheers!
> Paul
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Force-Producer-to-fail-when-blocked-by-slow-consumer-tp4710264p4710339.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

RE: Force Producer to fail when blocked by slow consumer?

Paulon

Hey,

 

The problem is that a call to session.close() ends up getting stuck during a rollback

 

After this call in ActiveMQSession.rollback()

this.connection.syncSendPacket(info);

It seems to block in FutureResponse.getResult()

e = (Response)this.responseSlot.take();

 

responseSlot only seems to have a size of 1 and it looks like the timedout request didn’t return the response slot it had taken?

 

Cheers,

Paul



 

 

From: christopher.l.shannon [via ActiveMQ] [mailto:ml-node+[hidden email]]
Sent: 04 April 2016 18:15
To: O'Neill, Paul <[hidden email]>
Subject: Re: Force Producer to fail when blocked by slow consumer?

 

Yes those settings are configured on the server side because it's the
server that decides when to block.  The broker is only going to throw an
exception to the client if those settings have been enabled otherwise the
client is just going to wait for space.

As you pointed out I think the only other thing you can do on the client
side is that timeout setting, but you said are seeing it hang on close.  It
does seem weird to me that it would hang on that close request and not be
able to process that so there could be a bug there.

If you could submit a small unit test that shows the close hanging when the
producer is blocked that would be great and someone could look into it to
see if there is some improvement that could be done to fix that.
Otherwise, when I get some time I can try and reproduce the issue and
create the test case.


On Mon, Apr 4, 2016 at 12:24 PM, Paulon <[hidden email]> wrote:


> Hey,
>
> are they not all configured on the MQ server side though?
>
> I don't have access to that as its on the Customer side (we are delivering
> to them)
> Is there anything that can be configured on the Producer side?
>
> Cheers!
> Paul
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Force-Producer-to-fail-when-blocked-by-slow-consumer-tp4710264p4710339.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

 


If you reply to this email, your message will be added to the discussion below:

http://activemq.2283324.n4.nabble.com/Force-Producer-to-fail-when-blocked-by-slow-consumer-tp4710264p4710341.html

To unsubscribe from Force Producer to fail when blocked by slow consumer?, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

RE: Force Producer to fail when blocked by slow consumer?

Paulon
In reply to this post by christopher.l.shannon

Actually on further digging it seems that the problem is the rollback() call is not using the sendTimeout, so I think it too is being blocked by the server/slow consumer.

 

If a send timeout is set on the connectionFactory, shouldn’t all connections/sends be using it?

 

To recreate:

Create a queue with small thresholds for disk/memory usage and setup a producer publishing to the queue with a connectionTimeout of say 2s.

As soon as the queue starts blocking the timeout exception will be thrown, if you then try to close the resources it will hang indefinitely.

 

 

From: christopher.l.shannon [via ActiveMQ] [mailto:ml-node+[hidden email]]
Sent: 04 April 2016 18:15
To: O'Neill, Paul <[hidden email]>
Subject: Re: Force Producer to fail when blocked by slow consumer?

 

Yes those settings are configured on the server side because it's the
server that decides when to block.  The broker is only going to throw an
exception to the client if those settings have been enabled otherwise the
client is just going to wait for space.

As you pointed out I think the only other thing you can do on the client
side is that timeout setting, but you said are seeing it hang on close.  It
does seem weird to me that it would hang on that close request and not be
able to process that so there could be a bug there.

If you could submit a small unit test that shows the close hanging when the
producer is blocked that would be great and someone could look into it to
see if there is some improvement that could be done to fix that.
Otherwise, when I get some time I can try and reproduce the issue and
create the test case.


On Mon, Apr 4, 2016 at 12:24 PM, Paulon <[hidden email]> wrote:


> Hey,
>
> are they not all configured on the MQ server side though?
>
> I don't have access to that as its on the Customer side (we are delivering
> to them)
> Is there anything that can be configured on the Producer side?
>
> Cheers!
> Paul
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Force-Producer-to-fail-when-blocked-by-slow-consumer-tp4710264p4710339.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

 


If you reply to this email, your message will be added to the discussion below:

http://activemq.2283324.n4.nabble.com/Force-Producer-to-fail-when-blocked-by-slow-consumer-tp4710264p4710341.html

To unsubscribe from Force Producer to fail when blocked by slow consumer?, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: Force Producer to fail when blocked by slow consumer?

KenHall
In reply to this post by christopher.l.shannon
Hi Christopher,

I'm working with Paul. I've created a small test to illustrate the issue.
The zip I have attached includes three classes in a zip of an Eclipse project. Call the main method of RunTest to run the test. The activemq.xml used is in the root directory of the project.

The test was run using an out of the box install of ActiveMQ 5.13.2 on Windows.

The program will timeout after 51 messages and attempt to close the session.

It eventually locks in this line:

            this.connection.syncSendPacket(info);

in the rollback method of the TransactionContext class.

We know we can get round this by making changes to the broker configuration but we do not own this in our business scenario so we must be able to terminate our producer independently of the broker.

We believe that by default the rollback should follow the same timeout  as the connection.MQCloseIssue.7z