Monitoring producer flow control.

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Monitoring producer flow control.

Kevin Burton
I’m looking at implementing producer flow control so that I don’t fill up
the queues on my broker.

It doesn’t look like there’s any way I can see that a client is blocking,
waiting for resources to be released.

Maybe one strategy could be to put a stopwatch around each send() and then
I can see that I have some outstanding that have been open for a long time.


--

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring producer flow control.

Tim Bain
There's a log line in the broker whenever PFC kicks in; we watched the logs
for that line and fire off an email to get someone to investigate.  Would
that meet your needs?
On Apr 16, 2015 10:10 PM, "Kevin Burton" <[hidden email]> wrote:

> I’m looking at implementing producer flow control so that I don’t fill up
> the queues on my broker.
>
> It doesn’t look like there’s any way I can see that a client is blocking,
> waiting for resources to be released.
>
> Maybe one strategy could be to put a stopwatch around each send() and then
> I can see that I have some outstanding that have been open for a long time.
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring producer flow control.

Tim Bain
Also keep in mind that send() is a blocking call, so the stopwatch idea
will only tell you when PFC has occurred and then cleared up (once send()
returns), but won't alert you to a deadlock that won't clear up on its own.
On Apr 17, 2015 6:06 AM, "Tim Bain" <[hidden email]> wrote:

> There's a log line in the broker whenever PFC kicks in; we watched the
> logs for that line and fire off an email to get someone to investigate.
> Would that meet your needs?
> On Apr 16, 2015 10:10 PM, "Kevin Burton" <[hidden email]> wrote:
>
>> I’m looking at implementing producer flow control so that I don’t fill up
>> the queues on my broker.
>>
>> It doesn’t look like there’s any way I can see that a client is blocking,
>> waiting for resources to be released.
>>
>> Maybe one strategy could be to put a stopwatch around each send() and then
>> I can see that I have some outstanding that have been open for a long
>> time.
>>
>>
>> --
>>
>> Founder/CEO Spinn3r.com
>> Location: *San Francisco, CA*
>> blog: http://burtonator.wordpress.com
>> … or check out my Google+ profile
>> <https://plus.google.com/102718274791889610666/posts>
>> <http://spinn3r.com>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring producer flow control.

Kevin Burton
In reply to this post by Tim Bain
Ah. yeah. I just saw that.  I wrote a test for it and it triggered the log
line.  I might look at that line and try to figure out the best way to
track that?  Maybe a JMX integer that’s integrated for each time it
happens?

I might not need it in the short term but probably will in the long term.
If there’s no other way to do it and I get caught up in my patches I’ll try
to create one.

On Fri, Apr 17, 2015 at 5:06 AM, Tim Bain <[hidden email]> wrote:

> There's a log line in the broker whenever PFC kicks in; we watched the logs
> for that line and fire off an email to get someone to investigate.  Would
> that meet your needs?
> On Apr 16, 2015 10:10 PM, "Kevin Burton" <[hidden email]> wrote:
>
> > I’m looking at implementing producer flow control so that I don’t fill up
> > the queues on my broker.
> >
> > It doesn’t look like there’s any way I can see that a client is blocking,
> > waiting for resources to be released.
> >
> > Maybe one strategy could be to put a stopwatch around each send() and
> then
> > I can see that I have some outstanding that have been open for a long
> time.
> >
> >
> > --
> >
> > Founder/CEO Spinn3r.com
> > Location: *San Francisco, CA*
> > blog: http://burtonator.wordpress.com
> > … or check out my Google+ profile
> > <https://plus.google.com/102718274791889610666/posts>
> > <http://spinn3r.com>
> >
>



--

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring producer flow control.

Kevin Burton
In reply to this post by Tim Bain
Yes.  The way I was going to do it was have a gauge that measures how long
blocking calls are open for.

I’ll see it in our monitoring system because all of a sudden the send
latencies will keep rising per second.

On Fri, Apr 17, 2015 at 5:10 AM, Tim Bain <[hidden email]> wrote:

> Also keep in mind that send() is a blocking call, so the stopwatch idea
> will only tell you when PFC has occurred and then cleared up (once send()
> returns), but won't alert you to a deadlock that won't clear up on its own.
> On Apr 17, 2015 6:06 AM, "Tim Bain" <[hidden email]> wrote:
>
> > There's a log line in the broker whenever PFC kicks in; we watched the
> > logs for that line and fire off an email to get someone to investigate.
> > Would that meet your needs?
> > On Apr 16, 2015 10:10 PM, "Kevin Burton" <[hidden email]> wrote:
> >
> >> I’m looking at implementing producer flow control so that I don’t fill
> up
> >> the queues on my broker.
> >>
> >> It doesn’t look like there’s any way I can see that a client is
> blocking,
> >> waiting for resources to be released.
> >>
> >> Maybe one strategy could be to put a stopwatch around each send() and
> then
> >> I can see that I have some outstanding that have been open for a long
> >> time.
> >>
> >>
> >> --
> >>
> >> Founder/CEO Spinn3r.com
> >> Location: *San Francisco, CA*
> >> blog: http://burtonator.wordpress.com
> >> … or check out my Google+ profile
> >> <https://plus.google.com/102718274791889610666/posts>
> >> <http://spinn3r.com>
> >>
> >
>



--

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring producer flow control.

ceposta
In reply to this post by Kevin Burton
there is a way to check the destination to see if PFC is enabled by
checking the JMX:

https://github.com/apache/activemq/blob/master/activemq-broker/src/main/java/org/apache/activemq/broker/jmx/DestinationView.java#L423


On Thu, Apr 16, 2015 at 7:25 PM, Kevin Burton <[hidden email]> wrote:

> I’m looking at implementing producer flow control so that I don’t fill up
> the queues on my broker.
>
> It doesn’t look like there’s any way I can see that a client is blocking,
> waiting for resources to be released.
>
> Maybe one strategy could be to put a stopwatch around each send() and then
> I can see that I have some outstanding that have been open for a long time.
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>



--
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring producer flow control.

gtully
In reply to this post by Kevin Burton
https://issues.apache.org/jira/browse/AMQ-4635 may lelp.
there is also a destination statistic for blocked sends that could be monitored
and there is an advisory that fires when a dest is full.

On 17 April 2015 at 03:25, Kevin Burton <[hidden email]> wrote:

> I’m looking at implementing producer flow control so that I don’t fill up
> the queues on my broker.
>
> It doesn’t look like there’s any way I can see that a client is blocking,
> waiting for resources to be released.
>
> Maybe one strategy could be to put a stopwatch around each send() and then
> I can see that I have some outstanding that have been open for a long time.
>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring producer flow control.

Tim Bain
And of course both of those are documented at
http://activemq.apache.org/jmx.html...  Er, nope, that page hasn't been
updated in ages, despite new features like these being added.

How can we improve the process to increase the likelihood that JMX changes
make it onto the webpage?  Maybe a post-commit trigger that looks for
commits of files ending in MBeanView and emails the committer?  And/or
maybe we can automate the generation of the list of bean types and
properties by using reflection (or by using JMX against a running nightly
build, though reflection sounds easier)?  (If we made it part of the
release process to generate the HTML and update the webpage, we wouldn't
need the post-commit nagging emails.)

That content also needs version info: when each bean or property was added
and when it was removed; the generation script could grab each prior
version JAR and see what beans were present in that earlier version (i.e.
removed from a later version) or missing in that earlier version (i.e.
added in a later version).
https://issues.apache.org/jira/browse/AMQ-4635 may lelp.
there is also a destination statistic for blocked sends that could be
monitored
and there is an advisory that fires when a dest is full.

On 17 April 2015 at 03:25, Kevin Burton <[hidden email]> wrote:
> I’m looking at implementing producer flow control so that I don’t fill up
> the queues on my broker.
>
> It doesn’t look like there’s any way I can see that a client is blocking,
> waiting for resources to be released.
>
> Maybe one strategy could be to put a stopwatch around each send() and then
> I can see that I have some outstanding that have been open for a long
time.

>
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
Reply | Threaded
Open this post in threaded view
|

Re: Monitoring producer flow control.

Tim Bain
https://issues.apache.org/jira/browse/AMQ-5741

On Wed, Apr 22, 2015 at 7:12 AM, Tim Bain <[hidden email]> wrote:

> And of course both of those are documented at
> http://activemq.apache.org/jmx.html...  Er, nope, that page hasn't been
> updated in ages, despite new features like these being added.
>
> How can we improve the process to increase the likelihood that JMX changes
> make it onto the webpage?  Maybe a post-commit trigger that looks for
> commits of files ending in MBeanView and emails the committer?  And/or
> maybe we can automate the generation of the list of bean types and
> properties by using reflection (or by using JMX against a running nightly
> build, though reflection sounds easier)?  (If we made it part of the
> release process to generate the HTML and update the webpage, we wouldn't
> need the post-commit nagging emails.)
>
> That content also needs version info: when each bean or property was added
> and when it was removed; the generation script could grab each prior
> version JAR and see what beans were present in that earlier version (i.e.
> removed from a later version) or missing in that earlier version (i.e.
> added in a later version).
> https://issues.apache.org/jira/browse/AMQ-4635 may lelp.
> there is also a destination statistic for blocked sends that could be
> monitored
> and there is an advisory that fires when a dest is full.
>
> On 17 April 2015 at 03:25, Kevin Burton <[hidden email]> wrote:
> > I’m looking at implementing producer flow control so that I don’t fill up
> > the queues on my broker.
> >
> > It doesn’t look like there’s any way I can see that a client is blocking,
> > waiting for resources to be released.
> >
> > Maybe one strategy could be to put a stopwatch around each send() and
> then
> > I can see that I have some outstanding that have been open for a long
> time.
> >
> >
> > --
> >
> > Founder/CEO Spinn3r.com
> > Location: *San Francisco, CA*
> > blog: http://burtonator.wordpress.com
> > … or check out my Google+ profile
> > <https://plus.google.com/102718274791889610666/posts>
> > <http://spinn3r.com>
>