ActiveMQ: producer flow control and temp store

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

ActiveMQ: producer flow control and temp store

art.licis
Dear community,

We want to enable producer flow control to protect a broker from running out
of memory if slow topic consumers appear (we had OOMs through several runs
on our staging environment, and so far PFC off is one of the versions why
this has happened). However, we would want to keep our previous approach: if
memory not enough, use temp store for our topic messages so that the
producer keeps producing and fast consumers are up-to-date. So we expect
(almost) no backpressure if ActiveMQ has enough CPU and IO to write those
messages to disk.

I can see that simply enabling producer flow control keeps utilizing temp
store:


<destinationPolicy>
  <policyMap>
    <policyEntries>
      <policyEntry topic=">" producerFlowControl="true">


<systemUsage>
  <systemUsage>
    <memoryUsage>
      <memoryUsage limit="2 gb"/>
    </memoryUsage>
    <storeUsage>
      <storeUsage limit="100 gb"/>
    </storeUsage>
    <tempUsage>
      <tempUsage limit="10 gb"/>
    </tempUsage>
  </systemUsage>
</systemUsage>


However, our concerns are that single case of slow consumer can flood most
of the available heap with messages before flowing to temp store. The
following might seem as a perfect solution:
*<policyEntry topic=">" producerFlowControl="true" memoryLimit="20 mb">*

However, with this configuration ActiveMQ start throttling producers as soon
as topic memory limit reached, without going to temp store.

Any chance there's some trick to limit memory per destination, and at the
same time keep using temp store?

Thank you,
Art



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ: producer flow control and temp store

Tim Bain
Art,

I've never seen a way to do that, but it sounds like a great idea.

Would you be willing to submit an enhancement request for it in JIRA?

Thanks,
Tim

On Thu, Aug 9, 2018, 8:54 AM art.licis <[hidden email]> wrote:

> Dear community,
>
> We want to enable producer flow control to protect a broker from running
> out
> of memory if slow topic consumers appear (we had OOMs through several runs
> on our staging environment, and so far PFC off is one of the versions why
> this has happened). However, we would want to keep our previous approach:
> if
> memory not enough, use temp store for our topic messages so that the
> producer keeps producing and fast consumers are up-to-date. So we expect
> (almost) no backpressure if ActiveMQ has enough CPU and IO to write those
> messages to disk.
>
> I can see that simply enabling producer flow control keeps utilizing temp
> store:
>
>
> <destinationPolicy>
>   <policyMap>
>     <policyEntries>
>       <policyEntry topic=">" producerFlowControl="true">
>
>
> <systemUsage>
>   <systemUsage>
>     <memoryUsage>
>       <memoryUsage limit="2 gb"/>
>     </memoryUsage>
>     <storeUsage>
>       <storeUsage limit="100 gb"/>
>     </storeUsage>
>     <tempUsage>
>       <tempUsage limit="10 gb"/>
>     </tempUsage>
>   </systemUsage>
> </systemUsage>
>
>
> However, our concerns are that single case of slow consumer can flood most
> of the available heap with messages before flowing to temp store. The
> following might seem as a perfect solution:
> *<policyEntry topic=">" producerFlowControl="true" memoryLimit="20 mb">*
>
> However, with this configuration ActiveMQ start throttling producers as
> soon
> as topic memory limit reached, without going to temp store.
>
> Any chance there's some trick to limit memory per destination, and at the
> same time keep using temp store?
>
> Thank you,
> Art
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ: producer flow control and temp store

art.licis
Hi Tim,

Sorry, I'm a bit late with this :)
It's great you like the idea! I'll do my quick research and I'd love to
contribute, so I'll make a JIRA and hopefully a PR.

- Art



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html