Quantcast

[DISCUSS] Artemis IOPS Limiter strategy

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
41 messages Options
123
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[DISCUSS] Artemis IOPS Limiter strategy

nigro_franz
This post was updated on .
Hi guys!

After the changes to TimedBuffer to make the Journal capable to adapt to the current load, I've had instructive feedbacks/advices about IOPS limits that could be reached on too high and long burst (depending on disk capacity).

There is the need to separate cleanly the concerns related to disk capacity and load requested of the system: a configurable IOLimiter could help on it.

A first attempt to do it is in this branch: https://github.com/franz1981/activemq-artemis/blob/iops_limiter

The possible configuration of such IOPS limiter could be:

1) no limiter (default): the disk naturally create backpressure on too high loads
2) balanced: "best effort" IO limiter based on IOPS computed by the broker tuning tools (ie SyncCalculation) https://github.com/franz1981/activemq-artemis/blob/iops_limiter/artemis-journal/src/main/java/org/apache/activemq/artemis/core/io/buffer/BestEffortIOLimiter.java

I was thinking to add a strict one too, but AFAIK strict limiters tends to over-compensate the controlled systems, killing its latencies/throughput when it is not be supposed to be done...just my 2 cents.

wdyt?

Franz
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] Artemis IOPS Limiter strategy

clebertsuconic
Is that really needed ?

You already have flow control on protocols and producers won't send data
beyond the capacity until you have enough capacity.

The timed buffer is just batching writes to use he io Better.


How this would differ in simple words ?



On Wed, May 10, 2017 at 4:43 AM nigro_franz <[hidden email]> wrote:

> Hi guys!
>
> After the changes to TimedBuffer to make the Journal capable to adapt to
> the
> current load, I've had instructive feedbacks/advices about IOPS limits that
> could be reached on too high and long burst (depending on disk capacity).
>
> There is the need to separate cleanly the concerns related to disk capacity
> and load requested of the system: a configurable IOLimiter could help on
> it.
>
> A first attemp to do it is in this branch:
> https://github.com/franz1981/activemq-artemis/blob/iops_limiter
>
> The possible configuration of such IOPS limiter could be:
>
> 1) no limiter (default): the disk naturally create backpressure on too high
> loads
> 2) balanced: "best effort" IO limiter based on IOPS computed by the broker
> tuning tools (ie SyncCalculation)
>
> https://github.com/franz1981/activemq-artemis/blob/iops_limiter/artemis-journal/src/main/java/org/apache/activemq/artemis/core/io/buffer/BestEffortIOLimiter.java
>
> I was thinking to add a strict one too, but AFAIK strict limiters tends to
> over-compensate the controlled systems, killing its latencies/throughput
> when it is not be supposed to be done...just my 2 cents.
>
> wdyt?
>
> Franz
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis-IOPS-Limiter-strategy-tp4725875.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>
--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] Artemis IOPS Limiter strategy

nigro_franz
You're right, but AFAIK flow control mechanisms can react only on already happened events: they cannot argue what could be the cause and avoid/reducing it before it will happen.
It is only a guess by my side...am I wrong?

AFAIK the majority of journals/WALs doesn't have IOPS limiters (AFAIK only for random access writes): strictly speaking seems not a "must have" for  the common case.

But consider VM-like (eg Amazon machines) environments with users that pay a certain amount of IOPS and get slowed down when exceeded or when is best to have generally slower (throughput) systems but with low variance in latency side.
A specialized IOPS limiter can address these cases, trying to compensate too many IOPS in the meantime, before the disk will do it for you in a worst manner....makes more sense?

We need hardware engineers specialized in storage solutions here to ask to!!!!!!! :P
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] Artemis IOPS Limiter strategy

clebertsuconic
In reply to this post by nigro_franz
The TimedBuffer actually had this functionality. I didn't realize this
semantic change on the timed waits. I'm strongly thinking about
reverting to the previous functionality.

This could even be the case of -1 the current releases, as this will
just keep writing bursts beyond the disk capacity.

That's why we calculate the IOPs the disk can perform when we create a
journal, but the TimedBuffer is just ignoring such setting after your
changes.

On Wed, May 10, 2017 at 4:30 AM, nigro_franz <[hidden email]> wrote:

> Hi guys!
>
> After the changes to TimedBuffer to make the Journal capable to adapt to the
> current load, I've had instructive feedbacks/advices about IOPS limits that
> could be reached on too high and long burst (depending on disk capacity).
>
> There is the need to separate cleanly the concerns related to disk capacity
> and load requested of the system: a configurable IOLimiter could help on it.
>
> A first attemp to do it is in this branch:
> https://github.com/franz1981/activemq-artemis/blob/iops_limiter
>
> The possible configuration of such IOPS limiter could be:
>
> 1) no limiter (default): the disk naturally create backpressure on too high
> loads
> 2) balanced: "best effort" IO limiter based on IOPS computed by the broker
> tuning tools (ie SyncCalculation)
> https://github.com/franz1981/activemq-artemis/blob/iops_limiter/artemis-journal/src/main/java/org/apache/activemq/artemis/core/io/buffer/BestEffortIOLimiter.java
>
> I was thinking to add a strict one too, but AFAIK strict limiters tends to
> over-compensate the controlled systems, killing its latencies/throughput
> when it is not be supposed to be done...just my 2 cents.
>
> wdyt?
>
> Franz
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis-IOPS-Limiter-strategy-tp4725875.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] Artemis IOPS Limiter strategy

clebertsuconic
I actually don't get your concern... looking at the TimedBuffer (after
your changes), there still a check about the timeout:

final boolean checkpoint = System.nanoTime() > lastFlushTime + timeout;
if (checkpoint || currentPendingSyncs >= estimatedOptimalBatch) {


That is, we won't flush the write unless the timeout is achieved...



that means the timeout calculated during the server.create (which is
using SyncCalculation) is still used... And this *should* work as the
IO limiter.



What am I missing here?

On Wed, May 10, 2017 at 1:13 PM, Clebert Suconic
<[hidden email]> wrote:

> The TimedBuffer actually had this functionality. I didn't realize this
> semantic change on the timed waits. I'm strongly thinking about
> reverting to the previous functionality.
>
> This could even be the case of -1 the current releases, as this will
> just keep writing bursts beyond the disk capacity.
>
> That's why we calculate the IOPs the disk can perform when we create a
> journal, but the TimedBuffer is just ignoring such setting after your
> changes.
>
> On Wed, May 10, 2017 at 4:30 AM, nigro_franz <[hidden email]> wrote:
>> Hi guys!
>>
>> After the changes to TimedBuffer to make the Journal capable to adapt to the
>> current load, I've had instructive feedbacks/advices about IOPS limits that
>> could be reached on too high and long burst (depending on disk capacity).
>>
>> There is the need to separate cleanly the concerns related to disk capacity
>> and load requested of the system: a configurable IOLimiter could help on it.
>>
>> A first attemp to do it is in this branch:
>> https://github.com/franz1981/activemq-artemis/blob/iops_limiter
>>
>> The possible configuration of such IOPS limiter could be:
>>
>> 1) no limiter (default): the disk naturally create backpressure on too high
>> loads
>> 2) balanced: "best effort" IO limiter based on IOPS computed by the broker
>> tuning tools (ie SyncCalculation)
>> https://github.com/franz1981/activemq-artemis/blob/iops_limiter/artemis-journal/src/main/java/org/apache/activemq/artemis/core/io/buffer/BestEffortIOLimiter.java
>>
>> I was thinking to add a strict one too, but AFAIK strict limiters tends to
>> over-compensate the controlled systems, killing its latencies/throughput
>> when it is not be supposed to be done...just my 2 cents.
>>
>> wdyt?
>>
>> Franz
>>
>>
>>
>> --
>> View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis-IOPS-Limiter-strategy-tp4725875.html
>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>
>
>
> --
> Clebert Suconic



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] Artemis IOPS Limiter strategy

nigro_franz
Hi Clebert!

Before reverting anything I think could be good to perform some tests to be sure of regressions, AFAIK the are only performance improving, but the final word I think will go to proper load tests.

Instead, I've already done sanity tests trying to push IOPS over the limits with the 2 TimedBuffer implementations (old and new one) and both suffer from the same issue, simply in different ways, but TBH, as I've said before, is not a big issue: it's more a possible chance of improvement from what we have now.

The test I'm talking is in this branch (using the original TimedBuffer): https://github.com/franz1981/activemq-artemis/tree/timed_buffer_iops_test
Is the shouldNotFlushUntilTimeout test, but I can build another couples of tests that do something similar and producing the same effect.

What the test verify is straightforward: a TimedBuffer can't accept 2 flushes to happen before the timeout expiration, because this would force a disk to perform too many IOPS than expected...and the test fails for both the TimedBuffer.

As I've said I propose to wait proper load tests before reverting it, considering that the test shows that the original version was suffering the same issue.
What do you think?

Franz


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] Artemis IOPS Limiter strategy

clebertsuconic
that's fair...

Panic is over.. when I first read your message I thought you had
completely changed semantics.. and it's not that deep.



On Wed, May 10, 2017 at 4:08 PM, nigro_franz <[hidden email]> wrote:

> Hi Clebert!
>
> Before reverting anything I think could be good to perform some tests to be
> sure of regressions, AFAIK the are only performance improving, but the final
> word I think will go to proper load tests.
>
> Instead, I've already done sanity tests trying to push IOPS over the limits
> with the 2 TimedBuffer implementations (old and new one) and both suffer
> from the same issue, simply in different ways, but TBH, as I've said before,
> is not a big issue: it's more a possible chance of improvement from what we
> have now.
>
> The test I'm talking is in this branch (using the original TimedBuffer):
> https://github.com/franz1981/activemq-artemis/tree/timed_buffer_iops_test
> Is the  shouldNotFlushUntilTimeout test
> <https://github.com/franz1981/activemq-artemis/blob/12c34475f16a73d669c3307999e6fa03601c6cb3/tests/unit-tests/src/test/java/org/apache/activemq/artemis/tests/unit/core/journal/impl/TimedBufferTest.java#L137-L137>
> , but I can build another couples of tests that do something similar and
> producing the same effect.
>
> What the test verify is straightforward: a TimedBuffer can't accept 2
> flushes to happen before the timeout expiration, because this would force a
> disk to perform too many IOPS than expected...and the test fails for both
> the TimedBuffer.
>
> As I've said I propose to wait proper load tests before reverting it,
> considering that the test shows that the original version was suffering the
> same issue.
> What do you think?
>
> Franz
>
>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis-IOPS-Limiter-strategy-tp4725875p4725924.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] Artemis IOPS Limiter strategy

nigro_franz
happy to hear that..I've had the same panic ahahah
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] Artemis IOPS Limiter strategy

nigro_franz
I've put the first attempt of relaxed/best effort IOPS limiter that works both for ASYNCIO and NIO too.

(The second commit on https://github.com/franz1981/activemq-artemis/tree/iops_limiter)

The 2 main differences with the original one embedded in the TimedBuffer are:

1) it performs adapting compensation instead of upfront limitation, using the current IOPS perceived
2) it performs compensation against ALL flush operations like the forced ones due to a filled batch buffer, not handled in the original one

In practical terms it doesn't overcompensate when the system is not really under stress, but It doesn't let the disk run out of IOPS due to forced batch buffer flushes.

I've added in that experimental branch a real-time HdrHistogram instrumentation to record the distributions of:

- flushed batch size in bytes (to understand the effectiveness of the coalescing heuristic)
- "perceived" flush latency
- compensations done by the limiter

After some measurements, I've noticed that NIO and ASYNCIO behave very differently.
ASYNCIO with maxIO>1 performs async flushes (histograms can't lie), while NIO si always synchronous and blocking, making the compensations more effective.
Further investigations could will the best strategy to do it.

If you have any idea, curiosity and/or advices about it please share it :)

Franz
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] Artemis IOPS Limiter strategy

clebertsuconic
You could replace the TimedBuffer.sleep for this implementation of
yours. it has the same semantic...


For the implementation (Mapped) that's not using the TimedBuffer you
may use this directly.

On Fri, May 12, 2017 at 9:36 AM, nigro_franz <[hidden email]> wrote:

> I've put the first attempt of relaxed/best effort IOPS limiter that works
> both for ASYNCIO and NIO too.
>
> (The second commit on
> https://github.com/franz1981/activemq-artemis/tree/iops_limiter)
>
> The 2 main differences with the original one embedded in the TimedBuffer
> are:
>
> 1) it performs adapting compensation instead of upfront limitation, using
> the current IOPS perceived
> 2) it performs compensation against ALL flush operations like the forced
> ones due to a filled batch buffer, not handled in the original one
>
> In practical terms it doesn't overcompensate when the system is not really
> under stress, but It doesn't let the disk run out of IOPS due to forced
> batch buffer flushes.
>
> I've added in that experimental branch a real-time HdrHistogram
> instrumentation to record the distributions of:
>
> - flushed batch size in bytes (to understand the effectiveness of the
> coalescing heuristic)
> - "perceived" flush latency
> - compensations done by the limiter
>
> After some measurements, I've noticed that NIO and ASYNCIO behave very
> differently.
> ASYNCIO with maxIO>1 performs async flushes (histograms can't lie), while
> NIO si always synchronous and blocking, making the compensations more
> effective.
> Further investigations could will the best strategy to do it.
>
> If you have any idea, curiosity and/or advices about it please share it :)
>
> Franz
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis-IOPS-Limiter-strategy-tp4725875p4726037.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] Artemis IOPS Limiter strategy

clebertsuconic
although.. we currently configure the time an operation can be
performed on disk...

you will need to convert the NanoSeconds back to the IOPS you need..
or change your implementation to cope with what we configure now...



We could endup deprecating the previous configuration and use a new
attribute, and multiple the nanoseconds we currently configure to the
new property.

On Fri, May 12, 2017 at 10:20 AM, Clebert Suconic
<[hidden email]> wrote:

> You could replace the TimedBuffer.sleep for this implementation of
> yours. it has the same semantic...
>
>
> For the implementation (Mapped) that's not using the TimedBuffer you
> may use this directly.
>
> On Fri, May 12, 2017 at 9:36 AM, nigro_franz <[hidden email]> wrote:
>> I've put the first attempt of relaxed/best effort IOPS limiter that works
>> both for ASYNCIO and NIO too.
>>
>> (The second commit on
>> https://github.com/franz1981/activemq-artemis/tree/iops_limiter)
>>
>> The 2 main differences with the original one embedded in the TimedBuffer
>> are:
>>
>> 1) it performs adapting compensation instead of upfront limitation, using
>> the current IOPS perceived
>> 2) it performs compensation against ALL flush operations like the forced
>> ones due to a filled batch buffer, not handled in the original one
>>
>> In practical terms it doesn't overcompensate when the system is not really
>> under stress, but It doesn't let the disk run out of IOPS due to forced
>> batch buffer flushes.
>>
>> I've added in that experimental branch a real-time HdrHistogram
>> instrumentation to record the distributions of:
>>
>> - flushed batch size in bytes (to understand the effectiveness of the
>> coalescing heuristic)
>> - "perceived" flush latency
>> - compensations done by the limiter
>>
>> After some measurements, I've noticed that NIO and ASYNCIO behave very
>> differently.
>> ASYNCIO with maxIO>1 performs async flushes (histograms can't lie), while
>> NIO si always synchronous and blocking, making the compensations more
>> effective.
>> Further investigations could will the best strategy to do it.
>>
>> If you have any idea, curiosity and/or advices about it please share it :)
>>
>> Franz
>>
>>
>>
>>
>> --
>> View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis-IOPS-Limiter-strategy-tp4725875p4726037.html
>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>
>
>
> --
> Clebert Suconic



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] Artemis IOPS Limiter strategy

nigro_franz
I was thinking of a similar solution but I've discovered that couldn't work (in the old or the new TimedBuffer too), because of the TimedBuffer::checkSize method that could force a flush if the batch buffer if not big enough to receive new data, going IOPS.
Sadly TimedBuffer::checkSize is outside any timeout, but depends on the writers.

That's why I've implemented the "compensation" right after any flush, in order to work with forced flushes too:

https://github.com/franz1981/activemq-artemis/blob/4b831021dab3e0dd276f477e3ea665e11ab54d0e/artemis-journal/src/main/java/org/apache/activemq/artemis/core/io/buffer/TimedBuffer.java#L338

Doing it on TimedBuffer::flush all the flushes on disk will be compensated (half of the story: ASYNCIO is async so depends on libAIO partially!)
Regarding the IOPS computation I've built this, as you've suggested: https://github.com/franz1981/activemq-artemis/blob/4b831021dab3e0dd276f477e3ea665e11ab54d0e/artemis-journal/src/main/java/org/apache/activemq/artemis/core/io/buffer/TimedBuffer.java#L119

The performance seems pretty good, it compensates well but it is faster than the original version, limiting IOPS too!

Thanks,
Franz
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] Artemis IOPS Limiter strategy

Michael André Pearce
Do you have a before and after high load end 2 end test? Where disk io is maxed out.

Reason I ask is taking 2.0.0 (amq ga) we have a much more steady latency under load.

Where as 2.1.0 we see very periodic spikes. (We saw this prior to our autoboxing change btw)



Sent from my iPhone

> On 12 May 2017, at 15:48, nigro_franz <[hidden email]> wrote:
>
> I was thinking of a similar solution but I've discovered that couldn't work
> (in the old or the new TimedBuffer too), because of the
> TimedBuffer::checkSize method that could force a flush if the batch buffer
> if not big enough to receive new data, going IOPS.
> Sadly TimedBuffer::checkSize is outside any timeout, but depends on the
> writers.
>
> That's why I've implemented the "compensation" right after any flush, in
> order to work with forced flushes too:
>
> https://github.com/franz1981/activemq-artemis/blob/4b831021dab3e0dd276f477e3ea665e11ab54d0e/artemis-journal/src/main/java/org/apache/activemq/artemis/core/io/buffer/TimedBuffer.java#L338
>
> Doing it on TimedBuffer::flush all the flushes on disk will be compensated
> (half of the story: ASYNCIO is async so depends on libAIO partially!)
> Regarding the IOPS computation I've built this, as you've suggested:
> https://github.com/franz1981/activemq-artemis/blob/4b831021dab3e0dd276f477e3ea665e11ab54d0e/artemis-journal/src/main/java/org/apache/activemq/artemis/core/io/buffer/TimedBuffer.java#L119
>
> The performance seems pretty good, it compensates well but it is faster than
> the original version, limiting IOPS too!
>
> Thanks,
> Franz
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis-IOPS-Limiter-strategy-tp4725875p4726057.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] Artemis IOPS Limiter strategy

clebertsuconic
In reply to this post by nigro_franz
There is a class we use on producer.  TokenLimiter.  Perhaps you could
reuse that one ?


On Fri, May 12, 2017 at 11:00 AM nigro_franz <[hidden email]> wrote:

> I was thinking of a similar solution but I've discovered that couldn't work
> (in the old or the new TimedBuffer too), because of the
> TimedBuffer::checkSize method that could force a flush if the batch buffer
> if not big enough to receive new data, going IOPS.
> Sadly TimedBuffer::checkSize is outside any timeout, but depends on the
> writers.
>
> That's why I've implemented the "compensation" right after any flush, in
> order to work with forced flushes too:
>
>
> https://github.com/franz1981/activemq-artemis/blob/4b831021dab3e0dd276f477e3ea665e11ab54d0e/artemis-journal/src/main/java/org/apache/activemq/artemis/core/io/buffer/TimedBuffer.java#L338
>
> Doing it on TimedBuffer::flush all the flushes on disk will be compensated
> (half of the story: ASYNCIO is async so depends on libAIO partially!)
> Regarding the IOPS computation I've built this, as you've suggested:
>
> https://github.com/franz1981/activemq-artemis/blob/4b831021dab3e0dd276f477e3ea665e11ab54d0e/artemis-journal/src/main/java/org/apache/activemq/artemis/core/io/buffer/TimedBuffer.java#L119
>
> The performance seems pretty good, it compensates well but it is faster
> than
> the original version, limiting IOPS too!
>
> Thanks,
> Franz
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis-IOPS-Limiter-strategy-tp4725875p4726057.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>
--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] Artemis IOPS Limiter strategy

Michael André Pearce
As it seems I can't send images to mail list, just sent to you both via email. Some graphs we have comparing versions.

Not sure what changes might cause it.

> On 12 May 2017, at 17:37, Clebert Suconic <[hidden email]> wrote:
>
> There is a class we use on producer.  TokenLimiter.  Perhaps you could
> reuse that one ?
>
>
>> On Fri, May 12, 2017 at 11:00 AM nigro_franz <[hidden email]> wrote:
>>
>> I was thinking of a similar solution but I've discovered that couldn't work
>> (in the old or the new TimedBuffer too), because of the
>> TimedBuffer::checkSize method that could force a flush if the batch buffer
>> if not big enough to receive new data, going IOPS.
>> Sadly TimedBuffer::checkSize is outside any timeout, but depends on the
>> writers.
>>
>> That's why I've implemented the "compensation" right after any flush, in
>> order to work with forced flushes too:
>>
>>
>> https://github.com/franz1981/activemq-artemis/blob/4b831021dab3e0dd276f477e3ea665e11ab54d0e/artemis-journal/src/main/java/org/apache/activemq/artemis/core/io/buffer/TimedBuffer.java#L338
>>
>> Doing it on TimedBuffer::flush all the flushes on disk will be compensated
>> (half of the story: ASYNCIO is async so depends on libAIO partially!)
>> Regarding the IOPS computation I've built this, as you've suggested:
>>
>> https://github.com/franz1981/activemq-artemis/blob/4b831021dab3e0dd276f477e3ea665e11ab54d0e/artemis-journal/src/main/java/org/apache/activemq/artemis/core/io/buffer/TimedBuffer.java#L119
>>
>> The performance seems pretty good, it compensates well but it is faster
>> than
>> the original version, limiting IOPS too!
>>
>> Thanks,
>> Franz
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis-IOPS-Limiter-strategy-tp4725875p4726057.html
>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>>
> --
> Clebert Suconic
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] Artemis IOPS Limiter strategy

clebertsuconic
I'm considering only keeping the pooled buffer part and switch back to the
ole sleep or an improved sleep we had.



On Fri, May 12, 2017 at 12:49 PM Michael André Pearce <
[hidden email]> wrote:

> As it seems I can't send images to mail list, just sent to you both via
> email. Some graphs we have comparing versions.
>
> Not sure what changes might cause it.
>
> > On 12 May 2017, at 17:37, Clebert Suconic <[hidden email]>
> wrote:
> >
> > There is a class we use on producer.  TokenLimiter.  Perhaps you could
> > reuse that one ?
> >
> >
> >> On Fri, May 12, 2017 at 11:00 AM nigro_franz <[hidden email]>
> wrote:
> >>
> >> I was thinking of a similar solution but I've discovered that couldn't
> work
> >> (in the old or the new TimedBuffer too), because of the
> >> TimedBuffer::checkSize method that could force a flush if the batch
> buffer
> >> if not big enough to receive new data, going IOPS.
> >> Sadly TimedBuffer::checkSize is outside any timeout, but depends on the
> >> writers.
> >>
> >> That's why I've implemented the "compensation" right after any flush, in
> >> order to work with forced flushes too:
> >>
> >>
> >>
> https://github.com/franz1981/activemq-artemis/blob/4b831021dab3e0dd276f477e3ea665e11ab54d0e/artemis-journal/src/main/java/org/apache/activemq/artemis/core/io/buffer/TimedBuffer.java#L338
> >>
> >> Doing it on TimedBuffer::flush all the flushes on disk will be
> compensated
> >> (half of the story: ASYNCIO is async so depends on libAIO partially!)
> >> Regarding the IOPS computation I've built this, as you've suggested:
> >>
> >>
> https://github.com/franz1981/activemq-artemis/blob/4b831021dab3e0dd276f477e3ea665e11ab54d0e/artemis-journal/src/main/java/org/apache/activemq/artemis/core/io/buffer/TimedBuffer.java#L119
> >>
> >> The performance seems pretty good, it compensates well but it is faster
> >> than
> >> the original version, limiting IOPS too!
> >>
> >> Thanks,
> >> Franz
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis-IOPS-Limiter-strategy-tp4725875p4726057.html
> >> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
> >>
> > --
> > Clebert Suconic
>
--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] Artemis IOPS Limiter strategy

clebertsuconic
As the only thing that could affect this is the Change on timed buffer.
Afaik


On Fri, May 12, 2017 at 12:57 PM Clebert Suconic <[hidden email]>
wrote:

> I'm considering only keeping the pooled buffer part and switch back to the
> ole sleep or an improved sleep we had.
>
>
>
> On Fri, May 12, 2017 at 12:49 PM Michael André Pearce <
> [hidden email]> wrote:
>
>> As it seems  I can't send images to mail list, just sent to  you both via
>> email. Some graphs we have comparing versions.
>>
>> Not sure what changes might cause it.
>>
>> > On 12 May 2017, at 17:37, Clebert Suconic <[hidden email]>
>> wrote:
>> >
>> > There is a class we use on producer.  TokenLimiter.  Perhaps you could
>> > reuse that one ?
>> >
>> >
>> >> On Fri, May 12, 2017 at 11:00 AM nigro_franz <[hidden email]>
>> wrote:
>> >>
>> >> I was thinking of a similar solution but I've discovered that couldn't
>> work
>> >> (in the old or the new TimedBuffer too), because of the
>> >> TimedBuffer::checkSize method that could force a flush if the batch
>> buffer
>> >> if not big enough to receive new data, going IOPS.
>> >> Sadly TimedBuffer::checkSize is outside any timeout, but depends on the
>> >> writers.
>> >>
>> >> That's why I've implemented the "compensation" right after any flush,
>> in
>> >> order to work with forced flushes too:
>> >>
>> >>
>> >>
>> https://github.com/franz1981/activemq-artemis/blob/4b831021dab3e0dd276f477e3ea665e11ab54d0e/artemis-journal/src/main/java/org/apache/activemq/artemis/core/io/buffer/TimedBuffer.java#L338
>> >>
>> >> Doing it on TimedBuffer::flush all the flushes on disk will be
>> compensated
>> >> (half of the story: ASYNCIO is async so depends on libAIO partially!)
>> >> Regarding the IOPS computation I've built this, as you've suggested:
>> >>
>> >>
>> https://github.com/franz1981/activemq-artemis/blob/4b831021dab3e0dd276f477e3ea665e11ab54d0e/artemis-journal/src/main/java/org/apache/activemq/artemis/core/io/buffer/TimedBuffer.java#L119
>> >>
>> >> The performance seems pretty good, it compensates well but it is faster
>> >> than
>> >> the original version, limiting IOPS too!
>> >>
>> >> Thanks,
>> >> Franz
>> >>
>> >>
>> >>
>> >> --
>> >> View this message in context:
>> >>
>> http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis-IOPS-Limiter-strategy-tp4725875p4726057.html
>> >> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>> >>
>> > --
>> > Clebert Suconic
>>
> --
> Clebert Suconic
>
--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] Artemis IOPS Limiter strategy

jbertram
In reply to this post by Michael André Pearce
> As it seems I can't send images to mail list...

FWIW, you can add images to gist [1] and then send links to the list.  I just tried it myself and it works as advertised.


Justin

[1] https://gist.github.com/remarkablemark/feff40b0a522f0c41c4eff0b77ea1d47

----- Original Message -----
From: "Michael André Pearce" <[hidden email]>
To: [hidden email]
Sent: Friday, May 12, 2017 11:48:50 AM
Subject: Re: [DISCUSS] Artemis IOPS Limiter strategy

As it seems I can't send images to mail list, just sent to you both via email. Some graphs we have comparing versions.

Not sure what changes might cause it.

> On 12 May 2017, at 17:37, Clebert Suconic <[hidden email]> wrote:
>
> There is a class we use on producer.  TokenLimiter.  Perhaps you could
> reuse that one ?
>
>
>> On Fri, May 12, 2017 at 11:00 AM nigro_franz <[hidden email]> wrote:
>>
>> I was thinking of a similar solution but I've discovered that couldn't work
>> (in the old or the new TimedBuffer too), because of the
>> TimedBuffer::checkSize method that could force a flush if the batch buffer
>> if not big enough to receive new data, going IOPS.
>> Sadly TimedBuffer::checkSize is outside any timeout, but depends on the
>> writers.
>>
>> That's why I've implemented the "compensation" right after any flush, in
>> order to work with forced flushes too:
>>
>>
>> https://github.com/franz1981/activemq-artemis/blob/4b831021dab3e0dd276f477e3ea665e11ab54d0e/artemis-journal/src/main/java/org/apache/activemq/artemis/core/io/buffer/TimedBuffer.java#L338
>>
>> Doing it on TimedBuffer::flush all the flushes on disk will be compensated
>> (half of the story: ASYNCIO is async so depends on libAIO partially!)
>> Regarding the IOPS computation I've built this, as you've suggested:
>>
>> https://github.com/franz1981/activemq-artemis/blob/4b831021dab3e0dd276f477e3ea665e11ab54d0e/artemis-journal/src/main/java/org/apache/activemq/artemis/core/io/buffer/TimedBuffer.java#L119
>>
>> The performance seems pretty good, it compensates well but it is faster
>> than
>> the original version, limiting IOPS too!
>>
>> Thanks,
>> Franz
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis-IOPS-Limiter-strategy-tp4725875p4726057.html
>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>>
> --
> Clebert Suconic
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] Artemis IOPS Limiter strategy

Michael André Pearce
In reply to this post by clebertsuconic
I've left for the day now. If I get time over the weekend I'll try see if I can make a build of 2.1.0 without that change and see if it makes any difference.

Can someone point me to the PR for that change, so I know what I'm unpicking locally?

Sent from my iPhone

> On 12 May 2017, at 17:58, Clebert Suconic <[hidden email]> wrote:
>
> As the only thing that could affect this is the Change on timed buffer.
> Afaik
>
>
> On Fri, May 12, 2017 at 12:57 PM Clebert Suconic <[hidden email]>
> wrote:
>
>> I'm considering only keeping the pooled buffer part and switch back to the
>> ole sleep or an improved sleep we had.
>>
>>
>>
>> On Fri, May 12, 2017 at 12:49 PM Michael André Pearce <
>> [hidden email]> wrote:
>>
>>> As it seems  I can't send images to mail list, just sent to  you both via
>>> email. Some graphs we have comparing versions.
>>>
>>> Not sure what changes might cause it.
>>>
>>>> On 12 May 2017, at 17:37, Clebert Suconic <[hidden email]>
>>> wrote:
>>>>
>>>> There is a class we use on producer.  TokenLimiter.  Perhaps you could
>>>> reuse that one ?
>>>>
>>>>
>>>>> On Fri, May 12, 2017 at 11:00 AM nigro_franz <[hidden email]>
>>> wrote:
>>>>>
>>>>> I was thinking of a similar solution but I've discovered that couldn't
>>> work
>>>>> (in the old or the new TimedBuffer too), because of the
>>>>> TimedBuffer::checkSize method that could force a flush if the batch
>>> buffer
>>>>> if not big enough to receive new data, going IOPS.
>>>>> Sadly TimedBuffer::checkSize is outside any timeout, but depends on the
>>>>> writers.
>>>>>
>>>>> That's why I've implemented the "compensation" right after any flush,
>>> in
>>>>> order to work with forced flushes too:
>>>>>
>>>>>
>>>>>
>>> https://github.com/franz1981/activemq-artemis/blob/4b831021dab3e0dd276f477e3ea665e11ab54d0e/artemis-journal/src/main/java/org/apache/activemq/artemis/core/io/buffer/TimedBuffer.java#L338
>>>>>
>>>>> Doing it on TimedBuffer::flush all the flushes on disk will be
>>> compensated
>>>>> (half of the story: ASYNCIO is async so depends on libAIO partially!)
>>>>> Regarding the IOPS computation I've built this, as you've suggested:
>>>>>
>>>>>
>>> https://github.com/franz1981/activemq-artemis/blob/4b831021dab3e0dd276f477e3ea665e11ab54d0e/artemis-journal/src/main/java/org/apache/activemq/artemis/core/io/buffer/TimedBuffer.java#L119
>>>>>
>>>>> The performance seems pretty good, it compensates well but it is faster
>>>>> than
>>>>> the original version, limiting IOPS too!
>>>>>
>>>>> Thanks,
>>>>> Franz
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>>
>>> http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis-IOPS-Limiter-strategy-tp4725875p4726057.html
>>>>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>>>>>
>>>> --
>>>> Clebert Suconic
>>>
>> --
>> Clebert Suconic
>>
> --
> Clebert Suconic
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS] Artemis IOPS Limiter strategy

clebertsuconic
@Michael: this is the PR:

https://github.com/apache/activemq-artemis/pull/1256


This is the commit:
https://github.com/apache/activemq-artemis/commit/21c9ed85cf6b9a53debdd32747bd42b2e733da80



Notice that the commit ID from the PR may change after merge, as we
rebase before merging it.

On Fri, May 12, 2017 at 1:10 PM, Michael André Pearce
<[hidden email]> wrote:

> I've left for the day now. If I get time over the weekend I'll try see if I can make a build of 2.1.0 without that change and see if it makes any difference.
>
> Can someone point me to the PR for that change, so I know what I'm unpicking locally?
>
> Sent from my iPhone
>
>> On 12 May 2017, at 17:58, Clebert Suconic <[hidden email]> wrote:
>>
>> As the only thing that could affect this is the Change on timed buffer.
>> Afaik
>>
>>
>> On Fri, May 12, 2017 at 12:57 PM Clebert Suconic <[hidden email]>
>> wrote:
>>
>>> I'm considering only keeping the pooled buffer part and switch back to the
>>> ole sleep or an improved sleep we had.
>>>
>>>
>>>
>>> On Fri, May 12, 2017 at 12:49 PM Michael André Pearce <
>>> [hidden email]> wrote:
>>>
>>>> As it seems  I can't send images to mail list, just sent to  you both via
>>>> email. Some graphs we have comparing versions.
>>>>
>>>> Not sure what changes might cause it.
>>>>
>>>>> On 12 May 2017, at 17:37, Clebert Suconic <[hidden email]>
>>>> wrote:
>>>>>
>>>>> There is a class we use on producer.  TokenLimiter.  Perhaps you could
>>>>> reuse that one ?
>>>>>
>>>>>
>>>>>> On Fri, May 12, 2017 at 11:00 AM nigro_franz <[hidden email]>
>>>> wrote:
>>>>>>
>>>>>> I was thinking of a similar solution but I've discovered that couldn't
>>>> work
>>>>>> (in the old or the new TimedBuffer too), because of the
>>>>>> TimedBuffer::checkSize method that could force a flush if the batch
>>>> buffer
>>>>>> if not big enough to receive new data, going IOPS.
>>>>>> Sadly TimedBuffer::checkSize is outside any timeout, but depends on the
>>>>>> writers.
>>>>>>
>>>>>> That's why I've implemented the "compensation" right after any flush,
>>>> in
>>>>>> order to work with forced flushes too:
>>>>>>
>>>>>>
>>>>>>
>>>> https://github.com/franz1981/activemq-artemis/blob/4b831021dab3e0dd276f477e3ea665e11ab54d0e/artemis-journal/src/main/java/org/apache/activemq/artemis/core/io/buffer/TimedBuffer.java#L338
>>>>>>
>>>>>> Doing it on TimedBuffer::flush all the flushes on disk will be
>>>> compensated
>>>>>> (half of the story: ASYNCIO is async so depends on libAIO partially!)
>>>>>> Regarding the IOPS computation I've built this, as you've suggested:
>>>>>>
>>>>>>
>>>> https://github.com/franz1981/activemq-artemis/blob/4b831021dab3e0dd276f477e3ea665e11ab54d0e/artemis-journal/src/main/java/org/apache/activemq/artemis/core/io/buffer/TimedBuffer.java#L119
>>>>>>
>>>>>> The performance seems pretty good, it compensates well but it is faster
>>>>>> than
>>>>>> the original version, limiting IOPS too!
>>>>>>
>>>>>> Thanks,
>>>>>> Franz
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>>
>>>> http://activemq.2283324.n4.nabble.com/DISCUSS-Artemis-IOPS-Limiter-strategy-tp4725875p4726057.html
>>>>>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>>>>>>
>>>>> --
>>>>> Clebert Suconic
>>>>
>>> --
>>> Clebert Suconic
>>>
>> --
>> Clebert Suconic



--
Clebert Suconic
123
Loading...