CPU core/thread scaling test

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|

CPU core/thread scaling test

jgoodyear
Hi All,

It was mentioned on another thread that ActiveMQ has hard challenges
with CPU core/thread scaling - I was wondering if there was a test
case/script published some where that shows this issue occurring?

Cheers,
Jamie
Reply | Threaded
Open this post in threaded view
|

Re: CPU core/thread scaling test

rajdavies
Hi Jamie,

you could look at the unit tests in the perf package: https://git-wip-us.apache.org/repos/asf?p=activemq.git;a=tree;f=activemq-unit-tests/src/test/java/org/apache/activemq/perf;h=4562127ff13ba3a10e1cadee9b81a899b6c66a9e;hb=refs/heads/activemq-5.11.x

These run clients and broker in the same VM - so you might want to break out the client bits and just run them against a stand-alone  ActiveMQ broker.

27 March 2015 10:14
Hi All,

It was mentioned on another thread that ActiveMQ has hard challenges
with CPU core/thread scaling - I was wondering if there was a test
case/script published some where that shows this issue occurring?

Cheers,
Jamie
Reply | Threaded
Open this post in threaded view
|

Re: CPU core/thread scaling test

jgoodyear
Perfect, is one of these in particular the one to review to see the CPU core/thread limitation issue?

Thanks,
Jamie

On Fri, Mar 27, 2015 at 7:59 AM, Rob Davies <[hidden email]> wrote:
Hi Jamie,

you could look at the unit tests in the perf package: https://git-wip-us.apache.org/repos/asf?p=activemq.git;a=tree;f=activemq-unit-tests/src/test/java/org/apache/activemq/perf;h=4562127ff13ba3a10e1cadee9b81a899b6c66a9e;hb=refs/heads/activemq-5.11.x

These run clients and broker in the same VM - so you might want to break out the client bits and just run them against a stand-alone  ActiveMQ broker.

27 March 2015 10:14
Hi All,

It was mentioned on another thread that ActiveMQ has hard challenges
with CPU core/thread scaling - I was wondering if there was a test
case/script published some where that shows this issue occurring?

Cheers,
Jamie

Reply | Threaded
Open this post in threaded view
|

Re: CPU core/thread scaling test

rajdavies
I'd look at the non-persistent Topic tests first.

27 March 2015 10:33
Perfect, is one of these in particular the one to review to see the CPU core/thread limitation issue?

Thanks,
Jamie


27 March 2015 10:29
Hi Jamie,

you could look at the unit tests in the perf package: https://git-wip-us.apache.org/repos/asf?p=activemq.git;a=tree;f=activemq-unit-tests/src/test/java/org/apache/activemq/perf;h=4562127ff13ba3a10e1cadee9b81a899b6c66a9e;hb=refs/heads/activemq-5.11.x

These run clients and broker in the same VM - so you might want to break out the client bits and just run them against a stand-alone  ActiveMQ broker.

27 March 2015 10:14
Hi All,

It was mentioned on another thread that ActiveMQ has hard challenges
with CPU core/thread scaling - I was wondering if there was a test
case/script published some where that shows this issue occurring?

Cheers,
Jamie
Reply | Threaded
Open this post in threaded view
|

Re: CPU core/thread scaling test

gtully
In reply to this post by jgoodyear
you can also use the producer/consumer examples from a distro.
Last time I was trying to saturate a network I uses some of the scripts at
https://github.com/gtully/broker-run/blob/master/scripts/clients.sh

I have not been in there for a while but you may find it useful to run
parallel load over multiple destinations.

On 27 March 2015 at 10:14, Jamie G. <[hidden email]> wrote:
> Hi All,
>
> It was mentioned on another thread that ActiveMQ has hard challenges
> with CPU core/thread scaling - I was wondering if there was a test
> case/script published some where that shows this issue occurring?
>
> Cheers,
> Jamie
Reply | Threaded
Open this post in threaded view
|

Re: CPU core/thread scaling test

jgoodyear
Thank you for the links - I'll try setting this up on my test lab.

Cheers,
Jamie

On Fri, Mar 27, 2015 at 9:59 AM, Gary Tully <[hidden email]> wrote:

> you can also use the producer/consumer examples from a distro.
> Last time I was trying to saturate a network I uses some of the scripts at
> https://github.com/gtully/broker-run/blob/master/scripts/clients.sh
>
> I have not been in there for a while but you may find it useful to run
> parallel load over multiple destinations.
>
> On 27 March 2015 at 10:14, Jamie G. <[hidden email]> wrote:
>> Hi All,
>>
>> It was mentioned on another thread that ActiveMQ has hard challenges
>> with CPU core/thread scaling - I was wondering if there was a test
>> case/script published some where that shows this issue occurring?
>>
>> Cheers,
>> Jamie
Reply | Threaded
Open this post in threaded view
|

Re: CPU core/thread scaling test

James Carman
Is the expected behavior that even under high load AMQ would only be
using one core/CPU, or less than it could/should be in some way?

On Fri, Mar 27, 2015 at 8:37 AM, Jamie G. <[hidden email]> wrote:

> Thank you for the links - I'll try setting this up on my test lab.
>
> Cheers,
> Jamie
>
> On Fri, Mar 27, 2015 at 9:59 AM, Gary Tully <[hidden email]> wrote:
>> you can also use the producer/consumer examples from a distro.
>> Last time I was trying to saturate a network I uses some of the scripts at
>> https://github.com/gtully/broker-run/blob/master/scripts/clients.sh
>>
>> I have not been in there for a while but you may find it useful to run
>> parallel load over multiple destinations.
>>
>> On 27 March 2015 at 10:14, Jamie G. <[hidden email]> wrote:
>>> Hi All,
>>>
>>> It was mentioned on another thread that ActiveMQ has hard challenges
>>> with CPU core/thread scaling - I was wondering if there was a test
>>> case/script published some where that shows this issue occurring?
>>>
>>> Cheers,
>>> Jamie
Reply | Threaded
Open this post in threaded view
|

Re: CPU core/thread scaling test

Jean-Baptiste Onofré
In reply to this post by jgoodyear
Good point Jamie.

I'm a bit surprised, and I worked with highly loaded ActiveMQ (on a two
machines) without seeing big issues in term of CPU and thread scaling
(of course, I did some tuning on kahadb and JVM/GC, but nothing major).

Regards
JB

On 03/27/2015 11:14 AM, Jamie G. wrote:
> Hi All,
>
> It was mentioned on another thread that ActiveMQ has hard challenges
> with CPU core/thread scaling - I was wondering if there was a test
> case/script published some where that shows this issue occurring?
>
> Cheers,
> Jamie
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: CPU core/thread scaling test

chirino
In reply to this post by James Carman
What ideally should happen is that under high load the network should
be the bottleneck.  But right now CPU is the bottleneck.

On Fri, Mar 27, 2015 at 8:53 AM, James Carman
<[hidden email]> wrote:

> Is the expected behavior that even under high load AMQ would only be
> using one core/CPU, or less than it could/should be in some way?
>
> On Fri, Mar 27, 2015 at 8:37 AM, Jamie G. <[hidden email]> wrote:
>> Thank you for the links - I'll try setting this up on my test lab.
>>
>> Cheers,
>> Jamie
>>
>> On Fri, Mar 27, 2015 at 9:59 AM, Gary Tully <[hidden email]> wrote:
>>> you can also use the producer/consumer examples from a distro.
>>> Last time I was trying to saturate a network I uses some of the scripts at
>>> https://github.com/gtully/broker-run/blob/master/scripts/clients.sh
>>>
>>> I have not been in there for a while but you may find it useful to run
>>> parallel load over multiple destinations.
>>>
>>> On 27 March 2015 at 10:14, Jamie G. <[hidden email]> wrote:
>>>> Hi All,
>>>>
>>>> It was mentioned on another thread that ActiveMQ has hard challenges
>>>> with CPU core/thread scaling - I was wondering if there was a test
>>>> case/script published some where that shows this issue occurring?
>>>>
>>>> Cheers,
>>>> Jamie



--
Hiram Chirino
Engineering | Red Hat, Inc.
[hidden email] | fusesource.com | redhat.com
skype: hiramchirino | twitter: @hiramchirino
Reply | Threaded
Open this post in threaded view
|

Re: CPU core/thread scaling test

jgoodyear
Hmm, I'd want to look a bit closer at the network stack -- I've
observed on other projects NICs off loading heavily onto the CPU at
higher push rates (see Freed-up CPU cycles
http://en.wikipedia.org/wiki/TCP_offload_engine).

-Jamie

On Fri, Mar 27, 2015 at 10:41 AM, Hiram Chirino <[hidden email]> wrote:

> What ideally should happen is that under high load the network should
> be the bottleneck.  But right now CPU is the bottleneck.
>
> On Fri, Mar 27, 2015 at 8:53 AM, James Carman
> <[hidden email]> wrote:
>> Is the expected behavior that even under high load AMQ would only be
>> using one core/CPU, or less than it could/should be in some way?
>>
>> On Fri, Mar 27, 2015 at 8:37 AM, Jamie G. <[hidden email]> wrote:
>>> Thank you for the links - I'll try setting this up on my test lab.
>>>
>>> Cheers,
>>> Jamie
>>>
>>> On Fri, Mar 27, 2015 at 9:59 AM, Gary Tully <[hidden email]> wrote:
>>>> you can also use the producer/consumer examples from a distro.
>>>> Last time I was trying to saturate a network I uses some of the scripts at
>>>> https://github.com/gtully/broker-run/blob/master/scripts/clients.sh
>>>>
>>>> I have not been in there for a while but you may find it useful to run
>>>> parallel load over multiple destinations.
>>>>
>>>> On 27 March 2015 at 10:14, Jamie G. <[hidden email]> wrote:
>>>>> Hi All,
>>>>>
>>>>> It was mentioned on another thread that ActiveMQ has hard challenges
>>>>> with CPU core/thread scaling - I was wondering if there was a test
>>>>> case/script published some where that shows this issue occurring?
>>>>>
>>>>> Cheers,
>>>>> Jamie
>
>
>
> --
> Hiram Chirino
> Engineering | Red Hat, Inc.
> [hidden email] | fusesource.com | redhat.com
> skype: hiramchirino | twitter: @hiramchirino
Reply | Threaded
Open this post in threaded view
|

Re: CPU core/thread scaling test

James Carman
In reply to this post by chirino
Oh, I misunderstood.  Gotcha.  I was thinking the idea was that AMQ
wasn't doing a good job spreading itself across the available
cores/CPUs.  I'm picking up what you're putting down now.  Thanks,
Hiram.

On Fri, Mar 27, 2015 at 9:11 AM, Hiram Chirino <[hidden email]> wrote:

> What ideally should happen is that under high load the network should
> be the bottleneck.  But right now CPU is the bottleneck.
>
> On Fri, Mar 27, 2015 at 8:53 AM, James Carman
> <[hidden email]> wrote:
>> Is the expected behavior that even under high load AMQ would only be
>> using one core/CPU, or less than it could/should be in some way?
>>
>> On Fri, Mar 27, 2015 at 8:37 AM, Jamie G. <[hidden email]> wrote:
>>> Thank you for the links - I'll try setting this up on my test lab.
>>>
>>> Cheers,
>>> Jamie
>>>
>>> On Fri, Mar 27, 2015 at 9:59 AM, Gary Tully <[hidden email]> wrote:
>>>> you can also use the producer/consumer examples from a distro.
>>>> Last time I was trying to saturate a network I uses some of the scripts at
>>>> https://github.com/gtully/broker-run/blob/master/scripts/clients.sh
>>>>
>>>> I have not been in there for a while but you may find it useful to run
>>>> parallel load over multiple destinations.
>>>>
>>>> On 27 March 2015 at 10:14, Jamie G. <[hidden email]> wrote:
>>>>> Hi All,
>>>>>
>>>>> It was mentioned on another thread that ActiveMQ has hard challenges
>>>>> with CPU core/thread scaling - I was wondering if there was a test
>>>>> case/script published some where that shows this issue occurring?
>>>>>
>>>>> Cheers,
>>>>> Jamie
>
>
>
> --
> Hiram Chirino
> Engineering | Red Hat, Inc.
> [hidden email] | fusesource.com | redhat.com
> skype: hiramchirino | twitter: @hiramchirino
Reply | Threaded
Open this post in threaded view
|

Re: CPU core/thread scaling test

rajdavies
In reply to this post by jgoodyear
I've compared ActiveMQ on the same network/machines against other Java brokers - and ActiveMQ always has a higher load on CPU. My suspicion is around  thread context switching in the broker - but a fresh set of eyes is greatly appreciated!

27 March 2015 13:20
Hmm, I'd want to look a bit closer at the network stack -- I've
observed on other projects NICs off loading heavily onto the CPU at
higher push rates (see Freed-up CPU cycles
http://en.wikipedia.org/wiki/TCP_offload_engine).

-Jamie
27 March 2015 12:37
Thank you for the links - I'll try setting this up on my test lab.

Cheers,
Jamie
27 March 2015 12:29
you can also use the producer/consumer examples from a distro.
Last time I was trying to saturate a network I uses some of the scripts at
https://github.com/gtully/broker-run/blob/master/scripts/clients.sh

I have not been in there for a while but you may find it useful to run
parallel load over multiple destinations.
27 March 2015 10:14
Hi All,

It was mentioned on another thread that ActiveMQ has hard challenges
with CPU core/thread scaling - I was wondering if there was a test
case/script published some where that shows this issue occurring?

Cheers,
Jamie
Reply | Threaded
Open this post in threaded view
|

Re: CPU core/thread scaling test

artnaseef
A higher load on CPU versus other brokers under the exact same load?

Are there any reports with details, such as the topology of brokers and ActiveMQ clients?

It would be great to have a baseline to start from, and some detail to review.  For example, not being able to saturate a network connection with a single client connection to ActiveMQ would not be a surprise.

Things I would be interested to know about these tests:
* Numbers of brokers and broker topology
* Numbers of ActiveMQ producers and consumers, and their topology wrt the ActiveMQ broker topology
* Types and sizes of messages
* ActiveMQ broker configuration
Reply | Threaded
Open this post in threaded view
|

Re: CPU core/thread scaling test

rajdavies
This was a while ago - but single broker, 20 producers, 20 consumers, 1k messages - different message config - including flow control on/off.
Actually thinking about it - would be worth using Hiram's performance tests he put together - see - https://github.com/chirino/jms-benchmark

27 March 2015 16:37
A higher load on CPU versus other brokers under the exact same load?

Are there any reports with details, such as the topology of brokers and
ActiveMQ clients?

It would be great to have a baseline to start from, and some detail to
review. For example, not being able to saturate a network connection with a
single client connection to ActiveMQ would not be a surprise.

Things I would be interested to know about these tests:
* Numbers of brokers and broker topology
* Numbers of ActiveMQ producers and consumers, and their topology wrt the
ActiveMQ broker topology
* Types and sizes of messages
* ActiveMQ broker configuration



--
View this message in context: http://activemq.2283324.n4.nabble.com/CPU-core-thread-scaling-test-tp4693913p4693951.html
Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
27 March 2015 14:18
I've compared ActiveMQ on the same network/machines against other Java brokers - and ActiveMQ always has a higher load on CPU. My suspicion is around  thread context switching in the broker - but a fresh set of eyes is greatly appreciated!

27 March 2015 13:20
Hmm, I'd want to look a bit closer at the network stack -- I've
observed on other projects NICs off loading heavily onto the CPU at
higher push rates (see Freed-up CPU cycles
http://en.wikipedia.org/wiki/TCP_offload_engine).

-Jamie
27 March 2015 13:11
What ideally should happen is that under high load the network should
be the bottleneck. But right now CPU is the bottleneck.

On Fri, Mar 27, 2015 at 8:53 AM, James Carman



27 March 2015 12:53
Is the expected behavior that even under high load AMQ would only be
using one core/CPU, or less than it could/should be in some way?
Reply | Threaded
Open this post in threaded view
|

Re: CPU core/thread scaling test

artnaseef
Oh, very nice.  I was thinking ActiveMQ would do well to have its own load tests from which baseline measurements could be taken.  Perhaps with a separate source code repository dedicated to the purpose?
Reply | Threaded
Open this post in threaded view
|

Re: CPU core/thread scaling test

ceposta
Yes but don't name the repository activemq-6 :)

*ducks head*



On Friday, March 27, 2015, artnaseef <[hidden email]> wrote:

> Oh, very nice.  I was thinking ActiveMQ would do well to have its own load
> tests from which baseline measurements could be taken.  Perhaps with a
> separate source code repository dedicated to the purpose?
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/CPU-core-thread-scaling-test-tp4693913p4694001.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>


--
*Christian Posta*
twitter: @christianposta
http://www.christianposta.com/blog
http://fabric8.io
Reply | Threaded
Open this post in threaded view
|

Re: CPU core/thread scaling test

artnaseef
That would be wise - both the naming, and the ducking!