NIO transport performance for ActiveMQ

[johnbing]

Hi,
I am using 5.8.0 version of ActiveMQ. I am trying to use NIO to improve performance and scalability of my ActiveMQ broker but it is not working well. I was expecting that it should reduce number of threads but I have not observed any changes in number of threads. It is even reducing the throughput.
I have tested in persistence and non-persistence mode with queues as well as topics.
 
I have made changes in broker's configuration file as specified http://activemq.apache.org/configuring-transports.html

Can somebody help me here.

Thanks,
John

[anujkhandelwal]

Hey,
I have also not seen any improvement in performance using NIO transport, it shows almost the same throughput. Number of threads are also not reducing.
If someone is using NIO, please reply.

Thanks,
Anuj

[ceposta]

Where are the bottlnecks in your system? What have you observed to be the
limiters of your current config (before switching to NIO)?


On Fri, Aug 30, 2013 at 3:06 AM, johnbing <[hidden email]> wrote:

> Hi,
> I am using 5.8.0 version of ActiveMQ. I am trying to use NIO to improve
> performance and scalability of my ActiveMQ broker but it is not working
> well. I was expecting that it should reduce number of threads but I have
> not
> observed any changes in number of threads. It is even reducing the
> throughput.
> I have tested in persistence and non-persistence mode with queues as well
> as
> topics.
>
> I have made changes in broker's configuration file as specified
> http://activemq.apache.org/configuring-transports.html
> <http://activemq.apache.org/configuring-transports.html>
>
> Can somebody help me here.
>
> Thanks,
> John
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/NIO-transport-performance-for-ActiveMQ-tp4670977.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

[johnbing]

Hey,

I am seeing the throughput  drop using NIO. With 1 producer and 1 consumer in non-persistence mode using queues I got 47K msgs/sec using NIO while 53K msgs/sec using TCP. My major concern is number of threads, there is no decrement in number of threads. At least NIO supposed to reduce threads, it should not create one thread per connection.

Thanks,
John

[anujkhandelwal]

Exactly !! Throughput is not the major concern but NIO is supposed to decrease no. of threads because it is the main functionality of NIO. Can someone from ActiveMQ team help me with this.


Thanks,
Anuj

[ceposta]

Yes, using the NIO transport vs regular blocking TCP will reduce the number
of threads required per connection over large numbers of connections. I
don't think you'll see much benefit with just one or two connections.

If you have a case where you have lots of connections, and you don't see a
difference between using regular TCP blocking transport, then please post
the thread trace (pastebin).


On Wed, Sep 4, 2013 at 3:49 AM, anujkhandelwal <[hidden email]>wrote:

> Exactly !! Throughput is not the major concern but NIO is supposed to
> decrease no. of threads because it is the main functionality of NIO. Can
> someone from ActiveMQ team help me with this.
>
>
> Thanks,
> Anuj
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/NIO-transport-performance-for-ActiveMQ-tp4670977p4671148.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

[samdowning]

Hi,
I am using NIO transport in my project but the number of threads are not decreasing.
Can somebody tell in what scenarios this kind problem occurs?



Thanks,
Sam

[johnbing]

Yes I have also tested it with 20 producers consumers. But still not seeing any reduction in the number of threads. I have tested NIO with STOMP as well as JMS producers.

USING(Language)          No. of threads before running producer   No. of threads after running producers(~20)
STOMP (PYTHON producer)                        63                                                  72
STOMP +NIO(PYTHON producer)               67                                             82
TCP(JAVA producer)                                58                                                  86
NIO(JAVA producer)                                58                                                  87

Thanks,
Anuj

[ceposta]

johnbing==Anuj? same person posting all these under diff alias? funny
shit....

so like i said, post your stack trace.. and also your config (broker + JVM
args + client settings, etc), and we can help out more. NIO doesn't "reduce
threads" magically in the overall broker... it reduces the number needed to
sustain large numbers of connections. If you have other settings that
create more threads in the broker core, then it's a wash right?




On Fri, Sep 6, 2013 at 2:23 AM, johnbing <[hidden email]> wrote:

> Yes I have also tested it with 20 producers consumers. But still not seeing
> any reduction in the number of threads. I have tested NIO with STOMP as
> well
> as JMS producers.
>
> USING(Language)          No. of threads before running producer   No. of
> threads after running producers(~20)
> STOMP (PYTHON producer)                         63
> 72
> STOMP +NIO(PYTHON producer)                     67
> 82
> TCP(JAVA producer)                                      58
> 86
> NIO(JAVA producer)                                      58
> 87
>
> Thanks,
> Anuj
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/NIO-transport-performance-for-ActiveMQ-tp4670977p4671250.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

[johnbing]

Hi ceposta,

I am sorry for this John and Anuj confusion. I was copying and pasting anuj's reply in this thread to get the context and forgot to remove the thanks part. I am not Anuj, don't even know who is the guy.

Regarding NIO I am attaching my broker configuration file and thread logs for you reference.

Thread_Trace.txt
NIO_broker_configuration.txt

Thanks,
John

[Steven Turner]

Hey ceposta,

Why nio reduces number of threads for large number of connections and not for small number, I think major advantage of nio transport is that it does not create one thread per connection as TCP does. So i think even if there are 5 clients it should reduce number of threads.
Another thing is even if it reduces clients with large number, can you please specific about numbers, After how many clients we can see the significant thread reduction.

Thanks,
Steven

[Steven Turner]

Hey john,

I think configurations are fine, not sure why threads are not reducing.
Even I am using the same configuration but not saw any improvement in throughput. Can Someone from developers team help me with this.


Thanks,
Steven

[ceposta]

I'm not seeing any issues with the previous thread usage.

ActiveMQ will use a pool of threads to handle the sockets for NIO, it won't
necessarily use a thread per socket.

As I said previously, and still maintain, please post *stack traces* or
analysis of thread usage for large numbers of connections and *show* that
you're not getting better thread usage *for connections*. It doesn't make
sense to use NIO and expect KahaDB threads to reduce.




On Mon, Sep 23, 2013 at 2:01 AM, Steven Turner <[hidden email]>wrote:

> Hey ceposta,
>
> Why nio reduces number of threads for large number of connections and not
> for small number, I think major advantage of nio transport is that it does
> not create one thread per connection as TCP does. So i think even if there
> are 5 clients it should reduce number of threads.
> Another thing is even if it reduces clients with large number, can you
> please specific about numbers, After how many clients we can see the
> significant thread reduction.
>
> Thanks,
> Steven
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/NIO-transport-performance-for-ActiveMQ-tp4670977p4671734.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

[johnbing]

Hi Ceposta,

When I connect my clients using NIO transport I can see NIO worker threads in my logs.
Like:
    40  5           3           3           0           0 ActiveMQ InactivityMonitor WriteCheckTimer
    41  5          25          25          10          10 ActiveMQ BrokerService[amqDevBroker] Task-3
    42  5          52          52          40          40 ActiveMQ NIO Worker 3
    43  5          22          22          10          10 ActiveMQ BrokerService[amqDevBroker] Task-4
    44  5          21          21          20          20 ActiveMQ BrokerService[amqDevBroker] Task-6
    45  5          21          21          10          10 ActiveMQ BrokerService[amqDevBroker] Task-5
    46  5          29          29          20          20 ActiveMQ NIO Worker 4
    47  5          18          18          10          10 ActiveMQ NIO Worker 5
    48  5           3           3           0           0 ActiveMQ InactivityMonitor Worker
    49  5          16          16          10          10 ActiveMQ NIO Worker 6
    60  5          71          33          50          20 ActiveMQ NIO Worker 5
    62  5          51          27          40          20 ActiveMQ BrokerService[amqDevBroker] Task-8
    63  5          49          31          40          30 ActiveMQ NIO Worker 6
    64  5          43          29          30          30 ActiveMQ BrokerService[amqDevBroker] Task-9
    65  5          34          18          20          10 ActiveMQ BrokerService[amqDevBroker] Task-10
    67  5          36          30          20          20 ActiveMQ NIO Worker 8
    68  5          26          19          10          10 ActiveMQ BrokerService[amqDevBroker] Task-11
    69  5          24          20          10          10 ActiveMQ BrokerService[amqDevBroker] Task-12
    70  5          23          23          10          10 ActiveMQ NIO Worker 9
    71  5          11          11           0           0 ActiveMQ NIO Worker 10
    72  5           1           1           0           0 ActiveMQ NIO Worker 11

NIO worker threads are increasing as soon as I connects a client. Even if I am using large number(20) of connections I cannot see any thread reduction compare to TCP. These threads are continuously increasing with client, which should not happen in case of NIO.
So I want to know that saturation value, after which number of NIO threads become constant.

Thanks,
John

[paulgale]

Can you post your configuration (activemq.xml) and your launching script?

On Tue, Sep 24, 2013 at 11:05 AM, johnbing <[hidden email]> wrote:

> Hi Ceposta,
>
> When I connect my clients using NIO transport I can see NIO worker threads
> in my logs.
> Like:
>     40  5           3           3           0           0 ActiveMQ
> InactivityMonitor WriteCheckTimer
>     41  5          25          25          10          10 ActiveMQ
> BrokerService[amqDevBroker] Task-3
>     42  5          52          52          40          40 ActiveMQ NIO
> Worker 3
>     43  5          22          22          10          10 ActiveMQ
> BrokerService[amqDevBroker] Task-4
>     44  5          21          21          20          20 ActiveMQ
> BrokerService[amqDevBroker] Task-6
>     45  5          21          21          10          10 ActiveMQ
> BrokerService[amqDevBroker] Task-5
>     46  5          29          29          20          20 ActiveMQ NIO
> Worker 4
>     47  5          18          18          10          10 ActiveMQ NIO
> Worker 5
>     48  5           3           3           0           0 ActiveMQ
> InactivityMonitor Worker
>     49  5          16          16          10          10 ActiveMQ NIO
> Worker 6
>     60  5          71          33          50          20 ActiveMQ NIO
> Worker 5
>     62  5          51          27          40          20 ActiveMQ
> BrokerService[amqDevBroker] Task-8
>     63  5          49          31          40          30 ActiveMQ NIO
> Worker 6
>     64  5          43          29          30          30 ActiveMQ
> BrokerService[amqDevBroker] Task-9
>     65  5          34          18          20          10 ActiveMQ
> BrokerService[amqDevBroker] Task-10
>     67  5          36          30          20          20 ActiveMQ NIO
> Worker 8
>     68  5          26          19          10          10 ActiveMQ
> BrokerService[amqDevBroker] Task-11
>     69  5          24          20          10          10 ActiveMQ
> BrokerService[amqDevBroker] Task-12
>     70  5          23          23          10          10 ActiveMQ NIO
> Worker 9
>     71  5          11          11           0           0 ActiveMQ NIO
> Worker 10
>     72  5           1           1           0           0 ActiveMQ NIO
> Worker 11
>
> NIO worker threads are increasing as soon as I connects a client. Even if I
> am using large number(20) of connections I cannot see any thread reduction
> compare to TCP. These threads are continuously increasing with client, which
> should not happen in case of NIO.
> So I want to know that saturation value, after which number of NIO threads
> become constant.
>
> Thanks,
> John
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/NIO-transport-performance-for-ActiveMQ-tp4670977p4671796.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.

[johnbing]

Hi,

Please find attached configuration file.NIO_broker_configuration.txt


My launcher file contains :

max_heap_size  = "2700M"
max_perm_size  = "128M"

<broker>
    name               = "mqbroker"
    activemq_jar       = "apache-activemq-5.8.0/bin/activemq.jar"
    log_dir            = "/logs/"
    default_broker_cfg = "amq_broker_config.xml"
</broker>

[ceposta]

How many connections are opened during your stack trace? I only see 10 NIO
threads at that particular snapshot.


On Wed, Sep 25, 2013 at 3:21 AM, johnbing <[hidden email]> wrote:

> Hi,
>
> Please find attached configuration file. NIO_broker_configuration.txt
> <
> http://activemq.2283324.n4.nabble.com/file/n4671813/NIO_broker_configuration.txt
> >
>
>
> My launcher file contains :
>
> max_heap_size  = "2700M"
> max_perm_size  = "128M"
>
> <broker>
>     name               = "mqbroker"
>     activemq_jar       = "apache-activemq-5.8.0/bin/activemq.jar"
>     log_dir            = "/logs/"
>     default_broker_cfg = "amq_broker_config.xml"
> </broker>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/NIO-transport-performance-for-ActiveMQ-tp4670977p4671813.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

[johnbing]

During this stack trace 20 clients were there, but I sent you snapshot of small portion of the stack trace, if you want I can send the whole stack trace for 20 clients. But as I specified NIO threads are continuously increasing with number of clients, So is there a threshold value, or some estimate after which I can expect constant NIO threads.

If my understanding is correct NIO threads are not threads per connection. These are worker threads which are based on operations performed by NIO transport. It has nothing to do with number of clients.

Please correct me if I am wrong.

Thanks,
John

[ceposta]

You are correct. It's a pool of threads that expand and contract. There
isn't a necessary 1-to-1.


On Wed, Sep 25, 2013 at 6:58 AM, johnbing <[hidden email]> wrote:

> During this stack trace 20 clients were there, but I sent you snapshot of
> small portion of the stack trace, if you want I can send the whole stack
> trace for 20 clients. But as I specified NIO threads are continuously
> increasing with number of clients, So is there a threshold value, or some
> estimate after which I can expect constant NIO threads.
>
> If my understanding is correct NIO threads are not threads per connection.
> These are worker threads which are based on operations performed by NIO
> transport. It has nothing to do with number of clients.
>
> Please correct me if I am wrong.
>
> Thanks,
> John
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/NIO-transport-performance-for-ActiveMQ-tp4670977p4671828.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta

[johnbing]

Hey,

I have searched but not able to find any results or any graphs for ActiveMQ performance. I mean do we have any numbers for ActiveMQ performance ( throughput, memory uses , threads etc.)
Most of the other frameworks always share performance analysis and graphs.

Thanks,
John