Is creating thousands of connections to a single AMQ broker node and keeping them open an anti-pattern?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Is creating thousands of connections to a single AMQ broker node and keeping them open an anti-pattern?

Shobhana
We use a single AMQ broker (using 5.14.1 version) node to exchange MQTT messages between our server and apps running on mobile devices (both Android and iOS).
The app establishes a connection with AMQ broker using Eclipse Paho client lib. To be able to receive messages as soon as they are published, the app keeps this connection open all the time. Even when the app is closed, a background service gets started (in case of Android) which establishes a connection with the broker.
In effect, if there are 100000 users using the app, there would be 100000 connections to AMQ broker. Is there any known issue in using AMQ broker this way?
Reply | Threaded
Open this post in threaded view
|

Re: Is creating thousands of connections to a single AMQ broker node and keeping them open an anti-pattern?

Tim Bain
If you have many connections from a single client process, then yes, that's
an antipattern. Connection setup/teardown is somewhat expensive (and
everyone will have to do it at the same time if the broker restarts), so
you should be minimizing the number if possible by using a pooling
connection factory. The broker also has to do a small amount of work to
send keepalive messages and check for idle connections, if you're using
keepalives; that's probably not a huge drain on system resources, but
there's no reason to do more of it than you have to.

If you have 100,000 separate client processes, then there's nothing you can
do to improve that as long as the broker process is holding up OK. (If not,
increase RAM and/or put it on a machine with beefier CPUs, as needed.) The
antipattern is many connections from a single process, not having lots of
long-running idle connections (though you will incur the costs I described
above).

Tim

On May 19, 2017 3:53 AM, "Shobhana" <[hidden email]> wrote:

> We use a single AMQ broker (using 5.14.1 version) node to exchange MQTT
> messages between our server and apps running on mobile devices (both
> Android
> and iOS).
> The app establishes a connection with AMQ broker using Eclipse Paho client
> lib. To be able to receive messages as soon as they are published, the app
> keeps this connection open all the time. Even when the app is closed, a
> background service gets started (in case of Android) which establishes a
> connection with the broker.
> In effect, if there are 100000 users using the app, there would be 100000
> connections to AMQ broker. Is there any known issue in using AMQ broker
> this
> way?
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Is-creating-thousands-of-connections-to-a-
> single-AMQ-broker-node-and-keeping-them-open-an-anti-patte-tp4726391.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Is creating thousands of connections to a single AMQ broker node and keeping them open an anti-pattern?

clebertsuconic
Why don't you look at ActivMQ Artemis.

It's using NIO/epoll so it won't use as much memory as you keep lots
of active connections.



On Fri, May 19, 2017 at 8:42 AM, Tim Bain <[hidden email]> wrote:

> If you have many connections from a single client process, then yes, that's
> an antipattern. Connection setup/teardown is somewhat expensive (and
> everyone will have to do it at the same time if the broker restarts), so
> you should be minimizing the number if possible by using a pooling
> connection factory. The broker also has to do a small amount of work to
> send keepalive messages and check for idle connections, if you're using
> keepalives; that's probably not a huge drain on system resources, but
> there's no reason to do more of it than you have to.
>
> If you have 100,000 separate client processes, then there's nothing you can
> do to improve that as long as the broker process is holding up OK. (If not,
> increase RAM and/or put it on a machine with beefier CPUs, as needed.) The
> antipattern is many connections from a single process, not having lots of
> long-running idle connections (though you will incur the costs I described
> above).
>
> Tim
>
> On May 19, 2017 3:53 AM, "Shobhana" <[hidden email]> wrote:
>
>> We use a single AMQ broker (using 5.14.1 version) node to exchange MQTT
>> messages between our server and apps running on mobile devices (both
>> Android
>> and iOS).
>> The app establishes a connection with AMQ broker using Eclipse Paho client
>> lib. To be able to receive messages as soon as they are published, the app
>> keeps this connection open all the time. Even when the app is closed, a
>> background service gets started (in case of Android) which establishes a
>> connection with the broker.
>> In effect, if there are 100000 users using the app, there would be 100000
>> connections to AMQ broker. Is there any known issue in using AMQ broker
>> this
>> way?
>>
>>
>>
>> --
>> View this message in context: http://activemq.2283324.n4.
>> nabble.com/Is-creating-thousands-of-connections-to-a-
>> single-AMQ-broker-node-and-keeping-them-open-an-anti-patte-tp4726391.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: Is creating thousands of connections to a single AMQ broker node and keeping them open an anti-pattern?

Shobhana
In reply to this post by Tim Bain
Tim, each client process will open just one connection.
There can be hundreds of thousands of different client processes (as many as no of users using our app) connected to the broker at the same time.

In another thread (http://activemq.2283324.n4.nabble.com/ActiveMQ-broker-becomes-unresponsive-after-sometime-td4725278.html#a4726390) I had mentioned that broker becomes unresponsive after some time; wanted to chcek if our usage itself was wrong; hence posted this thread.
Reply | Threaded
Open this post in threaded view
|

Re: Is creating thousands of connections to a single AMQ broker node and keeping them open an anti-pattern?

Shobhana
In reply to this post by clebertsuconic
@clebertsuconic, ActiveMQ also supports NIO and we have already configured to use it. How is this different from that supported in Artemis?
Reply | Threaded
Open this post in threaded view
|

Re: Is creating thousands of connections to a single AMQ broker node and keeping them open an anti-pattern?

clebertsuconic
Artemis has a new architecture and is more effective on keeping
managing connections. Starting from the Journal where we do everything
asynchronous using a lot less threads and more performant. It's the
reason why Artemis merged ActiveMQ.. and it's been quite improved at
2.1.0.. It's worth to at least check it.

On Fri, May 19, 2017 at 1:59 PM, Shobhana <[hidden email]> wrote:
> @clebertsuconic, ActiveMQ also supports NIO and we have already configured to
> use it. How is this different from that supported in Artemis?
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Is-creating-thousands-of-connections-to-a-single-AMQ-broker-node-and-keeping-them-open-an-anti-patte-tp4726391p4726401.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: Is creating thousands of connections to a single AMQ broker node and keeping them open an anti-pattern?

Tim Bain
In reply to this post by Shobhana
I don't have personal experience running with that number of connected
clients (I've only used ActiveMQ with a few dozen to a few hundred), but if
it's a problem, the JVisualVM CPU sampling we talked about in that thread
should make that clear pretty quickly.

Tim

On May 19, 2017 12:11 PM, "Shobhana" <[hidden email]> wrote:

> Tim, each client process will open just one connection.
> There can be hundreds of thousands of different client processes (as many
> as
> no of users using our app) connected to the broker at the same time.
>
> In another thread
> (http://activemq.2283324.n4.nabble.com/ActiveMQ-broker-
> becomes-unresponsive-after-sometime-td4725278.html#a4726390)
> I had mentioned that broker becomes unresponsive after some time; wanted to
> chcek if our usage itself was wrong; hence posted this thread.
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Is-creating-thousands-of-connections-to-a-
> single-AMQ-broker-node-and-keeping-them-open-an-anti-
> patte-tp4726391p4726400.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>