Large number of queues (HowTo)

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Large number of queues (HowTo)

Jose Luna-2

Hello,

We have the requirement of creating a very large number of queues (tens of thousands) in a single broker.  My searches turned up several others trying to do something similar, but with no results. We were recently able to accomplish this, so I decided to write a little howto. All of this information can be found in the mailing list, wiki, or XML reference, but it wasn't easy to pull it all together.  Hopefully this will help someone.  Also, please let me know if there are any inaccuracies, or if anything can be added.

== Thread Count ==
=== "QueueThread" ===
One of the issues that you'll come across is receiving the out of memory error "unable to create new native thread" when you create thousands of queues.  A "QueueThread" thread is created for every new queue.  The optimizedDispatch property for the queue policyEntry seemed to address our needs, the description states "don't use a separate thread for dispatching from a Queue"  (http://activemq.apache.org/per-destination-policies.html). However, turning this on just seemed to create "TempQueue" threads instead, still creating a thread for each queue.  Digging a little deeper, we find that using OptimizedDispatch creates the thread using a TaskRunner, which should pool the threads.   After further research, we found that the property org.apache.activemq.UseDedicatedTaskRunner should be set to false so that the task runner actually pools the threads (http://activemq.apache.org/javalangoutofmemory.html --- this document also provides other useful info, like
 how to lower the stack size for each thread).  

Example Policy Entry in INSTALL_DIR/conf/activemq.conf:
<policyEntries>
    <policyEntry queue=">" memoryLimit="15mb" optimizedDispatch='true'/>
</policyEntries>

You can either change the 'UseDedicatedTaskRunner' property in your startup script, INSTALL_DIR/bin/activemq:
ACTIVEMQ_OPTS="-Xmx512M -Dorg.apache.activemq.UseDedicatedTaskRunner=false"
Or you can set ACTIVEMQ_OPTS in /etc/activemq.conf .

Note: It would be very useful it the documentation listed above referenced each other, as they are most useful when used together.

=== "Checkpoint" thread ===
If you're using persistent storage, a "Checkpoint" thread is also created for every queue that is created. These are used to write persistent messages to the store.  Fortunately this is already created with a task runner, so setting the UseDedicatedTaskRunner property to false will prevent these thread from overhwelming your system.

=== Connection related threads ===
If you're also dealing with high thread count due to a very high number of connections, see  http://fusesource.com/wiki/display/ProdInfo/Understanding+the+Threads+Allocated+in+ActiveMQ.  Also, consider reducing the stack size for each thread (the JVM option -Xss).

== "too many open files" ==
ActiveMQ uses the amqPersistenceAdapter by default for persistent messages.  Unfortunately, this persistence adapter (as well as the kahaPersistenceAdapter) opens a file descriptor for each queue.  When creating large numbers of queues, you'll quickly run into the limit for your OS.   There are two possible solutions to this:  1.) raise the per-process open file limit for your OS 2.) Use a jdbc persistent adapter.
The first can simply be googled for your OS.  The second uses a connection pool to your DB of choice, and no longer requires an open file for each queue created.

An unrelated issue is running out of file descriptors because there are too many connections.  (In linux, and other OSes, a file descriptor is used for open sockets).  I don't know of any other way to handle that, other than increasing the limit for your process.

== Conclusion ==
We were able to create more than 20k queues using ~30 threads in a test environment with a 512MB heap.  Environment: ActiveMQ 5.2.0, SUN Java 1.6.0, Ubuntu 9.04 .  Our testing hasn't been exhaustive, but everything is running smoothly so far.

JLuna


     
Reply | Threaded
Open this post in threaded view
|

Re: Large number of queues (HowTo)

rajdavies
Awesome! - thanks for getting this info together! will put this on the  
Apache site


On 18 May 2009, at 19:27, Jose Luna wrote:

>
> Hello,
>
> We have the requirement of creating a very large number of queues  
> (tens of thousands) in a single broker.  My searches turned up  
> several others trying to do something similar, but with no results.  
> We were recently able to accomplish this, so I decided to write a  
> little howto. All of this information can be found in the mailing  
> list, wiki, or XML reference, but it wasn't easy to pull it all  
> together.  Hopefully this will help someone.  Also, please let me  
> know if there are any inaccuracies, or if anything can be added.
>
> == Thread Count ==
> === "QueueThread" ===
> One of the issues that you'll come across is receiving the out of  
> memory error "unable to create new native thread" when you create  
> thousands of queues.  A "QueueThread" thread is created for every  
> new queue.  The optimizedDispatch property for the queue policyEntry  
> seemed to address our needs, the description states "don't use a  
> separate thread for dispatching from a Queue"  (http://activemq.apache.org/per-destination-policies.html 
> ). However, turning this on just seemed to create "TempQueue"  
> threads instead, still creating a thread for each queue.  Digging a  
> little deeper, we find that using OptimizedDispatch creates the  
> thread using a TaskRunner, which should pool the threads.   After  
> further research, we found that the property  
> org.apache.activemq.UseDedicatedTaskRunner should be set to false so  
> that the task runner actually pools the threads (http://activemq.apache.org/javalangoutofmemory.html 
>  --- this document also provides other useful info, like
> how to lower the stack size for each thread).
>
> Example Policy Entry in INSTALL_DIR/conf/activemq.conf:
> <policyEntries>
>    <policyEntry queue=">" memoryLimit="15mb"  
> optimizedDispatch='true'/>
> </policyEntries>
>
> You can either change the 'UseDedicatedTaskRunner' property in your  
> startup script, INSTALL_DIR/bin/activemq:
> ACTIVEMQ_OPTS="-Xmx512M -
> Dorg.apache.activemq.UseDedicatedTaskRunner=false"
> Or you can set ACTIVEMQ_OPTS in /etc/activemq.conf .
>
> Note: It would be very useful it the documentation listed above  
> referenced each other, as they are most useful when used together.
>
> === "Checkpoint" thread ===
> If you're using persistent storage, a "Checkpoint" thread is also  
> created for every queue that is created. These are used to write  
> persistent messages to the store.  Fortunately this is already  
> created with a task runner, so setting the UseDedicatedTaskRunner  
> property to false will prevent these thread from overhwelming your  
> system.
>
> === Connection related threads ===
> If you're also dealing with high thread count due to a very high  
> number of connections, see  http://fusesource.com/wiki/display/ProdInfo/Understanding+the+Threads+Allocated+in+ActiveMQ 
> .  Also, consider reducing the stack size for each thread (the JVM  
> option -Xss).
>
> == "too many open files" ==
> ActiveMQ uses the amqPersistenceAdapter by default for persistent  
> messages.  Unfortunately, this persistence adapter (as well as the  
> kahaPersistenceAdapter) opens a file descriptor for each queue.  
> When creating large numbers of queues, you'll quickly run into the  
> limit for your OS.   There are two possible solutions to this:  1.)  
> raise the per-process open file limit for your OS 2.) Use a jdbc  
> persistent adapter.
> The first can simply be googled for your OS.  The second uses a  
> connection pool to your DB of choice, and no longer requires an open  
> file for each queue created.
>
> An unrelated issue is running out of file descriptors because there  
> are too many connections.  (In linux, and other OSes, a file  
> descriptor is used for open sockets).  I don't know of any other way  
> to handle that, other than increasing the limit for your process.
>
> == Conclusion ==
> We were able to create more than 20k queues using ~30 threads in a  
> test environment with a 512MB heap.  Environment: ActiveMQ 5.2.0,  
> SUN Java 1.6.0, Ubuntu 9.04 .  Our testing hasn't been exhaustive,  
> but everything is running smoothly so far.
>
> JLuna
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Large number of queues (HowTo)

DataMover
In reply to this post by Jose Luna-2
Any chance we can get a hold of your test code.
1. If I can test against a known configuration that is known to work I can find what I'm doing wrong faster
2. I see that most most posts about working setups are NOT Centos, in fact I have seen two posts that use Ubuntu successfully. Am I to change my linux distro?
3. How many is "many" connections or machines that connect to the server? At more than fifty we see issues.
4. How would having a total of 256k ram effect this?

Jose Luna-2 wrote
Hello,

We have the requirement of creating a very large number of queues (tens of thousands) in a single broker.  My searches turned up several others trying to do something similar, but with no results. We were recently able to accomplish this, so I decided to write a little howto. All of this information can be found in the mailing list, wiki, or XML reference, but it wasn't easy to pull it all together.  Hopefully this will help someone.  Also, please let me know if there are any inaccuracies, or if anything can be added.

== Thread Count ==
=== "QueueThread" ===
One of the issues that you'll come across is receiving the out of memory error "unable to create new native thread" when you create thousands of queues.  A "QueueThread" thread is created for every new queue.  The optimizedDispatch property for the queue policyEntry seemed to address our needs, the description states "don't use a separate thread for dispatching from a Queue"  (http://activemq.apache.org/per-destination-policies.html). However, turning this on just seemed to create "TempQueue" threads instead, still creating a thread for each queue.  Digging a little deeper, we find that using OptimizedDispatch creates the thread using a TaskRunner, which should pool the threads.   After further research, we found that the property org.apache.activemq.UseDedicatedTaskRunner should be set to false so that the task runner actually pools the threads (http://activemq.apache.org/javalangoutofmemory.html --- this document also provides other useful info, like
 how to lower the stack size for each thread).  

Example Policy Entry in INSTALL_DIR/conf/activemq.conf:
<policyEntries>
    <policyEntry queue=">" memoryLimit="15mb" optimizedDispatch='true'/>
</policyEntries>

You can either change the 'UseDedicatedTaskRunner' property in your startup script, INSTALL_DIR/bin/activemq:
ACTIVEMQ_OPTS="-Xmx512M -Dorg.apache.activemq.UseDedicatedTaskRunner=false"
Or you can set ACTIVEMQ_OPTS in /etc/activemq.conf .

Note: It would be very useful it the documentation listed above referenced each other, as they are most useful when used together.

=== "Checkpoint" thread ===
If you're using persistent storage, a "Checkpoint" thread is also created for every queue that is created. These are used to write persistent messages to the store.  Fortunately this is already created with a task runner, so setting the UseDedicatedTaskRunner property to false will prevent these thread from overhwelming your system.

=== Connection related threads ===
If you're also dealing with high thread count due to a very high number of connections, see  http://fusesource.com/wiki/display/ProdInfo/Understanding+the+Threads+Allocated+in+ActiveMQ.  Also, consider reducing the stack size for each thread (the JVM option -Xss).

== "too many open files" ==
ActiveMQ uses the amqPersistenceAdapter by default for persistent messages.  Unfortunately, this persistence adapter (as well as the kahaPersistenceAdapter) opens a file descriptor for each queue.  When creating large numbers of queues, you'll quickly run into the limit for your OS.   There are two possible solutions to this:  1.) raise the per-process open file limit for your OS 2.) Use a jdbc persistent adapter.
The first can simply be googled for your OS.  The second uses a connection pool to your DB of choice, and no longer requires an open file for each queue created.

An unrelated issue is running out of file descriptors because there are too many connections.  (In linux, and other OSes, a file descriptor is used for open sockets).  I don't know of any other way to handle that, other than increasing the limit for your process.

== Conclusion ==
We were able to create more than 20k queues using ~30 threads in a test environment with a 512MB heap.  Environment: ActiveMQ 5.2.0, SUN Java 1.6.0, Ubuntu 9.04 .  Our testing hasn't been exhaustive, but everything is running smoothly so far.

JLuna


     
Reply | Threaded
Open this post in threaded view
|

Re: Large number of queues (HowTo)

Jose Luna-2

> Any chance we can get a hold of your test code.
We use stomp, so our tests are very short python scripts using stomp.py:
http://www.briggs.net.nz/log/projects/stomppy/
They are simple scripts that create thousands of queues from a single connection, send messages to those queues, and also send messages to composite queues representing large numbers of actual queues.  

> 2. I see that most most posts about working setups are NOT Centos, in fact I
> have seen two posts that use Ubuntu successfully. Am I to change my linux
> distro?
I am not aware of any problems with CentOS5 and ActiveMQ (although I am certainly no expert with ActiveMQ).  Our production machine is RHEL5.  We test on Ubuntu for convenience, but we also test on a dev server running CentOS5 to emulate the production environment.  We've encountered no problems on CentOS5 or RHEL5.

> 3. How many is "many" connections or machines that connect to the server? At
> more than fifty we see issues.

We have not done any tests with a significantly large number of
connections, I only included that information in the HowTo because I encountered it in my research and it is relevant to the resource usage discussed.

> 4. How would having a total of 256k ram effect this?

I can't really attest to the consequences of 256 MB of ram.  The memory usage will vary widely depending on the use case.  If this is what you have available on the heap, it should be entirely sufficient for thousands of queues  (if you follow the instructions to limit the number of threads).  ActiveMQ is pretty memory intensive, but there are  lot of ways to tune it.   I think the best thing to do is run some tests that follow your particular use case.   If you find that you have memory-related issues, read the doc on memory usage I linked to in my original post.  Also, you should really read through all of the relevant elements and properties in the activemq config schema (http://activemq.apache.org/xml-reference.html).  A couple of examples off the top of my head (that may not be relevant to you):
- You can set how many messages are pulled from persistent storage and put into memory at a time, when a queue is being read.
- If you're using composite queues, you can choose whether a copy of the message is made when forwarding the message.