Slow and fast messages

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Slow and fast messages

MaryAuaun
I have an application that sends requests through a (persistent) queue, with multiple machines as consumers that can process the requests.  To send 1000 requests takes 40 seconds, which is very slow.  Is there any way to speed this up?



Also...
Some of the messages require just a little work and get processed in 1/100 seconds.  Others may take a few minutes of processing time.  Is there a way to get a large pre-fetch and do some kind of "put back" if the consumer determines it will be spending a few minutes on 1 request?

My producers can tag the messages before submitting them with a "processing time estimate".  Is there a way to evenly distribute the tasks to different consumers efficiently?
(When I had a large pre-fetch, one consumer would possibly take 20 messages that required 2-3 minutes each; it would be better to only get one of those large messages, or lots of smaller messages)
Reply | Threaded
Open this post in threaded view
|

Re: Slow and fast messages

MaryAuaun
I also tried switching to non-persistent, and added the following options:

                connectionFactory.setUseAsyncSend(true); // tbw to test speed
                connectionFactory.setAlwaysSyncSend(false);
                connectionFactory.setSendAcksAsync(true);
                connectionFactory.setProducerWindowSize(100);


It now takes 27 seconds instead of 40 seconds to write 1000 messages, which is still very very slow!
Reply | Threaded
Open this post in threaded view
|

Re: Slow and fast messages

MaryAuaun
In reply to this post by MaryAuaun
Boy, this forum is great for making me get my thoughts together and answering my own questions, but if anyone knows any more, please chime in!

For writing 1000 messages

Originally 1000 messages took 40 seconds.

Persistent vs. Non-Persistent didn't make much difference in speed

These settings made a little differenceoptions:
                connectionFactory.setUseAsyncSend(true); // tbw to test speed
                connectionFactory.setAlwaysSyncSend(false);
                connectionFactory.setSendAcksAsync(true);
                connectionFactory.setProducerWindowSize(100);
      (some of these settings are poorly document or not documented at all!)
  Now 1000 messages takes 27 seconds!

I removed the "transacted" and commits, that made it much faster
   1000 messages take about 6 seconds, even with Persistent on.

That's nos 166 messages / sec, which is not bad.  If there are other improvements I can make, please let me know!
Reply | Threaded
Open this post in threaded view
|

Re: Slow and fast messages

cschneider
I recently did perf measurements using activemq. You can find the results and configs here
http://www.liquid-reality.de/pages/viewpage.action?pageId=5865562

For simple one way pure jms i got about 5500 messages/s

Christian

Von meinem iPhone gesendet

Am 12.07.2011 um 00:55 schrieb MaryAuaun <[hidden email]>:

> Boy, this forum is great for making me get my thoughts together and answering
> my own questions, but if anyone knows any more, please chime in!
>
> For writing 1000 messages
>
> Originally 1000 messages took 40 seconds.
>
> Persistent vs. Non-Persistent didn't make much difference in speed
>
> These settings made a little differenceoptions:
>                connectionFactory.setUseAsyncSend(true); // tbw to test
> speed
>                connectionFactory.setAlwaysSyncSend(false);
>                connectionFactory.setSendAcksAsync(true);
>                connectionFactory.setProducerWindowSize(100);
>      (some of these settings are poorly document or not documented at all!)
>  Now 1000 messages takes 27 seconds!
>
> I removed the "transacted" and commits, that made it much faster
>   1000 messages take about 6 seconds, even with Persistent on.
>
> That's nos 166 messages / sec, which is not bad.  If there are other
> improvements I can make, please let me know!
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Slow-and-fast-messages-tp3660295p3661071.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Slow and fast messages

mickhayes
In reply to this post by MaryAuaun
Hi,
Chapter 13 "Tuning ActiveMQ for Performance" of the book "ActiveMQ in Action" details a variety of techniques for speeding things up.
For your setup, you should try to answer the question "where is the time being spent?".
For an idea of potential speed, the published results of various SPECjms2007 runs can be found here: http://www.spec.org/jms2007/results/jms2007.html:

E.g. http://www.spec.org/jms2007/results/res2010q3/jms2007-20100802-00021.png shows a sustained messaging rate of the order of 1,000,000 messages per second for that combination of HW/OS/SW.

All the best,
Mick Hayes

Michael Hayes B.Sc. (NUI), M.Sc. (DCU), SCSA SCNA
Reply | Threaded
Open this post in threaded view
|

Re: Slow and fast messages

TomLiebl
In reply to this post by MaryAuaun
Hi MaryAuaun - Regarding your comment that you "removed the transacted and
commits", what exactly did you do? It may be helpful for others (like
myself) to get some more detail there, in case we can make the same change
and benefit from the performance improvement.

--
View this message in context: http://activemq.2283324.n4.nabble.com/Slow-and-fast-messages-tp3660295p3662982.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

RE: Slow and fast messages

oseymen
In reply to this post by mickhayes
There is also a very detailed tuning guide from FuseSource here (need to register first): http://fusesource.com/require_registration_for?url=http://fusesource.com/docs/broker/5.5/tuning/tuning.pdf

Ozan


-----Original Message-----
From: mickhayes [mailto:[hidden email]]
Sent: 12 July 2011 07:09
To: [hidden email]
Subject: Re: Slow and fast messages

Hi,
Chapter 13 "Tuning ActiveMQ for Performance" of the book "ActiveMQ in Action" details a variety of techniques for speeding things up.
For your setup, you should try to answer the question "where is the time being spent?".
For an idea of potential speed, the published results of various SPECjms2007 runs can be found here:  http://www.spec.org/jms2007/results/jms2007.html
http://www.spec.org/jms2007/results/jms2007.html :

E.g.
http://www.spec.org/jms2007/results/res2010q3/jms2007-20100802-00021.png
http://www.spec.org/jms2007/results/res2010q3/jms2007-20100802-00021.png
shows a sustained messaging rate of the order of 1,000,000 messages per second for that combination of HW/OS/SW.

All the best,
Mick Hayes



-----
Michael Hayes B.Sc. (NUI), M.Sc. (DCU), SCSA SCNA

--
View this message in context: http://activemq.2283324.n4.nabble.com/Slow-and-fast-messages-tp3660295p3661531.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email ______________________________________________________________________

This e-mail and any attached files are intended for the named addressee only. It contains information which may be confidential and legally privileged and also protected by copyright. Unless you are the named addressee (or authorised to receive for the addressee) you may not copy or use it or disclose it to anyone else. If you received it in error please notify the sender immediately and then delete it from your system.

Please be advised that the views and opinions expressed in this e-mail may not reflect the views and opinions of The Digital Property Group Limited or any of its subsidiary companies.

We make every effort to keep our network free from viruses. However, you do need to check this e-mail and any attachments to it for viruses as we can take no responsibility for any computer virus which may be transferred by way of this e-mail. We reserve the right to monitor all e-mail communications.

The Digital Property Group Limited is a Daily Mail and General Trust plc company.
Registered Office: Northcliffe House, 2 Derry Street, London, W8 5TT.
Registered in England & Wales No: 02290527  VAT no. 243571174

______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
______________________________________________________________________
Reply | Threaded
Open this post in threaded view
|

Re: Slow and fast messages

whaley
In reply to this post by MaryAuaun
Two questions/suggestions regarding your AMQ setup:

1) What is the average size (or range of sizes) of the actual message being sent from your producer?  If you are dealing with larger messages you may wish to consider sending Blob Messages instead of putting the entire payload inside of the message itself - see http://activemq.apache.org/blob-messages.html for more detail.  Determining what a "large" message is may vary, but once you start approaching 1MB you should definitely consider Blob Messages.

2) What is the network latency between your producer and your broker?  If it is substantial enough to affect the speed at which you can produce a message and receive acknowledgement from the broker, you could use a Network of Brokers between your producer and consumer such that one AMQ instance lives on the same host or at least on the same subnet as your producer and that instance is responsible for forwarding messages to another AMQ instance that your consumer consumes from.  See http://activemq.apache.org/networks-of-brokers.html for more.

On Jul 11, 2011, at 4:55 PM, MaryAuaun wrote:

> Boy, this forum is great for making me get my thoughts together and answering
> my own questions, but if anyone knows any more, please chime in!
>
> For writing 1000 messages
>
> Originally 1000 messages took 40 seconds.
>
> Persistent vs. Non-Persistent didn't make much difference in speed
>
> These settings made a little differenceoptions:
>                connectionFactory.setUseAsyncSend(true); // tbw to test
> speed
>                connectionFactory.setAlwaysSyncSend(false);
>                connectionFactory.setSendAcksAsync(true);
>                connectionFactory.setProducerWindowSize(100);
>      (some of these settings are poorly document or not documented at all!)
>  Now 1000 messages takes 27 seconds!
>
> I removed the "transacted" and commits, that made it much faster
>   1000 messages take about 6 seconds, even with Persistent on.
>
> That's nos 166 messages / sec, which is not bad.  If there are other
> improvements I can make, please let me know!
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Slow-and-fast-messages-tp3660295p3661071.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: Slow and fast messages

MaryAuaun
In reply to this post by TomLiebl
javax.jms.Connection connection = ...
connection.createSession(false, parms);   // the first parameter is "transacted?", I used false

And I commented out calls to session.commit()  (session is declared as "javax.jms.Session")
Reply | Threaded
Open this post in threaded view
|

Re: Slow and fast messages

MaryAuaun
In reply to this post by whaley
@whaley, those are good things to consider.  I'm beginning to suspect serialization.  The messages are probably 2k Bytes max, but it does include a hash table (typically containing 2-3 items in the table!).

The network is local, two machines on the same switch, or sometimes sender and receiver are on the same machine, so I'm suspecting it's not network latency.