Questions to broker architecture and setup, memory configuration

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Questions to broker architecture and setup, memory configuration

Tom_Z
Hi there,

as an ActiveMQ starter I would like you to give me some feedback on my setup.

Scenario is:
- currently single broker setup with standalone ActiveMQ 5.8 using LevelDB, JRE 7, Protocol for all connected systems will be OpenWire
- 2 backend systems, let's call them B1 and B2, acting as producer and consumer in sequential, non-multithreaded working mode
- B1 and B2 do produce different kinds of messages into their own individual queue (two 'in'-queues from the client perspective)
- these systems can rely on a stable network connection and can be seen as a fast producer and consumer running all over the day
- on the other side we do have a client application C with up to 250 instances working in parallel as a consumer on the two before mentioned out-queues
- each client C does produce different kinds of response messages to different 'out'-queues
- the clients do not have a stable network connection and messages can be stuck in the queue for a while resulting in large out queues, at least quite a bundle of messages for some of the consumers can remain for a longer time
- expected data volume: 50.000 messages a day, average size 10kb (range is from 1kb to 1MB)
- all messages must be persisted
- all queues and virtual destinations are predefined in activemq.xml

Message processing:
- B1 produces into QIB1 (in-queue for B1) and consumes from QO1, QO2, QO3 (queues-out 1 to 3), spread all over the day
- B2 usually produces large amounts of messages only once a day into QIB2, but consumes continiously returning messages from QO4
- every client C consumes request messages designated to that individual client through JMSXGroupID from QIB1 and QIB2 and produces response messages to QO1,2,3,4  
- in the opposite direction B1 and B2 are able to process messages for theirselves delivered to their designated queues
- message order is relevant in both directions (in and out)
- for testing purposes und to be able to redeliver messages to clients in case of loss at client side we have defined "Composite Destinations", so that each message is copied (forwardOnly="false") into an in- respectively out-archive queue, so that messages can be pushed back into the relevant queues (we are a little concerned about the quality of implementation on the client side and finding bugs processing messages in production and like to be prepared for such cases, archiving will be deactivated later)

1. Is that a suitable setup to use ActiveMQ in production?

2. Is a composite destination from queue to queue okay? In the documentation often topics are used, from the archive queues will never be consumed - only copied, deleted or purged and viewed in the queue browser.

3. Is it possible to add an expiry date while forwarding to the archive queue/topic (Apache Camel is not in use)?

4. One open question is whether to use a single out queue (easier for client app C) and to use filtered destinations, filtering must then base on a application specific message property to distribute the message types to the different out-queues, is that possible? Does it perhaps perform better, on client and server side?

5. I still have to take a deeper look at the memory settings. The queues do have quite different characteristics, from being constantly emptied (out queues) to holding large amounts of messages for lots of clients for some time being able to deliver at any time (in queues). I guess at least regarding to both out queues containing messages for up to 250 different message groups I have to do some finetuning?

6. What are the side effects of too little memory for queue? Out-queues are not time or performance critical and B1 and B2 should grab all messages regardless of the message group, in-queues must deliver fast to the client according to the requested message group.

I found a note: "like message selector matching, grouping based on JMSXGroupID occurs before dispatch on messages in memory. With the default maxPageSize setting, large backlogs of messages destined for one group they can block receipt of messages to other groups if they don't all fit in memory."

That means if memory is not enough for all(!) groups/messages in the in-queue, a client possibly receives only a subset of messages for his message group?

Thanks for any hints and advice,
Tom
Reply | Threaded
Open this post in threaded view
|

Re: Questions to broker architecture and setup, memory configuration

ceposta
So this a little TL;DR... but I try answer some of your questions inline...

On Mon, Mar 31, 2014 at 11:53 AM, Tom_Z <[hidden email]> wrote:

> Hi there,
>
> as an ActiveMQ starter I would like you to give me some feedback on my
> setup.
>
> Scenario is:
> - currently single broker setup with standalone ActiveMQ 5.8 using LevelDB,
> JRE 7, Protocol for all connected systems will be OpenWire
> - 2 backend systems, let's call them B1 and B2, acting as producer and
> consumer in sequential, non-multithreaded working mode
> - B1 and B2 do produce different kinds of messages into their own individual
> queue (two 'in'-queues from the client perspective)
> - these systems can rely on a stable network connection and can be seen as a
> fast producer and consumer running all over the day
> - on the other side we do have a client application C with up to 250
> instances working in parallel as a consumer on the two before mentioned
> out-queues
> - each client C does produce different kinds of response messages to
> different 'out'-queues
> - the clients do not have a stable network connection and messages can be
> stuck in the queue for a while resulting in large out queues, at least quite
> a bundle of messages for some of the consumers can remain for a longer time
> - expected data volume: 50.000 messages a day, average size 10kb (range is
> from 1kb to 1MB)
> - all messages must be persisted
> - all queues and virtual destinations are predefined in activemq.xml
>
> Message processing:
> - B1 produces into QIB1 (in-queue for B1) and consumes from QO1, QO2, QO3
> (queues-out 1 to 3), spread all over the day
> - B2 usually produces large amounts of messages only once a day into QIB2,
> but consumes continiously returning messages from QO4
> - every client C consumes request messages designated to that individual
> client through JMSXGroupID from QIB1 and QIB2 and produces response messages
> to QO1,2,3,4
> - in the opposite direction B1 and B2 are able to process messages for
> theirselves delivered to their designated queues
> - message order is relevant in both directions (in and out)
> - for testing purposes und to be able to redeliver messages to clients in
> case of loss at client side we have defined "Composite Destinations", so
> that each message is copied (forwardOnly="false") into an in- respectively
> out-archive queue, so that messages can be pushed back into the relevant
> queues (we are a little concerned about the quality of implementation on the
> client side and finding bugs processing messages in production and like to
> be prepared for such cases, archiving will be deactivated later)
>
> 1. Is that a suitable setup to use ActiveMQ in production?

If I read correctly, you have only one instance running? No.. this is
not a suitable setup for production. What happens if/when that one
instance fails? You should think about having at least a Slave broker
in warm standy.

>
> 2. Is a composite destination from queue to queue okay? In the documentation
> often topics are used, from the archive queues will never be consumed - only
> copied, deleted or purged and viewed in the queue browser.

The messages will never be consumed?
This will not work for long periods, you will run out of disk space.
Why not just store to a separate, longer-term storage? Sounds like
you're trying to use ActiveMQ for both a real-time even broker and
some kind of longer-term audit log... not a good idea.

>
> 3. Is it possible to add an expiry date while forwarding to the archive
> queue/topic (Apache Camel is not in use)?
>
> 4. One open question is whether to use a single out queue (easier for client
> app C) and to use filtered destinations, filtering must then base on a
> application specific message property to distribute the message types to the
> different out-queues, is that possible? Does it perhaps perform better, on
> client and server side?

Depends on how many messages and how much you're filtering...If the
producers know the partition ahead of time, then just stick with that
until it doesn't scale/work.

>
> 5. I still have to take a deeper look at the memory settings. The queues do
> have quite different characteristics, from being constantly emptied (out
> queues) to holding large amounts of messages for lots of clients for some
> time being able to deliver at any time (in queues). I guess at least
> regarding to both out queues containing messages for up to 250 different
> message groups I have to do some finetuning?
>
> 6. What are the side effects of too little memory for queue?
A couple things... message groups are honored based on messages that
have been paged into memory.

http://activemq.apache.org/message-groups.html

If you have too little memory allocated to broker or destinations,
then messages will need to be paged in from disk (which can be
slower).


Out-queues are

> not time or performance critical and B1 and B2 should grab all messages
> regardless of the message group, in-queues must deliver fast to the client
> according to the requested message group.
>
> I found a note: "like message selector matching, grouping based on
> JMSXGroupID occurs before dispatch on messages in memory. With the default
> maxPageSize setting, large backlogs of messages destined for one group they
> can block receipt of messages to other groups if they don't all fit in
> memory."
>
> That means if memory is not enough for all(!) groups/messages in the
> in-queue, a client possibly receives only a subset of messages for his
> message group?
>
> Thanks for any hints and advice,
> Tom
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Questions-to-broker-architecture-and-setup-memory-configuration-tp4679749.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.



--
Christian Posta
http://www.christianposta.com/blog
twitter: @christianposta
Reply | Threaded
Open this post in threaded view
|

Re: Questions to broker architecture and setup, memory configuration

Tom_Z
ceposta wrote
So this a little TL;DR... but I try answer some of your questions inline...
Hi Christian, sorry, what does that mean? :-) TL, DR?

ceposta wrote
If I read correctly, you have only one instance running? No.. this is
not a suitable setup for production. What happens if/when that one
instance fails? You should think about having at least a Slave broker
in warm standy.
Sorry, that was misleading. I only wanted to express not to use a network of brokers. In production environment we will add a slave instance.

ceposta wrote
The messages will never be consumed?
This will not work for long periods, you will run out of disk space.
Why not just store to a separate, longer-term storage? Sounds like
you're trying to use ActiveMQ for both a real-time even broker and
some kind of longer-term audit log... not a good idea.
I'm aware that this is not in sense of messaging but as I stated this will be an temporary feature with purging in between. We do not want to establish more infrastructure beside the broker and like to be able to easily push back to queues. What long term alternative is able to do so? As questioned in no 3, ideally we would add an expiry date, so that archive is not growing but has quite a constant number of messages until finally deactivated. We can add this at client side as message property, but server side would be preferable.

ceposta wrote
>
> 3. Is it possible to add an expiry date while forwarding to the archive
> queue/topic (Apache Camel is not in use)?
>
> 4. One open question is whether to use a single out queue (easier for client
> app C) and to use filtered destinations, filtering must then base on a
> application specific message property to distribute the message types to the
> different out-queues, is that possible? Does it perhaps perform better, on
> client and server side?

Depends on how many messages and how much you're filtering...If the
producers know the partition ahead of time, then just stick with that
until it doesn't scale/work.
That is not so easy to answer, but I try to do an estimated guess: 50.000 messages per day, let's 35.000 in the out-queues split QO1=20.000, QO2=10.000, QO3=4.000, QO4=1.000. Would filtering in that rough case a better way?

ceposta wrote
A couple things... message groups are honored based on messages that
have been paged into memory.

http://activemq.apache.org/message-groups.html

If you have too little memory allocated to broker or destinations,
then messages will need to be paged in from disk (which can be
slower).
That means for a requested message group some messages may be paged in memory but other non-paged messages will be read from disk but all together will be delivered to the client in the same connection or can only those messages be delivered for a message group that are already paged in memory?

Thanks,
Tom