conduit subscriptions

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

conduit subscriptions

MatsH
Hey,

We use both topics and queues alongside selectors, ActiveMQ 5.2, and we
were intending to go live using a network of brokers and the clients
using the failover: protocol.

Unfortunately, due to how conduit subscriptions work we will start with
a single broker for now, until we can figure out a good way of doing it.
The problem is that conduit subscriptions make it difficult to use
either topics or queues depending on if you have it set to on or off.

For instance, with conduit subscriptions turned off, you get duplicate
messages. This quickly gets out of hand if you add more brokers. If you
have two consumers on broker A and send a message to their topic on
broker B, each consumer receives the message twice. If you add another
broker and send the message to one of the brokers that is remote to the
consumers each consumer now gets 6 messages. Add another consumer and
each consumer now gets the message 12 times, etc. It very quickly gets
out of hand, so having conduit subscriptions turned off is a no go.

On the other hand, with conduit subscriptions turned on you now run into
problems with queues. With two consumers on broker A and one consumer on
broker B the messages get distributed unevenly, if 20 messages are sent
to broker B its lone consumer gets 10 messages, and the other 10 are
sent to broker A where each consumer only gets 5 messages each.
Furthermore if you are using selectors (which we are) then if it so
happened that the 10 messages that went to the consumers on broker A
didn't match the selectors then those 10 messages would just be stuck on
the queue on broker A. So having conduit subscriptions turned on is
firstly impractical since the messages to queues are distributed
unevenly, and if you are using selectors it doesn't work at all.

So the situation we are in is that we can't have conduit subscriptions
turned on because of our queues and we can't have it turned off because
of our topics. Yes we could refactor and stop using selectors, but our
messages would still be distributed unevenly.

It strikes me that if conduit subscriptions could be turned on and off
independently between queues and topics things would work; we could have
it turned on for topics and off for queues, and everything would work.

For those of you that have looked more at the ActiveMQ internals than I
have, would this be a difficult change to make?


--
Mats Henrikson
Software Engineer
SunGard Kiodex

Think before you print


Reply | Threaded
Open this post in threaded view
|

Re: conduit subscriptions

joe_fernandez
CONTENTS DELETED
The author has deleted this message.
Reply | Threaded
Open this post in threaded view
|

RE: conduit subscriptions

MatsH
> What about using virtual topics instead of topics? This way, all your
> consumers consume from queues.

Hmm, interesting. I hadn't looked into VirtualTopics before, it would
depend on how they work under the hood I guess. I'll have to experiment
with them a bit. Thanks for the suggestion!

Mats


Reply | Threaded
Open this post in threaded view
|

Re: conduit subscriptions

gtully
In reply to this post by MatsH
along with virtual topics, consider adding two network connectors, one that
matches topics, the other that matches queues. In this way you can have your
required configuration for the conduit variable

Also, consider a move to a 5.3-SNAPSHOT, where selectors are propagated
(with conduit subs off) http://issues.apache.org/activemq/browse/AMQ-2104

2009/8/11 <[hidden email]>

> Hey,
>
> We use both topics and queues alongside selectors, ActiveMQ 5.2, and we
> were intending to go live using a network of brokers and the clients
> using the failover: protocol.
>
> Unfortunately, due to how conduit subscriptions work we will start with
> a single broker for now, until we can figure out a good way of doing it.
> The problem is that conduit subscriptions make it difficult to use
> either topics or queues depending on if you have it set to on or off.
>
> For instance, with conduit subscriptions turned off, you get duplicate
> messages. This quickly gets out of hand if you add more brokers. If you
> have two consumers on broker A and send a message to their topic on
> broker B, each consumer receives the message twice. If you add another
> broker and send the message to one of the brokers that is remote to the
> consumers each consumer now gets 6 messages. Add another consumer and
> each consumer now gets the message 12 times, etc. It very quickly gets
> out of hand, so having conduit subscriptions turned off is a no go.
>
> On the other hand, with conduit subscriptions turned on you now run into
> problems with queues. With two consumers on broker A and one consumer on
> broker B the messages get distributed unevenly, if 20 messages are sent
> to broker B its lone consumer gets 10 messages, and the other 10 are
> sent to broker A where each consumer only gets 5 messages each.
> Furthermore if you are using selectors (which we are) then if it so
> happened that the 10 messages that went to the consumers on broker A
> didn't match the selectors then those 10 messages would just be stuck on
> the queue on broker A. So having conduit subscriptions turned on is
> firstly impractical since the messages to queues are distributed
> unevenly, and if you are using selectors it doesn't work at all.
>
> So the situation we are in is that we can't have conduit subscriptions
> turned on because of our queues and we can't have it turned off because
> of our topics. Yes we could refactor and stop using selectors, but our
> messages would still be distributed unevenly.
>
> It strikes me that if conduit subscriptions could be turned on and off
> independently between queues and topics things would work; we could have
> it turned on for topics and off for queues, and everything would work.
>
> For those of you that have looked more at the ActiveMQ internals than I
> have, would this be a difficult change to make?
>
>
> --
> Mats Henrikson
> Software Engineer
> SunGard Kiodex
>
> Think before you print
>
>
>


--
http://blog.garytully.com

Open Source Integration
http://fusesource.com
Reply | Threaded
Open this post in threaded view
|

Re: conduit subscriptions

rajdavies
In reply to this post by MatsH

On 11 Aug 2009, at 17:14, <[hidden email]> <[hidden email]
 > wrote:

> Hey,
>
> We use both topics and queues alongside selectors, ActiveMQ 5.2, and  
> we
> were intending to go live using a network of brokers and the clients
> using the failover: protocol.
>
> Unfortunately, due to how conduit subscriptions work we will start  
> with
> a single broker for now, until we can figure out a good way of doing  
> it.
> The problem is that conduit subscriptions make it difficult to use
> either topics or queues depending on if you have it set to on or off.
>
> For instance, with conduit subscriptions turned off, you get duplicate
> messages. This quickly gets out of hand if you add more brokers. If  
> you
> have two consumers on broker A and send a message to their topic on
> broker B, each consumer receives the message twice. If you add another
> broker and send the message to one of the brokers that is remote to  
> the
> consumers each consumer now gets 6 messages. Add another consumer and
> each consumer now gets the message 12 times, etc. It very quickly gets
> out of hand, so having conduit subscriptions turned off is a no go.
>
> On the other hand, with conduit subscriptions turned on you now run  
> into
> problems with queues. With two consumers on broker A and one  
> consumer on
> broker B the messages get distributed unevenly, if 20 messages are  
> sent
> to broker B its lone consumer gets 10 messages, and the other 10 are
> sent to broker A where each consumer only gets 5 messages each.
> Furthermore if you are using selectors (which we are) then if it so
> happened that the 10 messages that went to the consumers on broker A
> didn't match the selectors then those 10 messages would just be  
> stuck on
> the queue on broker A. So having conduit subscriptions turned on is
> firstly impractical since the messages to queues are distributed
> unevenly, and if you are using selectors it doesn't work at all.
>
> So the situation we are in is that we can't have conduit subscriptions
> turned on because of our queues and we can't have it turned off  
> because
> of our topics. Yes we could refactor and stop using selectors, but our
> messages would still be distributed unevenly.
>
> It strikes me that if conduit subscriptions could be turned on and off
> independently between queues and topics things would work; we could  
> have
> it turned on for topics and off for queues, and everything would work.
>
> For those of you that have looked more at the ActiveMQ internals  
> than I
> have, would this be a difficult change to make?
>
>
> --
> Mats Henrikson
> Software Engineer
> SunGard Kiodex
>
> Think before you print
>
>


Use one network connection for Queues - one for Topics using included/
excluded destinations - switch off conduit subscriptions for the Queue  
network.

cheers,

Rob

Rob Davies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/