Network of brokers as a cluster

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Network of brokers as a cluster

jeremy11
Hi,
 
can a network of 2 brokers be used to form a cluster? or I need to use Master/slave or other option.
The need is for clustering and that a consumer can disconnect and  connect again to the second broker and to receive messages that were sent while it was down.
can messages pass to both of the brokers even all consumer are disconnected (in a virtual topic)
thanks.
Reply | Threaded
Open this post in threaded view
|

Re: Network of brokers as a cluster

Tim Bain
Yes, you can have a pair of brokers that are both active at the same time
and that both have their own independent message store (not master/slave
with a shared message store).  You'll need to configure the brokers so they
can talk to either other (either a duplex networkConnector from one to the
other, or two simplex networkConnectors one in each direction).  Your
client would connect to either broker, and messages would get routed to the
consumer wherever it is.

There are some configuration issues you'll need to think through, such as
the possibility for the messages to be stranded on the "other" broker
because it's already been routed through that broker (look at the Stuck
Messages section of http://activemq.apache.org/networks-of-brokers.html for
more info about your options here) and the possibility for forwarding
between the brokers to result in more network hops than your networkTTL
will allow (you might want to use a large number for that value to avoid
the problem).

Also, if there will be any other brokers in the network, you'll either need
them to connect to your pair with the static: transport or you'll need to
use static:failover: with maxReconnectAttempts=0 and without setting
priorityBackup=true, since priorityBackup doesn't work for broker-to-broker
failover transports.  Or just use the masterslave: transport (which is not
the same as a master/slave broker pair, so don't get confused there), since
it sets the maxReconnectAttempts URI option for you.

Also keep in mind that when a broker goes down, all messages on that broker
are either lost (if they are non-persistent) or held until the broker comes
back up and can try to deliver them (if they are persistent).  If you can't
live with that, then master/slave is really what you want, not a cluster.

On Thu, Nov 6, 2014 at 10:17 AM, jeremy11 <[hidden email]> wrote:

> Hi,
>
> can a network of 2 brokers be used to form a cluster? or I need to use
> Master/slave or other option.
> The need is for clustering and that a consumer can disconnect and  connect
> again to the second broker and to receive messages that were sent while it
> was down.
> can messages pass to both of the brokers even all consumer are disconnected
> (in a virtual topic)
> thanks.
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Network-of-brokers-as-a-cluster-tp4687028.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Network of brokers as a cluster

jeremy11
thanks for the quick answer. my problem with the virtual topic with 2 brokers is this:
consumer has a failover url of the 2 brokers, brokers are connected in duplex mode, replayWhenNoConsumers="true"

1. I have a consumer connected to broker1
2. each message that is sent to broker 1 or 2 is received by the consumer on broker 1
3. consumer goes offline
4. messages (persist) are sent to broker 1 when consumer is down.
5. broker 1 goes down
6. consumer goes on line and connects to broker 2 - does not receive the messages that were sent to broker 1 when the consumer was offline
7. broker 1 goes online and only then the consumer which is connected to broker 2 gets the messages

thanks again!
Reply | Threaded
Open this post in threaded view
|

Re: Network of brokers as a cluster

Tim Bain
What you're describing sounds like what I warned about in my last
paragraph: messages are getting routed to broker1 while the consumer is
offline, which means they exist in broker1's message store and not anywhere
else in the network of brokers.  When broker1 goes offline, those messages
don't exist anywhere in the network of brokers, so they can't be delivered
when the consumer reconnects.  Once broker1 comes back online, they
reappear in the network of brokers and are delivered to the consumer (even
if the consumer is still on broker2).

If you can't afford to have previously-sent messages not delivered (and
potentially delivered out of order, since messages sent after broker1 goes
down will be delivered while it's down) while a broker is offline, you need
to use the master/slave broker configuration instead of a cluster.

On Thu, Nov 6, 2014 at 10:49 AM, jeremy11 <[hidden email]> wrote:

> thanks for the quick answer. my problem with the virtual topic with 2
> brokers
> is this:
> consumer has a failover url of the 2 brokers, brokers are connected in
> duplex mode, replayWhenNoConsumers="true"
>
> 1. I have a consumer connected to broker1
> 2. each message that is sent to broker 1 or 2 is received by the consumer
> on
> broker 1
> 3. consumer goes offline
> 4. messages (persist) are sent to broker 1 when consumer is down.
> 5. broker 1 goes down
> 6. consumer goes on line and connects to broker 2 - does not receive the
> messages that were sent to broker 1 when the consumer was offline
> 7. broker 1 goes online and only then the consumer which is connected to
> broker 2 gets the messages
>
> thanks again!
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Network-of-brokers-as-a-cluster-tp4687028p4687030.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Network of brokers as a cluster

jeremy11
Hi,

I guess this is a real problem with networks of broker and virtual topics, I thought it could solve the problem of messages getting stuck when one of the brokers go down.
I guess I will go on a simple topic and if my consumer goes down and reconnects he will only gets new messages (not  durable).

thanks for the answers
Reply | Threaded
Open this post in threaded view
|

Re: Network of brokers as a cluster

Tim Bain
As far as I know, this behavior would happen with simple topics and simple
queues just as much as with virtual topics; this is a result of ActiveMQ's
fundamental architecture, not something specific to virtual topics.  The
architecture of ActiveMQ ensures that any broker only knows about its own
messages and not those held by any other broker.  This allows for
complicated, flexible networks of brokers that would be difficult or
impossible otherwise, but because brokers don't have access to messages
elsewhere in the network, they can't replay messages if a broker goes down.

If you want reliability of message delivery in the face of broker failure
(instead of only reliability of the routing infrastructure but temporary
unavailability of messages), you should use the master/slave paradigm; it
was developed to handle exactly the situation you're looking at.

Also, you can have a durable subscription on a simple topic, which will
allow the consumer to get all messages (including all messages sent while
the consumer is disconnected), so I'm not sure why you're saying that a
non-durable topic subscription is your best remaining option; I'd think
that a durable subscription would still be preferable to a non-durable one
for the use case you described, whether or not you use a master/slave
broker pair to ensure that you can access all messages even when one broker
goes down.

Tim

On Thu, Nov 6, 2014 at 1:07 PM, jeremy11 <[hidden email]> wrote:

> Hi,
>
> I guess this is a real problem with networks of broker and virtual topics,
> I
> thought it could solve the problem of messages getting stuck when one of
> the
> brokers go down.
> I guess I will go on a simple topic and if my consumer goes down and
> reconnects he will only gets new messages (not  durable).
>
> thanks for the answers
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Network-of-brokers-as-a-cluster-tp4687028p4687033.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Network of brokers as a cluster

jeremy11
Hi,

I also tested durable subscriber on a network of 2 brokers and you are correct this case can also happen, from the documents on durable subscription: "However, if the subscriber disconnects and reconnects to broker A, any messages sent by P while the subscriber was away will be stuck on B until the subscriber reconnects to B" (from http://activemq.apache.org/how-do-i-use-durable-subscribers-in-a-network-of-brokers.html ).
so going to master/slave solution seems to be one solution but in this case you should have:
1. shared file system  or,
2. shared db or,
3. use Replicated LevelDB Store - the documentation says it is recommended to use  at least 3 replica nodes

I can't assume I will have shared file system or shared db in production and I dont want more than 2 brokers so if I can leave with the fact that if the subscriber goes down and goes up again it should make sync (with db/rest server etc) and only then reconnects to the broker I think the solution of simple topic might be enough with the failover of consumer and producer to each of the brokers to ensure that if one broker goes down the consumer and producer will automatically connects to the second broker.

thanks again Tim  
Reply | Threaded
Open this post in threaded view
|

Re: Network of brokers as a cluster

Tim Bain
As I understand it, the text you quoted only applies if
replayWhenNoConsumers=false.  With replayWhenNoConsumers=true (as you have
it), you should get all messages delivered from all brokers that are up,
but you still won't get messages from brokers that are down until they come
back up.  So I think the same conclusion holds, only the text you quoted
isn't what proves it.

We made essentially the same decision: our messages are non-persistent for
performance reasons, so we can't use any of the three options you mentioned
to get master/slave functionality because they all ultimately rely on a
persistence store.  There was some work done on a share-nothing
master-slave capability, but it was buggy and has since been removed.  So
we're just using pairs of independent brokers, and we've engineered the
system to tolerate not getting some messages (or getting them
late/out-of-order) when a broker failure occurs.

On Fri, Nov 7, 2014 at 3:20 AM, jeremy11 <[hidden email]> wrote:

> Hi,
>
> I also tested durable subscriber on a network of 2 brokers and you are
> correct this case can also happen, from the documents on durable
> subscription: "However, if the subscriber disconnects and reconnects to
> broker A, any messages sent by P while the subscriber was away will be
> stuck
> on B until the subscriber reconnects to B" (from
>
> http://activemq.apache.org/how-do-i-use-durable-subscribers-in-a-network-of-brokers.html
> ).
> so going to master/slave solution seems to be one solution but in this case
> you should have:
> 1. shared file system  or,
> 2. shared db or,
> 3. use Replicated LevelDB Store - the documentation says it is recommended
> to use  at least 3 replica nodes
>
> I can't assume I will have shared file system or shared db in production
> and
> I dont want more than 2 brokers so if I can leave with the fact that if the
> subscriber goes down and goes up again it should make sync (with db/rest
> server etc) and only then reconnects to the broker I think the solution of
> simple topic might be enough with the failover of consumer and producer to
> each of the brokers to ensure that if one broker goes down the consumer and
> producer will automatically connects to the second broker.
>
> thanks again Tim
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Network-of-brokers-as-a-cluster-tp4687028p4687051.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>