ActiveMQ 5 Multi-Datacenter Camel Routes

classic Classic list List threaded Threaded
5 messages Options
wc
Reply | Threaded
Open this post in threaded view
|

ActiveMQ 5 Multi-Datacenter Camel Routes

wc
We have the following configuration.

3 brokers in Datacenter A in a master/slave configuration using a shared
file system. 3 brokers in Datacenter B in a master/slave configuration
using a shared file system.

There are producers sending messages to topics at Datacenter A. There are
Camel routes routing those messages from the topics to separate local
queues, 1 for local processing and 1 for remote processing, where another
set of Camel routes route the messages from the local remote queue to
remote queues at Datacenter B.

The same configuration exists at Datacenter B where the routing for the
remote works in the opposite direction to Datacenter A.

This configuration is intended to give us an active/active multi-datacenter
cluster.

Even though this configuration seems to work, we are wondering if this
would be the recommended way to do this. I am thinking in terms of using
"network of brokers", but reading the documentation, it doesn't seem like
that would accomplish what we are accomplishing with the Camel routes.

We do have problems from time to time, but we do not know for sure if the
Camel routes are the cause.

Any insight would be appreciated.
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ 5 Multi-Datacenter Camel Routes

Tim Bain
William,

If I understand you correctly, you're using Camel to clone the message so
you get one per data center, and then using Camel to move the message from
the data center at which it was produced to the other one. And you're doing
this so that you can have all messages reach consumers in both data
centers, though I'm a little confused about why you're routing messages
sent to topics (with pub/sub semantics, supporting multiple consumers) to a
single local queue in each data center (which would allow only one consumer
per data center to process the message). Did I get that right?

If so, it does seem like a network of brokers would handle this natively
without all the Camel routes. You'd ensure that messages and subscriptions
for the topic in question is forwarded across the network bridge, and all
consumers in either data center would get delivered a copy of the message.
What is it that you're accomplishing with the Camel routes that would not
be accomplished with this setup?

Tim

On Tue, Sep 17, 2019 at 11:11 AM William Culler <[hidden email]>
wrote:

> We have the following configuration.
>
> 3 brokers in Datacenter A in a master/slave configuration using a shared
> file system. 3 brokers in Datacenter B in a master/slave configuration
> using a shared file system.
>
> There are producers sending messages to topics at Datacenter A. There are
> Camel routes routing those messages from the topics to separate local
> queues, 1 for local processing and 1 for remote processing, where another
> set of Camel routes route the messages from the local remote queue to
> remote queues at Datacenter B.
>
> The same configuration exists at Datacenter B where the routing for the
> remote works in the opposite direction to Datacenter A.
>
> This configuration is intended to give us an active/active multi-datacenter
> cluster.
>
> Even though this configuration seems to work, we are wondering if this
> would be the recommended way to do this. I am thinking in terms of using
> "network of brokers", but reading the documentation, it doesn't seem like
> that would accomplish what we are accomplishing with the Camel routes.
>
> We do have problems from time to time, but we do not know for sure if the
> Camel routes are the cause.
>
> Any insight would be appreciated.
>
wc
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ 5 Multi-Datacenter Camel Routes

wc
Hello Tim.

Thanks for taking the time for a response.

You are correct.  The setup is being used to keep the 2 datacenters
completely in sync.  It allows us to switch our reporting frontend to the
other datacenter at any time if we are having issues at the opposite
datacenter.  The messages are consumed from ActiveMQ to populate a
relational reporting database that exists independently at each datacenter.

I'll give you a little more background.  My team (the data engineering
team) did not implement the configuration,  We have been brought in because
of some performance issues, etc. and it appears that since it has KahaDB,
the powers that be assume that it's a database and should fall under our
team.

In any event, we have been researching ActiveMQ and our configuration to
help out.

I believe that the thought process was that the apps would work as follows:

They would produce to the topic where there are 5 "streams", which is what
our company calls each individual application server, and the application
would open 25 threads per topic, there are 10 topics, essentially one topic
for each message type.  So we're looking at 1250 threads open producing
messages to ActiveMQ.  I assume they used a topic and then used the camel
routes to intercept those messages and write them to the local and local
remote queues as we have the same number of threads consuming messages from
the local queue and then 25 consumers set on the camel routes that route
the messages from the local remote queue to the opposite datacenter.  We
need to be able to consume messages in parallel and it doesn't appear that
is possible using a topic in ActiveMQ 5.X as it is JMS 1.1 compliant.  It
appears that Virtual Topics are used to get around this.  Correct me if  I
am wrong on that.

Are you suggesting that we could use a "network of brokers" with Virtual
Topics to still get the functionality we have with the camel routes?

William

On Sun, Sep 22, 2019 at 1:39 AM Tim Bain <[hidden email]> wrote:

> William,
>
> If I understand you correctly, you're using Camel to clone the message so
> you get one per data center, and then using Camel to move the message from
> the data center at which it was produced to the other one. And you're doing
> this so that you can have all messages reach consumers in both data
> centers, though I'm a little confused about why you're routing messages
> sent to topics (with pub/sub semantics, supporting multiple consumers) to a
> single local queue in each data center (which would allow only one consumer
> per data center to process the message). Did I get that right?
>
> If so, it does seem like a network of brokers would handle this natively
> without all the Camel routes. You'd ensure that messages and subscriptions
> for the topic in question is forwarded across the network bridge, and all
> consumers in either data center would get delivered a copy of the message.
> What is it that you're accomplishing with the Camel routes that would not
> be accomplished with this setup?
>
> Tim
>
> On Tue, Sep 17, 2019 at 11:11 AM William Culler <[hidden email]>
> wrote:
>
> > We have the following configuration.
> >
> > 3 brokers in Datacenter A in a master/slave configuration using a shared
> > file system. 3 brokers in Datacenter B in a master/slave configuration
> > using a shared file system.
> >
> > There are producers sending messages to topics at Datacenter A. There are
> > Camel routes routing those messages from the topics to separate local
> > queues, 1 for local processing and 1 for remote processing, where another
> > set of Camel routes route the messages from the local remote queue to
> > remote queues at Datacenter B.
> >
> > The same configuration exists at Datacenter B where the routing for the
> > remote works in the opposite direction to Datacenter A.
> >
> > This configuration is intended to give us an active/active
> multi-datacenter
> > cluster.
> >
> > Even though this configuration seems to work, we are wondering if this
> > would be the recommended way to do this. I am thinking in terms of using
> > "network of brokers", but reading the documentation, it doesn't seem like
> > that would accomplish what we are accomplishing with the Camel routes.
> >
> > We do have problems from time to time, but we do not know for sure if the
> > Camel routes are the cause.
> >
> > Any insight would be appreciated.
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ 5 Multi-Datacenter Camel Routes

Tim Bain
Yes, if you're looking for competing consumer semantics but for each
message to be consumed exactly once in each data center, then a network of
brokers with a virtual topic atop the two queues would be my first thought
for how to do this. I've not personally done any bridging of virtual
destinations across a NoB, but I think that as long as you forward the
queues between the brokers and don't forward the topic, it should work as
your current setup does without the need for any Camel configuration.

Tim

On Mon, Sep 23, 2019 at 10:21 PM William Culler <[hidden email]>
wrote:

> Hello Tim.
>
> Thanks for taking the time for a response.
>
> You are correct.  The setup is being used to keep the 2 datacenters
> completely in sync.  It allows us to switch our reporting frontend to the
> other datacenter at any time if we are having issues at the opposite
> datacenter.  The messages are consumed from ActiveMQ to populate a
> relational reporting database that exists independently at each datacenter.
>
> I'll give you a little more background.  My team (the data engineering
> team) did not implement the configuration,  We have been brought in because
> of some performance issues, etc. and it appears that since it has KahaDB,
> the powers that be assume that it's a database and should fall under our
> team.
>
> In any event, we have been researching ActiveMQ and our configuration to
> help out.
>
> I believe that the thought process was that the apps would work as follows:
>
> They would produce to the topic where there are 5 "streams", which is what
> our company calls each individual application server, and the application
> would open 25 threads per topic, there are 10 topics, essentially one topic
> for each message type.  So we're looking at 1250 threads open producing
> messages to ActiveMQ.  I assume they used a topic and then used the camel
> routes to intercept those messages and write them to the local and local
> remote queues as we have the same number of threads consuming messages from
> the local queue and then 25 consumers set on the camel routes that route
> the messages from the local remote queue to the opposite datacenter.  We
> need to be able to consume messages in parallel and it doesn't appear that
> is possible using a topic in ActiveMQ 5.X as it is JMS 1.1 compliant.  It
> appears that Virtual Topics are used to get around this.  Correct me if  I
> am wrong on that.
>
> Are you suggesting that we could use a "network of brokers" with Virtual
> Topics to still get the functionality we have with the camel routes?
>
> William
>
> On Sun, Sep 22, 2019 at 1:39 AM Tim Bain <[hidden email]> wrote:
>
> > William,
> >
> > If I understand you correctly, you're using Camel to clone the message so
> > you get one per data center, and then using Camel to move the message
> from
> > the data center at which it was produced to the other one. And you're
> doing
> > this so that you can have all messages reach consumers in both data
> > centers, though I'm a little confused about why you're routing messages
> > sent to topics (with pub/sub semantics, supporting multiple consumers)
> to a
> > single local queue in each data center (which would allow only one
> consumer
> > per data center to process the message). Did I get that right?
> >
> > If so, it does seem like a network of brokers would handle this natively
> > without all the Camel routes. You'd ensure that messages and
> subscriptions
> > for the topic in question is forwarded across the network bridge, and all
> > consumers in either data center would get delivered a copy of the
> message.
> > What is it that you're accomplishing with the Camel routes that would not
> > be accomplished with this setup?
> >
> > Tim
> >
> > On Tue, Sep 17, 2019 at 11:11 AM William Culler <[hidden email]>
> > wrote:
> >
> > > We have the following configuration.
> > >
> > > 3 brokers in Datacenter A in a master/slave configuration using a
> shared
> > > file system. 3 brokers in Datacenter B in a master/slave configuration
> > > using a shared file system.
> > >
> > > There are producers sending messages to topics at Datacenter A. There
> are
> > > Camel routes routing those messages from the topics to separate local
> > > queues, 1 for local processing and 1 for remote processing, where
> another
> > > set of Camel routes route the messages from the local remote queue to
> > > remote queues at Datacenter B.
> > >
> > > The same configuration exists at Datacenter B where the routing for the
> > > remote works in the opposite direction to Datacenter A.
> > >
> > > This configuration is intended to give us an active/active
> > multi-datacenter
> > > cluster.
> > >
> > > Even though this configuration seems to work, we are wondering if this
> > > would be the recommended way to do this. I am thinking in terms of
> using
> > > "network of brokers", but reading the documentation, it doesn't seem
> like
> > > that would accomplish what we are accomplishing with the Camel routes.
> > >
> > > We do have problems from time to time, but we do not know for sure if
> the
> > > Camel routes are the cause.
> > >
> > > Any insight would be appreciated.
> > >
> >
>
wc
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ 5 Multi-Datacenter Camel Routes

wc
Thanks Tim.

We're going to try a couple different solutions.

On Fri, Sep 27, 2019 at 1:13 AM Tim Bain <[hidden email]> wrote:

> Yes, if you're looking for competing consumer semantics but for each
> message to be consumed exactly once in each data center, then a network of
> brokers with a virtual topic atop the two queues would be my first thought
> for how to do this. I've not personally done any bridging of virtual
> destinations across a NoB, but I think that as long as you forward the
> queues between the brokers and don't forward the topic, it should work as
> your current setup does without the need for any Camel configuration.
>
> Tim
>
> On Mon, Sep 23, 2019 at 10:21 PM William Culler <[hidden email]>
> wrote:
>
> > Hello Tim.
> >
> > Thanks for taking the time for a response.
> >
> > You are correct.  The setup is being used to keep the 2 datacenters
> > completely in sync.  It allows us to switch our reporting frontend to the
> > other datacenter at any time if we are having issues at the opposite
> > datacenter.  The messages are consumed from ActiveMQ to populate a
> > relational reporting database that exists independently at each
> datacenter.
> >
> > I'll give you a little more background.  My team (the data engineering
> > team) did not implement the configuration,  We have been brought in
> because
> > of some performance issues, etc. and it appears that since it has KahaDB,
> > the powers that be assume that it's a database and should fall under our
> > team.
> >
> > In any event, we have been researching ActiveMQ and our configuration to
> > help out.
> >
> > I believe that the thought process was that the apps would work as
> follows:
> >
> > They would produce to the topic where there are 5 "streams", which is
> what
> > our company calls each individual application server, and the application
> > would open 25 threads per topic, there are 10 topics, essentially one
> topic
> > for each message type.  So we're looking at 1250 threads open producing
> > messages to ActiveMQ.  I assume they used a topic and then used the camel
> > routes to intercept those messages and write them to the local and local
> > remote queues as we have the same number of threads consuming messages
> from
> > the local queue and then 25 consumers set on the camel routes that route
> > the messages from the local remote queue to the opposite datacenter.  We
> > need to be able to consume messages in parallel and it doesn't appear
> that
> > is possible using a topic in ActiveMQ 5.X as it is JMS 1.1 compliant.  It
> > appears that Virtual Topics are used to get around this.  Correct me if
> I
> > am wrong on that.
> >
> > Are you suggesting that we could use a "network of brokers" with Virtual
> > Topics to still get the functionality we have with the camel routes?
> >
> > William
> >
> > On Sun, Sep 22, 2019 at 1:39 AM Tim Bain <[hidden email]> wrote:
> >
> > > William,
> > >
> > > If I understand you correctly, you're using Camel to clone the message
> so
> > > you get one per data center, and then using Camel to move the message
> > from
> > > the data center at which it was produced to the other one. And you're
> > doing
> > > this so that you can have all messages reach consumers in both data
> > > centers, though I'm a little confused about why you're routing messages
> > > sent to topics (with pub/sub semantics, supporting multiple consumers)
> > to a
> > > single local queue in each data center (which would allow only one
> > consumer
> > > per data center to process the message). Did I get that right?
> > >
> > > If so, it does seem like a network of brokers would handle this
> natively
> > > without all the Camel routes. You'd ensure that messages and
> > subscriptions
> > > for the topic in question is forwarded across the network bridge, and
> all
> > > consumers in either data center would get delivered a copy of the
> > message.
> > > What is it that you're accomplishing with the Camel routes that would
> not
> > > be accomplished with this setup?
> > >
> > > Tim
> > >
> > > On Tue, Sep 17, 2019 at 11:11 AM William Culler <[hidden email]>
> > > wrote:
> > >
> > > > We have the following configuration.
> > > >
> > > > 3 brokers in Datacenter A in a master/slave configuration using a
> > shared
> > > > file system. 3 brokers in Datacenter B in a master/slave
> configuration
> > > > using a shared file system.
> > > >
> > > > There are producers sending messages to topics at Datacenter A. There
> > are
> > > > Camel routes routing those messages from the topics to separate local
> > > > queues, 1 for local processing and 1 for remote processing, where
> > another
> > > > set of Camel routes route the messages from the local remote queue to
> > > > remote queues at Datacenter B.
> > > >
> > > > The same configuration exists at Datacenter B where the routing for
> the
> > > > remote works in the opposite direction to Datacenter A.
> > > >
> > > > This configuration is intended to give us an active/active
> > > multi-datacenter
> > > > cluster.
> > > >
> > > > Even though this configuration seems to work, we are wondering if
> this
> > > > would be the recommended way to do this. I am thinking in terms of
> > using
> > > > "network of brokers", but reading the documentation, it doesn't seem
> > like
> > > > that would accomplish what we are accomplishing with the Camel
> routes.
> > > >
> > > > We do have problems from time to time, but we do not know for sure if
> > the
> > > > Camel routes are the cause.
> > > >
> > > > Any insight would be appreciated.
> > > >
> > >
> >
>