Network of brokers and destination policies

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
rth
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Network of brokers and destination policies

rth
We currently have a single broker that is configured with several <destinationPolicy> entries (all for topics). We will be a network of brokers.

My question is this: how should I deal with the <destinationPolicy> entries? Do they need to be replicated across all brokers' configuration files? Across some subset of brokers (say, brokers from which applications will be consuming the message?) Something different?

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Network of brokers and destination policies

Tim Bain
There's no one-size-fits-all answer here. The basic thing to know is that
destination policies are specific to a given broker and the presence of a
given policy on one broker doesn't cause another broker to automatically
apply the same policy.

Beyond that, how you configure each broker in an NoB depends heavily on how
that broker is being used, what its performance capabilities are, and
ultimately how you want it to behave.

With all of that said, I'd say that more often than not you'll use the same
policies on all brokers, but that's not an across-the-board rule.

If you have a specific policy for which you have questions about how best
to apply it in order to achieve a specific goal, please ask and we can give
a recommendation for that particular policy.

Tim

On Aug 2, 2017 3:12 PM, "rth" <[hidden email]> wrote:

> We currently have a single broker that is configured with several
> <destinationPolicy> entries (all for topics). We will be a network of
> brokers.
>
> My question is this: how should I deal with the <destinationPolicy>
> entries?
> Do they need to be replicated across all brokers' configuration files?
> Across some subset of brokers (say, brokers from which applications will be
> consuming the message?) Something different?
>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Network-of-brokers-and-destination-policies-tp4729188.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
rth
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Network of brokers and destination policies

rth
Thank you very much Tim. That is enough information to get me started
thinking about this. It wasn't clear to me if the brokers had separate
policies or if the same policy was communicated between brokers via
administrative messages.

I will almost certainly take you up on the offer of more specific questions
in the future.


On Wed, Aug 2, 2017 at 10:10 PM, Tim Bain <[hidden email]> wrote:

> There's no one-size-fits-all answer here. The basic thing to know is that
> destination policies are specific to a given broker and the presence of a
> given policy on one broker doesn't cause another broker to automatically
> apply the same policy.
>
> Beyond that, how you configure each broker in an NoB depends heavily on how
> that broker is being used, what its performance capabilities are, and
> ultimately how you want it to behave.
>
> With all of that said, I'd say that more often than not you'll use the same
> policies on all brokers, but that's not an across-the-board rule.
>
> If you have a specific policy for which you have questions about how best
> to apply it in order to achieve a specific goal, please ask and we can give
> a recommendation for that particular policy.
>
> Tim
>
> On Aug 2, 2017 3:12 PM, "rth" <[hidden email]> wrote:
>
> > We currently have a single broker that is configured with several
> > <destinationPolicy> entries (all for topics). We will be a network of
> > brokers.
> >
> > My question is this: how should I deal with the <destinationPolicy>
> > entries?
> > Do they need to be replicated across all brokers' configuration files?
> > Across some subset of brokers (say, brokers from which applications will
> be
> > consuming the message?) Something different?
> >
> >
> >
> >
> >
> > --
> > View this message in context: http://activemq.2283324.n4.
> > nabble.com/Network-of-brokers-and-destination-policies-tp4729188.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
rth
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Network of brokers and destination policies

rth
In reply to this post by Tim Bain
Here's a simplified diagram of my broker network:



Application 1 and Application 2 both both use messaging internally. Their messages do not need to be distributed across the broker network.

The applications consume messages from the clients that all connect to Broker 3. However, processing of those messages is relatively slow. Therefore, Broker 1 and Broker 2 are configured with destination policies for the external destinations so they do not consume 100% of the brokers' resources, and we enable flow control so the clients will slow down production of those messages.

If I understand you correctly, we must also configure destination policies for those external destinations on Broker 3. Otherwise, when Application 1 pushes back, Broker 3 would use 100% of its resources for that destination, right?

Furthermore, I cannot just copy the destination policies: I need to determine what percentage of Broker 3's resources will be used by Application 1 and Application 2 messages.

Does that sound right?
 
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Network of brokers and destination policies

Tim Bain
What resources are you concerned about? Are you trying to load-balance the
work across the two applications (i.e. you're concerned about CPU and about
pre-allocating messages to a particular application), or are you worried
about running out of space in the brokers' message stores? Or something
else?

If you're concerned about filling the message stores (since you mentioned
PFC), then yes you do need to configure policies on Broker 3 as well. But I
didn't follow the final bit about percentages for Broker 1 and Broker 2.

Tim

On Aug 10, 2017 9:53 AM, "rth" <[hidden email]> wrote:

> Here's a simplified diagram of my broker network:
>
> <http://activemq.2283324.n4.nabble.com/file/n4729601/Broker_Network.png>
>
> Application 1 and Application 2 both both use messaging internally. Their
> messages do not need to be distributed across the broker network.
>
> The applications consume messages from the clients that all connect to
> Broker 3. However, processing of those messages is relatively slow.
> Therefore, Broker 1 and Broker 2 are configured with destination policies
> for the external destinations so they do not consume 100% of the brokers'
> resources, and we enable flow control so the clients will slow down
> production of those messages.
>
> If I understand you correctly, we must also configure destination policies
> for those external destinations on Broker 3. Otherwise, when Application 1
> pushes back, Broker 3 would use 100% of its resources for that destination,
> right?
>
> Furthermore, I cannot just copy the destination policies: I need to
> determine what percentage of Broker 3's resources will be used by
> Application 1 and Application 2 messages.
>
> Does that sound right?
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Network-of-brokers-and-destination-policies-
> tp4729188p4729601.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
rth
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Network of brokers and destination policies

rth
I'm primarily concerned about running out of space in broker memory. And
we're not load-balancing between the applications; they are completely
different, but both consume from some of the same topics that the clients
publish to.

Application 1 can fall behind when processing some of those messages. With
flow control enabled, and no destination policies, then both brokers 1 and
3 could dedicate 100% of their memory to those topic, thus starving all the
other messages.

Anyway, you answered my question, I think: if we have destination policies
in broker 1 to prevent that problem, we need a similar destination policy
in broker 3.

Thanks again for the help.



On Thu, Aug 10, 2017 at 11:00 PM, Tim Bain <[hidden email]> wrote:

> What resources are you concerned about? Are you trying to load-balance the
> work across the two applications (i.e. you're concerned about CPU and about
> pre-allocating messages to a particular application), or are you worried
> about running out of space in the brokers' message stores? Or something
> else?
>
> If you're concerned about filling the message stores (since you mentioned
> PFC), then yes you do need to configure policies on Broker 3 as well. But I
> didn't follow the final bit about percentages for Broker 1 and Broker 2.
>
> Tim
>
> On Aug 10, 2017 9:53 AM, "rth" <[hidden email]> wrote:
>
> > Here's a simplified diagram of my broker network:
> >
> > <http://activemq.2283324.n4.nabble.com/file/n4729601/Broker_Network.png>
> >
> > Application 1 and Application 2 both both use messaging internally. Their
> > messages do not need to be distributed across the broker network.
> >
> > The applications consume messages from the clients that all connect to
> > Broker 3. However, processing of those messages is relatively slow.
> > Therefore, Broker 1 and Broker 2 are configured with destination policies
> > for the external destinations so they do not consume 100% of the brokers'
> > resources, and we enable flow control so the clients will slow down
> > production of those messages.
> >
> > If I understand you correctly, we must also configure destination
> policies
> > for those external destinations on Broker 3. Otherwise, when Application
> 1
> > pushes back, Broker 3 would use 100% of its resources for that
> destination,
> > right?
> >
> > Furthermore, I cannot just copy the destination policies: I need to
> > determine what percentage of Broker 3's resources will be used by
> > Application 1 and Application 2 messages.
> >
> > Does that sound right?
> >
> >
> >
> >
> > --
> > View this message in context: http://activemq.2283324.n4.
> > nabble.com/Network-of-brokers-and-destination-policies-
> > tp4729188p4729601.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Network of brokers and destination policies

Tim Bain
Sounds like you're on the right path.

If you're worried about being constrained by your available memory but
don't want to restrict your producers and the fast application, you could
send the messages to a persistence store on disk instead, to allow you to
store more of them.

Tim

On Aug 11, 2017 12:10 PM, "Robert Huffman" <[hidden email]> wrote:

> I'm primarily concerned about running out of space in broker memory. And
> we're not load-balancing between the applications; they are completely
> different, but both consume from some of the same topics that the clients
> publish to.
>
> Application 1 can fall behind when processing some of those messages. With
> flow control enabled, and no destination policies, then both brokers 1 and
> 3 could dedicate 100% of their memory to those topic, thus starving all the
> other messages.
>
> Anyway, you answered my question, I think: if we have destination policies
> in broker 1 to prevent that problem, we need a similar destination policy
> in broker 3.
>
> Thanks again for the help.
>
>
>
> On Thu, Aug 10, 2017 at 11:00 PM, Tim Bain <[hidden email]> wrote:
>
> > What resources are you concerned about? Are you trying to load-balance
> the
> > work across the two applications (i.e. you're concerned about CPU and
> about
> > pre-allocating messages to a particular application), or are you worried
> > about running out of space in the brokers' message stores? Or something
> > else?
> >
> > If you're concerned about filling the message stores (since you mentioned
> > PFC), then yes you do need to configure policies on Broker 3 as well.
> But I
> > didn't follow the final bit about percentages for Broker 1 and Broker 2.
> >
> > Tim
> >
> > On Aug 10, 2017 9:53 AM, "rth" <[hidden email]> wrote:
> >
> > > Here's a simplified diagram of my broker network:
> > >
> > > <http://activemq.2283324.n4.nabble.com/file/n4729601/
> Broker_Network.png>
> > >
> > > Application 1 and Application 2 both both use messaging internally.
> Their
> > > messages do not need to be distributed across the broker network.
> > >
> > > The applications consume messages from the clients that all connect to
> > > Broker 3. However, processing of those messages is relatively slow.
> > > Therefore, Broker 1 and Broker 2 are configured with destination
> policies
> > > for the external destinations so they do not consume 100% of the
> brokers'
> > > resources, and we enable flow control so the clients will slow down
> > > production of those messages.
> > >
> > > If I understand you correctly, we must also configure destination
> > policies
> > > for those external destinations on Broker 3. Otherwise, when
> Application
> > 1
> > > pushes back, Broker 3 would use 100% of its resources for that
> > destination,
> > > right?
> > >
> > > Furthermore, I cannot just copy the destination policies: I need to
> > > determine what percentage of Broker 3's resources will be used by
> > > Application 1 and Application 2 messages.
> > >
> > > Does that sound right?
> > >
> > >
> > >
> > >
> > > --
> > > View this message in context: http://activemq.2283324.n4.
> > > nabble.com/Network-of-brokers-and-destination-policies-
> > > tp4729188p4729601.html
> > > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> > >
> >
>
Loading...