Programatically setting up a proxying broker

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
23 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Programatically setting up a proxying broker

Mark Raynsford
'Lo.

I'm experimenting with configuring a broker P that will simply forward
all messages to a remote broker R. There will not be any subscribers
directly accessing P (but multiple producers will send messages to
queues and topics on P to be forwarded to R).

I have the following small example program that starts up two brokers
and attempts to get one to forward messages to the other.
Unfortunately... The subscriber to the receiving broker never sees any
of the messages. I'm not sure if I'm misunderstanding how message
relaying is supposed to work, or if I've made some trivial mistake
somewhere:

  http://ataxia.io7m.com/2017/07/29/ForwardingServers.java

Any insights would be appreciated.

--
Mark Raynsford | http://www.io7m.com

attachment0 (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Programatically setting up a proxying broker

Tim Bain
If you use the TCP transport for the networkConnector between the brokers,
does it work? I've got a half-memory of a mailing list post where someone
found that the NIO transport didn't work in a networkConnector, but I can't
find it now so it's possible I'm misremembering.

Tim

On Jul 29, 2017 2:00 PM, "Mark Raynsford" <[hidden email]>
wrote:

> 'Lo.
>
> I'm experimenting with configuring a broker P that will simply forward
> all messages to a remote broker R. There will not be any subscribers
> directly accessing P (but multiple producers will send messages to
> queues and topics on P to be forwarded to R).
>
> I have the following small example program that starts up two brokers
> and attempts to get one to forward messages to the other.
> Unfortunately... The subscriber to the receiving broker never sees any
> of the messages. I'm not sure if I'm misunderstanding how message
> relaying is supposed to work, or if I've made some trivial mistake
> somewhere:
>
>   http://ataxia.io7m.com/2017/07/29/ForwardingServers.java
>
> Any insights would be appreciated.
>
> --
> Mark Raynsford | http://www.io7m.com
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Programatically setting up a proxying broker

Mark Raynsford
On 2017-07-29T18:57:30 -0600
Tim Bain <[hidden email]> wrote:

> If you use the TCP transport for the networkConnector between the brokers,
> does it work? I've got a half-memory of a mailing list post where someone
> found that the NIO transport didn't work in a networkConnector, but I can't
> find it now so it's possible I'm misremembering.
>
> Tim

'Lo.

Switching out the NIO connector for the plain TCP connector made no
difference, unfortunately.

--
Mark Raynsford | http://www.io7m.com

attachment0 (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Programatically setting up a proxying broker

Mark Raynsford
Slightly shorter and simpler example here:

  http://ataxia.io7m.com/2017/07/30/ForwardingServers.java

I've tried to step through the broker code in a debugger, but I'm not
familiar with the ActiveMQ sources, and regardless it's pretty tough to
use a debugger to find problems of the class "nothing happens". :)

--
Mark Raynsford | http://www.io7m.com

attachment0 (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Programatically setting up a proxying broker

Mark Raynsford
Hm, is anyone out there actually using ActiveMQ in this manner?

The reason I'm doing this is that I'm considering deploying message
brokers in a manner analogous to the way in which one would
typically deploy mail servers: Programs on a server communicate with an
MTA (such as Postfix or qmail) on the same machine, and that MTA acts
as a dumb relay that simply sends all messages to a centrally
administered MTA/MX. I'd deploy ActiveMQ instances on each server that
simply send all of their messages on to a central ActiveMQ instance (or
series of load-balanced instances).

I've had a look at Artemis, and it seems like the "core bridging"
configuration requires you to manually name the queues that you want
forwarded:

  https://activemq.apache.org/artemis/docs/2.1.0/core-bridges.html

Given that Artemis seems to require you to specify queues to be
forwarded, and given that plain ActiveMQ doesn't seem to want to
forward messages no matter what I tell it, does this suggest that
people don't typically deploy brokers in this way?

--
Mark Raynsford | http://www.io7m.com

attachment0 (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Programatically setting up a proxying broker

Tim Bain
People do what you're trying to do, so this is a use case that works (or at
least, did work at some point in the past).

My advice when message forwarding isn't working in a network of brokers is
to use JMX to figure out what's going on in each broker. I'd start with the
proxy broker and confirm that a subscriber representing the main broker is
created on your topic. If you don't see it, then something is wrong with
how your statically included destinations are being applied. If you do, see
if the JMX stats indicate that messages are being dispatched to it. Then
check the same things on the main broker, except that the consumer is your
actual consumer.

Since you've programmatically disabled JMX, you'll need to enable it to be
able to do those things.

BTW, why are you statically including your destinations rather than doing
it dynamically with a wildcard? Seems like that would be a simpler
confirmation to implement and maintain.

Tim

On Jul 31, 2017 3:25 AM, "Mark Raynsford" <[hidden email]>
wrote:

> Hm, is anyone out there actually using ActiveMQ in this manner?
>
> The reason I'm doing this is that I'm considering deploying message
> brokers in a manner analogous to the way in which one would
> typically deploy mail servers: Programs on a server communicate with an
> MTA (such as Postfix or qmail) on the same machine, and that MTA acts
> as a dumb relay that simply sends all messages to a centrally
> administered MTA/MX. I'd deploy ActiveMQ instances on each server that
> simply send all of their messages on to a central ActiveMQ instance (or
> series of load-balanced instances).
>
> I've had a look at Artemis, and it seems like the "core bridging"
> configuration requires you to manually name the queues that you want
> forwarded:
>
>   https://activemq.apache.org/artemis/docs/2.1.0/core-bridges.html
>
> Given that Artemis seems to require you to specify queues to be
> forwarded, and given that plain ActiveMQ doesn't seem to want to
> forward messages no matter what I tell it, does this suggest that
> people don't typically deploy brokers in this way?
>
> --
> Mark Raynsford | http://www.io7m.com
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Programatically setting up a proxying broker

Mark Raynsford
On 2017-07-31T06:55:07 -0600
Tim Bain <[hidden email]> wrote:

> People do what you're trying to do, so this is a use case that works (or at
> least, did work at some point in the past).

Right.

> My advice when message forwarding isn't working in a network of brokers is
> to use JMX to figure out what's going on in each broker. I'd start with the
> proxy broker and confirm that a subscriber representing the main broker is
> created on your topic. If you don't see it, then something is wrong with
> how your statically included destinations are being applied. If you do, see
> if the JMX stats indicate that messages are being dispatched to it. Then
> check the same things on the main broker, except that the consumer is your
> actual consumer.

I can confirm via JMX that there's no subscriber on the proxy broker.

> Since you've programmatically disabled JMX, you'll need to enable it to be
> able to do those things.

Got it.

> BTW, why are you statically including your destinations rather than doing
> it dynamically with a wildcard? Seems like that would be a simpler
> confirmation to implement and maintain.

To some extent, it's because I'm not entirely sure what I'm doing. :)

The documentation is a little vague on why one might want dynamic vs
static. The documentation for the static configuration seemed to fit
what I was trying to do:

  http://activemq.apache.org/networks-of-brokers.html#NetworksofBrokers-Purestaticnetworks

"If you wish to completely protect the broker from any influence of
consumers on the remote broker, or if you wish to use the brokers as a
simple proxy and forward all messages to the remote side no matter if
there are consumers there or not, static networks are something you
should consider."

I'll give the dynamic configuration with a wildcard a shot now.

--
Mark Raynsford | http://www.io7m.com

attachment0 (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Programatically setting up a proxying broker

Mark Raynsford
Still no luck, unfortunately. This is the test program I'm using now:

  http://ataxia.io7m.com/2017/07/31/ForwardingServers.java

I use setDynamicallyIncludedDestinations() with a wildcard topic ">". I
still see no subscriptions on the proxy broker from the receiver broker.

--
Mark Raynsford | http://www.io7m.com

attachment0 (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Programatically setting up a proxying broker

Tim Bain
OK, no subscriber on the proxy broker certainly explains why no messages
are making it to the main broker, so let's dig into that.

Do you see (via JMX) a networkConnector between the two brokers? And do you
see advisory messages passing from the main broker to the proxy when a
subscriber connects? With dynamically included destinations, if the
advisory messages aren't flowing, you're not going to get any inter-broker
subscriptions established.

Also, I'd suggest that you set up the main broker as a standalone broker
instance. This is how you're going to run eventually anyway, and you'll
have an easier time troubleshooting problems if you have access to log
files, simpler JMX configuration, etc., which may make it easier to work
through whatever's going wrong now.

BTW, I may have led you down the wrong path on the mechanism for including
destinations, because I forgot a very important advantage that statically
included destinations have over dynamically included ones: messages will be
forwarded to the main broker even when there are no consumers attached to
the main broker. Of course, this advantage only really matters for queues,
because for topics a message will be discarded if there are no subscribers,
and so all you're changing is where the message gets discarded. But if you
plan to have queues in addition to the topic shown in your example, this
might make a difference to you. But if you're going to be topic-only, then
there's no disadvantage to using dynamically included destinations per your
current setup.

Tim

On Mon, Jul 31, 2017 at 8:28 AM, Mark Raynsford <
[hidden email]> wrote:

> Still no luck, unfortunately. This is the test program I'm using now:
>
>   http://ataxia.io7m.com/2017/07/31/ForwardingServers.java
>
> I use setDynamicallyIncludedDestinations() with a wildcard topic ">". I
> still see no subscriptions on the proxy broker from the receiver broker.
>
> --
> Mark Raynsford | http://www.io7m.com
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Programatically setting up a proxying broker

Mark Raynsford
'Ello.

For the purposes of keeping everything consistent, I've pushed my
example program to a GitHub repository:

  https://github.com/io7m/activemq-examples

See:

  https://github.com/io7m/activemq-examples/blob/develop/src/main/java/com/io7m/activemq/examples/ForwardingServers.java

I've also included some packet captures in the event that they're
necessary.

On 2017-07-31T21:54:56 -0600
Tim Bain <[hidden email]> wrote:
>
> OK, no subscriber on the proxy broker certainly explains why no messages
> are making it to the main broker, so let's dig into that.
>
> Do you see (via JMX) a networkConnector between the two brokers? And do you
> see advisory messages passing from the main broker to the proxy when a
> subscriber connects? With dynamically included destinations, if the
> advisory messages aren't flowing, you're not going to get any inter-broker
> subscriptions established.

Via JMX, I see a networkConnector on the proxy broker, but none on the
receiver broker:

  http://ataxia.io7m.com/2017/08/01/jmx.png

I've confirmed via watching the network interface with Wireshark that
at no point do either of the brokers send any packets to each other
whatsoever.

A thought suddenly occurred to me... Am I configuring the network
connector on the right broker? My thinking was that the proxy is the
one that should have the network connector configuration, because
messages sent to the proxy should be relayed to the receiver broker.
However, the reverse also makes sense... The receiver broker could be
the one that requires the network connector configuration because it's
subscribing to messages being sent to the proxy.

> Also, I'd suggest that you set up the main broker as a standalone broker
> instance.

I'm actually not sure that the main broker will be standalone, it'll
likely still be embedded. Logging is no problem, in any case, as the
hosting example program is using Logback with everything down to TRACE
level being logged. I'm not sure if a standalone broker gets me more
logging than this?

I'd like to stick to doing things programatically, as I think it gives
other people more of a chance to reproduce any problems I have (they
automatically get the same versions of everything I'm using via Maven
and so on).

> BTW, I may have led you down the wrong path on the mechanism for including
> destinations, because I forgot a very important advantage that statically
> included destinations have over dynamically included ones: messages will be
> forwarded to the main broker even when there are no consumers attached to
> the main broker.

Got it.

I'm not sure yet if I will be using queues, so I'll stick to using the
static configuration as it provides both.

--
Mark Raynsford | http://www.io7m.com

attachment0 (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Programatically setting up a proxying broker

tpavelka
I know nothing about static destinations but one thing I noticed in your Github code: when you add network and transport connectors, you do not start them. I tested this with one of my sample ActiveMQ programs and if I don't start the connectors then there are no messages flowing.

Try to use:
org.apache.activemq.broker.TransportConnector#start
and
org.apache.activemq.network.NetworkConnector#start

HTH,
Tomas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Programatically setting up a proxying broker

Mark Raynsford
'Ello.

On 2017-08-02T00:19:48 -0700
tpavelka <[hidden email]> wrote:

> I know nothing about static destinations but one thing I noticed in your
> Github code: when you add network and transport connectors, you do not start
> them. I tested this with one of my sample ActiveMQ programs and if I don't
> start the connectors then there are no messages flowing.

I see the following in the log without any start() calls:

08:06:50.280 INFO [main] org.apache.activemq.broker.TransportConnector: Connector tcp://[fd38:73b9:8748:8f82:0:0:0:2]:61616 started
08:06:50.370 INFO [main] org.apache.activemq.network.NetworkConnector: Network Connector com.io7m.activemq.examples.ForwardingServers$StaticNetworkConnector@5579bb86 started

... So I don't think that's the problem here. I've added a possibly-redundant
start() call to the network connector anyway. As I initialize the transport
connectors with URIs, I don't actually get direct access to instances upon which
I can call start(), so I've not added anything there.

--
Mark Raynsford | http://www.io7m.com

attachment0 (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Programatically setting up a proxying broker

tpavelka
Looks like this was a dead end. I found the reason I needed to start the connectors was because I added them after the broker was already running. I guess if you add them before you start the broker then you don't need to start them explicitly...

Tomas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Fixed] Programatically setting up a proxying broker

Mark Raynsford
In reply to this post by Mark Raynsford
I discovered what the problem was.

I'd created my own subclass of NetworkConnector, because none of the
existing implementations seemed to be what I needed. It turns out that
I actually needed to use DiscoveryNetworkConnector; the abstract
NetworkConnector base class doesn't actually do any useful work.

--
Mark Raynsford | http://www.io7m.com

attachment0 (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Fixed] Programatically setting up a proxying broker

Tim Bain
OK, glad to hear that it's working properly now.

But I'm surprised to hear that you think the existing classes don't do what
you want; what are you looking for that you couldn't get?

Tim

On Aug 3, 2017 4:38 AM, "Mark Raynsford" <[hidden email]>
wrote:

> I discovered what the problem was.
>
> I'd created my own subclass of NetworkConnector, because none of the
> existing implementations seemed to be what I needed. It turns out that
> I actually needed to use DiscoveryNetworkConnector; the abstract
> NetworkConnector base class doesn't actually do any useful work.
>
> --
> Mark Raynsford | http://www.io7m.com
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Fixed] Programatically setting up a proxying broker

Mark Raynsford
On 2017-08-03T06:29:48 -0600
Tim Bain <[hidden email]> wrote:

> OK, glad to hear that it's working properly now.
>
> But I'm surprised to hear that you think the existing classes don't do what
> you want; what are you looking for that you couldn't get?

Sorry, that was phrased poorly. What I meant was that the abstract
NetworkConnector class doesn't actually seem to implement much by
itself; all of the useful functionality (such as actually creating
connections to brokers) appears to be contained within the subclasses
such as DiscoveryNetworkConnector. I didn't realize this when I
subclassed it and so ended up with a connector that didn't really do
anything.

Thanks for your time!

--
Mark Raynsford | http://www.io7m.com

attachment0 (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Fixed] Programatically setting up a proxying broker

Tim Bain
My question was why you needed to subclass it in the first place.

On Aug 3, 2017 6:51 AM, "Mark Raynsford" <[hidden email]>
wrote:

> On 2017-08-03T06:29:48 -0600
> Tim Bain <[hidden email]> wrote:
>
> > OK, glad to hear that it's working properly now.
> >
> > But I'm surprised to hear that you think the existing classes don't do
> what
> > you want; what are you looking for that you couldn't get?
>
> Sorry, that was phrased poorly. What I meant was that the abstract
> NetworkConnector class doesn't actually seem to implement much by
> itself; all of the useful functionality (such as actually creating
> connections to brokers) appears to be contained within the subclasses
> such as DiscoveryNetworkConnector. I didn't realize this when I
> subclassed it and so ended up with a connector that didn't really do
> anything.
>
> Thanks for your time!
>
> --
> Mark Raynsford | http://www.io7m.com
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Fixed] Programatically setting up a proxying broker

Mark Raynsford
On 2017-08-03T08:57:38 -0600
Tim Bain <[hidden email]> wrote:
>
> My question was why you needed to subclass it in the first place.

Ah, that was down to an error in thinking on my part when looking at
the available network connectors. It went something like this: "Hm, need
a NetworkConnector. Don't seem to need 'discovery', don't seem to need
'multicast', don't seem to need LDAP... Guess I'm supposed to provide
my own subclass". I only realized my mistake when I tracked down which
subclass is instantiated for the broker when running the binary
distribution from the command line (DiscoveryNetworkConnector).

--
Mark Raynsford | http://www.io7m.com

attachment0 (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Fixed] Programatically setting up a proxying broker

clebertsuconic
There is a pending pull request in activemq artemis to add that feature.

On Thu, Aug 3, 2017 at 11:51 AM Mark Raynsford <
[hidden email]> wrote:

> On 2017-08-03T08:57:38 -0600
> Tim Bain <[hidden email]> wrote:
> >
> > My question was why you needed to subclass it in the first place.
>
> Ah, that was down to an error in thinking on my part when looking at
> the available network connectors. It went something like this: "Hm, need
> a NetworkConnector. Don't seem to need 'discovery', don't seem to need
> 'multicast', don't seem to need LDAP... Guess I'm supposed to provide
> my own subclass". I only realized my mistake when I tracked down which
> subclass is instantiated for the broker when running the binary
> distribution from the command line (DiscoveryNetworkConnector).
>
> --
> Mark Raynsford | http://www.io7m.com
>
--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Fixed] Programatically setting up a proxying broker

Tim Bain
In reply to this post by Mark Raynsford
Ah, that makes sense. Those class names are a little confusing, and so is
the term "discovery" since it sounds like it should mean "runtime discovery
of dynamic information" but static discovery actually means "discovery from
a config file."

Since it sounds like you aren't planning to continue using your subclass,
then no problem.

Tim

On Aug 3, 2017 9:51 AM, "Mark Raynsford" <[hidden email]>
wrote:

> On 2017-08-03T08:57:38 -0600
> Tim Bain <[hidden email]> wrote:
> >
> > My question was why you needed to subclass it in the first place.
>
> Ah, that was down to an error in thinking on my part when looking at
> the available network connectors. It went something like this: "Hm, need
> a NetworkConnector. Don't seem to need 'discovery', don't seem to need
> 'multicast', don't seem to need LDAP... Guess I'm supposed to provide
> my own subclass". I only realized my mistake when I tracked down which
> subclass is instantiated for the broker when running the binary
> distribution from the command line (DiscoveryNetworkConnector).
>
> --
> Mark Raynsford | http://www.io7m.com
>
12
Loading...