Duplicate messages received with ActiveMQ 5.13.2

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Duplicate messages received with ActiveMQ 5.13.2

tpavelka
I have a problem where I produce a TextMessage to a topic and the consumer receives it twice. I only have one system where I can reproduce this and I would appreciate any tips on how to debug this further.

I have four JVMs, let's call them A, B, C and D. Each one runs an ActiveMQ broker. The connection topology looks like this:

A -> B
C -> B
D -> B

I.e. B is the only one with broker that runs transport connectors.

Let's say that I have topic T, where C and D have subscribers with AUTO_ACKNOWLEDGE.

- If A sends a text message to T, then C receives it twice
- If D is not connected then C receives it only once.

I put some debug logs in C and so far I have learned the following about the two instances received:

1) The messages arrive about 200 ms apart.
2) Both messages have the same id as returned by javax.jms.Message#getJMSMessageID
3) Both have the same connection as reported by org.apache.activemq.command.Message#getConnection
4) Both have redelivered as false as reported by org.apache.activemq.command.ActiveMQMessage#getJMSRedelivered
5) Both have redeliver counter = 0 as reported by org.apache.activemq.command.Message#getRedeliveryCounter
6) The messages have different broker intime, as reported by org.apache.activemq.command.Message#getBrokerInTime - The first has 1500305481165 and the second has 1500305481166.

I have tried to reproduce this elsewhere without any success and I am near the end of my wits.

Thanks for any debugging tips.

Tomas  
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Duplicate messages received with ActiveMQ 5.13.2

Tim Bain
Do you have conduitSubscriptions set to false for your networkConnectors?

Or are your topics virtual and you're dynamically including both the topic
and the consumer queues?

Both of those could result in duplicate messages on a topic.

Tim

On Jul 18, 2017 6:36 AM, "tpavelka" <[hidden email]> wrote:

> I have a problem where I produce a TextMessage to a topic and the consumer
> receives it twice. I only have one system where I can reproduce this and I
> would appreciate any tips on how to debug this further.
>
> I have four JVMs, let's call them A, B, C and D. Each one runs an ActiveMQ
> broker. The connection topology looks like this:
>
> A -> B
> C -> B
> D -> B
>
> I.e. B is the only one with broker that runs transport connectors.
>
> Let's say that I have topic T, where C and D have subscribers with
> AUTO_ACKNOWLEDGE.
>
> - If A sends a text message to T, then C receives it twice
> - If D is not connected then C receives it only once.
>
> I put some debug logs in C and so far I have learned the following about
> the
> two instances received:
>
> 1) The messages arrive about 200 ms apart.
> 2) Both messages have the same id as returned by
> javax.jms.Message#getJMSMessageID
> 3) Both have the same connection as reported by
> org.apache.activemq.command.Message#getConnection
> 4) Both have redelivered as false as reported by
> org.apache.activemq.command.ActiveMQMessage#getJMSRedelivered
> 5) Both have redeliver counter = 0 as reported by
> org.apache.activemq.command.Message#getRedeliveryCounter
> 6) The messages have different broker intime, as reported by
> org.apache.activemq.command.Message#getBrokerInTime - The first has
> 1500305481165 and the second has 1500305481166.
>
> I have tried to reproduce this elsewhere without any success and I am near
> the end of my wits.
>
> Thanks for any debugging tips.
>
> Tomas
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-tp4728627.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Duplicate messages received with ActiveMQ 5.13.2

tpavelka
I am using just regular topics. In my example above C and D connect to their embedded brokers via VM transport, create a consumer and register a listener.

I have debugged it just to make sure and conduitSubscriptions are set to false, i.e. org.apache.activemq.network.NetworkBridgeFactory#createBridge returns an instance of org.apache.activemq.network.DurableConduitBridge.

Is there anything I could do to trace the flow to see where the message gets duplicated?

Thanks,
Tomas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Duplicate messages received with ActiveMQ 5.13.2

Tim Bain
If you set conduitSubscriptions to true, does the behavior become the
expected behavior? If so, you can read a description of why having it false
would cause this to happen halfway down
http://activemq.apache.org/networks-of-brokers.html.

You can determine where the duplication occurs by watching the enqueue
counts on each broker when you send a single message. You can see those
counts in the web console or via JMX.

Tim

On Jul 19, 2017 1:17 AM, "tpavelka" <[hidden email]> wrote:

> I am using just regular topics. In my example above C and D connect to
> their
> embedded brokers via VM transport, create a consumer and register a
> listener.
>
> I have debugged it just to make sure and conduitSubscriptions are set to
> false, i.e. org.apache.activemq.network.NetworkBridgeFactory#createBridge
> returns an instance of org.apache.activemq.network.DurableConduitBridge.
>
> Is there anything I could do to trace the flow to see where the message
> gets
> duplicated?
>
> Thanks,
> Tomas
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-
> tp4728627p4728671.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Duplicate messages received with ActiveMQ 5.13.2

tpavelka
Sorry, I messed up my boolean logic when I wrote my reply. I meant to write that I have verified that my conduitSubscriptions is set to TRUE so this should not be the cause. By mistake I have typed false...

I'll try to watch the the enqueue counts to see if I can figure out the origin.

Thanks,
Tomas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Duplicate messages received with ActiveMQ 5.13.2

tpavelka
I can now reliably reproduce the duplication and I can prevent it. However, the behavior leaves me a bit puzzled:

The issue is that the broker adds the message twice to the subscription, via org.apache.activemq.broker.region.TopicSubscription#add

There is a check for duplicate messages, org.apache.activemq.broker.region.TopicSubscription#isDuplicate which only works if org.apache.activemq.broker.region.TopicSubscription#enableAudit is enabled (set to true).

enableAutdit is false by default and thus I am getting duplicate messages. However, if I add a policy that applies to the topic, then policies have org.apache.activemq.broker.region.policy.PolicyEntry#enableAudit which is true by default, so any policy applied to the destination will turn audit on and prevent duplicated messages.

I have a couple of questions:

1) Is getting duplicate messages expected behavior, if conduitSubscribers is set to true? (i.e. when I am using org.apache.activemq.network.DurableConduitBridge)?

2) Why does the auditing default change when policies are applied? Or, what is the reason that TopicSubscription has audit off by default?

Thanks,
Tomas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Duplicate messages received with ActiveMQ 5.13.2

Tim Bain
I can't answer question #2, so I asked the dev list if someone could answer
it.

For #1, what you're seeing is not expected when conduitSubscriptions is in
use. In fact, this behavior is expected when someone fails to use
conduitSubscriptions, so it's odd that you're seeing it with
conduitSubscriptions enabled. Your earlier messages sounded like you
debugged the C-B and D-B links to confirm that conduitSubscriptions was in
used there, but have you confirmed that conduitSubscriptions is also being
applied on the A-B link? That's the only one where it matters.

Tim

Did you confirm which broker is the one where the duplication occurs? Also,
does broker D also get two copies of the message, or does it get one or
zero?

On Jul 25, 2017 6:40 AM, "tpavelka" <[hidden email]> wrote:

> I can now reliably reproduce the duplication and I can prevent it. However,
> the behavior leaves me a bit puzzled:
>
> The issue is that the broker adds the message twice to the subscription,
> via
> org.apache.activemq.broker.region.TopicSubscription#add
>
> There is a check for duplicate messages,
> org.apache.activemq.broker.region.TopicSubscription#isDuplicate which only
> works if org.apache.activemq.broker.region.TopicSubscription#enableAudit
> is
> enabled (set to true).
>
> enableAutdit is false by default and thus I am getting duplicate messages.
> However, if I add a policy that applies to the topic, then policies have
> org.apache.activemq.broker.region.policy.PolicyEntry#enableAudit which is
> true by default, so any policy applied to the destination will turn audit
> on
> and prevent duplicated messages.
>
> I have a couple of questions:
>
> 1) Is getting duplicate messages expected behavior, if conduitSubscribers
> is
> set to true? (i.e. when I am using
> org.apache.activemq.network.DurableConduitBridge)?
>
> 2) Why does the auditing default change when policies are applied? Or, what
> is the reason that TopicSubscription has audit off by default?
>
> Thanks,
> Tomas
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-
> tp4728627p4728871.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Duplicate messages received with ActiveMQ 5.13.2

tpavelka
I plan to debug further, but right now I have statistics for one message sent from A to topic T. This is best shown in a table, but I don't want to mess up formatting in email clients, so here it is in CSV:

With audit on:

audit on,A,B,C,D,,
Enqueue count,1,2,1,1
Dequeue count,2,2,1,1
Dispatch count,2,2,1,1
Forward count,2,2,0,0
Inflight count,0,0,0,0
Consumer count,2,2,2,2

With audit off:

audit off,A,B,C,D
Enqueue count,1,2,2,2
Dequeue count,2,4,2,2
Dispatch count,2,4,2,2
Forward count,2,4,0,0
Inflight count,0,0,0,0
Consumer count,2,2,2,2

I can reliably reproduce this now, but I am using some proprietary wrappers which I can't share. If I can't find the root cause in reasonable time I can try to reproduce this in plain ActiveMQ.

Thanks for the help!

Tomas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Duplicate messages received with ActiveMQ 5.13.2

Tim Bain
Those stats clearly show that the duplication occurs when forwarding from A
to B, so my question of whether conduitSubscriptions is in use for that
link is even more pertinent. My guess is that you'll find that it's not,
and that enabling it fixes this behavior.

Tim

On Jul 26, 2017 7:13 AM, "tpavelka" <[hidden email]> wrote:

> I plan to debug further, but right now I have statistics for one message
> sent
> from A to topic T. This is best shown in a table, but I don't want to mess
> up formatting in email clients, so here it is in CSV:
>
> With audit on:
>
> audit on,A,B,C,D,,
> Enqueue count,1,2,1,1
> Dequeue count,2,2,1,1
> Dispatch count,2,2,1,1
> Forward count,2,2,0,0
> Inflight count,0,0,0,0
> Consumer count,2,2,2,2
>
> With audit off:
>
> audit off,A,B,C,D
> Enqueue count,1,2,2,2
> Dequeue count,2,4,2,2
> Dispatch count,2,4,2,2
> Forward count,2,4,0,0
> Inflight count,0,0,0,0
> Consumer count,2,2,2,2
>
> I can reliably reproduce this now, but I am using some proprietary wrappers
> which I can't share. If I can't find the root cause in reasonable time I
> can
> try to reproduce this in plain ActiveMQ.
>
> Thanks for the help!
>
> Tomas
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-
> tp4728627p4728926.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Duplicate messages received with ActiveMQ 5.13.2

tpavelka
I double checked just to make sure:

There is this code in org.apache.activemq.network.NetworkBridgeFactory

    public static DemandForwardingBridge createBridge(NetworkBridgeConfiguration configuration,
                                                      Transport localTransport, Transport remoteTransport,
                                                      final NetworkBridgeListener listener) {
        DemandForwardingBridge result = null;
        if (configuration.isConduitSubscriptions()) {
            // dynamicOnly determines whether durables are auto bridged
            result = new DurableConduitBridge(configuration, localTransport, remoteTransport);
        } else {
            result = new DemandForwardingBridge(configuration, localTransport, remoteTransport);
        }
        if (listener != null) {
            result.setNetworkBridgeListener(listener);
        }
        return result;
    }

I put a breakpoint in and made it display the result of configuration.isConduitSubscriptions(), it shows as true for every call for every end point.

Can you think of any other place that I may have missed that could change the conduitSubscribers behavior?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Duplicate messages received with ActiveMQ 5.13.2

Tim Bain
I don't know of another place where it would be configured, so I think the
more likely explanation is that there's a bug causing the second
subscription to not be conduited onto the first.

But let's first let's confirm that we actually have two subscriptions on A
(because if we don't, then something else is going on). Can you please
attach a JMX viewer such as JConsole to broker A, and navigate the MBeans
to find your topic, and drill into it within the tree to see how many
subscriptions are associated with it? You should see a single subscription
if conduitSubscriptions is working properly, but I expect that you'll see
two.

Tim

On Jul 26, 2017 8:04 AM, "tpavelka" <[hidden email]> wrote:

> I double checked just to make sure:
>
> There is this code in org.apache.activemq.network.NetworkBridgeFactory
>
>     public static DemandForwardingBridge
> createBridge(NetworkBridgeConfiguration configuration,
>                                                       Transport
> localTransport, Transport remoteTransport,
>                                                       final
> NetworkBridgeListener listener) {
>         DemandForwardingBridge result = null;
>         if (configuration.isConduitSubscriptions()) {
>             // dynamicOnly determines whether durables are auto bridged
>             result = new DurableConduitBridge(configuration,
> localTransport,
> remoteTransport);
>         } else {
>             result = new DemandForwardingBridge(configuration,
> localTransport, remoteTransport);
>         }
>         if (listener != null) {
>             result.setNetworkBridgeListener(listener);
>         }
>         return result;
>     }
>
> I put a breakpoint in and made it display the result of
> configuration.isConduitSubscriptions(), it shows as true for every call
> for
> every end point.
>
> Can you think of any other place that I may have missed that could change
> the conduitSubscribers behavior?
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-
> tp4728627p4728931.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Duplicate messages received with ActiveMQ 5.13.2

tpavelka
The tables I sent before have a row named "Number of consumers". These are from org.apache.activemq.broker.jmx.DestinationViewMBean#getConsumerCount

I also put a breakpoint in org.apache.activemq.broker.region.Topic#dispatch
And logged the number of consumers (org.apache.activemq.broker.region.Topic#consumers.length()), on every broker (including A) it was always 2 (of type org.apache.activemq.broker.region.TopicSubscription).

Is this what you mean when you say "we have two subscriptions on A"?

Also, can you confirm that my understanding of conduitSubscriptions is correct?
The way I understand it is that if there is a broker (X) who sends a message to another broker (Y) that has multiple subscriptions for it then X tracks it as just one subscription so that it does not have to send the message multiple times.

I am trying to debug where exactly does the decision not to conduit get made but I am still kind of lost in the code.

Tomas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Duplicate messages received with ActiveMQ 5.13.2

tpavelka
BTW here is where I am getting stuck:
I was assuming that brokers must somehow learn about consumers on other brokers. I looked at the code of org.apache.activemq.network.ConduitBridge and later org.apache.activemq.network.DemandForwardingBridge and it seemed to me that the means by which the brokers learn about consumers is by sending and receiving instances of org.apache.activemq.command.ConsumerInfo.
So I put a breakpoint in org.apache.activemq.transport.TransportSupport#doConsume and I log every instance of ConsumerInfo. But I am only seeing ConsumerInfo's for some advisory topics and not for the topic which I am sending messages to.
So either I don't have the tracing setup right or there is some other channel via which brokers exchange information about consumers.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Duplicate messages received with ActiveMQ 5.13.2

Tim Bain
In reply to this post by tpavelka
Sorry, I was so focused on the difference between the enqueue and dequeue
counts that I missed that row in your earlier post.

So that confirms that despite using a DurableConduitBridge, the
subscriptions are not actually being combined, which is indeed what I was
asking. (And yes, your understanding of what happens when subscriptions are
conduited is exactly right.)

The only way that brokers know to forward messages across a dynamically
included networkConnector is by getting messages on the appropriate
advisory topic for a given destination. I've never looked at what type
those messages have, but ConsumerInfo sounds plausible. But if you see the
consumer count increasing without hitting your breakpoint, then that
doesn't match my mental model of how the network bridges work.

I probably won't have time to go through the code before the weekend, but
if you don't have it figured out by then, I can see if I can pinpoint where
you'd want to set your breakpoint to catch the decision about whether to
conduit or not.

Tim

On Jul 27, 2017 7:37 AM, "tpavelka" <[hidden email]> wrote:

> The tables I sent before have a row named "Number of consumers". These are
> from org.apache.activemq.broker.jmx.DestinationViewMBean#getConsumerCount
>
> I also put a breakpoint in org.apache.activemq.broker.
> region.Topic#dispatch
> And logged the number of consumers
> (org.apache.activemq.broker.region.Topic#consumers.length()), on every
> broker (including A) it was always 2 (of type
> org.apache.activemq.broker.region.TopicSubscription).
>
> Is this what you mean when you say "we have two subscriptions on A"?
>
> Also, can you confirm that my understanding of conduitSubscriptions is
> correct?
> The way I understand it is that if there is a broker (X) who sends a
> message
> to another broker (Y) that has multiple subscriptions for it then X tracks
> it as just one subscription so that it does not have to send the message
> multiple times.
>
> I am trying to debug where exactly does the decision not to conduit get
> made
> but I am still kind of lost in the code.
>
> Tomas
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-
> tp4728627p4728962.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Duplicate messages received with ActiveMQ 5.13.2

tpavelka
I figured out why I am not seeing ConsumerInfo coming: it comes in wrapped in ActiveMQMessage (an advisory message) and the ConsumerInfo is contained within as a dataStructure.

I don't know why conduit is not happening but I can now trace the code flow to within ConduitBridge.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Duplicate messages received with ActiveMQ 5.13.2

Tim Bain
Can you set a breakpoint at ConduitBridge:60 and see whether
info.isNetworkSubscription() evaluates to true (i.e. whether you go into
the then block, and therefore return false)? If I'm understanding that code
correctly, I think that may be the reason why the connections aren't being
conduited.

Tim

On Thu, Jul 27, 2017 at 8:54 AM, tpavelka <[hidden email]> wrote:

> I figured out why I am not seeing ConsumerInfo coming: it comes in wrapped
> in
> ActiveMQMessage (an advisory message) and the ConsumerInfo is contained
> within as a dataStructure.
>
> I don't know why conduit is not happening but I can now trace the code flow
> to within ConduitBridge.
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-
> tp4728627p4728966.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Duplicate messages received with ActiveMQ 5.13.2

tpavelka
In my broker "B" which is the hub in the hub & spoke topology, and in org.apache.activemq.network.ConduitBridge#addToAlreadyInterestedConsumers

There is this code:

protected boolean addToAlreadyInterestedConsumers(ConsumerInfo info) {
        // search through existing subscriptions and see if we have a match
        if (info.isNetworkSubscription()) {
            return false;
        }
        boolean matched = false;

        for (DemandSubscription ds : subscriptionMapByLocalId.values()) {
            DestinationFilter filter = DestinationFilter.parseFilter(ds.getLocalInfo().getDestination());
            if (canConduit(ds) && filter.matches(info.getDestination())) {


I get two ConsumerInfo instances (as expected, from "C" and "D"), each of these have isNetworkSubscription = false

On top of that, canConduit(ds) is true but filter.matches(info.getDestination()) evaluates to false in both cases.

I would expect that when the first one comes, the subscriptionMapByLocalId would get updated and the second one would result in a conduit. But that does not happen.

I suspect that the glitch is in subscriptionMapByLocalId not getting updated when it should...

Tomas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Duplicate messages received with ActiveMQ 5.13.2

tpavelka
I got a little further today:
The reason that conduit does not happen is that even though ConsumerInfos come twice for the same topic, they come over different transport connections and thus come to different bridges. For conduit to happen it must happen within one bridge.
Also, I noticed that when A,C and D (spokes) connect to B (hub) they do so over duplex connections. I noticed some special processing for duplex connections, namely that it creates additional bridges, in org.apache.activemq.broker.TransportConnection#processBrokerInfo.

Tomas
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Duplicate messages received with ActiveMQ 5.13.2

Tim Bain
Wait, there are multiple transport connections between A and B? Why? That
right there seems like a problem, and might be the root cause of the
behavior you're seeing.

The code I see in that processBrokerInfo() method creates the inverse side
of the duplex network connection (under the hood it's not a single two-way
connection, but rather two one-way connections), but I don't see anything
that seems concerning about what it's doing.

Tim

On Mon, Jul 31, 2017 at 8:10 AM, tpavelka <[hidden email]> wrote:

> I got a little further today:
> The reason that conduit does not happen is that even though ConsumerInfos
> come twice for the same topic, they come over different transport
> connections and thus come to different bridges. For conduit to happen it
> must happen within one bridge.
> Also, I noticed that when A,C and D (spokes) connect to B (hub) they do so
> over duplex connections. I noticed some special processing for duplex
> connections, namely that it creates additional bridges, in
> org.apache.activemq.broker.TransportConnection#processBrokerInfo.
>
> Tomas
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Duplicate-messages-received-with-ActiveMQ-5-13-2-
> tp4728627p4729068.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Duplicate messages received with ActiveMQ 5.13.2

tpavelka
I finally understand where the conduit should happen. I was of the impression that the conduit happens in broker B (hub) when subscriptions come from C and D. But that can't happen because there are two bridges (B-C and B-D) and conduit needs to happen within one bridge.
The conduit needs to happen in the bridge A-B. The reason it does not happen is that the two ConsumerInfos that come (over one transport connection) have isNetworkSubscription() returning true.

I suspect the networkSubscription is true because the subscription passed through two brokers (B, C or B, D). Moreover, if I skip the embedded broker in C and D then networkSubscription is false and conduit happens within the A-B bridge and there is no message duplication.

Moreover, if I don't use the embedded broker in A and use it in C and D, then there is no conduit but also no duplication.

So this topology:

spoke [broker] -> hub [broker] <- spoke [broker]
                                          <- spoke [broker]

Leads to message duplication,

But those two don't:

spoke             -> hub [broker] <- spoke [broker]
                                          <- spoke [broker]

spoke [broker] -> hub [broker] <- spoke
                                          <- spoke
12
Loading...