Virtual topics, custom prefix limitations

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Virtual topics, custom prefix limitations

Devlin
When customizing virtual topic consumer "prefix", can I use wildcards?

I know this works:

<virtualDestinations>
    <virtualTopic name="*.*.*.virtual" prefix="Consumers.*." />
</virtualDestinations>

..not sure if this would work:

<virtualDestinations>
    <virtualTopic name="*.*.*.virtual" prefix="*.*.*.virtual.*"/>
</virtualDestinations>
Reply | Threaded
Open this post in threaded view
|

Re: Virtual topics, custom prefix limitations

Tim Bain
This blog post (
http://workingwithqueues.blogspot.com/2012/05/activemq-virtual-topics-or-virtual.html?m=1)
seems to indicate that you can.  Please let us know if it works.

On Oct 4, 2016 8:31 AM, "Devlin" <[hidden email]> wrote:

> When customizing virtual topic consumer "prefix", can I use wildcards?
>
> I know this works:
>
> <virtualDestinations>
>     <virtualTopic name="*.*.*.virtual" prefix="Consumers.*." />
> </virtualDestinations>
>
> ..not sure if this /would/ work:
>
> <virtualDestinations>
>     <virtualTopic name="*.*.*.virtual" prefix="*.*.*.virtual.*"/>
> </virtualDestinations>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Virtual-topics-custom-prefix-limitations-tp4717481.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Virtual topics, custom prefix limitations

Devlin
This post was updated on .
Hi Tim,

We managed to get working virtual topics in a 2 broker network, but only for cases where producer and consumers are using the same broker; consumers connected to the broker where the producer is not present did not receive messages. We confirmed the network functions as expected for standard topics/queues, but not for virtual topics.

I found this in the docs, looks interesting, but appears to be for 5.13, we're on 5.11.
<broker name="remoteBroker"
           useVirtualDestSubs="true"
           useVirtualDestSubsOnCreation="true"> 



Broker Config
=============================================================

       <destinationInterceptors>
            <virtualDestinationInterceptor>
                <virtualDestinations>
                    <virtualTopic name="*.*.*._vtopic.>"  prefix="*.*.*._vsub.*."  selectorAware="true"/>
                </virtualDestinations>
            </virtualDestinationInterceptor>
        </destinationInterceptors>

        <transportConnectors>
           <transportConnector
                name="owire"
                uri="tcp://0.0.0.0:61616"
                updateClusterClients="true"
                updateClusterClientsOnRemove="true"/>
        </transportConnectors>

        <networkConnectors>
            <networkConnector
                name="queueNetwork"
                userName="${auth.user}"
                password="${auth.password}"                      
                uri="${network.uri.1}?${network.options}"
                consumerTTL="1"
                messageTTL="100"
                conduitSubscriptions="false"
                decreaseNetworkConsumerPriority="true"
                suppressDuplicateQueueSubscriptions="true">
                <dynamicallyIncludedDestinations>
                    <queue physicalName=">"/>
                </dynamicallyIncludedDestinations>
                <excludedDestinations>
                    <queue physicalName="*.*.*._vsub.>"/>
                </excludedDestinations>
            </networkConnector>           
            <networkConnector
                name="topicNetwork"
                userName="${auth.user}"
                password="${auth.password}"                      
                uri="${network.uri.1}?${network.options}"
                consumerTTL="1"
                messageTTL="100"                
                decreaseNetworkConsumerPriority="true">
                <dynamicallyIncludedDestinations>
                    <topic physicalName=">"/>
                </dynamicallyIncludedDestinations>
            </networkConnector> 
        </networkConnectors>

        <destinationPolicy>
          <policyMap>
            <policyEntries>
              <policyEntry
                  topic=">"
                  producerFlowControl="true"
                  gcInactiveDestinations="true"
                  inactiveTimoutBeforeGC="600000">
                <pendingMessageLimitStrategy>
                  <constantPendingMessageLimitStrategy
                      limit="5000"/>
                </pendingMessageLimitStrategy>
              </policyEntry>
              <policyEntry
                  queue=">"
                  producerFlowControl="true"
                  memoryLimit="20mb"
                  enableAudit="false"
                  gcInactiveDestinations="true"
                  inactiveTimoutBeforeGC="600000">
                  <networkBridgeFilterFactory>
                    <conditionalNetworkBridgeFilterFactory
                      replayWhenNoConsumers="true"
                      replayDelay="1000" />
                  </networkBridgeFilterFactory>
              </policyEntry>
            </policyEntries>
          </policyMap>
        </destinationPolicy>

Reply | Threaded
Open this post in threaded view
|

Re: Virtual topics, custom prefix limitations

Tim Bain
The description on the Network of Brokers wiki page says that what you're
trying to do won't work without those flags set, so you should upgrade if
this is a feature you need to use.

Tim

On Oct 13, 2016 6:33 PM, "Devlin" <[hidden email]> wrote:

> Hi Tim,
>
> We managed to get working virtual topics in a 2 broker network, but only
> for
> cases where producer and consumers are using the same broker; consumers
> connected to the broker where the producer is not present did not receive
> messages. We confirmed the network functions as expected for standard
> topics/queues, but not for virtual topics.
>
> I found this in the docs, looks interesting, not sure if it would help, but
> regardless, it's for 5.13, we're on 5.11.
>  <broker name="remoteBroker" useVirtualDestSubs="true"
> useVirtualDestSubsOnCreation="true">
>      .....
>   </broker>
>
> Here's our config:
>        <destinationInterceptors>
>             <virtualDestinationInterceptor>
>                 <virtualDestinations>
>                     <virtualTopic name="*.*.*._vtopic.>"
> prefix="*.*.*._vsub.*."  selectorAware="true"/>
>                 </virtualDestinations>
>             </virtualDestinationInterceptor>
>         </destinationInterceptors>
>
>         <transportConnectors>
>            <transportConnector
>                 name="owire"
>                 uri="tcp://0.0.0.0:61616"
>                 updateClusterClients="true"
>                 updateClusterClientsOnRemove="true"/>
>         </transportConnectors>
>
>         <networkConnectors>
>             <networkConnector
>                 name="queueNetwork"
>                 userName="${auth.user}"
>                 password="${auth.password}"
>                 uri="${network.uri.1}?${network.options}"
>                 consumerTTL="1"
>                 messageTTL="100"
>                 conduitSubscriptions="false"
>                 decreaseNetworkConsumerPriority="true"
>                 suppressDuplicateQueueSubscriptions="true">
>                 <dynamicallyIncludedDestinations>
>                     <queue physicalName=">"/>
>                 </dynamicallyIncludedDestinations>
>                 <excludedDestinations>
>                     <queue physicalName="*.*.*._vsub.>"/>
>                 </excludedDestinations>
>             </networkConnector>
>             <networkConnector
>                 name="topicNetwork"
>                 userName="${auth.user}"
>                 password="${auth.password}"
>                 uri="${network.uri.1}?${network.options}"
>                 consumerTTL="1"
>                 messageTTL="100"
>                 decreaseNetworkConsumerPriority="true">
>                 <dynamicallyIncludedDestinations>
>                     <topic physicalName=">"/>
>                 </dynamicallyIncludedDestinations>
>             </networkConnector>
>         </networkConnectors>
>
>         <destinationPolicy>
>           <policyMap>
>             <policyEntries>
>               <policyEntry
>                   topic=">"
>                   producerFlowControl="true"
>                   gcInactiveDestinations="true"
>                   inactiveTimoutBeforeGC="600000">
>                 <pendingMessageLimitStrategy>
>                   <constantPendingMessageLimitStrategy
>                       limit="5000"/>
>                 </pendingMessageLimitStrategy>
>               </policyEntry>
>               <policyEntry
>                   queue=">"
>                   producerFlowControl="true"
>                   memoryLimit="20mb"
>                   enableAudit="false"
>                   gcInactiveDestinations="true"
>                   inactiveTimoutBeforeGC="600000">
>                   <networkBridgeFilterFactory>
>                     <conditionalNetworkBridgeFilterFactory
>                       replayWhenNoConsumers="true"
>                       replayDelay="1000" />
>                   </networkBridgeFilterFactory>
>               </policyEntry>
>             </policyEntries>
>           </policyMap>
>         </destinationPolicy>
>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Virtual-topics-custom-prefix-limitations-tp4717481p4717887.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Virtual topics, custom prefix limitations

Devlin
Tim,

The documentation on this topic (haha!) was never 100% clear to me. Here's what we want to achieve given our architecture:

Topology
--------
12 broker network (full-mesh)
All destinations dynamic (no static definitions)
Clients connect and reconnect with random brokers (preference to local)
AMQ 6.2.1 (preping for 6.3)

What we need
---------------
+ Virtual topics working in above architecture.
* Virtual topics to support queue *and* normal topic subscribers (can live without this if duplication cannot be avoided)

It's my understanding AMQ 6.3 still uses ActiveMQ 5.11, bummer if that's true, so as you said, we can't use useVirtualDestSubs broker params.

Question
--------
If we statically-defined all virtual topics (we don't have many), can we meet the requirements above?
Reply | Threaded
Open this post in threaded view
|

Re: Virtual topics, custom prefix limitations

Tim Bain
I believe that listing all virtual topics in the
staticallyIncludedDestinations element would work.  Please let us know if
it works, so that the next person knows whether this is an option for them.

Tim

On Oct 14, 2016 9:07 PM, "Devlin" <[hidden email]> wrote:

> Tim,
>
> The documentation on this topic (haha!) was never 100% clear to me. Here's
> what we want to achieve given our architecture:
>
> Topology
> --------
> 12 broker network (full-mesh)
> All destinations dynamic (no static definitions)
> Clients connect and reconnect with random brokers (preference to local)
> AMQ 6.2.1 (preping for 6.3)
>
> What we need
> ---------------
> + Virtual topics working in above architecture.
> * Virtual topics to support queue *and* normal topic subscribers (can live
> without this if duplication cannot be avoided)
>
> It's my understanding AMQ 6.3 still uses ActiveMQ 5.11, bummer if that's
> true, so as you said, we can't use useVirtualDestSubs broker params.
>
> Question
> --------
> If we statically-defined all virtual topics (we don't have many), can we
> meet the requirements above?
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Virtual-topics-custom-prefix-limitations-tp4717481p4717957.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Virtual topics, custom prefix limitations

Devlin
Ok, just to be clear, we define the virtual topic statically (the one used by the producer), not the individual consumer queues. I hope it's the former because client queues for virtual topics are named using "app-version" convention for grouping related consumers; we can't define those names statically as they're too volatile. Will report back on this.
Reply | Threaded
Open this post in threaded view
|

Re: Virtual topics, custom prefix limitations

Tim Bain
Correct, statically include the topics, not the queues.

On Oct 15, 2016 9:27 AM, "Devlin" <[hidden email]> wrote:

> Ok, just to be clear, we define the virtual topic statically (the one used
> by
> the producer), not the individual consumer queues. I hope it's the former
> because client queues for virtual topics are named using "app-version"
> convention for grouping related consumers; we can't define those names
> statically as they're too volatile. Will report back on this.
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Virtual-topics-custom-prefix-limitations-tp4717481p4717968.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Virtual topics, custom prefix limitations

Tim Bain
BTW, do you have consumers consuming directly from the virtual topic, or
only via queues?  If you have no topic consumers, I believe you can just
remove your excludedDestinations entry and everything will work as
intended.  As I understand it,  the problems only occur when you have a mix
of queue and topic consumers.

On Oct 15, 2016 6:53 PM, "Tim Bain" <[hidden email]> wrote:

> Correct, statically include the topics, not the queues.
>
> On Oct 15, 2016 9:27 AM, "Devlin" <[hidden email]> wrote:
>
>> Ok, just to be clear, we define the virtual topic statically (the one
>> used by
>> the producer), not the individual consumer queues. I hope it's the former
>> because client queues for virtual topics are named using "app-version"
>> convention for grouping related consumers; we can't define those names
>> statically as they're too volatile. Will report back on this.
>>
>>
>>
>> --
>> View this message in context: http://activemq.2283324.n4.nab
>> ble.com/Virtual-topics-custom-prefix-limitations-tp4717481p4717968.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Virtual topics, custom prefix limitations

Devlin
We may need to support standard subscribers to the virtual topic, but it's not a drop-dead requirement.
Having said that, we verified the broker network is working for standard queues/topics using the above configuration, but not virtual topics, even after removing <excludedDestinations>. VT's are only working when producers AND consumers are all connected to the same broker.
Reply | Threaded
Open this post in threaded view
|

Re: Virtual topics, custom prefix limitations

Tim Bain
I could have sworn I had read something on the wiki that indicated this
used to work (and that failing to exclude the queues would result in
duplicate messages), but I can't find it so maybe I'm thinking of something
else.  Thanks for humoring my faulty memory.

So what happened when you included the topics in the
staticallyIncludedDestinations element?

On Oct 15, 2016 11:18 PM, "Devlin" <[hidden email]> wrote:

We may need to support standard subscribers to the virtual topic, but it's
not a drop-dead requirement.
Having said that, we verified the broker network is working for standard
queues/topics using the above configuration, but not virtual topics, even
after removing <excludedDestinations>. VT's are only working when producers
AND consumers are all connected to the same broker.



--
View this message in context: http://activemq.2283324.n4.
nabble.com/Virtual-topics-custom-prefix-limitations-tp4717481p4717975.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.