Virtual destination subscription durability and message recovery

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Virtual destination subscription durability and message recovery

Devlin
We've performed extensive testing with virtual destinations on 5.11 network
of brokers, it works well with two exceptions:

#1 - subscription durability when consumers go offline,

#2 - producers publishing before consumers register their subscription

We manage to get around #1 by configuring producers to publish messages
using PERSISTENT delivery mode, then we disabled cleanup of inactive virtual
queues using wildcard destination policies. Now, when consumer who
previously created a virtual queue subscription go offline, newly-published
messages are retained on those queues until they are consumed, or expire;
that's good.

But #2 has proved challenging; we thought about using retroactive recovery
policies, but after reading the documentation we discovered it's not
reliable in broker networks. We're not sure how to proceed; unless there's a
way to retain messages in memory, or on disk, reliably for short periods of
time (basically recovery policy) and for an unknown number of durable
consumers, I suspect there is no solution to this problem.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Virtual destination subscription durability and message recovery

gtully
#2 is a problem - when the name of the subscription queue will be unknown.
If you know the names in advance then statically creating the destinations
will work.

You could go down the plugin route and implement/manage a shared
retroactive queue SRQ

Intercept send to the virtual topic to forward a copy to the SRQ and set an
appropriate expiry.
Intercept ack to the SRQ to ignore any remote standard ack; on expiry the
ack will have the broker context.
Intercept subscribe to the virtual topic to use a composite destination to
add SRQ to the subscription.

The down side is that a repeated connect/disconnect would get duplicates
from the SRQ. A failover client would suppress but they would generally
need to be handled and ignored.

It would be a nice feature. Essentially SRQ is some sort of time bounded
queue. A more general bounded queue would be better (bounded on size,
memory or time) but for your use case maybe the plugin is the way to go.


On Tue, 14 Nov 2017 at 00:02 Devlin <[hidden email]> wrote:

> We've performed extensive testing with virtual destinations on 5.11 network
> of brokers, it works well with two exceptions:
>
> #1 - subscription durability when consumers go offline,
>
> #2 - producers publishing before consumers register their subscription
>
> We manage to get around #1 by configuring producers to publish messages
> using PERSISTENT delivery mode, then we disabled cleanup of inactive
> virtual
> queues using wildcard destination policies. Now, when consumer who
> previously created a virtual queue subscription go offline, newly-published
> messages are retained on those queues until they are consumed, or expire;
> that's good.
>
> But #2 has proved challenging; we thought about using retroactive recovery
> policies, but after reading the documentation we discovered it's not
> reliable in broker networks. We're not sure how to proceed; unless there's
> a
> way to retain messages in memory, or on disk, reliably for short periods of
> time (basically recovery policy) and for an unknown number of durable
> consumers, I suspect there is no solution to this problem.
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Virtual destination subscription durability and message recovery

Devlin
Thanks, gtully!

Is our implementation for #1 correct? The assumption is by disabling removal
of inactive virtual destination queues, we effectively make them static
queues the moment they are created. So, assuming a producer publishes to
"topic://virtualTopics.mytopic" on BrokerA, if BrokerB has
queue://virtualTopics.mytopic.sub1" but no active consumers, will BrokerA
forward messages to BrokerB if the brokers are networked?



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Virtual destination subscription durability and message recovery

Tim Bain
Networks of brokers with only dynamically included destinations will only
forward messages when a consumer is present; otherwise, the messages will
stay on the broker to which they were published (BrokerA, in your example)
until a consumer connects to another broker in the network. But if you do
one of the things that Gary mentioned and you statically include your
destinations, then yes, messages will flow across the network as specified
by the statically included destinations, even if no consumers are present.

Tim

On Wed, Nov 15, 2017 at 9:17 PM, Devlin <[hidden email]> wrote:

> Thanks, gtully!
>
> Is our implementation for #1 correct? The assumption is by disabling
> removal
> of inactive virtual destination queues, we effectively make them static
> queues the moment they are created. So, assuming a producer publishes to
> "topic://virtualTopics.mytopic" on BrokerA, if BrokerB has
> queue://virtualTopics.mytopic.sub1" but no active consumers, will BrokerA
> forward messages to BrokerB if the brokers are networked?
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>