Duplicating messages published to a Topic to a Queue

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Duplicating messages published to a Topic to a Queue

Lachezar Dobrev
  Hello all,

  I have a design problem: a topic is clustered in a network of brokers to
allow multiple nodes to handle events occurring in other nodes: i.e. a user
logs in on one node, but all nodes *might* need to know that.
  A new requirement has surfaced, that copies of these messages need to be
sent to an external system (not a messaging system). The trivial solution
is to add a new subscriber to the Topic that handles the delivery, but
since this is a distributed system of equivalent nodes that ends up sending
the message multiple times, one per node in the cluster.

  My current solution is to use Springframework's infrastructure (which the
Application is based on), and override the JMSTemplate with a custom one
that sends a copy of every message sent to the Topic to a special Queue
that the deliverer is handling. This seems to work just fine.

  I'm wondering if there is any out-of-the-box solution to this problem
provided by Artemis. Like routing/duplication/monitoring that I can turn on
and get the same (or similar) functionality.

  Lachezar
Reply | Threaded
Open this post in threaded view
|

Re: Duplicating messages published to a Topic to a Queue

Martyn Taylor
I don't have enough information to understand you problem fully.  But
perhaps you could use a combination of duplicate detection and diverts
could solve your problem.

https://activemq.apache.org/artemis/docs/1.4.0/duplicate-detection.htm
https://activemq.apache.org/artemis/docs/1.4.0/diverts.html

Cheers
Martyn



On Mon, Oct 3, 2016 at 2:54 PM, Lachezar Dobrev <[hidden email]> wrote:

>   Hello all,
>
>   I have a design problem: a topic is clustered in a network of brokers to
> allow multiple nodes to handle events occurring in other nodes: i.e. a user
> logs in on one node, but all nodes *might* need to know that.
>   A new requirement has surfaced, that copies of these messages need to be
> sent to an external system (not a messaging system). The trivial solution
> is to add a new subscriber to the Topic that handles the delivery, but
> since this is a distributed system of equivalent nodes that ends up sending
> the message multiple times, one per node in the cluster.
>
>   My current solution is to use Springframework's infrastructure (which the
> Application is based on), and override the JMSTemplate with a custom one
> that sends a copy of every message sent to the Topic to a special Queue
> that the deliverer is handling. This seems to work just fine.
>
>   I'm wondering if there is any out-of-the-box solution to this problem
> provided by Artemis. Like routing/duplication/monitoring that I can turn on
> and get the same (or similar) functionality.
>
>   Lachezar
>
Reply | Threaded
Open this post in threaded view
|

Re: Duplicating messages published to a Topic to a Queue

Lachezar Dobrev
  Thanks!
  Diverting was exactly what I was looking for. Creating a non-exclusive
divert from the topic to the queue did the trick without any class changes.
Very nice. Using the Spring support for declarative configuration
management it is pretty easy to declare diverts.
  A few things bit me on the way though:
  - diver name must be specified, otherwise the diver is ignored with a log
message (easily missed)
  - to divert topic named "topic.whatever" the address must be
"jms.topic.topic.whatever"
  - to divert to a query named "query.whatever" the forwarding address must
be "jms.queue.queue.whatever"

  Otherwise the diverts work pretty well, I even managed to catch one queue
message being processed on a different node than the one sending the topic
message, which is exactly what I was hoping for.

  Thanks again for the directions.

2016-10-07 13:10 GMT+03:00 Martyn Taylor <[hidden email]>:

> I don't have enough information to understand you problem fully.  But
> perhaps you could use a combination of duplicate detection and diverts
> could solve your problem.
>
> https://activemq.apache.org/artemis/docs/1.4.0/duplicate-detection.htm
> https://activemq.apache.org/artemis/docs/1.4.0/diverts.html
>
> Cheers
> Martyn
>
>
>
> On Mon, Oct 3, 2016 at 2:54 PM, Lachezar Dobrev <[hidden email]>
> wrote:
>
> >   Hello all,
> >
> >   I have a design problem: a topic is clustered in a network of brokers
> to
> > allow multiple nodes to handle events occurring in other nodes: i.e. a
> user
> > logs in on one node, but all nodes *might* need to know that.
> >   A new requirement has surfaced, that copies of these messages need to
> be
> > sent to an external system (not a messaging system). The trivial solution
> > is to add a new subscriber to the Topic that handles the delivery, but
> > since this is a distributed system of equivalent nodes that ends up
> sending
> > the message multiple times, one per node in the cluster.
> >
> >   My current solution is to use Springframework's infrastructure (which
> the
> > Application is based on), and override the JMSTemplate with a custom one
> > that sends a copy of every message sent to the Topic to a special Queue
> > that the deliverer is handling. This seems to work just fine.
> >
> >   I'm wondering if there is any out-of-the-box solution to this problem
> > provided by Artemis. Like routing/duplication/monitoring that I can turn
> on
> > and get the same (or similar) functionality.
> >
> >   Lachezar
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Duplicating messages published to a Topic to a Queue

Martyn Taylor
Good to hear,

Re: Your comments

The "jms.topic." and "jms.queue." prefix requirements are causing a lot of
confusion for many users, hopefully it'll be resolved soon, as part of:
https://issues.apache.org/jira/browse/ARTEMIS-780

Regarding the divert name, we should probably throw an exception on boot to
make this clearer.  I've opened a JIRA here:
https://issues.apache.org/jira/browse/ARTEMIS-802

Thanks




On Thu, Oct 13, 2016 at 1:50 PM, Lachezar Dobrev <[hidden email]> wrote:

>   Thanks!
>   Diverting was exactly what I was looking for. Creating a non-exclusive
> divert from the topic to the queue did the trick without any class changes.
> Very nice. Using the Spring support for declarative configuration
> management it is pretty easy to declare diverts.
>   A few things bit me on the way though:
>   - diver name must be specified, otherwise the diver is ignored with a log
> message (easily missed)
>   - to divert topic named "topic.whatever" the address must be
> "jms.topic.topic.whatever"
>   - to divert to a query named "query.whatever" the forwarding address must
> be "jms.queue.queue.whatever"
>
>   Otherwise the diverts work pretty well, I even managed to catch one queue
> message being processed on a different node than the one sending the topic
> message, which is exactly what I was hoping for.
>
>   Thanks again for the directions.
>
> 2016-10-07 13:10 GMT+03:00 Martyn Taylor <[hidden email]>:
>
> > I don't have enough information to understand you problem fully.  But
> > perhaps you could use a combination of duplicate detection and diverts
> > could solve your problem.
> >
> > https://activemq.apache.org/artemis/docs/1.4.0/duplicate-detection.htm
> > https://activemq.apache.org/artemis/docs/1.4.0/diverts.html
> >
> > Cheers
> > Martyn
> >
> >
> >
> > On Mon, Oct 3, 2016 at 2:54 PM, Lachezar Dobrev <[hidden email]>
> > wrote:
> >
> > >   Hello all,
> > >
> > >   I have a design problem: a topic is clustered in a network of brokers
> > to
> > > allow multiple nodes to handle events occurring in other nodes: i.e. a
> > user
> > > logs in on one node, but all nodes *might* need to know that.
> > >   A new requirement has surfaced, that copies of these messages need to
> > be
> > > sent to an external system (not a messaging system). The trivial
> solution
> > > is to add a new subscriber to the Topic that handles the delivery, but
> > > since this is a distributed system of equivalent nodes that ends up
> > sending
> > > the message multiple times, one per node in the cluster.
> > >
> > >   My current solution is to use Springframework's infrastructure (which
> > the
> > > Application is based on), and override the JMSTemplate with a custom
> one
> > > that sends a copy of every message sent to the Topic to a special Queue
> > > that the deliverer is handling. This seems to work just fine.
> > >
> > >   I'm wondering if there is any out-of-the-box solution to this problem
> > > provided by Artemis. Like routing/duplication/monitoring that I can
> turn
> > on
> > > and get the same (or similar) functionality.
> > >
> > >   Lachezar
> > >
> >
>