destination unspecified - can ActiveMQ set it?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

destination unspecified - can ActiveMQ set it?

Vince Cole
Is it possible for a producer to send a message to ActiveMQ, with an unspecified destination?

The intention is that ActiveMQ will (via a plugin) inspect the message (content, headers and/or properties) and according to some business rules, it will decide which destination(s) the message must be sent to.

Based on the rules, there could be zero, one or many destinations.

None of this behaviour should depend on any consumers being present, connected or subscribed.

I have started looking into Camel and Selectors, but it looks like there is a lot of information there and I want to make sure that I start looking in the right place and am not barking up the wrong tree.

Many thanks
Reply | Threaded
Open this post in threaded view
|

Re: destination unspecified - can ActiveMQ set it?

tabish121@gmail.com
On 04/28/2017 10:46 AM, Vince Cole wrote:

> Is it possible for a producer to send a message to ActiveMQ, with an
> unspecified destination?
>
> The intention is that ActiveMQ will (via a plugin) inspect the message
> (content, headers and/or properties) and according to some business rules,
> it will decide which destination(s) the message must be sent to.
>
> Based on the rules, there could be zero, one or many destinations.
>
> None of this behaviour should depend on any consumers being present,
> connected or subscribed.
>
> I have started looking into Camel and Selectors, but it looks like there is
> a lot of information there and I want to make sure that I start looking in
> the right place and am not barking up the wrong tree.
>
> Many thanks
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/destination-unspecified-can-ActiveMQ-set-it-tp4725338.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
No you cannot just send a message at the broker without giving it a
destination.  You can add Camel routes or broker plugins to reroute the
message once it arrives but you need to give the message an original
destination to be sent to.


--
Tim Bish
twitter: @tabish121
blog: http://timbish.blogspot.com/

Reply | Threaded
Open this post in threaded view
|

Re: destination unspecified - can ActiveMQ set it?

Vince Cole
Thanks,

So the producer needs to set a destination.
That's OK, I guess it can be a 'special' destination, from which the broker will perform routing to the appropriately determined destinations.

PS the destinations won't be known in advance - they could be constructed by concatenating various parts of the message.

My question is then: Can all of this be achieved via Selectors or do Selectors require consumers, or pre-configured routes, to work?

Similarly, would the Camel plugin be suitable for this?

Finally, what effect would either of these have on performance/throughput?

Many thanks once more in anticipation.
Reply | Threaded
Open this post in threaded view
|

Re: destination unspecified - can ActiveMQ set it?

Tim Bain
Your requirement that messages be routed in the absence of
consumers/subscriptions (presumably that means you want the messages to
stick around till a consumer shows up for them) drives you to needing to
use queues, and selectors on queues could only allow you to route the
original message to a single (possibly arbitrarily-chosen) consumer.

The use cases you described include the ability to route a single message
to multiple arbitrary destinations, which can only be done by publishing
additional clones of the original via something like Camel.  (If you know
the rules for how to fan them out in advance, you could use virtual
destinations to do that instead of Camel, but from your description, I
think you need more flexibility than that.)

Camel is built with throughput in mind (it's a pipelined architecture with
the ability to execute separate stages concurrently for different messages
in order to keep throughput high), so although I've only run Camel as a
standalone consumer rather than an embedded one, I would expect that you
won't see too much of a performance overhead from running some embedded
routes, as long as the business logic you're using to decide what to do
isn't too heavyweight.  The one potential impact is that if your messages
are very large, fanning out multiple copies of them could add somewhat to
the memory/storage load on the broker, but if your message sizes are
reasonable then I don't think you'll experience unreasonable impacts.

One observation: you'll see a lower performance impact if you can get all
the information that you'd want your Camel route to evaluate and set them
as header fields on the message when it's first published, so Camel doesn't
have to go parsing the body of each message in order to grab that content.
If you don't control the producers, that may not be possible, but if it is
I'd definitely do that work there to keep the load on the broker lower.

Tim

On Fri, Apr 28, 2017 at 1:23 PM, Vince Cole <[hidden email]>
wrote:

> Thanks,
>
> So the producer needs to set a destination.
> That's OK, I guess it can be a 'special' destination, from which the broker
> will perform routing to the appropriately determined destinations.
>
> PS the destinations won't be known in advance - they could be constructed
> by
> concatenating various parts of the message.
>
> My question is then: Can all of this be achieved via Selectors or do
> Selectors require consumers, or pre-configured routes, to work?
>
> Similarly, would the Camel plugin be suitable for this?
>
> Finally, what effect would either of these have on performance/throughput?
>
> Many thanks once more in anticipation.
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/destination-unspecified-can-ActiveMQ-set-
> it-tp4725338p4725358.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: destination unspecified - can ActiveMQ set it?

Vince Cole
HI Tim

That's awesome, very helpful advice.

Thanks!
Reply | Threaded
Open this post in threaded view
|

Re: destination unspecified - can ActiveMQ set it?

Vince Cole
In reply to this post by Tim Bain
Hi Tim

OK - just one more question:
If having a single consumer per queue wasn't a problem, would Selectors still be a workable solution?
Or can they only be used if you have consumers (i.e. is it the consumers who have to specify them)?

OK OK - another:
If it was OK to lose messages when consumers weren't attached, then topics could be used.
Would Selectors work in the same way with topics?

thanks again in advance
Reply | Threaded
Open this post in threaded view
|

Re: destination unspecified - can ActiveMQ set it?

Tim Bain
Selectors are a way for a consumer to tell the broker, "Don't send me
messages like this when you do normal dispatch for this destination."

For a queue, all messages are essentially communal between the set of
consumers, so the skipped messages stay on the queue for some other
consumer (one with a matching selector) to consume them. It's very
important that you ensure that the set of selectors from the different
consumers covers all messages that will be sent, because any messages that
don't match any consumer will sit on the top of the queue and if there are
enough of them they can prevent messages from being dispatched to any
consumer and work will stop on the queue.

For a topic, each consumer has their own copy of the message stream, so
messages that don't match their consumer's selector are simply discarded
rather than left behind. This means it's not critical to ensure that the
set of selectors fully covers the set of messages.

So yes, they're specified by the consumers, but they will have their
desired effect even though they haven't been specified when the first
message is sent.

As long as your routing rules can be computed at consumer startup (i.e.
there's nothing dependent on application state), selectors could give you
what you need in the more-limited scenarios you described.

One caution, though: Art Naseef always warned that selectors were an easy
way to kill broker performance, because they must be evaluated on the
broker for each message for each consumer. So if you are going to have
large numbers of consumers and/or large numbers of messages (and
unfortunately I don't have good heuristics for what constitutes "large
numbers" here), there might be a performance impact to using the selector
approach. With that said, lots of people use selectors in ActiveMQ without
performance problems, so if you think, "we don't have all THAT many
messages and consumers," you'll probably be fine.

Tim

On May 2, 2017 3:54 AM, "Vince Cole" <[hidden email]> wrote:

> Hi Tim
>
> OK - just one more question:
> If having a single consumer per queue wasn't a problem, would Selectors
> still be a workable solution?
> Or can they only be used if you have consumers (i.e. is it the consumers
> who
> have to specify them)?
>
> OK OK - another:
> If it was OK to lose messages when consumers weren't attached, then topics
> could be used.
> Would Selectors work in the same way with topics?
>
> thanks again in advance
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/destination-unspecified-can-ActiveMQ-set-
> it-tp4725338p4725420.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>