Scaling and duplicate message detection

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Scaling and duplicate message detection

funnysage
Hi All,

I am new to ActiveMQ. I have architectural question.

I have a Spring boot service that I want to scale horizontally in AWS.
This service goes to a partner API on a scheduled basis.
So we have a scheduler set up in Spring boot application.

The read from the partner API is based on number of events to be read,
watermark.
Once the read is complete, we get a new watermark. We update the watermark
in the DB.

I wanted to have multiple containers that can go and read the messages using
these schedulers.
Now as these schedulers can wake up at the same time, there is a likelihood
that the same messages will be read at the same time.
My restriction is that I cannot process a given message one more time.
I do not want to use a DB locking mechanism (if I can avoid it the better).
I dnot have an option to have distributed cache or for that matter install
any other enterprise wide product.

I came across this feature within ActiveMQ. Perhaps I can use that.
https://activemq.apache.org/artemis/docs/latest/duplicate-detection.html

I understand that I need to add something to the message header which is
unique.
But I am not sure how to generate a UUID for a given message that I have
read?
Also not sure if the UUID generated will be consistent across all the
containers?
Also as the messages are going to be textual in nature can just get the
hashcode for this message and use it for duplicate message detection?

Thanking you in advance.




--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Scaling and duplicate message detection

jbertram
If you put a message into a queue (i.e. instead of the database) then the
broker will make sure that only one consumer get it (assuming you're using
anycast routing). This replaces the DB locking mechanism you want to avoid.

The duplicate-detection functionality that you referenced is for the
*broker* to detect duplicates sent by clients.  It's not for clients to
detect duplicates from the broker (as that is already guaranteed by using
anycast routing).

Hope that helps.


Justin

On Fri, Jan 11, 2019 at 8:24 AM funnysage <[hidden email]> wrote:

> Hi All,
>
> I am new to ActiveMQ. I have architectural question.
>
> I have a Spring boot service that I want to scale horizontally in AWS.
> This service goes to a partner API on a scheduled basis.
> So we have a scheduler set up in Spring boot application.
>
> The read from the partner API is based on number of events to be read,
> watermark.
> Once the read is complete, we get a new watermark. We update the watermark
> in the DB.
>
> I wanted to have multiple containers that can go and read the messages
> using
> these schedulers.
> Now as these schedulers can wake up at the same time, there is a likelihood
> that the same messages will be read at the same time.
> My restriction is that I cannot process a given message one more time.
> I do not want to use a DB locking mechanism (if I can avoid it the better).
> I dnot have an option to have distributed cache or for that matter install
> any other enterprise wide product.
>
> I came across this feature within ActiveMQ. Perhaps I can use that.
> https://activemq.apache.org/artemis/docs/latest/duplicate-detection.html
>
> I understand that I need to add something to the message header which is
> unique.
> But I am not sure how to generate a UUID for a given message that I have
> read?
> Also not sure if the UUID generated will be consistent across all the
> containers?
> Also as the messages are going to be textual in nature can just get the
> hashcode for this message and use it for duplicate message detection?
>
> Thanking you in advance.
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Scaling and duplicate message detection

funnysage
Hi Justin,

Thanks for the reply. Yes I understand that detection is at broker level and
that is how it should be.
Now what I have learnt is that we are not using Artemis but ActiveMQ 5.18.
So will this feature be there in 5.18 too? Or I have to some other approach.

Thanks



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Scaling and duplicate message detection

jbertram
I don't believe the 5.x broker has duplicate detection functionality.


Justin

On Fri, Jan 18, 2019 at 3:00 PM funnysage <[hidden email]> wrote:

> Hi Justin,
>
> Thanks for the reply. Yes I understand that detection is at broker level
> and
> that is how it should be.
> Now what I have learnt is that we are not using Artemis but ActiveMQ 5.18.
> So will this feature be there in 5.18 too? Or I have to some other
> approach.
>
> Thanks
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>