Increasing Duplicate ID Cache size

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Increasing Duplicate ID Cache size

amaramrahul
Hi,

Due to the requirement of being able to store a large number of messages
in the queue without duplicates, I am considering increasing the
duplicate id cache size (parameter: id-cache-size) from its default
value of 2000 to a million. While I know that this would require
additional memory, I am not sure of what other complications could I run
into. I have looked up the forums but couldn't find any information
about increasing the duplicate id cache size.

Is it ok to increase the duplicate id cache size or could it have some
serious performance implications on ActiveMQ broker? Kindly advise.

Thanks,

Rahul.


--
Email: [hidden email]
Mobile: +91 9866302502
Skype: rahulamaram
WebEx Personal Room: https://lithium.webex.com/join/rahul.amaramlithium.com

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Increasing Duplicate ID Cache size

clebertsuconic
I believe you are talking about ActiveMQ Artemis.

the duplicate detection is intended for in transit messages.. that
means.. in case you retry a send (say because of an error), the
duplicate would be ignored.. and you would then achieve Exactly Once
semantic without using transactions.
You could do something similar on your consumer side as well...


for that, you don't need to have your entire set of messages on the
duplicate cache.. just enough to guarantee your Exactly Once
semantics.. .1 Million wouldn't be efficient for 2 reasons: (The
HashMap would be too big), and the journal performance (IO) would also
be affected by the duplicate caches.




On Fri, Jul 7, 2017 at 7:13 AM, Rahul Amaram <[hidden email]> wrote:

> Hi,
>
> Due to the requirement of being able to store a large number of messages in
> the queue without duplicates, I am considering increasing the duplicate id
> cache size (parameter: id-cache-size) from its default value of 2000 to a
> million. While I know that this would require additional memory, I am not
> sure of what other complications could I run into. I have looked up the
> forums but couldn't find any information about increasing the duplicate id
> cache size.
>
> Is it ok to increase the duplicate id cache size or could it have some
> serious performance implications on ActiveMQ broker? Kindly advise.
>
> Thanks,
>
> Rahul.
>
>
> --
> Email: [hidden email]
> Mobile: +91 9866302502
> Skype: rahulamaram
> WebEx Personal Room: https://lithium.webex.com/join/rahul.amaramlithium.com
>



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Increasing Duplicate ID Cache size

Martin Schlapfer-2
In reply to this post by amaramrahul

-------- Original message --------From: Clebert Suconic <[hidden email]> Date: 7/7/17  7:56 PM  (GMT+01:00) To: [hidden email] Subject: Re: Increasing Duplicate ID Cache size
I believe you are talking about ActiveMQ Artemis.

the duplicate detection is intended for in transit messages.. that
means.. in case you retry a send (say because of an error), the
duplicate would be ignored.. and you would then achieve Exactly Once
semantic without using transactions.
You could do something similar on your consumer side as well...


for that, you don't need to have your entire set of messages on the
duplicate cache.. just enough to guarantee your Exactly Once
semantics.. .1 Million wouldn't be efficient for 2 reasons: (The
HashMap would be too big), and the journal performance (IO) would also
be affected by the duplicate caches.




On Fri, Jul 7, 2017 at 7:13 AM, Rahul Amaram <[hidden email]> wrote:

> Hi,
>
> Due to the requirement of being able to store a large number of messages in
> the queue without duplicates, I am considering increasing the duplicate id
> cache size (parameter: id-cache-size) from its default value of 2000 to a
> million. While I know that this would require additional memory, I am not
> sure of what other complications could I run into. I have looked up the
> forums but couldn't find any information about increasing the duplicate id
> cache size.
>
> Is it ok to increase the duplicate id cache size or could it have some
> serious performance implications on ActiveMQ broker? Kindly advise.
>
> Thanks,
>
> Rahul.
>
>
> --
> Email: [hidden email]
> Mobile: +91 9866302502
> Skype: rahulamaram
> WebEx Personal Room: https://lithium.webex.com/join/rahul.amaramlithium.com
>



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Increasing Duplicate ID Cache size

Martin Schlapfer-2
In reply to this post by amaramrahul
www 122
-------- Original message --------From: Clebert Suconic <[hidden email]> Date: 7/7/17  7:56 PM  (GMT+01:00) To: [hidden email] Subject: Re: Increasing Duplicate ID Cache size WwwšŸ˜†šŸ˜€šŸ˜€I believe you are talking about ActiveMQ Artemis.

the duplicate detection is intended for in transit messages.. that
means.. in case you retry a send (say because of an error), the
duplicate would be ignored.. and you would then achieve Exactly Once
semantic without using transactions.
You could do something similar on your consumer side as well...


for that, you don't need to have your entire set of messages on the
duplicate cache.. just enough to guarantee your Exactly Once
semantics.. .1 Million wouldn't be efficient for 2 reasons: (The
HashMap would be too big), and the journal performance (IO) would also
be affected by the duplicate caches.




On Fri, Jul 7, 2017 at 7:13 AM, Rahul Amaram <[hidden email]> wrote:

> Hi,
>
> Due to the requirement of being able to store a large number of messages in
> the queue without duplicates, I am considering increasing the duplicate id
> cache size (parameter: id-cache-size) from its default value of 2000 to a
> million. While I know that this would require additional memory, I am not
> sure of what other complications could I run into. I have looked up the
> forums but couldn't find any information about increasing the duplicate id
> cache size.
>
> Is it ok to increase the duplicate id cache size or could it have some
> serious performance implications on ActiveMQ broker? Kindly advise.
>
> Thanks,
>
> Rahul.
>
>
> --
> Email: [hidden email]
> Mobile: +91 9866302502
> Skype: rahulamaram
> WebEx Personal Room: https://lithium.webex.com/join/rahul.amaramlithium.com
>



--
Clebert Suconic
Loading...