Does a persistent message stay (keep) on the broker forever ?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Does a persistent message stay (keep) on the broker forever ?

Abimael
Hi
Perhaps my question is simple . I tried to look for information on ApacheMQ documentaion but I 'm not able to do.
Scenario :
 DurableTopic configured via webadmin;
 Persistent Messages sent by Producer with no time to live defined.
 Consumer consumer messages deliveried assync.

 I noticed that the Kahadb files does not change . I mean , the Kahadb files does not decrease.
 I guess that the messages are kept there forever , even they are consumer by the Consumer.
But I could not check such "guess" in the documentation.
 I understand that Persistent messages are kept to avoid be lost by a broker outage , but I do not know if they are kept AFTER consumer consumes .
 
 I am planning some disk space and I do not know if the space will increase indefinitely . I guess it will not, but I want to be sure and understand "how" it is managed.

Thanks in advance
Reply | Threaded
Open this post in threaded view
|

Re: Does a persistent message stay (keep) on the broker forever ?

Matt Pavlovich-2
Abimael-

ActiveMQ does not permanently store messages on its one.  KahaDB will
create a full-sized db-1.log (default is 32Mb) at start-up.  It stores
messages in there, and as messages are created and consumed, it will
create new db-*.log files as needed, and delete db-*.log files as
messages are consumed.  For your test, try running 10,000+ of your test
messages.  You should see a db-2.log file created and the db-1.log file
deleted once all the messages are consumed.

See the SystemUsage configuration options for limiting the disk space
that you want to allow AMQ to use.  A good rule of thumb is space for at
least 3-4 days worth of data (depending on your messaging pattern).

Hope this helps!
Matt Pavlovich

On 1/24/12 1:32 PM, Abimael wrote:

> Hi
> Perhaps my question is simple . I tried to look for information on ApacheMQ
> documentaion but I 'm not able to do.
> Scenario :
>   DurableTopic configured via webadmin;
>   Persistent Messages sent by Producer with no time to live defined.
>   Consumer consumer messages deliveried assync.
>
>   I noticed that the Kahadb files does not change . I mean , the Kahadb files
> does not decrease.
>   I guess that the messages are kept there forever , even they are consumer
> by the Consumer.
> But I could not check such "guess" in the documentation.
>   I understand that Persistent messages are kept to avoid be lost by a broker
> outage , but I do not know if they are kept AFTER consumer consumes .
>
>   I am planning some disk space and I do not know if the space will increase
> indefinitely . I guess it will not, but I want to be sure and understand
> "how" it is managed.
>
> Thanks in advance
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Does-a-persistent-message-stay-keep-on-the-broker-forever-tp4324883p4324883.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: Does a persistent message stay (keep) on the broker forever ?

Abimael
@Matt Pavlovich-2

 Thank you.
  Yes, while I was waiting for the response, I did a test with more than 500000 messages  with the Consumer application STOPPED.
I noticed that there was  db-XX.log files (several of them) . Once I started Consumer, all of them were deleted , and just one were kept : the latest one .

  Also , I had configured SystemUsage to have 1G in the "store tag " , that I understand it is the correct tag for the disk storage space . Am I correct ?

  I agree that I have space at least for 3 -4 days.. Now, with your explanation, I will be able to estimate space .

 
 Thank you very much
Reply | Threaded
Open this post in threaded view
|

Re: Does a persistent message stay (keep) on the broker forever ?

Matt Pavlovich-2
Glad to hear.  Yes, the "store" limits the disk space that the broker
will use.  Once that limit is reached, the broker will kick in "producer
flow control", where it will not ack messages to producers.  This allows
the consumer to "catch up" and drain the queues.

You shouldn't have to tweak things like cursors and turn off producer
flow control in most scenarios, but if you want some nice bedtime
reading you can get more detail here:

http://activemq.apache.org/producer-flow-control.html

Matt Pavlovich

On 1/26/12 11:53 AM, Abimael wrote:

> @Matt Pavlovich-2
>
>   Thank you.
>    Yes, while I was waiting for the response, I did a test with more than
> 500000 messages  with the Consumer application STOPPED.
> I noticed that there was  db-XX.log files (several of them) . Once I started
> Consumer, all of them were deleted , and just one were kept : the latest one
> .
>
>    Also , I had configured SystemUsage to have 1G in the "store tag " , that
> I understand it is the correct tag for the disk storage space . Am I correct
> ?
>
>    I agree that I have space at least for 3 -4 days.. Now, with your
> explanation, I will be able to estimate space .
>
>
>   Thank you very much
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Does-a-persistent-message-stay-keep-on-the-broker-forever-tp4324883p4331174.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: Does a persistent message stay (keep) on the broker forever ?

ceposta
In reply to this post by Abimael
Abimael,

Just to follow up on what Matt said, the messages in the log files will only be deleted once they're *all* consumed. If you have a log file where all but one message is consumed, the log file will stay around until the last message is consumed.

Thanks,
Christian
Reply | Threaded
Open this post in threaded view
|

Re: Does a persistent message stay (keep) on the broker forever ?

gtully
There is a little write up on the detlail of kahadb gc at:
http://activemq.apache.org/why-do-kahadb-log-files-remain-after-cleanup.html

On 26 January 2012 23:38, ceposta <[hidden email]> wrote:

> Abimael,
>
> Just to follow up on what Matt said, the messages in the log files will only
> be deleted once they're *all* consumed. If you have a log file where all but
> one message is consumed, the log file will stay around until the last
> message is consumed.
>
> Thanks,
> Christian
>
> -----
> http://www.christianposta.com
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Does-a-persistent-message-stay-keep-on-the-broker-forever-tp4324883p4332185.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.



--
http://fusesource.com
http://blog.garytully.com
Reply | Threaded
Open this post in threaded view
|

Re: Does a persistent message stay (keep) on the broker forever ?

ceposta
Oh perfect, thanks Gary!
Reply | Threaded
Open this post in threaded view
|

Re: Does a persistent message stay (keep) on the broker forever ?

Abimael
Thank you everybody ..
Have a nice weekend.