Managing persistent store space

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Managing persistent store space

petrk
This post was updated on .
Hi

I have a single instance of ActiveMQ v5.9 using persistent store and we are constantly running out of space.
We use persistent messages with KahaDB and have 50gb store defined.
The queues directory only actually contains 26GB, but ActiveMQ admin says "Store percent used 94", is that right?
Browsing the number of pending messages in queues, there is no way there is 26GB or 50GB worth of data!

I have noticed that we have lots of ActiveMQ.Advisory.Consumer.Queue.* topics, which we don't use, but we use the advisoryWhenFull="true" in conjuction with sendFailIfNoSpace="true" so that we get notified when messages can't be queued. We also expire messages after 30 days, in hope they will get deleted. If I disable advisory messages am I going to lose these two functions?

Also, if I disable advisory messages, will activeMQ delete all the advisory topics or do I have to delete them one by one?

Any ideas?

Here are the relevant bits from our activemq.xml:
<bean id="store" class="org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter" >
     <property name="directory" value="/opt/xyz/queues" />
     <property name="journalMaxFileLength" value="3145728" />
     <property name="checkpointInterval" value="5000" />
     <property name="cleanupInterval" value="5000" />
</bean>

<broker xmlns="http://activemq.apache.org/schema/core" 
        useJmx="true" brokerName="localhost" dataDirectory="/opt/xyz/queues"
         persistenceAdapter="#store">
        <destinationPolicy>
            <policyMap>
              <policyEntries>
                <policyEntry topic=">">
                                        <deadLetterStrategy>
                        <sharedDeadLetterStrategy processExpired="false"/>
                </deadLetterStrategy>
                        </policyEntry>
                        <policyEntry queue=">" cursorMemoryHighWaterMark="90" maxPageSize="10000" queuePrefetch="10000" producerFlowControl="false" advisoryWhenFull="true">
            <pendingQueuePolicy>
                <storeCursor/>
            </pendingQueuePolicy>
                </policyEntry>
              </policyEntries>
            </policyMap>
        </destinationPolicy>
          <systemUsage>
            <systemUsage sendFailIfNoSpace="true">
                <memoryUsage>
                    <memoryUsage limit = "8 gb"/>
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="50 gb" store="#store"/>
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="10 gb"/>
                </tempUsage>
            </systemUsage>
        </systemUsage>
        <plugins>
                <timeStampingBrokerPlugin zeroExpirationOverride="2419200000"/>
                <discardingDLQBrokerPlugin dropAll="true" dropTemporaryTopics="true" dropTemporaryQueues="true" />
        </plugins>

    </broker>
Reply | Threaded
Open this post in threaded view
|

Re: Managing persistent store space

Andreas Gies
Hi

we had a similar issue in our ActiveMQ broker. The reason was that we
had a topic that we use for guaranteed message delivery with message
persistence and durable subscribers. Some durable subscribers might be
offline and we spotted that even with a few MB worth of messages our
persistent store was growing to an unexplainable size.

The reason underneath was that we were using one kahadb instance for ALL
our messaging traffic. Since kahadb has a write only journal, the worst
case would be that our messages waiting for delivery would be spread
across journal files interleaved with all the other journal entries that
refer to already processed messages.

Gary has written a very good blog entry that describes our problem
nicely :
http://blog.garytully.com/2011/11/activemq-multiple-kahadb-instances.html

At the end we are now running with a kahadb split in 3 parts. 2 parts
serving a destination with occasinally offline consumers each and a
third part handling "the normal" JMS use case.

That solved our store problem pretty well.

For reference this is the interesting part of our config:

<amq:persistenceAdapter>
           <amq:mKahaDB
             directory="amq/mkahadb"
             journalMaxFileLength="16m">

             <amq:filteredPersistenceAdapters>

               <amq:filteredKahaDB
topic="our.occasianally.disconnected.topic">
                 <amq:persistenceAdapter>
                   <amq:kahaDB
                     directory="amq/kahadb/topic"
                     journalMaxFileLength="16m"
                   />
                 </amq:persistenceAdapter>
               </amq:filteredKahaDB>

               <amq:filteredKahaDB
queue="our.occasionally.disconnected.queue">
                 <amq:persistenceAdapter>
                   <amq:kahaDB
                     directory="amq/kahadb/queue"
                     journalMaxFileLength="16m"
                   />
                 </amq:persistenceAdapter>
               </amq:filteredKahaDB>

               <amq:filteredKahaDB>
                 <amq:persistenceAdapter>
                   <amq:kahaDB
                     directory="amq/kahadb/default"
                     journalMaxFileLength="16m"
                   />
                 </amq:persistenceAdapter>
               </amq:filteredKahaDB>

             </amq:filteredPersistenceAdapters>

           </amq:mKahaDB>
         </amq:persistenceAdapter>

Best regards
Andreas


On 26/09/14 17:57, petrk wrote:

> Hi
>
> I have a single instance of ActiveMQ v5.9 using persistent store and we are
> constantly running out of space.
> We use persistent messages with KahaDB and have 50gb store defined.
> The queues directory only actually contains 26GB, but ActiveMQ admin says
> "Store percent used 94", is that right?
> Browsing the number of pending messages in queues, there is no way there is
> 26GB or 50GB worth of data!
>
> I have noticed that we have lots of ActiveMQ.Advisory.Consumer.Queue.*
> topics, which we don't use, but we use the advisoryWhenFull="true" in
> conjuction with sendFailIfNoSpace="true" so that we get notified when
> messages can't be queued. We also expire messages after 30 days, in hope
> they will get deleted. If I disable advisory message am I going to lose
> these two functions?
>
> Any ideas?
>
> Here are the relevant bits from our activemq.xml:
> <bean id="store"
> class="org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter" >
>       <property name="directory" value="/opt/xyz/queues" />
>       <property name="journalMaxFileLength" value="3145728" />
>       <property name="checkpointInterval" value="5000" />
>       <property name="cleanupInterval" value="5000" />
> </bean>
>
> <broker xmlns="http://activemq.apache.org/schema/core"
> useJmx="true" brokerName="localhost" dataDirectory="/opt/xyz/queues"
>           persistenceAdapter="#store">
>          <destinationPolicy>
>              <policyMap>
>                <policyEntries>
>                   <policyEntry topic=">">
> <deadLetterStrategy>
>                          <sharedDeadLetterStrategy processExpired="false"/>
>                   </deadLetterStrategy>
> </policyEntry>
> <policyEntry queue=">" cursorMemoryHighWaterMark="90" maxPageSize="10000"
> queuePrefetch="10000" producerFlowControl="false" advisoryWhenFull="true">
>               <pendingQueuePolicy>
>                   <storeCursor/>
>               </pendingQueuePolicy>
> </policyEntry>
>                </policyEntries>
>              </policyMap>
>          </destinationPolicy>
>            <systemUsage>
>              <systemUsage sendFailIfNoSpace="true">
>                  <memoryUsage>
>                      <memoryUsage limit = "8 gb"/>
>                  </memoryUsage>
>                  <storeUsage>
>                      <storeUsage limit="50 gb" store="#store"/>
>                  </storeUsage>
>                  <tempUsage>
>                      <tempUsage limit="10 gb"/>
>                  </tempUsage>
>              </systemUsage>
>          </systemUsage>
> <plugins>
>                  <timeStampingBrokerPlugin
> zeroExpirationOverride="2419200000"/>
>                  <discardingDLQBrokerPlugin dropAll="true"
> dropTemporaryTopics="true" dropTemporaryQueues="true" />
>          </plugins>
>
>      </broker>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Managing-persistent-store-space-tp4685913.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

--


    Andreas Gies

WoQ -- Way of Quality GmbH

Geschäftsführer & CTO

/eMail:/[hidden email] <mailto:[hidden email]>

/Tel:/ +49 151 23470823

/Fax:/ +49 1805 006534 2114

/Twitter:/ andreasgies /Skype:/ giessonic

/LinkedIn:/ <http://de.linkedin.com/pub/andreas-gies/0/594/aa5/>
(http://de.linkedin.com/pub/andreas-gies/0/594/aa5/)

/Xing:/ <http://www.xing.com/profile/Andreas_Gies>
(http://www.xing.com/profile/Andreas_Gies)

/Blog:/ <http://www.wayofquality.de/index.php/en/blog>
(http://www.wayofquality.de/index.php/en/blog)

/Github:/ <https://github.com/atooni> (https://github.com/atooni)

/Amtsgericht Landshut:/HRB 8352//

//

/Ust.-Id.:/ DE274771254


      Haftungsausschluss

Diese Email kann vertrauliche und/oder rechtlich geschützte
Informationen enthalten und ist ausschließlich für den/die benannten
Adressaten bestimmt. Sollten Sie nicht der beabsichtigte Empfänger sein
oder diese Email irrtümlich erhalten haben, ist es Ihnen nicht gestattet
diese Mail oder einen Teil davon ohne unsere Erlaubnis zu verbreiten, zu
kopieren, unbefugt weiterzuleiten oder zu behalten. Informieren Sie
bitte sofort den Absender telefonisch oder per Email und löschen Sie
diese Email und alle Kopien aus Ihrem System. Wir haften nicht für die
Unversehrtheit von Emails, nachdem sie unseren Einflussbereich verlassen
haben.


      Disclaimer

This email may contain confidential and/or privileged information and is
intended solely for the attention and use of the named addressee(s). If
you are not the intended recipient, or a person responsible for
delivering it to the intended recipient, you are not authorized to and
must not disclose, copy, distribute, or retain this message or any part
of it without our authority. Please contact the sender by call or reply
email immediately and destroy all copies and the original message. We
are not responsible for the integrity of emails after they have left our
sphere of control.

//
Reply | Threaded
Open this post in threaded view
|

Re: Managing persistent store space

petrk
That is an interesting solution as we do have two streams of messages on our queues, fast one with lots of data and a slow one with more smaller sized messages.

Does anyone know if the kaha db adapter supports use of wildcards (i.e. "OrdersForDevice*"), as we have dynamic queuenames. would something like this work:

<persistenceAdapter>
    <mKahaDB directory="${activemq.base}/data/kahadb">
      <filteredPersistenceAdapters>
       <filteredKahaDB queue="OrdersForDevice*">
        <persistenceAdapter>
          <kahaDB />
        </persistenceAdapter>
       </filteredKahaDB>
       <filteredKahaDB>
        <persistenceAdapter>
          <kahaDB />
        </persistenceAdapter>
       </filteredKahaDB>
      </filteredPersistenceAdapters>
    </mKahaDB>
</persistenceAdapter>
Reply | Threaded
Open this post in threaded view
|

Re: Managing persistent store space

Andreas Gies
Hi again,

as far as I understood (and believe I have observed it in our case) you
can use wildcards
to qualify the kahadb instance. The instance with no destination
qualification is the default
kahadb instance.

Best regards
Andreas

On 29/09/14 14:56, petrk wrote:

> That is an interesting solution as we do have two streams of messages on our
> queues, fast one with lots of data and a slow one with more smaller sized
> messages.
>
> Does anyone know if the kaha db adapter supports use of wildcards (i.e.
> "OrdersForDevice*"), as we have dynamic queuenames. would something like
> this work:
>
> <persistenceAdapter>
>      <mKahaDB directory="${activemq.base}/data/kahadb">
>        <filteredPersistenceAdapters>
>         <filteredKahaDB queue="OrdersForDevice*">
>          <persistenceAdapter>
>            <kahaDB />
>          </persistenceAdapter>
>         </filteredKahaDB>
>         <filteredKahaDB>
>          <persistenceAdapter>
>            <kahaDB />
>          </persistenceAdapter>
>         </filteredKahaDB>
>        </filteredPersistenceAdapters>
>      </mKahaDB>
> </persistenceAdapter>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Managing-persistent-store-space-tp4685913p4685981.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

--


    Andreas Gies

WoQ -- Way of Quality GmbH

Geschäftsführer & CTO

/eMail:/[hidden email] <mailto:[hidden email]>

/Tel:/ +49 151 23470823

/Fax:/ +49 1805 006534 2114

/Twitter:/ andreasgies /Skype:/ giessonic

/LinkedIn:/ <http://de.linkedin.com/pub/andreas-gies/0/594/aa5/>
(http://de.linkedin.com/pub/andreas-gies/0/594/aa5/)

/Xing:/ <http://www.xing.com/profile/Andreas_Gies>
(http://www.xing.com/profile/Andreas_Gies)

/Blog:/ <http://www.wayofquality.de/index.php/en/blog>
(http://www.wayofquality.de/index.php/en/blog)

/Github:/ <https://github.com/atooni> (https://github.com/atooni)

/Amtsgericht Landshut:/HRB 8352//

//

/Ust.-Id.:/ DE274771254


      Haftungsausschluss

Diese Email kann vertrauliche und/oder rechtlich geschützte
Informationen enthalten und ist ausschließlich für den/die benannten
Adressaten bestimmt. Sollten Sie nicht der beabsichtigte Empfänger sein
oder diese Email irrtümlich erhalten haben, ist es Ihnen nicht gestattet
diese Mail oder einen Teil davon ohne unsere Erlaubnis zu verbreiten, zu
kopieren, unbefugt weiterzuleiten oder zu behalten. Informieren Sie
bitte sofort den Absender telefonisch oder per Email und löschen Sie
diese Email und alle Kopien aus Ihrem System. Wir haften nicht für die
Unversehrtheit von Emails, nachdem sie unseren Einflussbereich verlassen
haben.


      Disclaimer

This email may contain confidential and/or privileged information and is
intended solely for the attention and use of the named addressee(s). If
you are not the intended recipient, or a person responsible for
delivering it to the intended recipient, you are not authorized to and
must not disclose, copy, distribute, or retain this message or any part
of it without our authority. Please contact the sender by call or reply
email immediately and destroy all copies and the original message. We
are not responsible for the integrity of emails after they have left our
sphere of control.

//
Reply | Threaded
Open this post in threaded view
|

Re: Managing persistent store space

petrk
In reply to this post by petrk
I have one issue though, as per Limiting disk store usage with Active MQ I need to explicitly specify a persistence adapter as a bean within the configuration file and reference it from the broker and the storage definition otherwise the store size gets ignored.

and apperars I can only specify Multi kahaDB persistence adapter from inside of the broker config.

Has anyone tried to combine  Multi kahaDB persistence adapter whilst specifying store usage?
Reply | Threaded
Open this post in threaded view
|

Re: Managing persistent store space

petrk
Right I have managed to set up Multi kahaDB persistence adapter from inside of the broker config:

    <bean id="mainStore" class="org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter" >
       <property name="journalMaxFileLength" value="3145728" />
       <property name="checkpointInterval" value="5000" />
       <property name="cleanupInterval" value="5000" />
       <property name="directory" value="/opt/xyz/queues" /> 
    </bean>

    <bean id="otherStore" class="org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter" >
       <property name="journalMaxFileLength" value="3145728" />
       <property name="checkpointInterval" value="5000" />
       <property name="cleanupInterval" value="5000" />
       <property name="directory" value="/opt/xyz/queues/Other" />
    </bean>

    <bean id="filteredMainStore" class="org.apache.activemq.store.kahadb.FilteredKahaDBPersistenceAdapter" >
       <property name="persistenceAdapter" ref="mainStore" />
    </bean>

    <bean id="filteredOtherStore" class="org.apache.activemq.store.kahadb.FilteredKahaDBPersistenceAdapter" >
       <property name="persistenceAdapter" ref="otherStore" />
       <property name="queue" value="Other*" />
    </bean>

    <bean id="store" class="org.apache.activemq.store.kahadb.MultiKahaDBPersistenceAdapter" >
      <property name="directory" value="/opt/xyz/queues" />
      <property name="filteredPersistenceAdapters">
                         <list>
                                <ref bean="filteredOtherStore" />
                                <ref bean="filteredMainStore" />
                        </list>
                </property>
   </bean>

However, the existing queues before the above persistance adapter changes are no longer visible/don't exist. I need to be able to keep the existing queues if changing to this MultiKahaDBPersistenceAdapter, is there any way to do that?

when activeMQ starts up I can see the following log entry:

2014-10-01 16:16:35,840 | INFO  | Using Persistence Adapter: MultiKahaDBPersistenceAdapter[C:\apache-activemq-5.9.0\bin\activemq-data\mKahaDB][KahaDBPersistenceAdapter[C:\opt\RaxStore\queues\StoreLogon\queue#3a#2f#2fStoreLogon#2a], KahaDBPersistenceAdapter[C:\opt\RaxStore\queues\0]] | org.apache.activemq.broker.BrokerService | main