KahaDB journal logs cleanup issue

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view

KahaDB journal logs cleanup issue

I have read http://activemq.apache.org/why-do-kahadb-log-files-remain-after-cleanup.html and I fairly understand some of the reasons why some journal logs may not get deleted :
(a) It contains a pending message for a destination or durable topic subscription
(b) It contains an ack for a message which is in an in-use data file - the ack cannot be removed as a recovery would then mark the message for redelivery
(c) The journal references a pending transaction
(d) It is a journal file, and there may be a pending write to it

Is the above list a complete list of reasons why journal files may not get deleted? Or are there any more possible reasons?

Since I have no control over offline subscribers (these are apps used by end users), I try to overcome the above scenarios with certain configurations :

To avoid issues due to (a), I have enabled following configurations in my broker xml :

To ensure offline durable subscribers don't cause piling up of these log files, I have enabled timeout for offline durable subscribers to 24 hours :

<broker xmlns="http://activemq.apache.org/schema/core" useJmx="false" brokerName="PrimaryBroker" deleteAllMessagesOnStartup="false" advisorySupport="false" schedulePeriodForDestinationPurge="600000" offlineDurableSubscriberTimeout="86400000" offlineDurableSubscriberTaskSchedule="3600000" dataDirectory="${activemq.data}">

I have also set message expiry as 12 hours :

    <discardingDLQBrokerPlugin dropAll="true" dropTemporaryTopics="true" dropTemporaryQueues="true" />
    <timeStampingBrokerPlugin ttlCeiling="43200000" zeroExpirationOverride="43200000"/>

I think the above configurations only help to overcome scenario (a). How do I overcome scenarios (b), (c) and (d)? Are there any configuration to :
a) delete old ack messages?
b) timeout pending transactions?

In the last 3 days run, I see that there are some journal logs which are were created on 17-Feb early morning. Why are these files still not getting deleted even after more than 72 hours? What could be the probable reasons? Since this is in production environment, I cannot enable debug logs to see which destination is holding up which journal file. Any help will be greatly appreciated.

P.S : We use AMQ 5.14.1 and exchange MQTT messages mostly (thousands of topics will be created on the fly and both persistent and non-persistent messages are exchanged over these topics) and a few JMS messages to a queue.

Reply | Threaded
Open this post in threaded view

Re: KahaDB journal logs cleanup issue

I am wondering what you ended up doing for the number 2, pending acks?

Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html