Log files not being cleaned up

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Log files not being cleaned up

Lionel van den Berg
Hi all,
ActiveMQ 5.14.4 running on Suse enterprise 11 sp4

I'm having trouble with the log files (like db-1.log) not being cleaned up
and after about 6 weeks of running ActiveMQ shuts down because there are
too many files in the directory (I think it was 3000), it cannot be
restarted we think due to a JVM limitation. We use a mix of topics and
queues, we can't guarantee that the consumer from the third party won't be
shutdown for an extended period. We are storing messages for 7 days, or at
least we think we are, I've copied the config further down. Don't assume I
know everything that we are doing with all of the config :). I've read
various posts but I'm not sure still whether there might be a parameter
that I can set that will trigger the cleanup.

It looks similar to this https://issues.apache.org/jira/browse/AMQ-5695 but
the last comment there is that it is resolved in 5.14.3, I'm running 5.14.4
on the server side at least, client libraries we did not change.

Does anyone have any ideas?



<!--
    Licensed to the Apache Software Foundation (ASF) under one or more
    contributor license agreements.  See the NOTICE file distributed with
    this work for additional information regarding copyright ownership.
    The ASF licenses this file to You under the Apache License, Version 2.0
    (the "License"); you may not use this file except in compliance with
    the License.  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
-->
<beans
  xmlns="http://www.springframework.org/schema/beans"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
  http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd">

    <!-- Allows us to use system properties as variables in this
configuration file -->
    <bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
      <property name="locations">
        <value>file:${activemq.conf}/credentials.properties</value>
      </property>
    </bean>

   <!-- Allows accessing the server log -->
    <bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
          lazy-init="false" scope="singleton"
          init-method="start" destroy-method="stop">
    </bean>

    <!-- The <broker> element is used to configure the ActiveMQ broker. -->
    <broker xmlns="http://activemq.apache.org/schema/core"
brokerName="AutomationServerThree" dataDirectory="${activemq.data}">

        <destinationPolicy>
          <policyMap>
            <policyEntries>
              <!-- policy for all topics -->
              <policyEntry topic=">" producerFlowControl="true"
useCache="true" expireMessagesPeriod="86400000" alwaysRetroactive="true">
                <pendingMessageLimitStrategy>
                  <constantPendingMessageLimitStrategy limit="1000"/>
                </pendingMessageLimitStrategy>
                <!-- discard expired messages -->
                <deadLetterStrategy>
                  <sharedDeadLetterStrategy processExpired="false"/>
                </deadLetterStrategy>
              </policyEntry>
              <!-- policy for all queues -->
              <policyEntry queue=">" producerFlowControl="true"
useCache="true" expireMessagesPeriod="86400000">
                <pendingMessageLimitStrategy>
                  <constantPendingMessageLimitStrategy limit="1000"/>
                </pendingMessageLimitStrategy>
                <!-- discard expired messages -->
                <deadLetterStrategy>
                  <sharedDeadLetterStrategy processExpired="false"/>
                </deadLetterStrategy>
              </policyEntry>
            </policyEntries>
          </policyMap>
        </destinationPolicy>


        <!--
            The managementContext is used to configure how ActiveMQ is
exposed in
            JMX. By default, ActiveMQ uses the MBean server that is started
by
            the JVM. For more information, see:

            http://activemq.apache.org/jmx.html
        -->
        <managementContext>
          <managementContext createConnector="true"/>
        </managementContext>

        <!--
            Configure message persistence for the broker. The default
persistence
            mechanism is the KahaDB store (identified by the kahaDB tag).
            For more information, see:

            http://activemq.apache.org/persistence.html
        -->
        <persistenceAdapter>
          <kahaDB directory="/media/SharedStorage/Binaries/activemq"
                  indexCacheSize="40000"
                  checkForCorruptJournalFiles="true">
            <locker>
              <shared-native-file-locker lockAcquireSleepInterval="5000" />
            </locker>
          </kahaDB>
        </persistenceAdapter>


        <!--
          The systemUsage controls the maximum amount of space the broker
will
          use before disabling caching and/or slowing down producers. For
more information, see:
          http://activemq.apache.org/producer-flow-control.html
        -->
        <systemUsage>
          <systemUsage>
            <memoryUsage>
              <memoryUsage percentOfJvmHeap="70" />
            </memoryUsage>
            <storeUsage>
              <storeUsage limit="100 gb"/>
            </storeUsage>
              <tempUsage>
                <tempUsage limit="50 gb"/>
            </tempUsage>
          </systemUsage>
        </systemUsage>

        <!--
            The transport connectors expose ActiveMQ over a given protocol
to
            clients and other brokers. For more information, see:

            http://activemq.apache.org/configuring-transports.html
        -->
        <transportConnectors>
          <!-- DOS protection, limit concurrent connections to 1000 and
frame size to 100MB -->
          <transportConnector name="openwire" uri="tcp://
0.0.0.0:61616?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600
"/>
          <transportConnector name="amqp" uri="amqp://
0.0.0.0:5672?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600
"/>
          <transportConnector name="stomp" uri="stomp://
0.0.0.0:61613?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600
"/>
          <transportConnector name="mqtt" uri="mqtt://
0.0.0.0:1883?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600
"/>
          <transportConnector name="ws" uri="ws://
0.0.0.0:61614?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600
"/>
        </transportConnectors>

        <!-- destroy the spring context on shutdown to stop jetty -->
        <shutdownHooks>
          <bean xmlns="http://www.springframework.org/schema/beans"
class="org.apache.activemq.hooks.SpringContextHook" />
        </shutdownHooks>

    </broker>

    <!--
        Enable web consoles, REST and Ajax APIs and demos
        The web consoles requires by default login, you can disable this in
the jetty.xml file

        Take a look at ${ACTIVEMQ_HOME}/conf/jetty.xml for more details
    -->
    <import resource="jetty.xml"/>

</beans>
Reply | Threaded
Open this post in threaded view
|

Re: Log files not being cleaned up

Tim Bain
Have you run the troubleshooting procedures in
http://activemq.apache.org/why-do-kahadb-log-files-remain-after-cleanup.html?
What did they indicate?

On Sep 12, 2017 8:27 PM, "Lionel van den Berg" <[hidden email]> wrote:

Hi all,
ActiveMQ 5.14.4 running on Suse enterprise 11 sp4

I'm having trouble with the log files (like db-1.log) not being cleaned up
and after about 6 weeks of running ActiveMQ shuts down because there are
too many files in the directory (I think it was 3000), it cannot be
restarted we think due to a JVM limitation. We use a mix of topics and
queues, we can't guarantee that the consumer from the third party won't be
shutdown for an extended period. We are storing messages for 7 days, or at
least we think we are, I've copied the config further down. Don't assume I
know everything that we are doing with all of the config :). I've read
various posts but I'm not sure still whether there might be a parameter
that I can set that will trigger the cleanup.

It looks similar to this https://issues.apache.org/jira/browse/AMQ-5695 but
the last comment there is that it is resolved in 5.14.3, I'm running 5.14.4
on the server side at least, client libraries we did not change.

Does anyone have any ideas?



<!--
    Licensed to the Apache Software Foundation (ASF) under one or more
    contributor license agreements.  See the NOTICE file distributed with
    this work for additional information regarding copyright ownership.
    The ASF licenses this file to You under the Apache License, Version 2.0
    (the "License"); you may not use this file except in compliance with
    the License.  You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

    Unless required by applicable law or agreed to in writing, software
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    limitations under the License.
-->
<beans
  xmlns="http://www.springframework.org/schema/beans"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
  http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd">

    <!-- Allows us to use system properties as variables in this
configuration file -->
    <bean
class="org.springframework.beans.factory.config.
PropertyPlaceholderConfigurer">
      <property name="locations">
        <value>file:${activemq.conf}/credentials.properties</value>
      </property>
    </bean>

   <!-- Allows accessing the server log -->
    <bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
          lazy-init="false" scope="singleton"
          init-method="start" destroy-method="stop">
    </bean>

    <!-- The <broker> element is used to configure the ActiveMQ broker. -->
    <broker xmlns="http://activemq.apache.org/schema/core"
brokerName="AutomationServerThree" dataDirectory="${activemq.data}">

        <destinationPolicy>
          <policyMap>
            <policyEntries>
              <!-- policy for all topics -->
              <policyEntry topic=">" producerFlowControl="true"
useCache="true" expireMessagesPeriod="86400000" alwaysRetroactive="true">
                <pendingMessageLimitStrategy>
                  <constantPendingMessageLimitStrategy limit="1000"/>
                </pendingMessageLimitStrategy>
                <!-- discard expired messages -->
                <deadLetterStrategy>
                  <sharedDeadLetterStrategy processExpired="false"/>
                </deadLetterStrategy>
              </policyEntry>
              <!-- policy for all queues -->
              <policyEntry queue=">" producerFlowControl="true"
useCache="true" expireMessagesPeriod="86400000">
                <pendingMessageLimitStrategy>
                  <constantPendingMessageLimitStrategy limit="1000"/>
                </pendingMessageLimitStrategy>
                <!-- discard expired messages -->
                <deadLetterStrategy>
                  <sharedDeadLetterStrategy processExpired="false"/>
                </deadLetterStrategy>
              </policyEntry>
            </policyEntries>
          </policyMap>
        </destinationPolicy>


        <!--
            The managementContext is used to configure how ActiveMQ is
exposed in
            JMX. By default, ActiveMQ uses the MBean server that is started
by
            the JVM. For more information, see:

            http://activemq.apache.org/jmx.html
        -->
        <managementContext>
          <managementContext createConnector="true"/>
        </managementContext>

        <!--
            Configure message persistence for the broker. The default
persistence
            mechanism is the KahaDB store (identified by the kahaDB tag).
            For more information, see:

            http://activemq.apache.org/persistence.html
        -->
        <persistenceAdapter>
          <kahaDB directory="/media/SharedStorage/Binaries/activemq"
                  indexCacheSize="40000"
                  checkForCorruptJournalFiles="true">
            <locker>
              <shared-native-file-locker lockAcquireSleepInterval="5000" />
            </locker>
          </kahaDB>
        </persistenceAdapter>


        <!--
          The systemUsage controls the maximum amount of space the broker
will
          use before disabling caching and/or slowing down producers. For
more information, see:
          http://activemq.apache.org/producer-flow-control.html
        -->
        <systemUsage>
          <systemUsage>
            <memoryUsage>
              <memoryUsage percentOfJvmHeap="70" />
            </memoryUsage>
            <storeUsage>
              <storeUsage limit="100 gb"/>
            </storeUsage>
              <tempUsage>
                <tempUsage limit="50 gb"/>
            </tempUsage>
          </systemUsage>
        </systemUsage>

        <!--
            The transport connectors expose ActiveMQ over a given protocol
to
            clients and other brokers. For more information, see:

            http://activemq.apache.org/configuring-transports.html
        -->
        <transportConnectors>
          <!-- DOS protection, limit concurrent connections to 1000 and
frame size to 100MB -->
          <transportConnector name="openwire" uri="tcp://
0.0.0.0:61616?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600
"/>
          <transportConnector name="amqp" uri="amqp://
0.0.0.0:5672?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600
"/>
          <transportConnector name="stomp" uri="stomp://
0.0.0.0:61613?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600
"/>
          <transportConnector name="mqtt" uri="mqtt://
0.0.0.0:1883?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600
"/>
          <transportConnector name="ws" uri="ws://
0.0.0.0:61614?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600
"/>
        </transportConnectors>

        <!-- destroy the spring context on shutdown to stop jetty -->
        <shutdownHooks>
          <bean xmlns="http://www.springframework.org/schema/beans"
class="org.apache.activemq.hooks.SpringContextHook" />
        </shutdownHooks>

    </broker>

    <!--
        Enable web consoles, REST and Ajax APIs and demos
        The web consoles requires by default login, you can disable this in
the jetty.xml file

        Take a look at ${ACTIVEMQ_HOME}/conf/jetty.xml for more details
    -->
    <import resource="jetty.xml"/>

</beans>
Reply | Threaded
Open this post in threaded view
|

Re: Log files not being cleaned up

GlenWard
In reply to this post by Lionel van den Berg
Facing the same problem. I had done all possible solutions but still problem
remains unsolved


________________________________
http://www.menstwilight.com/kamagra.html




--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Log files not being cleaned up

Tim Bain
The question I asked Lionel applies to you ad well.

On Sep 29, 2017 9:12 AM, "GlenWard" <[hidden email]> wrote:

> Facing the same problem. I had done all possible solutions but still
> problem
> remains unsolved
>
>
> ________________________________
> http://www.menstwilight.com/kamagra.html
>
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Log files not being cleaned up

Lionel van den Berg
In reply to this post by Tim Bain
Hi, thanks for the response.

Some questions on these points from the troubleshooting.


   1. *It contains a pending message for a destination or durable topic
   subscription*

This seems a little flawed, if a consumer who I have little control of is
mis-behaving then my ActiveMQ can end up shutting down and unrecoverable.
Is there some way of timing this out or similar?

*2. It contains an ack for a message which is in an in-use data file - the
ack cannot be removed as a recovery would then mark the message for
redelivery*

Same comment as 1.

*3. The journal references a pending transaction*

I'm not using transactions, but are there transactions under the hood?

*4. It is a journal file, and there may be a pending write to it*

Why would this be the case?

I'll see if I can change the logging settings, since the first occurrence
the number of log files does not seem to have been an issue. I have it
configured to keep messages for 7 days so regardless of the above
conditions I would have thought that at that expiry the log would be
cleaned up so we don't end up in such a situation where the system stops
and cannot restart.


On 14 September 2017 at 00:02, Tim Bain <[hidden email]> wrote:

> Have you run the troubleshooting procedures in
> http://activemq.apache.org/why-do-kahadb-log-files-
> remain-after-cleanup.html?
> What did they indicate?
>
> On Sep 12, 2017 8:27 PM, "Lionel van den Berg" <[hidden email]> wrote:
>
> Hi all,
> ActiveMQ 5.14.4 running on Suse enterprise 11 sp4
>
> I'm having trouble with the log files (like db-1.log) not being cleaned up
> and after about 6 weeks of running ActiveMQ shuts down because there are
> too many files in the directory (I think it was 3000), it cannot be
> restarted we think due to a JVM limitation. We use a mix of topics and
> queues, we can't guarantee that the consumer from the third party won't be
> shutdown for an extended period. We are storing messages for 7 days, or at
> least we think we are, I've copied the config further down. Don't assume I
> know everything that we are doing with all of the config :). I've read
> various posts but I'm not sure still whether there might be a parameter
> that I can set that will trigger the cleanup.
>
> It looks similar to this https://issues.apache.org/jira/browse/AMQ-5695
> but
> the last comment there is that it is resolved in 5.14.3, I'm running 5.14.4
> on the server side at least, client libraries we did not change.
>
> Does anyone have any ideas?
>
>
>
> <!--
>     Licensed to the Apache Software Foundation (ASF) under one or more
>     contributor license agreements.  See the NOTICE file distributed with
>     this work for additional information regarding copyright ownership.
>     The ASF licenses this file to You under the Apache License, Version 2.0
>     (the "License"); you may not use this file except in compliance with
>     the License.  You may obtain a copy of the License at
>
>     http://www.apache.org/licenses/LICENSE-2.0
>
>     Unless required by applicable law or agreed to in writing, software
>     distributed under the License is distributed on an "AS IS" BASIS,
>     WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> implied.
>     See the License for the specific language governing permissions and
>     limitations under the License.
> -->
> <beans
>   xmlns="http://www.springframework.org/schema/beans"
>   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>   xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd
>   http://activemq.apache.org/schema/core
> http://activemq.apache.org/schema/core/activemq-core.xsd">
>
>     <!-- Allows us to use system properties as variables in this
> configuration file -->
>     <bean
> class="org.springframework.beans.factory.config.
> PropertyPlaceholderConfigurer">
>       <property name="locations">
>         <value>file:${activemq.conf}/credentials.properties</value>
>       </property>
>     </bean>
>
>    <!-- Allows accessing the server log -->
>     <bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
>           lazy-init="false" scope="singleton"
>           init-method="start" destroy-method="stop">
>     </bean>
>
>     <!-- The <broker> element is used to configure the ActiveMQ broker. -->
>     <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="AutomationServerThree" dataDirectory="${activemq.data}">
>
>         <destinationPolicy>
>           <policyMap>
>             <policyEntries>
>               <!-- policy for all topics -->
>               <policyEntry topic=">" producerFlowControl="true"
> useCache="true" expireMessagesPeriod="86400000" alwaysRetroactive="true">
>                 <pendingMessageLimitStrategy>
>                   <constantPendingMessageLimitStrategy limit="1000"/>
>                 </pendingMessageLimitStrategy>
>                 <!-- discard expired messages -->
>                 <deadLetterStrategy>
>                   <sharedDeadLetterStrategy processExpired="false"/>
>                 </deadLetterStrategy>
>               </policyEntry>
>               <!-- policy for all queues -->
>               <policyEntry queue=">" producerFlowControl="true"
> useCache="true" expireMessagesPeriod="86400000">
>                 <pendingMessageLimitStrategy>
>                   <constantPendingMessageLimitStrategy limit="1000"/>
>                 </pendingMessageLimitStrategy>
>                 <!-- discard expired messages -->
>                 <deadLetterStrategy>
>                   <sharedDeadLetterStrategy processExpired="false"/>
>                 </deadLetterStrategy>
>               </policyEntry>
>             </policyEntries>
>           </policyMap>
>         </destinationPolicy>
>
>
>         <!--
>             The managementContext is used to configure how ActiveMQ is
> exposed in
>             JMX. By default, ActiveMQ uses the MBean server that is started
> by
>             the JVM. For more information, see:
>
>             http://activemq.apache.org/jmx.html
>         -->
>         <managementContext>
>           <managementContext createConnector="true"/>
>         </managementContext>
>
>         <!--
>             Configure message persistence for the broker. The default
> persistence
>             mechanism is the KahaDB store (identified by the kahaDB tag).
>             For more information, see:
>
>             http://activemq.apache.org/persistence.html
>         -->
>         <persistenceAdapter>
>           <kahaDB directory="/media/SharedStorage/Binaries/activemq"
>                   indexCacheSize="40000"
>                   checkForCorruptJournalFiles="true">
>             <locker>
>               <shared-native-file-locker lockAcquireSleepInterval="5000"
> />
>             </locker>
>           </kahaDB>
>         </persistenceAdapter>
>
>
>         <!--
>           The systemUsage controls the maximum amount of space the broker
> will
>           use before disabling caching and/or slowing down producers. For
> more information, see:
>           http://activemq.apache.org/producer-flow-control.html
>         -->
>         <systemUsage>
>           <systemUsage>
>             <memoryUsage>
>               <memoryUsage percentOfJvmHeap="70" />
>             </memoryUsage>
>             <storeUsage>
>               <storeUsage limit="100 gb"/>
>             </storeUsage>
>               <tempUsage>
>                 <tempUsage limit="50 gb"/>
>             </tempUsage>
>           </systemUsage>
>         </systemUsage>
>
>         <!--
>             The transport connectors expose ActiveMQ over a given protocol
> to
>             clients and other brokers. For more information, see:
>
>             http://activemq.apache.org/configuring-transports.html
>         -->
>         <transportConnectors>
>           <!-- DOS protection, limit concurrent connections to 1000 and
> frame size to 100MB -->
>           <transportConnector name="openwire" uri="tcp://
> 0.0.0.0:61616?maximumConnections=1000&amp;wireFormat.maxFrameSize=
> 104857600
> "/>
>           <transportConnector name="amqp" uri="amqp://
> 0.0.0.0:5672?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600
> "/>
>           <transportConnector name="stomp" uri="stomp://
> 0.0.0.0:61613?maximumConnections=1000&amp;wireFormat.maxFrameSize=
> 104857600
> "/>
>           <transportConnector name="mqtt" uri="mqtt://
> 0.0.0.0:1883?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600
> "/>
>           <transportConnector name="ws" uri="ws://
> 0.0.0.0:61614?maximumConnections=1000&amp;wireFormat.maxFrameSize=
> 104857600
> "/>
>         </transportConnectors>
>
>         <!-- destroy the spring context on shutdown to stop jetty -->
>         <shutdownHooks>
>           <bean xmlns="http://www.springframework.org/schema/beans"
> class="org.apache.activemq.hooks.SpringContextHook" />
>         </shutdownHooks>
>
>     </broker>
>
>     <!--
>         Enable web consoles, REST and Ajax APIs and demos
>         The web consoles requires by default login, you can disable this in
> the jetty.xml file
>
>         Take a look at ${ACTIVEMQ_HOME}/conf/jetty.xml for more details
>     -->
>     <import resource="jetty.xml"/>
>
> </beans>
>
Reply | Threaded
Open this post in threaded view
|

Re: Log files not being cleaned up

Tim Bain
Responses inline.

On Fri, Oct 20, 2017 at 5:46 AM, Lionel van den Berg <[hidden email]>
wrote:

> Hi, thanks for the response.
>
> Some questions on these points from the troubleshooting.
>
>
>    1. *It contains a pending message for a destination or durable topic
>    subscription*
>
> This seems a little flawed, if a consumer who I have little control of is
> mis-behaving then my ActiveMQ can end up shutting down and unrecoverable.
> Is there some way of timing this out or similar?
>

There are multiple ways of discarding messages that are not being consumed,
which are detailed at http://activemq.apache.org/slow-consumer-handling.html
(several of which it sounds like you're already using). Keep in mind that
unconsumed DLQ messages are unconsumed messages, so you'll want to make
sure you address those messages as well;
http://activemq.apache.org/message-redelivery-and-dlq-handling.html
contains additional information about handling messages in the context of
the DLQ. And no, I wouldn't say it's flawed, it just means you have to do
some configuration work that you haven't yet done.


> *2. It contains an ack for a message which is in an in-use data file - the
> ack cannot be removed as a recovery would then mark the message for
> redelivery*
>
> Same comment as 1.
>

Same response as for #1. There's one additional wrinkle (KahaDB keeps an
entire data file alive because of a single live message, which in turn
means you have to keep the acks for the later messages which are in later
data files), but that's been partially mitigated by the addition of the
ability to compact acks by replaying them into the current data file, which
should allow any data file that contains no live non-ack messages to be
GC'ed. So there's a small portion of this that's purely the result of
KahaDB's design as a non-compacting data store, but it's a problem only
when there's an old unacknowledged message, which takes us back to #1.


> *3. The journal references a pending transaction*
>
> I'm not using transactions, but are there transactions under the hood?
>

No, this would only apply if you were directly using transactions, so this
doesn't apply to you.


> *4. It is a journal file, and there may be a pending write to it*
>
> Why would this be the case?
>

If we haven't finished flushing the file, using a buffer-then-flush
paradigm. This will be an infrequent situation, and should only be a small
number of data files, so if you're having a problem with the number of
files kept, it's not because of this. It's just included in the list for
completeness.

I'll see if I can change the logging settings, since the first occurrence
> the number of log files does not seem to have been an issue. I have it
> configured to keep messages for 7 days so regardless of the above
> conditions I would have thought that at that expiry the log would be
> cleaned up so we don't end up in such a situation where the system stops
> and cannot restart.
>

If you are indeed configured as you describe, I would think that log
cleanup would indeed happen as you expect, which means that either there's
an undiscovered bug in our code or you're not configured the way you think
you are.

The page I linked to originally has instructions for how to determine which
destinations have messages that are preventing the KahaDB data files from
being deleted, which might let you investigate further (for example, by
looking at the messages and their attributes to see if timestamps are being
set correctly).

Tim
Reply | Threaded
Open this post in threaded view
|

Re: Log files not being cleaned up

Lionel van den Berg
Thanks for the detailed response. I generally agree it's not flawed and it
is likely my configuration, I'm trying to take steps to track down the
cause, but of course it is behaving now.

I do think some defensiveness to protect it from shutting down regardless
of configuration would be a good idea. Is there some system in place that
would allow me to be alerted of issues that could be catastrophic before
they happen?

On 21 October 2017 at 14:08, Tim Bain <[hidden email]> wrote:

> Responses inline.
>
> On Fri, Oct 20, 2017 at 5:46 AM, Lionel van den Berg <[hidden email]>
> wrote:
>
> > Hi, thanks for the response.
> >
> > Some questions on these points from the troubleshooting.
> >
> >
> >    1. *It contains a pending message for a destination or durable topic
> >    subscription*
> >
> > This seems a little flawed, if a consumer who I have little control of is
> > mis-behaving then my ActiveMQ can end up shutting down and unrecoverable.
> > Is there some way of timing this out or similar?
> >
>
> There are multiple ways of discarding messages that are not being consumed,
> which are detailed at http://activemq.apache.org/
> slow-consumer-handling.html
> (several of which it sounds like you're already using). Keep in mind that
> unconsumed DLQ messages are unconsumed messages, so you'll want to make
> sure you address those messages as well;
> http://activemq.apache.org/message-redelivery-and-dlq-handling.html
> contains additional information about handling messages in the context of
> the DLQ. And no, I wouldn't say it's flawed, it just means you have to do
> some configuration work that you haven't yet done.
>
>
> > *2. It contains an ack for a message which is in an in-use data file -
> the
> > ack cannot be removed as a recovery would then mark the message for
> > redelivery*
> >
> > Same comment as 1.
> >
>
> Same response as for #1. There's one additional wrinkle (KahaDB keeps an
> entire data file alive because of a single live message, which in turn
> means you have to keep the acks for the later messages which are in later
> data files), but that's been partially mitigated by the addition of the
> ability to compact acks by replaying them into the current data file, which
> should allow any data file that contains no live non-ack messages to be
> GC'ed. So there's a small portion of this that's purely the result of
> KahaDB's design as a non-compacting data store, but it's a problem only
> when there's an old unacknowledged message, which takes us back to #1.
>
>
> > *3. The journal references a pending transaction*
> >
> > I'm not using transactions, but are there transactions under the hood?
> >
>
> No, this would only apply if you were directly using transactions, so this
> doesn't apply to you.
>
>
> > *4. It is a journal file, and there may be a pending write to it*
> >
> > Why would this be the case?
> >
>
> If we haven't finished flushing the file, using a buffer-then-flush
> paradigm. This will be an infrequent situation, and should only be a small
> number of data files, so if you're having a problem with the number of
> files kept, it's not because of this. It's just included in the list for
> completeness.
>
> I'll see if I can change the logging settings, since the first occurrence
> > the number of log files does not seem to have been an issue. I have it
> > configured to keep messages for 7 days so regardless of the above
> > conditions I would have thought that at that expiry the log would be
> > cleaned up so we don't end up in such a situation where the system stops
> > and cannot restart.
> >
>
> If you are indeed configured as you describe, I would think that log
> cleanup would indeed happen as you expect, which means that either there's
> an undiscovered bug in our code or you're not configured the way you think
> you are.
>
> The page I linked to originally has instructions for how to determine which
> destinations have messages that are preventing the KahaDB data files from
> being deleted, which might let you investigate further (for example, by
> looking at the messages and their attributes to see if timestamps are being
> set correctly).
>
> Tim
>
Reply | Threaded
Open this post in threaded view
|

Re: Log files not being cleaned up

Tim Bain
The broker does already have a variety of checks that attempt to warn you
defensively about certain potential problems and to prevent you from
running into them (producer flow control, for example), but I'm not aware
of a check specifically on the number of live KahaDB log files. In part,
I'm not aware of any universal limitation on the number of files that a
broker (or any arbitrary process, for that matter) can open sequentially,
so I'm not sure that anyone would have programmed a check against that
number, nor that there's a universal number that could be applied to all
installations no matter what their environments. If you wanted to provide
more information about exactly what errors are seen when the broker "shuts
down because there are too many files in the directory (I think it was
3000), [and] cannot be restarted we think due to a JVM limitation," we
could see if maybe there is indeed some reasonable threshold that could be
universally applied.

Tim

On Mon, Nov 6, 2017 at 3:05 AM, Lionel van den Berg <[hidden email]>
wrote:

> Thanks for the detailed response. I generally agree it's not flawed and it
> is likely my configuration, I'm trying to take steps to track down the
> cause, but of course it is behaving now.
>
> I do think some defensiveness to protect it from shutting down regardless
> of configuration would be a good idea. Is there some system in place that
> would allow me to be alerted of issues that could be catastrophic before
> they happen?
>
> On 21 October 2017 at 14:08, Tim Bain <[hidden email]> wrote:
>
> > Responses inline.
> >
> > On Fri, Oct 20, 2017 at 5:46 AM, Lionel van den Berg <[hidden email]>
> > wrote:
> >
> > > Hi, thanks for the response.
> > >
> > > Some questions on these points from the troubleshooting.
> > >
> > >
> > >    1. *It contains a pending message for a destination or durable topic
> > >    subscription*
> > >
> > > This seems a little flawed, if a consumer who I have little control of
> is
> > > mis-behaving then my ActiveMQ can end up shutting down and
> unrecoverable.
> > > Is there some way of timing this out or similar?
> > >
> >
> > There are multiple ways of discarding messages that are not being
> consumed,
> > which are detailed at http://activemq.apache.org/
> > slow-consumer-handling.html
> > (several of which it sounds like you're already using). Keep in mind that
> > unconsumed DLQ messages are unconsumed messages, so you'll want to make
> > sure you address those messages as well;
> > http://activemq.apache.org/message-redelivery-and-dlq-handling.html
> > contains additional information about handling messages in the context of
> > the DLQ. And no, I wouldn't say it's flawed, it just means you have to do
> > some configuration work that you haven't yet done.
> >
> >
> > > *2. It contains an ack for a message which is in an in-use data file -
> > the
> > > ack cannot be removed as a recovery would then mark the message for
> > > redelivery*
> > >
> > > Same comment as 1.
> > >
> >
> > Same response as for #1. There's one additional wrinkle (KahaDB keeps an
> > entire data file alive because of a single live message, which in turn
> > means you have to keep the acks for the later messages which are in later
> > data files), but that's been partially mitigated by the addition of the
> > ability to compact acks by replaying them into the current data file,
> which
> > should allow any data file that contains no live non-ack messages to be
> > GC'ed. So there's a small portion of this that's purely the result of
> > KahaDB's design as a non-compacting data store, but it's a problem only
> > when there's an old unacknowledged message, which takes us back to #1.
> >
> >
> > > *3. The journal references a pending transaction*
> > >
> > > I'm not using transactions, but are there transactions under the hood?
> > >
> >
> > No, this would only apply if you were directly using transactions, so
> this
> > doesn't apply to you.
> >
> >
> > > *4. It is a journal file, and there may be a pending write to it*
> > >
> > > Why would this be the case?
> > >
> >
> > If we haven't finished flushing the file, using a buffer-then-flush
> > paradigm. This will be an infrequent situation, and should only be a
> small
> > number of data files, so if you're having a problem with the number of
> > files kept, it's not because of this. It's just included in the list for
> > completeness.
> >
> > I'll see if I can change the logging settings, since the first occurrence
> > > the number of log files does not seem to have been an issue. I have it
> > > configured to keep messages for 7 days so regardless of the above
> > > conditions I would have thought that at that expiry the log would be
> > > cleaned up so we don't end up in such a situation where the system
> stops
> > > and cannot restart.
> > >
> >
> > If you are indeed configured as you describe, I would think that log
> > cleanup would indeed happen as you expect, which means that either
> there's
> > an undiscovered bug in our code or you're not configured the way you
> think
> > you are.
> >
> > The page I linked to originally has instructions for how to determine
> which
> > destinations have messages that are preventing the KahaDB data files from
> > being deleted, which might let you investigate further (for example, by
> > looking at the messages and their attributes to see if timestamps are
> being
> > set correctly).
> >
> > Tim
> >
>