Failed to fill batch

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Failed to fill batch

darhon
I get a lot of errors in activemq.log:

2016-06-11 22:23:50,616 | ERROR | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@queue_name,batchResetNeeded=false,size=2437,cacheEnabled=false,maxBatchSize:200,hasSpace:true,pendingCachedIds.size:0,lastSyncCachedId:null,lastSyncCachedId-seq:null,lastAsyncCachedId:null,lastAsyncCachedId-seq:null,store=permits:10000,sd=nextSeq:257467,lastRet:MessageOrderCursor:[def:255030, low:0, high:0],pending:0 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | ActiveMQ BrokerService[erased] Task-7598
java.lang.ClassCastException
2016-06-11 22:23:50,618 | ERROR | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@queue_name,batchResetNeeded=false,size=2437,cacheEnabled=false,maxBatchSize:200,hasSpace:true,pendingCachedIds.size:0,lastSyncCachedId:null,lastSyncCachedId-seq:null,lastAsyncCachedId:null,lastAsyncCachedId-seq:null,store=permits:10000,sd=nextSeq:257467,lastRet:MessageOrderCursor:[def:255030, low:0, high:0],pending:0 - Failed to fill batch | org.apache.activemq.broker.region.cursors.AbstractStoreCursor | ActiveMQ BrokerService[erased] Task-7598
java.lang.RuntimeException: java.lang.ClassCastException
        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:409)[activemq-broker-5.13.3.jar:5.13.3]
        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:149)[activemq-broker-5.13.3.jar:5.13.3]
        at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:169)[activemq-broker-5.13.3.jar:5.13.3]
        at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1947)[activemq-broker-5.13.3.jar:5.13.3]
        at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2172)[activemq-broker-5.13.3.jar:5.13.3]
        at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1619)[activemq-broker-5.13.3.jar:5.13.3]
        at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)[activemq-client-5.13.3.jar:5.13.3]
        at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)[activemq-client-5.13.3.jar:5.13.3]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_71]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_71]
        at java.lang.Thread.run(Thread.java:745)[:1.8.0_71]
2016-06-11 22:23:50,619 | ERROR | Failed to page in more queue messages  | org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[erased] Task-7598
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassCastException
        at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:152)[activemq-broker-5.13.3.jar:5.13.3]
        at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:169)[activemq-broker-5.13.3.jar:5.13.3]
        at org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1947)[activemq-broker-5.13.3.jar:5.13.3]
        at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2172)[activemq-broker-5.13.3.jar:5.13.3]
        at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1619)[activemq-broker-5.13.3.jar:5.13.3]
        at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)[activemq-client-5.13.3.jar:5.13.3]
        at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)[activemq-client-5.13.3.jar:5.13.3]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_71]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_71]
        at java.lang.Thread.run(Thread.java:745)[:1.8.0_71]


but I can not understand what exactly causes this errors.

ActiveMQ version: 5.13.3
Configuration master-slave with shared KahaDB on GlusterFS

My current activemq.xml:

<beans
  xmlns="http://www.springframework.org/schema/beans"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
  http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">

    <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
        <property name="locations">
            <value>file:${activemq.conf}/credentials.properties</value>
        </property>
    </bean>

    <bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
          lazy-init="false" scope="singleton"
          init-method="start" destroy-method="stop">
    </bean>

    <broker xmlns="http://activemq.apache.org/schema/core" brokerName="erased" dataDirectory="/opt/activemq/data"
            schedulerSupport="true" persistent="true">

        <destinationPolicy>
            <policyMap>
              <policyEntries>
                <policyEntry topic=">" >
                  <pendingMessageLimitStrategy>
                    <constantPendingMessageLimitStrategy limit="2000"/>
                  </pendingMessageLimitStrategy>
                </policyEntry>
                <policyEntry queue=">" prioritizedMessages="true" useCache="false" expireMessagesPeriod="0" queuePrefetch="1" />
              </policyEntries>
            </policyMap>
        </destinationPolicy>

        <managementContext>
            <managementContext createConnector="true"/>
        </managementContext>

        <persistenceAdapter>
            <kahaDB directory="/opt/activemq/data/kahadb"/>
        </persistenceAdapter>

          <systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage percentOfJvmHeap="80" />
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="40 gb"/>
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="10 gb"/>
                </tempUsage>
            </systemUsage>
        </systemUsage>

        <transportConnectors>
            <transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=8000&amp;wireFormat.maxFrameSize=104857600"/>
            <transportConnector name="stomp+nio" uri="stomp+nio://0.0.0.0:61613?maximumConnections=8000&amp;wireFormat.maxFrameSize=104857600"/>
        </transportConnectors>

        <shutdownHooks>
            <bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook" />
        </shutdownHooks>

    </broker>

    <import resource="jetty.xml"/>

</beans>


Can anyone help to find the cause of the problem?
Reply | Threaded
Open this post in threaded view
|

Re: Failed to fill batch

Tim Bain
This has been seen before (e.g.
http://activemq.2283324.n4.nabble.com/ClassCastException-while-subscribing-to-a-topic-tp4710870.html,
among others), but because we're not able to see the stack trace for the
inner ClassCastException, we don't know what's causing it.

The post I linked to is in the case of a durable topic subscription,
whereas your example is with a queue, so it's something general and
cross-cutting, but we don't know what.

Are you able to reproduce this behavior on a broker whose operation you can
afford to interrupt briefly?  If so, please do the following:
* Download the source for the version of ActiveMQ you're using (5.13.3).
* In your IDE of choice, open the ActiveMQ source as new projects.
* Edit the ActiveMQ start script to cause the JVM to open a remote debug
port (
http://stackoverflow.com/questions/975271/remote-debugging-a-java-application
).
* Attach the debugger from your IDE of choice to the broker as a remote
process.  The exact steps will depend on your IDE, but are well-documented
for the main IDEs, so Google based on your IDE.
* Set a breakpoint on line 409 of
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.
* When you hit that breakpoint, use your debugger's tools to get the inner
exception (the one for the ClassCastException) and print its stack trace
(e.g. by calling printStackTrace() on it and then looking for that output
either in the IDE's console or in the ActiveMQ log files).  In Eclipse,
you'll use the Watch view; other IDEs may have different methods to do this.
* Show us that inner stack trace, which might let us figure out what's
going on.

Tim

On Jun 13, 2016 5:49 AM, "darhon" <[hidden email]> wrote:

> I get a lot of errors in activemq.log:
>
> 2016-06-11 22:23:50,616 | ERROR |
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch@queue_name
> ,batchResetNeeded=false,size=2437,cacheEnabled=false,maxBatchSize:200,hasSpace:true,pendingCachedIds.size:0,lastSyncCachedId:null,lastSyncCachedId-seq:null,lastAsyncCachedId:null,lastAsyncCachedId-seq:null,store=permits:10000,sd=nextSeq:257467,lastRet:MessageOrderCursor:[def:255030,
> low:0, high:0],pending:0 - Failed to fill batch |
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor | ActiveMQ
> BrokerService[erased] Task-7598
> java.lang.ClassCastException
> 2016-06-11 22:23:50,618 | ERROR |
> org.apache.activemq.broker.region.cursors.QueueStorePrefetch@queue_name
> ,batchResetNeeded=false,size=2437,cacheEnabled=false,maxBatchSize:200,hasSpace:true,pendingCachedIds.size:0,lastSyncCachedId:null,lastSyncCachedId-seq:null,lastAsyncCachedId:null,lastAsyncCachedId-seq:null,store=permits:10000,sd=nextSeq:257467,lastRet:MessageOrderCursor:[def:255030,
> low:0, high:0],pending:0 - Failed to fill batch |
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor | ActiveMQ
> BrokerService[erased] Task-7598
> java.lang.RuntimeException: java.lang.ClassCastException
>         at
>
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:409)[activemq-broker-5.13.3.jar:5.13.3]
>         at
>
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:149)[activemq-broker-5.13.3.jar:5.13.3]
>         at
>
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:169)[activemq-broker-5.13.3.jar:5.13.3]
>         at
>
> org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1947)[activemq-broker-5.13.3.jar:5.13.3]
>         at
>
> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2172)[activemq-broker-5.13.3.jar:5.13.3]
>         at
>
> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1619)[activemq-broker-5.13.3.jar:5.13.3]
>         at
>
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)[activemq-client-5.13.3.jar:5.13.3]
>         at
>
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)[activemq-client-5.13.3.jar:5.13.3]
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_71]
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_71]
>         at java.lang.Thread.run(Thread.java:745)[:1.8.0_71]
> 2016-06-11 22:23:50,619 | ERROR | Failed to page in more queue messages  |
> org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[erased]
> Task-7598
> java.lang.RuntimeException: java.lang.RuntimeException:
> java.lang.ClassCastException
>         at
>
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:152)[activemq-broker-5.13.3.jar:5.13.3]
>         at
>
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:169)[activemq-broker-5.13.3.jar:5.13.3]
>         at
>
> org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1947)[activemq-broker-5.13.3.jar:5.13.3]
>         at
>
> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2172)[activemq-broker-5.13.3.jar:5.13.3]
>         at
>
> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1619)[activemq-broker-5.13.3.jar:5.13.3]
>         at
>
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)[activemq-client-5.13.3.jar:5.13.3]
>         at
>
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)[activemq-client-5.13.3.jar:5.13.3]
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_71]
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_71]
>         at java.lang.Thread.run(Thread.java:745)[:1.8.0_71]
>
>
> but I can not understand what exactly causes this errors.
>
> ActiveMQ version: 5.13.3
> Configuration master-slave with shared KahaDB on GlusterFS
>
> My current activemq.xml:
>
> <beans
>   xmlns="http://www.springframework.org/schema/beans"
>   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>   xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd
>   http://activemq.apache.org/schema/core
> http://activemq.apache.org/schema/core/activemq-core.xsd">
>
>     <bean
>
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>         <property name="locations">
>             <value>file:${activemq.conf}/credentials.properties</value>
>         </property>
>     </bean>
>
>     <bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
>           lazy-init="false" scope="singleton"
>           init-method="start" destroy-method="stop">
>     </bean>
>
>     <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="erased" dataDirectory="/opt/activemq/data"
>             schedulerSupport="true" persistent="true">
>
>         <destinationPolicy>
>             <policyMap>
>               <policyEntries>
>                 <policyEntry topic=">" >
>                   <pendingMessageLimitStrategy>
>                     <constantPendingMessageLimitStrategy limit="2000"/>
>                   </pendingMessageLimitStrategy>
>                 </policyEntry>
>                 <policyEntry queue=">" prioritizedMessages="true"
> useCache="false" expireMessagesPeriod="0" queuePrefetch="1" />
>               </policyEntries>
>             </policyMap>
>         </destinationPolicy>
>
>         <managementContext>
>             <managementContext createConnector="true"/>
>         </managementContext>
>
>         <persistenceAdapter>
>             <kahaDB directory="/opt/activemq/data/kahadb"/>
>         </persistenceAdapter>
>
>           <systemUsage>
>             <systemUsage>
>                 <memoryUsage>
>                     <memoryUsage percentOfJvmHeap="80" />
>                 </memoryUsage>
>                 <storeUsage>
>                     <storeUsage limit="40 gb"/>
>                 </storeUsage>
>                 <tempUsage>
>                     <tempUsage limit="10 gb"/>
>                 </tempUsage>
>             </systemUsage>
>         </systemUsage>
>
>         <transportConnectors>
>             <transportConnector name="openwire"
> uri="tcp://
> 0.0.0.0:61616?maximumConnections=8000&amp;wireFormat.maxFrameSize=104857600
> "/>
>             <transportConnector name="stomp+nio"
> uri="stomp+nio://
> 0.0.0.0:61613?maximumConnections=8000&amp;wireFormat.maxFrameSize=104857600
> "/>
>         </transportConnectors>
>
>         <shutdownHooks>
>             <bean xmlns="http://www.springframework.org/schema/beans"
> class="org.apache.activemq.hooks.SpringContextHook" />
>         </shutdownHooks>
>
>     </broker>
>
>     <import resource="jetty.xml"/>
>
> </beans>
>
>
> Can anyone help to find the cause of the problem?
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Failed-to-fill-batch-tp4712924.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Failed to fill batch

christopher.l.shannon
You can also try setting this compiler flag: -XX:-OmitStackTraceInFastThrow
and seeing if that will show the inner stack trace as it might be an
optimization to hide it because log4j should usually log the inner stack
trace.

From Oracle:
"The compiler in the server VM now provides correct stack backtraces for
all "cold" built-in exceptions. For performance purposes, when such an
exception is thrown a few times, the method may be recompiled. After
recompilation, the compiler may choose a faster tactic using preallocated
exceptions that do not provide a stack trace. To disable completely the use
of preallocated exceptions, use this new flag:
 -XX:-OmitStackTraceInFastThrow."

On Tue, Jun 14, 2016 at 8:42 AM, Tim Bain <[hidden email]> wrote:

> This has been seen before (e.g.
>
> http://activemq.2283324.n4.nabble.com/ClassCastException-while-subscribing-to-a-topic-tp4710870.html
> ,
> among others), but because we're not able to see the stack trace for the
> inner ClassCastException, we don't know what's causing it.
>
> The post I linked to is in the case of a durable topic subscription,
> whereas your example is with a queue, so it's something general and
> cross-cutting, but we don't know what.
>
> Are you able to reproduce this behavior on a broker whose operation you can
> afford to interrupt briefly?  If so, please do the following:
> * Download the source for the version of ActiveMQ you're using (5.13.3).
> * In your IDE of choice, open the ActiveMQ source as new projects.
> * Edit the ActiveMQ start script to cause the JVM to open a remote debug
> port (
>
> http://stackoverflow.com/questions/975271/remote-debugging-a-java-application
> ).
> * Attach the debugger from your IDE of choice to the broker as a remote
> process.  The exact steps will depend on your IDE, but are well-documented
> for the main IDEs, so Google based on your IDE.
> * Set a breakpoint on line 409 of
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.
> * When you hit that breakpoint, use your debugger's tools to get the inner
> exception (the one for the ClassCastException) and print its stack trace
> (e.g. by calling printStackTrace() on it and then looking for that output
> either in the IDE's console or in the ActiveMQ log files).  In Eclipse,
> you'll use the Watch view; other IDEs may have different methods to do
> this.
> * Show us that inner stack trace, which might let us figure out what's
> going on.
>
> Tim
>
> On Jun 13, 2016 5:49 AM, "darhon" <[hidden email]> wrote:
>
> > I get a lot of errors in activemq.log:
> >
> > 2016-06-11 22:23:50,616 | ERROR |
> > org.apache.activemq.broker.region.cursors.QueueStorePrefetch@queue_name
> >
> ,batchResetNeeded=false,size=2437,cacheEnabled=false,maxBatchSize:200,hasSpace:true,pendingCachedIds.size:0,lastSyncCachedId:null,lastSyncCachedId-seq:null,lastAsyncCachedId:null,lastAsyncCachedId-seq:null,store=permits:10000,sd=nextSeq:257467,lastRet:MessageOrderCursor:[def:255030,
> > low:0, high:0],pending:0 - Failed to fill batch |
> > org.apache.activemq.broker.region.cursors.AbstractStoreCursor | ActiveMQ
> > BrokerService[erased] Task-7598
> > java.lang.ClassCastException
> > 2016-06-11 22:23:50,618 | ERROR |
> > org.apache.activemq.broker.region.cursors.QueueStorePrefetch@queue_name
> >
> ,batchResetNeeded=false,size=2437,cacheEnabled=false,maxBatchSize:200,hasSpace:true,pendingCachedIds.size:0,lastSyncCachedId:null,lastSyncCachedId-seq:null,lastAsyncCachedId:null,lastAsyncCachedId-seq:null,store=permits:10000,sd=nextSeq:257467,lastRet:MessageOrderCursor:[def:255030,
> > low:0, high:0],pending:0 - Failed to fill batch |
> > org.apache.activemq.broker.region.cursors.AbstractStoreCursor | ActiveMQ
> > BrokerService[erased] Task-7598
> > java.lang.RuntimeException: java.lang.ClassCastException
> >         at
> >
> >
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:409)[activemq-broker-5.13.3.jar:5.13.3]
> >         at
> >
> >
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:149)[activemq-broker-5.13.3.jar:5.13.3]
> >         at
> >
> >
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:169)[activemq-broker-5.13.3.jar:5.13.3]
> >         at
> >
> >
> org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1947)[activemq-broker-5.13.3.jar:5.13.3]
> >         at
> >
> >
> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2172)[activemq-broker-5.13.3.jar:5.13.3]
> >         at
> >
> >
> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1619)[activemq-broker-5.13.3.jar:5.13.3]
> >         at
> >
> >
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)[activemq-client-5.13.3.jar:5.13.3]
> >         at
> >
> >
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)[activemq-client-5.13.3.jar:5.13.3]
> >         at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_71]
> >         at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_71]
> >         at java.lang.Thread.run(Thread.java:745)[:1.8.0_71]
> > 2016-06-11 22:23:50,619 | ERROR | Failed to page in more queue messages
> |
> > org.apache.activemq.broker.region.Queue | ActiveMQ BrokerService[erased]
> > Task-7598
> > java.lang.RuntimeException: java.lang.RuntimeException:
> > java.lang.ClassCastException
> >         at
> >
> >
> org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:152)[activemq-broker-5.13.3.jar:5.13.3]
> >         at
> >
> >
> org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:169)[activemq-broker-5.13.3.jar:5.13.3]
> >         at
> >
> >
> org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1947)[activemq-broker-5.13.3.jar:5.13.3]
> >         at
> >
> >
> org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2172)[activemq-broker-5.13.3.jar:5.13.3]
> >         at
> >
> >
> org.apache.activemq.broker.region.Queue.iterate(Queue.java:1619)[activemq-broker-5.13.3.jar:5.13.3]
> >         at
> >
> >
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)[activemq-client-5.13.3.jar:5.13.3]
> >         at
> >
> >
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)[activemq-client-5.13.3.jar:5.13.3]
> >         at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)[:1.8.0_71]
> >         at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)[:1.8.0_71]
> >         at java.lang.Thread.run(Thread.java:745)[:1.8.0_71]
> >
> >
> > but I can not understand what exactly causes this errors.
> >
> > ActiveMQ version: 5.13.3
> > Configuration master-slave with shared KahaDB on GlusterFS
> >
> > My current activemq.xml:
> >
> > <beans
> >   xmlns="http://www.springframework.org/schema/beans"
> >   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> >   xsi:schemaLocation="http://www.springframework.org/schema/beans
> > http://www.springframework.org/schema/beans/spring-beans.xsd
> >   http://activemq.apache.org/schema/core
> > http://activemq.apache.org/schema/core/activemq-core.xsd">
> >
> >     <bean
> >
> >
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
> >         <property name="locations">
> >             <value>file:${activemq.conf}/credentials.properties</value>
> >         </property>
> >     </bean>
> >
> >     <bean id="logQuery"
> class="io.fabric8.insight.log.log4j.Log4jLogQuery"
> >           lazy-init="false" scope="singleton"
> >           init-method="start" destroy-method="stop">
> >     </bean>
> >
> >     <broker xmlns="http://activemq.apache.org/schema/core"
> > brokerName="erased" dataDirectory="/opt/activemq/data"
> >             schedulerSupport="true" persistent="true">
> >
> >         <destinationPolicy>
> >             <policyMap>
> >               <policyEntries>
> >                 <policyEntry topic=">" >
> >                   <pendingMessageLimitStrategy>
> >                     <constantPendingMessageLimitStrategy limit="2000"/>
> >                   </pendingMessageLimitStrategy>
> >                 </policyEntry>
> >                 <policyEntry queue=">" prioritizedMessages="true"
> > useCache="false" expireMessagesPeriod="0" queuePrefetch="1" />
> >               </policyEntries>
> >             </policyMap>
> >         </destinationPolicy>
> >
> >         <managementContext>
> >             <managementContext createConnector="true"/>
> >         </managementContext>
> >
> >         <persistenceAdapter>
> >             <kahaDB directory="/opt/activemq/data/kahadb"/>
> >         </persistenceAdapter>
> >
> >           <systemUsage>
> >             <systemUsage>
> >                 <memoryUsage>
> >                     <memoryUsage percentOfJvmHeap="80" />
> >                 </memoryUsage>
> >                 <storeUsage>
> >                     <storeUsage limit="40 gb"/>
> >                 </storeUsage>
> >                 <tempUsage>
> >                     <tempUsage limit="10 gb"/>
> >                 </tempUsage>
> >             </systemUsage>
> >         </systemUsage>
> >
> >         <transportConnectors>
> >             <transportConnector name="openwire"
> > uri="tcp://
> >
> 0.0.0.0:61616?maximumConnections=8000&amp;wireFormat.maxFrameSize=104857600
> > "/>
> >             <transportConnector name="stomp+nio"
> > uri="stomp+nio://
> >
> 0.0.0.0:61613?maximumConnections=8000&amp;wireFormat.maxFrameSize=104857600
> > "/>
> >         </transportConnectors>
> >
> >         <shutdownHooks>
> >             <bean xmlns="http://www.springframework.org/schema/beans"
> > class="org.apache.activemq.hooks.SpringContextHook" />
> >         </shutdownHooks>
> >
> >     </broker>
> >
> >     <import resource="jetty.xml"/>
> >
> > </beans>
> >
> >
> > Can anyone help to find the cause of the problem?
> >
> >
> >
> > --
> > View this message in context:
> >
> http://activemq.2283324.n4.nabble.com/Failed-to-fill-batch-tp4712924.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Failed to fill batch

darhon
In reply to this post by Tim Bain
Unfortunately, these errors was on production system, but I'll try to reproduce it in test environment.

At this moment errors disappeared after I cleaned my KahaDB from old messages (total size was about 8GB before cleaning because ActivemMQ 5.13.3 still doesn't properly cleanup  journal files) and the size of the database has become about 200-400MB
Reply | Threaded
Open this post in threaded view
|

Re: Failed to fill batch

Tim Bain
5.13.3 is supposed to allow durable subscription rewrites, which should
allow garbage collection of old data files if the subscription messages are
all that's keeping the file alive.  Having unconsumed real messages,
however, will still prevent garbage collection.

Do you have unconsumed real messages?  Or does your statement mean that
maybe the fix in 5.13.3 isn't working as designed?

Tim
On Jun 15, 2016 7:26 AM, "darhon" <[hidden email]> wrote:

> Unfortunately, these errors was on production system, but I'll try to
> reproduce it in test environment.
>
> At this moment errors disappeared after I cleaned my KahaDB from old
> messages (total size was about 8GB before cleaning because ActivemMQ 5.13.3
> still doesn't properly cleanup  journal files) and the size of the database
> has become about 200-400MB
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Failed-to-fill-batch-tp4712924p4713003.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>