Another NegativeArraySizeException problem - ActiveMQ 5.5.1.fuse-70-097

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Another NegativeArraySizeException problem - ActiveMQ 5.5.1.fuse-70-097

cristisor
This post was updated on .
Hello,

We are facing a really big problem in production. From time to time, the broker stops delivering the messages to some consumers which are located on the same machine as the broker. After a restart, the broker delivers some more messages but new messages can accumulate. The only way to fix the problem for another 2 - 3 weeks is to clear the kahaDB, but this shouldn't be happening in production.

Finally we managed to get a copy of the kahaDB before the support team restarted the broker, and when I deployed it into my broker, I received the following error at broker initialization:

2014/07/10 15:57:15,033 | INFO  | WrapperSimpleAppMain | MessageDatabase          | ahadb.MessageDatabase$Metadata  155 | KahaDB is version 4
2014/07/10 15:57:16,502 | ERROR | WrapperSimpleAppMain | BrokerService            | .activemq.broker.BrokerService  555 | Failed to start ActiveMQ JMS Message Broker (fpskylla_activemq, null). Reason: java.io.IOException: Invalid location: 1193:28702079, : java.lang.NegativeArraySizeException
java.io.IOException: Invalid location: 1193:28702079, : java.lang.NegativeArraySizeException
        at org.apache.kahadb.journal.DataFileAccessor.readRecord(DataFileAccessor.java:94)
        at org.apache.kahadb.journal.Journal.read(Journal.java:601)
        at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:898)
        at org.apache.activemq.store.kahadb.MessageDatabase.recoverProducerAudit(MessageDatabase.java:577)
        at org.apache.activemq.store.kahadb.MessageDatabase.recover(MessageDatabase.java:504)
        at org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:360)
        at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:408)
        at org.apache.activemq.store.kahadb.MessageDatabase.doStart(MessageDatabase.java:237)
        at org.apache.activemq.store.kahadb.KahaDBStore.doStart(KahaDBStore.java:176)
        at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:54)
        at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.start(KahaDBPersistenceAdapter.java:188)
        at org.apache.activemq.broker.BrokerService.start(BrokerService.java:512)
        at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:60)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1612)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1553)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1483)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:524)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:461)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:295)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:223)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:292)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:285)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:285)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:608)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:932)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:479)
        at org.apache.servicemix.Main.main(Main.java:69)
------------------------------------------------------------


The broker initialization fails and everything stops. If I restart the broker, everything goes ok and I see the following logging:

2014/07/10 16:06:53,330 | INFO  | WrapperSimpleAppMain | BrokerService            | .activemq.broker.BrokerService  508 | Using Persistence Adapter: KahaDBPersistenceAdapter[d:\activemq\apache-servicemix-3.6.0-fuse-00-89\activemq-data]
2014/07/10 16:06:53,963 | INFO  | WrapperSimpleAppMain | MessageDatabase          | ahadb.MessageDatabase$Metadata  155 | KahaDB is version 4
2014/07/10 16:06:55,387 | INFO  | WrapperSimpleAppMain | MessageDatabase          | q.store.kahadb.MessageDatabase  511 | Recovering from the journal ...
2014/07/10 16:06:55,393 | DEBUG | WrapperSimpleAppMain | MessageDatabase          | q.store.kahadb.MessageDatabase 1212 | message not found in sequence id index: ID:FR0502-1079-1398479702307-13:1:1:1:832
2014/07/10 16:06:56,902 | DEBUG | WrapperSimpleAppMain | MessageDatabase          | q.store.kahadb.MessageDatabase 1212 | message not found in sequence id index: ID:FR0502-1143-1403750130523-13:1:1:1:696
2014/07/10 16:06:57,552 | DEBUG | WrapperSimpleAppMain | MessageDatabase          | q.store.kahadb.MessageDatabase 1230 | no message sequence exists for id: ID:FR050204-1068-1404358678116-2:1:53:1:9 and sub: PosTransaction_necExportService:PosTransaction_necExportService
2014/07/10 16:06:57,552 | DEBUG | WrapperSimpleAppMain | MessageDatabase          | q.store.kahadb.MessageDatabase 1230 | no message sequence exists for id: ID:FR050204-1068-1404358678116-2:1:53:1:9 and sub: eReSdcBS-receiver_FR0502:eReSdcBS-receiver_FR0502_eres-postransaction-01-0502-@
2014/07/10 16:06:58,088 | DEBUG | WrapperSimpleAppMain | MessageDatabase          | q.store.kahadb.MessageDatabase 1212 | message not found in sequence id index: ID:FR0502-1280-1404441548401-13:1:1:1:896
2014/07/10 16:06:58,527 | INFO  | WrapperSimpleAppMain | MessageDatabase          | q.store.kahadb.MessageDatabase  524 | Recovery replayed 33353 operations from the journal in 3.152 seconds.
2014/07/10 16:06:58,542 | INFO  | WrapperSimpleAppMain | BrokerService            | .activemq.broker.BrokerService  785 | ActiveMQ 5.5.1.fuse-70-097 JMS Message Broker (fpskylla_activemq) is starting


If I delete the db.data and db.redo, forcing an index recreation, the broker starts without errors from the very beginning.

I spent a lot of time investigating this problem and I came across some known bugs (AMQ-3104, AMQ-3422, AMQ-3775, AMQ-3806, MB-1046), but I have the fixes in the ActiveMQ version that I am using. I am very confused by the fact that my ActiveMQ version, 5.5.1.fuse-70-097, contains fixes which should be found in the ActiveMQ 5.6.0. Also, the prefetchPolicy has been set to 1 and the auto acknowledge mode is on.

Do you think that a thread dump would help me? Any idea is really appreciated because we are stuck.

If somebody can analyze the kahaDB, I would be happy to zip it and send it.

Many thanks,
Cristian.
Reply | Threaded
Open this post in threaded view
|

Re: Another NegativeArraySizeException problem - ActiveMQ 5.5.1.fuse-70-097

cristisor
There might be a timeout problem when shutting down some Camel routes which work with ActiveMQ: http://camel.465427.n5.nabble.com/Camel-ActiveMQ-shutdown-timeout-problem-td5753919.html.

What do you think?