org.apache.activemq.kaha.impl.index.?

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

org.apache.activemq.kaha.impl.index.?

Jonathan Roby
Hi, i've just joined the list so apologies if i've not this correctly.


We run a number of islandora instances that use ActiveMQ 5.7.0 for message passing between components. Yesterday, one of our instances  seemed to stop working. after several restarts of components and the system itself, I noticed that when the components restarted, it always produced a nested exception and the very last error was org.apache.activemq.kaha.impl.index.BadMagicException.


I've googled this and taken the advice given, which is usually to empty/delete the data directory or the kahadb directory and it still isnt working. All the other components seem to be working fine.


Thank you for any help anyone can give me.

Jon


Here is the last part of the stack trace.


Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: null
    at org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.broker.BrokerService.start(BrokerService.java:578) ~[activemq-core-5.7.0.jar:5.7.0]
    at org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539) ~[activemq-core-5.7.0.jar:5.7.0]
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_162]
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_162]
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_162]
    at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
    at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
    at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
    at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
Reply | Threaded
Open this post in threaded view
|

Re: org.apache.activemq.kaha.impl.index.?

jbonofre
Hi,

Does the kahadb folder has been created with the same ActiveMQ version
(and you updated ActiveMQ version later) ?

It seems the kahadb index is not consistent.

Regards
JB

On 02/11/2019 18:14, Jonathan Roby wrote:

> Hi, i've just joined the list so apologies if i've not this correctly.
>
>
> We run a number of islandora instances that use ActiveMQ 5.7.0 for message passing between components. Yesterday, one of our instances  seemed to stop working. after several restarts of components and the system itself, I noticed that when the components restarted, it always produced a nested exception and the very last error was org.apache.activemq.kaha.impl.index.BadMagicException.
>
>
> I've googled this and taken the advice given, which is usually to empty/delete the data directory or the kahadb directory and it still isnt working. All the other components seem to be working fine.
>
>
> Thank you for any help anyone can give me.
>
> Jon
>
>
> Here is the last part of the stack trace.
>
>
> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: null
>     at org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.BrokerService.start(BrokerService.java:578) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539) ~[activemq-core-5.7.0.jar:5.7.0]
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_162]
>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_162]
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_162]
>     at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
>     at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>     at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>     at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: org.apache.activemq.kaha.impl.index.?

Jonathan Roby
Hi


as far as I know, the version of ActiveMQ hasn't been updated and i've now deleted the kahadb directory a few times (it regenerated) and even moved the broker path in case it was a drive error


Jon

________________________________
From: Jean-Baptiste Onofré <[hidden email]>
Sent: Saturday, November 2, 2019 12:16:31 PM
To: [hidden email]
Subject: Re: org.apache.activemq.kaha.impl.index.?

Hi,

Does the kahadb folder has been created with the same ActiveMQ version
(and you updated ActiveMQ version later) ?

It seems the kahadb index is not consistent.

Regards
JB

On 02/11/2019 18:14, Jonathan Roby wrote:

> Hi, i've just joined the list so apologies if i've not this correctly.
>
>
> We run a number of islandora instances that use ActiveMQ 5.7.0 for message passing between components. Yesterday, one of our instances  seemed to stop working. after several restarts of components and the system itself, I noticed that when the components restarted, it always produced a nested exception and the very last error was org.apache.activemq.kaha.impl.index.BadMagicException.
>
>
> I've googled this and taken the advice given, which is usually to empty/delete the data directory or the kahadb directory and it still isnt working. All the other components seem to be working fine.
>
>
> Thank you for any help anyone can give me.
>
> Jon
>
>
> Here is the last part of the stack trace.
>
>
> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: null
>     at org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.BrokerService.start(BrokerService.java:578) ~[activemq-core-5.7.0.jar:5.7.0]
>     at org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539) ~[activemq-core-5.7.0.jar:5.7.0]
>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_162]
>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_162]
>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_162]
>     at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
>     at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>     at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>     at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: org.apache.activemq.kaha.impl.index.?

jbonofre
Hi,

It's definitely something weird while reading file on the filesystem.

Is it a local filesystem or shared one (NFS, CIFS, ...) ?

Regards
JB

On 02/11/2019 18:18, Jonathan Roby wrote:

> Hi
>
>
> as far as I know, the version of ActiveMQ hasn't been updated and i've now deleted the kahadb directory a few times (it regenerated) and even moved the broker path in case it was a drive error
>
>
> Jon
>
> ________________________________
> From: Jean-Baptiste Onofré <[hidden email]>
> Sent: Saturday, November 2, 2019 12:16:31 PM
> To: [hidden email]
> Subject: Re: org.apache.activemq.kaha.impl.index.?
>
> Hi,
>
> Does the kahadb folder has been created with the same ActiveMQ version
> (and you updated ActiveMQ version later) ?
>
> It seems the kahadb index is not consistent.
>
> Regards
> JB
>
> On 02/11/2019 18:14, Jonathan Roby wrote:
>> Hi, i've just joined the list so apologies if i've not this correctly.
>>
>>
>> We run a number of islandora instances that use ActiveMQ 5.7.0 for message passing between components. Yesterday, one of our instances  seemed to stop working. after several restarts of components and the system itself, I noticed that when the components restarted, it always produced a nested exception and the very last error was org.apache.activemq.kaha.impl.index.BadMagicException.
>>
>>
>> I've googled this and taken the advice given, which is usually to empty/delete the data directory or the kahadb directory and it still isnt working. All the other components seem to be working fine.
>>
>>
>> Thank you for any help anyone can give me.
>>
>> Jon
>>
>>
>> Here is the last part of the stack trace.
>>
>>
>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: null
>>     at org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.BrokerService.start(BrokerService.java:578) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_162]
>>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_162]
>>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_162]
>>     at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
>>     at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>     at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>     at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>
>
> --
> Jean-Baptiste Onofré
> [hidden email]
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: org.apache.activemq.kaha.impl.index.?

jbonofre
I guess it's fixed with you complete remove the kahadb folder right ?

I remember couple of users mentioned similar issue on virtual machine.

Did you try ActiveMQ 5.15.10 ? We did a bunch of fixes and improvements
on this version around kahadb.

Regards
JB

On 02/11/2019 18:46, Jean-Baptiste Onofré wrote:

> Hi,
>
> It's definitely something weird while reading file on the filesystem.
>
> Is it a local filesystem or shared one (NFS, CIFS, ...) ?
>
> Regards
> JB
>
> On 02/11/2019 18:18, Jonathan Roby wrote:
>> Hi
>>
>>
>> as far as I know, the version of ActiveMQ hasn't been updated and i've now deleted the kahadb directory a few times (it regenerated) and even moved the broker path in case it was a drive error
>>
>>
>> Jon
>>
>> ________________________________
>> From: Jean-Baptiste Onofré <[hidden email]>
>> Sent: Saturday, November 2, 2019 12:16:31 PM
>> To: [hidden email]
>> Subject: Re: org.apache.activemq.kaha.impl.index.?
>>
>> Hi,
>>
>> Does the kahadb folder has been created with the same ActiveMQ version
>> (and you updated ActiveMQ version later) ?
>>
>> It seems the kahadb index is not consistent.
>>
>> Regards
>> JB
>>
>> On 02/11/2019 18:14, Jonathan Roby wrote:
>>> Hi, i've just joined the list so apologies if i've not this correctly.
>>>
>>>
>>> We run a number of islandora instances that use ActiveMQ 5.7.0 for message passing between components. Yesterday, one of our instances  seemed to stop working. after several restarts of components and the system itself, I noticed that when the components restarted, it always produced a nested exception and the very last error was org.apache.activemq.kaha.impl.index.BadMagicException.
>>>
>>>
>>> I've googled this and taken the advice given, which is usually to empty/delete the data directory or the kahadb directory and it still isnt working. All the other components seem to be working fine.
>>>
>>>
>>> Thank you for any help anyone can give me.
>>>
>>> Jon
>>>
>>>
>>> Here is the last part of the stack trace.
>>>
>>>
>>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: null
>>>     at org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.BrokerService.start(BrokerService.java:578) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_162]
>>>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_162]
>>>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_162]
>>>     at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
>>>     at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>>     at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>>     at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>>
>>
>> --
>> Jean-Baptiste Onofré
>> [hidden email]
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: org.apache.activemq.kaha.impl.index.?

Jonathan Roby
In reply to this post by jbonofre
hi, sorry for the delay


its a large storage array mounted onto the VM the software runs upon


Jon

________________________________
From: Jean-Baptiste Onofré <[hidden email]>
Sent: Saturday, November 2, 2019 12:46:22 PM
To: [hidden email]
Subject: Re: org.apache.activemq.kaha.impl.index.?

Hi,

It's definitely something weird while reading file on the filesystem.

Is it a local filesystem or shared one (NFS, CIFS, ...) ?

Regards
JB

On 02/11/2019 18:18, Jonathan Roby wrote:

> Hi
>
>
> as far as I know, the version of ActiveMQ hasn't been updated and i've now deleted the kahadb directory a few times (it regenerated) and even moved the broker path in case it was a drive error
>
>
> Jon
>
> ________________________________
> From: Jean-Baptiste Onofré <[hidden email]>
> Sent: Saturday, November 2, 2019 12:16:31 PM
> To: [hidden email]
> Subject: Re: org.apache.activemq.kaha.impl.index.?
>
> Hi,
>
> Does the kahadb folder has been created with the same ActiveMQ version
> (and you updated ActiveMQ version later) ?
>
> It seems the kahadb index is not consistent.
>
> Regards
> JB
>
> On 02/11/2019 18:14, Jonathan Roby wrote:
>> Hi, i've just joined the list so apologies if i've not this correctly.
>>
>>
>> We run a number of islandora instances that use ActiveMQ 5.7.0 for message passing between components. Yesterday, one of our instances  seemed to stop working. after several restarts of components and the system itself, I noticed that when the components restarted, it always produced a nested exception and the very last error was org.apache.activemq.kaha.impl.index.BadMagicException.
>>
>>
>> I've googled this and taken the advice given, which is usually to empty/delete the data directory or the kahadb directory and it still isnt working. All the other components seem to be working fine.
>>
>>
>> Thank you for any help anyone can give me.
>>
>> Jon
>>
>>
>> Here is the last part of the stack trace.
>>
>>
>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: null
>>     at org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.BrokerService.start(BrokerService.java:578) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539) ~[activemq-core-5.7.0.jar:5.7.0]
>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_162]
>>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_162]
>>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_162]
>>     at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
>>     at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>     at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>     at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>
>
> --
> Jean-Baptiste Onofré
> [hidden email]
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: org.apache.activemq.kaha.impl.index.?

Jonathan Roby
In reply to this post by jbonofre

No, its not fixed. I wiped out the data folder and the persistant folder. both were regenerated but still is failing


its weird as we have other instances set up exactly the same as this one that dont have this error. same versions, same everything


so it sounds like you might be correct about it being the filesystem. i hope not as that can be tricky to debug


Thank you for your help

Jon

________________________________
From: Jean-Baptiste Onofré <[hidden email]>
Sent: Saturday, November 2, 2019 12:48:29 PM
To: [hidden email]
Subject: Re: org.apache.activemq.kaha.impl.index.?

I guess it's fixed with you complete remove the kahadb folder right ?

I remember couple of users mentioned similar issue on virtual machine.

Did you try ActiveMQ 5.15.10 ? We did a bunch of fixes and improvements
on this version around kahadb.

Regards
JB

On 02/11/2019 18:46, Jean-Baptiste Onofré wrote:

> Hi,
>
> It's definitely something weird while reading file on the filesystem.
>
> Is it a local filesystem or shared one (NFS, CIFS, ...) ?
>
> Regards
> JB
>
> On 02/11/2019 18:18, Jonathan Roby wrote:
>> Hi
>>
>>
>> as far as I know, the version of ActiveMQ hasn't been updated and i've now deleted the kahadb directory a few times (it regenerated) and even moved the broker path in case it was a drive error
>>
>>
>> Jon
>>
>> ________________________________
>> From: Jean-Baptiste Onofré <[hidden email]>
>> Sent: Saturday, November 2, 2019 12:16:31 PM
>> To: [hidden email]
>> Subject: Re: org.apache.activemq.kaha.impl.index.?
>>
>> Hi,
>>
>> Does the kahadb folder has been created with the same ActiveMQ version
>> (and you updated ActiveMQ version later) ?
>>
>> It seems the kahadb index is not consistent.
>>
>> Regards
>> JB
>>
>> On 02/11/2019 18:14, Jonathan Roby wrote:
>>> Hi, i've just joined the list so apologies if i've not this correctly.
>>>
>>>
>>> We run a number of islandora instances that use ActiveMQ 5.7.0 for message passing between components. Yesterday, one of our instances  seemed to stop working. after several restarts of components and the system itself, I noticed that when the components restarted, it always produced a nested exception and the very last error was org.apache.activemq.kaha.impl.index.BadMagicException.
>>>
>>>
>>> I've googled this and taken the advice given, which is usually to empty/delete the data directory or the kahadb directory and it still isnt working. All the other components seem to be working fine.
>>>
>>>
>>> Thank you for any help anyone can give me.
>>>
>>> Jon
>>>
>>>
>>> Here is the last part of the stack trace.
>>>
>>>
>>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: null
>>>     at org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.BrokerService.start(BrokerService.java:578) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539) ~[activemq-core-5.7.0.jar:5.7.0]
>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_162]
>>>     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_162]
>>>     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_162]
>>>     at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
>>>     at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>>     at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>>     at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130) ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>>
>>
>> --
>> Jean-Baptiste Onofré
>> [hidden email]
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
Talend - A Cloud Data Integration Leader (modern ETL)<http://www.talend.com/>
www.talend.com
Talend Data Fabric offers a single suite of cloud apps for data integration and data integrity to help enterprises collect, govern, transform, and share data.


>>
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Talend - A Cloud Data Integration Leader (modern ETL)<http://www.talend.com/>
www.talend.com
Talend Data Fabric offers a single suite of cloud apps for data integration and data integrity to help enterprises collect, govern, transform, and share data.


Reply | Threaded
Open this post in threaded view
|

Re: org.apache.activemq.kaha.impl.index.?

Tim Bain
Following up on JB's line of questioning, is the directory only recreated
when (exactly when, to the second, with a matching timestamp) this broker
process is started? Might it be possible that a different broker process,
running elsewhere, might be creating the directory and its files?

If you reconfigure this broker to store the KahaDB data file on local disk
instead of your SAN (just temporarily), does the process start
successfully?

Tim

On Sat, Nov 2, 2019, 1:56 PM Jonathan Roby <[hidden email]>
wrote:

>
> No, its not fixed. I wiped out the data folder and the persistant folder.
> both were regenerated but still is failing
>
>
> its weird as we have other instances set up exactly the same as this one
> that dont have this error. same versions, same everything
>
>
> so it sounds like you might be correct about it being the filesystem. i
> hope not as that can be tricky to debug
>
>
> Thank you for your help
>
> Jon
>
> ________________________________
> From: Jean-Baptiste Onofré <[hidden email]>
> Sent: Saturday, November 2, 2019 12:48:29 PM
> To: [hidden email]
> Subject: Re: org.apache.activemq.kaha.impl.index.?
>
> I guess it's fixed with you complete remove the kahadb folder right ?
>
> I remember couple of users mentioned similar issue on virtual machine.
>
> Did you try ActiveMQ 5.15.10 ? We did a bunch of fixes and improvements
> on this version around kahadb.
>
> Regards
> JB
>
> On 02/11/2019 18:46, Jean-Baptiste Onofré wrote:
> > Hi,
> >
> > It's definitely something weird while reading file on the filesystem.
> >
> > Is it a local filesystem or shared one (NFS, CIFS, ...) ?
> >
> > Regards
> > JB
> >
> > On 02/11/2019 18:18, Jonathan Roby wrote:
> >> Hi
> >>
> >>
> >> as far as I know, the version of ActiveMQ hasn't been updated and i've
> now deleted the kahadb directory a few times (it regenerated) and even
> moved the broker path in case it was a drive error
> >>
> >>
> >> Jon
> >>
> >> ________________________________
> >> From: Jean-Baptiste Onofré <[hidden email]>
> >> Sent: Saturday, November 2, 2019 12:16:31 PM
> >> To: [hidden email]
> >> Subject: Re: org.apache.activemq.kaha.impl.index.?
> >>
> >> Hi,
> >>
> >> Does the kahadb folder has been created with the same ActiveMQ version
> >> (and you updated ActiveMQ version later) ?
> >>
> >> It seems the kahadb index is not consistent.
> >>
> >> Regards
> >> JB
> >>
> >> On 02/11/2019 18:14, Jonathan Roby wrote:
> >>> Hi, i've just joined the list so apologies if i've not this correctly.
> >>>
> >>>
> >>> We run a number of islandora instances that use ActiveMQ 5.7.0 for
> message passing between components. Yesterday, one of our instances  seemed
> to stop working. after several restarts of components and the system
> itself, I noticed that when the components restarted, it always produced a
> nested exception and the very last error was
> org.apache.activemq.kaha.impl.index.BadMagicException.
> >>>
> >>>
> >>> I've googled this and taken the advice given, which is usually to
> empty/delete the data directory or the kahadb directory and it still isnt
> working. All the other components seem to be working fine.
> >>>
> >>>
> >>> Thank you for any help anyone can give me.
> >>>
> >>> Jon
> >>>
> >>>
> >>> Here is the last part of the stack trace.
> >>>
> >>>
> >>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: null
> >>>     at
> org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerService.start(BrokerService.java:578)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.8.0_162]
> >>>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[na:1.8.0_162]
> >>>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_162]
> >>>     at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
> >>>     at
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340)
> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>>     at
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293)
> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>>     at
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130)
> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>>
> >>
> >> --
> >> Jean-Baptiste Onofré
> >> [hidden email]
> >> http://blog.nanthrax.net
> >> Talend - http://www.talend.com
> Talend - A Cloud Data Integration Leader (modern ETL)<
> http://www.talend.com/>
> www.talend.com
> Talend Data Fabric offers a single suite of cloud apps for data
> integration and data integrity to help enterprises collect, govern,
> transform, and share data.
>
>
> >>
> >
>
> --
> Jean-Baptiste Onofré
> [hidden email]
> http://blog.nanthrax.net
> Talend - http://www.talend.com
> Talend - A Cloud Data Integration Leader (modern ETL)<
> http://www.talend.com/>
> www.talend.com
> Talend Data Fabric offers a single suite of cloud apps for data
> integration and data integrity to help enterprises collect, govern,
> transform, and share data.
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: org.apache.activemq.kaha.impl.index.?

Jonathan Roby
Hi


I just checked and the directory is created exactly when the process tries to start. I've also checked and the ports used by the broker are not visible in the output of the netstat command.


I also tried moving the kahadb directory onto the local drive and still get the BadMagicException at the end of nested exception stacktrace


Thank you

Jon

________________________________
From: Tim Bain <[hidden email]>
Sent: Sunday, November 3, 2019 8:06:29 AM
To: ActiveMQ Users
Subject: Re: org.apache.activemq.kaha.impl.index.?

Following up on JB's line of questioning, is the directory only recreated
when (exactly when, to the second, with a matching timestamp) this broker
process is started? Might it be possible that a different broker process,
running elsewhere, might be creating the directory and its files?

If you reconfigure this broker to store the KahaDB data file on local disk
instead of your SAN (just temporarily), does the process start
successfully?

Tim

On Sat, Nov 2, 2019, 1:56 PM Jonathan Roby <[hidden email]>
wrote:

>
> No, its not fixed. I wiped out the data folder and the persistant folder.
> both were regenerated but still is failing
>
>
> its weird as we have other instances set up exactly the same as this one
> that dont have this error. same versions, same everything
>
>
> so it sounds like you might be correct about it being the filesystem. i
> hope not as that can be tricky to debug
>
>
> Thank you for your help
>
> Jon
>
> ________________________________
> From: Jean-Baptiste Onofré <[hidden email]>
> Sent: Saturday, November 2, 2019 12:48:29 PM
> To: [hidden email]
> Subject: Re: org.apache.activemq.kaha.impl.index.?
>
> I guess it's fixed with you complete remove the kahadb folder right ?
>
> I remember couple of users mentioned similar issue on virtual machine.
>
> Did you try ActiveMQ 5.15.10 ? We did a bunch of fixes and improvements
> on this version around kahadb.
>
> Regards
> JB
>
> On 02/11/2019 18:46, Jean-Baptiste Onofré wrote:
> > Hi,
> >
> > It's definitely something weird while reading file on the filesystem.
> >
> > Is it a local filesystem or shared one (NFS, CIFS, ...) ?
> >
> > Regards
> > JB
> >
> > On 02/11/2019 18:18, Jonathan Roby wrote:
> >> Hi
> >>
> >>
> >> as far as I know, the version of ActiveMQ hasn't been updated and i've
> now deleted the kahadb directory a few times (it regenerated) and even
> moved the broker path in case it was a drive error
> >>
> >>
> >> Jon
> >>
> >> ________________________________
> >> From: Jean-Baptiste Onofré <[hidden email]>
> >> Sent: Saturday, November 2, 2019 12:16:31 PM
> >> To: [hidden email]
> >> Subject: Re: org.apache.activemq.kaha.impl.index.?
> >>
> >> Hi,
> >>
> >> Does the kahadb folder has been created with the same ActiveMQ version
> >> (and you updated ActiveMQ version later) ?
> >>
> >> It seems the kahadb index is not consistent.
> >>
> >> Regards
> >> JB
> >>
> >> On 02/11/2019 18:14, Jonathan Roby wrote:
> >>> Hi, i've just joined the list so apologies if i've not this correctly.
> >>>
> >>>
> >>> We run a number of islandora instances that use ActiveMQ 5.7.0 for
> message passing between components. Yesterday, one of our instances  seemed
> to stop working. after several restarts of components and the system
> itself, I noticed that when the components restarted, it always produced a
> nested exception and the very last error was
> org.apache.activemq.kaha.impl.index.BadMagicException.
> >>>
> >>>
> >>> I've googled this and taken the advice given, which is usually to
> empty/delete the data directory or the kahadb directory and it still isnt
> working. All the other components seem to be working fine.
> >>>
> >>>
> >>> Thank you for any help anyone can give me.
> >>>
> >>> Jon
> >>>
> >>>
> >>> Here is the last part of the stack trace.
> >>>
> >>>
> >>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: null
> >>>     at
> org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerService.start(BrokerService.java:578)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.8.0_162]
> >>>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[na:1.8.0_162]
> >>>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_162]
> >>>     at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
> >>>     at
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340)
> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>>     at
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293)
> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>>     at
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130)
> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>>
> >>
> >> --
> >> Jean-Baptiste Onofré
> >> [hidden email]
> >> http://blog.nanthrax.net
> >> Talend - http://www.talend.com
> Talend - A Cloud Data Integration Leader (modern ETL)<
> http://www.talend.com/>
> www.talend.com<http://www.talend.com>
> Talend Data Fabric offers a single suite of cloud apps for data
> integration and data integrity to help enterprises collect, govern,
> transform, and share data.
>
>
> >>
> >
>
> --
> Jean-Baptiste Onofré
> [hidden email]
> http://blog.nanthrax.net
> Talend - http://www.talend.com
> Talend - A Cloud Data Integration Leader (modern ETL)<
> http://www.talend.com/>
> www.talend.com<http://www.talend.com>
> Talend Data Fabric offers a single suite of cloud apps for data
> integration and data integrity to help enterprises collect, govern,
> transform, and share data.
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: org.apache.activemq.kaha.impl.index.?

Jonathan Roby
Hi


Also, i'm sorry for the subject heading of my original email, I'm not sure what happened but its somewhat nonsensical.


Also, would anyone recommend I post this question to the developer mailing list, in case they might have other ideas?


I've asked our IT infrastructure team if they can check the VMs storage/machine layout for recent changes that could have caused this. hopefully t hey find something or I get a lucky break


Thank you everyone for your help so far

Jon

________________________________
From: Jonathan Roby
Sent: Sunday, November 3, 2019 9:02:56 AM
To: ActiveMQ Users
Subject: Re: org.apache.activemq.kaha.impl.index.?


Hi


I just checked and the directory is created exactly when the process tries to start. I've also checked and the ports used by the broker are not visible in the output of the netstat command.


I also tried moving the kahadb directory onto the local drive and still get the BadMagicException at the end of nested exception stacktrace


Thank you

Jon

________________________________
From: Tim Bain <[hidden email]>
Sent: Sunday, November 3, 2019 8:06:29 AM
To: ActiveMQ Users
Subject: Re: org.apache.activemq.kaha.impl.index.?

Following up on JB's line of questioning, is the directory only recreated
when (exactly when, to the second, with a matching timestamp) this broker
process is started? Might it be possible that a different broker process,
running elsewhere, might be creating the directory and its files?

If you reconfigure this broker to store the KahaDB data file on local disk
instead of your SAN (just temporarily), does the process start
successfully?

Tim

On Sat, Nov 2, 2019, 1:56 PM Jonathan Roby <[hidden email]>
wrote:

>
> No, its not fixed. I wiped out the data folder and the persistant folder.
> both were regenerated but still is failing
>
>
> its weird as we have other instances set up exactly the same as this one
> that dont have this error. same versions, same everything
>
>
> so it sounds like you might be correct about it being the filesystem. i
> hope not as that can be tricky to debug
>
>
> Thank you for your help
>
> Jon
>
> ________________________________
> From: Jean-Baptiste Onofré <[hidden email]>
> Sent: Saturday, November 2, 2019 12:48:29 PM
> To: [hidden email]
> Subject: Re: org.apache.activemq.kaha.impl.index.?
>
> I guess it's fixed with you complete remove the kahadb folder right ?
>
> I remember couple of users mentioned similar issue on virtual machine.
>
> Did you try ActiveMQ 5.15.10 ? We did a bunch of fixes and improvements
> on this version around kahadb.
>
> Regards
> JB
>
> On 02/11/2019 18:46, Jean-Baptiste Onofré wrote:
> > Hi,
> >
> > It's definitely something weird while reading file on the filesystem.
> >
> > Is it a local filesystem or shared one (NFS, CIFS, ...) ?
> >
> > Regards
> > JB
> >
> > On 02/11/2019 18:18, Jonathan Roby wrote:
> >> Hi
> >>
> >>
> >> as far as I know, the version of ActiveMQ hasn't been updated and i've
> now deleted the kahadb directory a few times (it regenerated) and even
> moved the broker path in case it was a drive error
> >>
> >>
> >> Jon
> >>
> >> ________________________________
> >> From: Jean-Baptiste Onofré <[hidden email]>
> >> Sent: Saturday, November 2, 2019 12:16:31 PM
> >> To: [hidden email]
> >> Subject: Re: org.apache.activemq.kaha.impl.index.?
> >>
> >> Hi,
> >>
> >> Does the kahadb folder has been created with the same ActiveMQ version
> >> (and you updated ActiveMQ version later) ?
> >>
> >> It seems the kahadb index is not consistent.
> >>
> >> Regards
> >> JB
> >>
> >> On 02/11/2019 18:14, Jonathan Roby wrote:
> >>> Hi, i've just joined the list so apologies if i've not this correctly.
> >>>
> >>>
> >>> We run a number of islandora instances that use ActiveMQ 5.7.0 for
> message passing between components. Yesterday, one of our instances  seemed
> to stop working. after several restarts of components and the system
> itself, I noticed that when the components restarted, it always produced a
> nested exception and the very last error was
> org.apache.activemq.kaha.impl.index.BadMagicException.
> >>>
> >>>
> >>> I've googled this and taken the advice given, which is usually to
> empty/delete the data directory or the kahadb directory and it still isnt
> working. All the other components seem to be working fine.
> >>>
> >>>
> >>> Thank you for any help anyone can give me.
> >>>
> >>> Jon
> >>>
> >>>
> >>> Here is the last part of the stack trace.
> >>>
> >>>
> >>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: null
> >>>     at
> org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerService.start(BrokerService.java:578)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at
> org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539)
> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[na:1.8.0_162]
> >>>     at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[na:1.8.0_162]
> >>>     at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[na:1.8.0_162]
> >>>     at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
> >>>     at
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340)
> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>>     at
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293)
> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>>     at
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130)
> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>>
> >>
> >> --
> >> Jean-Baptiste Onofré
> >> [hidden email]
> >> http://blog.nanthrax.net
> >> Talend - http://www.talend.com
> Talend - A Cloud Data Integration Leader (modern ETL)<
> http://www.talend.com/>
> www.talend.com<http://www.talend.com>
> Talend Data Fabric offers a single suite of cloud apps for data
> integration and data integrity to help enterprises collect, govern,
> transform, and share data.
>
>
> >>
> >
>
> --
> Jean-Baptiste Onofré
> [hidden email]
> http://blog.nanthrax.net
> Talend - http://www.talend.com
> Talend - A Cloud Data Integration Leader (modern ETL)<
> http://www.talend.com/>
> www.talend.com<http://www.talend.com>
> Talend Data Fabric offers a single suite of cloud apps for data
> integration and data integrity to help enterprises collect, govern,
> transform, and share data.
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: org.apache.activemq.kaha.impl.index.?

jbonofre
Hi Jonathan,

user mailing list is fine for this kind of question.

IMHO, I'm pretty sure it's related to the filesystem.

As Tim suggested, maybe you can try to put the kahadb directory on local
filesystem instead of the SAN. It could be an interesting test.

Regards
JB

On 03/11/2019 16:52, Jonathan Roby wrote:

> Hi
>
>
> Also, i'm sorry for the subject heading of my original email, I'm not sure what happened but its somewhat nonsensical.
>
>
> Also, would anyone recommend I post this question to the developer mailing list, in case they might have other ideas?
>
>
> I've asked our IT infrastructure team if they can check the VMs storage/machine layout for recent changes that could have caused this. hopefully t hey find something or I get a lucky break
>
>
> Thank you everyone for your help so far
>
> Jon
>
> ________________________________
> From: Jonathan Roby
> Sent: Sunday, November 3, 2019 9:02:56 AM
> To: ActiveMQ Users
> Subject: Re: org.apache.activemq.kaha.impl.index.?
>
>
> Hi
>
>
> I just checked and the directory is created exactly when the process tries to start. I've also checked and the ports used by the broker are not visible in the output of the netstat command.
>
>
> I also tried moving the kahadb directory onto the local drive and still get the BadMagicException at the end of nested exception stacktrace
>
>
> Thank you
>
> Jon
>
> ________________________________
> From: Tim Bain <[hidden email]>
> Sent: Sunday, November 3, 2019 8:06:29 AM
> To: ActiveMQ Users
> Subject: Re: org.apache.activemq.kaha.impl.index.?
>
> Following up on JB's line of questioning, is the directory only recreated
> when (exactly when, to the second, with a matching timestamp) this broker
> process is started? Might it be possible that a different broker process,
> running elsewhere, might be creating the directory and its files?
>
> If you reconfigure this broker to store the KahaDB data file on local disk
> instead of your SAN (just temporarily), does the process start
> successfully?
>
> Tim
>
> On Sat, Nov 2, 2019, 1:56 PM Jonathan Roby <[hidden email]>
> wrote:
>
>>
>> No, its not fixed. I wiped out the data folder and the persistant folder.
>> both were regenerated but still is failing
>>
>>
>> its weird as we have other instances set up exactly the same as this one
>> that dont have this error. same versions, same everything
>>
>>
>> so it sounds like you might be correct about it being the filesystem. i
>> hope not as that can be tricky to debug
>>
>>
>> Thank you for your help
>>
>> Jon
>>
>> ________________________________
>> From: Jean-Baptiste Onofré <[hidden email]>
>> Sent: Saturday, November 2, 2019 12:48:29 PM
>> To: [hidden email]
>> Subject: Re: org.apache.activemq.kaha.impl.index.?
>>
>> I guess it's fixed with you complete remove the kahadb folder right ?
>>
>> I remember couple of users mentioned similar issue on virtual machine.
>>
>> Did you try ActiveMQ 5.15.10 ? We did a bunch of fixes and improvements
>> on this version around kahadb.
>>
>> Regards
>> JB
>>
>> On 02/11/2019 18:46, Jean-Baptiste Onofré wrote:
>>> Hi,
>>>
>>> It's definitely something weird while reading file on the filesystem.
>>>
>>> Is it a local filesystem or shared one (NFS, CIFS, ...) ?
>>>
>>> Regards
>>> JB
>>>
>>> On 02/11/2019 18:18, Jonathan Roby wrote:
>>>> Hi
>>>>
>>>>
>>>> as far as I know, the version of ActiveMQ hasn't been updated and i've
>> now deleted the kahadb directory a few times (it regenerated) and even
>> moved the broker path in case it was a drive error
>>>>
>>>>
>>>> Jon
>>>>
>>>> ________________________________
>>>> From: Jean-Baptiste Onofré <[hidden email]>
>>>> Sent: Saturday, November 2, 2019 12:16:31 PM
>>>> To: [hidden email]
>>>> Subject: Re: org.apache.activemq.kaha.impl.index.?
>>>>
>>>> Hi,
>>>>
>>>> Does the kahadb folder has been created with the same ActiveMQ version
>>>> (and you updated ActiveMQ version later) ?
>>>>
>>>> It seems the kahadb index is not consistent.
>>>>
>>>> Regards
>>>> JB
>>>>
>>>> On 02/11/2019 18:14, Jonathan Roby wrote:
>>>>> Hi, i've just joined the list so apologies if i've not this correctly.
>>>>>
>>>>>
>>>>> We run a number of islandora instances that use ActiveMQ 5.7.0 for
>> message passing between components. Yesterday, one of our instances  seemed
>> to stop working. after several restarts of components and the system
>> itself, I noticed that when the components restarted, it always produced a
>> nested exception and the very last error was
>> org.apache.activemq.kaha.impl.index.BadMagicException.
>>>>>
>>>>>
>>>>> I've googled this and taken the advice given, which is usually to
>> empty/delete the data directory or the kahadb directory and it still isnt
>> working. All the other components seem to be working fine.
>>>>>
>>>>>
>>>>> Thank you for any help anyone can give me.
>>>>>
>>>>> Jon
>>>>>
>>>>>
>>>>> Here is the last part of the stack trace.
>>>>>
>>>>>
>>>>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: null
>>>>>     at
>> org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.BrokerService.start(BrokerService.java:578)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> ~[na:1.8.0_162]
>>>>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> ~[na:1.8.0_162]
>>>>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> ~[na:1.8.0_162]
>>>>>     at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
>>>>>     at
>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340)
>> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>>>>     at
>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293)
>> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>>>>     at
>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130)
>> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>>>>
>>>>
>>>> --
>>>> Jean-Baptiste Onofré
>>>> [hidden email]
>>>> http://blog.nanthrax.net
>>>> Talend - http://www.talend.com
>> Talend - A Cloud Data Integration Leader (modern ETL)<
>> http://www.talend.com/>
>> www.talend.com<http://www.talend.com>
>> Talend Data Fabric offers a single suite of cloud apps for data
>> integration and data integrity to help enterprises collect, govern,
>> transform, and share data.
>>
>>
>>>>
>>>
>>
>> --
>> Jean-Baptiste Onofré
>> [hidden email]
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>> Talend - A Cloud Data Integration Leader (modern ETL)<
>> http://www.talend.com/>
>> www.talend.com<http://www.talend.com>
>> Talend Data Fabric offers a single suite of cloud apps for data
>> integration and data integrity to help enterprises collect, govern,
>> transform, and share data.
>>
>>
>>
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: org.apache.activemq.kaha.impl.index.?

Jonathan Roby
Hi


I tried putting the kahadb directory on the local directory, it regenerated on startup but still gave me a BadMagicException

________________________________
From: Jean-Baptiste Onofré <[hidden email]>
Sent: Sunday, November 3, 2019 11:07:22 AM
To: [hidden email]
Subject: Re: org.apache.activemq.kaha.impl.index.?

Hi Jonathan,

user mailing list is fine for this kind of question.

IMHO, I'm pretty sure it's related to the filesystem.

As Tim suggested, maybe you can try to put the kahadb directory on local
filesystem instead of the SAN. It could be an interesting test.

Regards
JB

On 03/11/2019 16:52, Jonathan Roby wrote:

> Hi
>
>
> Also, i'm sorry for the subject heading of my original email, I'm not sure what happened but its somewhat nonsensical.
>
>
> Also, would anyone recommend I post this question to the developer mailing list, in case they might have other ideas?
>
>
> I've asked our IT infrastructure team if they can check the VMs storage/machine layout for recent changes that could have caused this. hopefully t hey find something or I get a lucky break
>
>
> Thank you everyone for your help so far
>
> Jon
>
> ________________________________
> From: Jonathan Roby
> Sent: Sunday, November 3, 2019 9:02:56 AM
> To: ActiveMQ Users
> Subject: Re: org.apache.activemq.kaha.impl.index.?
>
>
> Hi
>
>
> I just checked and the directory is created exactly when the process tries to start. I've also checked and the ports used by the broker are not visible in the output of the netstat command.
>
>
> I also tried moving the kahadb directory onto the local drive and still get the BadMagicException at the end of nested exception stacktrace
>
>
> Thank you
>
> Jon
>
> ________________________________
> From: Tim Bain <[hidden email]>
> Sent: Sunday, November 3, 2019 8:06:29 AM
> To: ActiveMQ Users
> Subject: Re: org.apache.activemq.kaha.impl.index.?
>
> Following up on JB's line of questioning, is the directory only recreated
> when (exactly when, to the second, with a matching timestamp) this broker
> process is started? Might it be possible that a different broker process,
> running elsewhere, might be creating the directory and its files?
>
> If you reconfigure this broker to store the KahaDB data file on local disk
> instead of your SAN (just temporarily), does the process start
> successfully?
>
> Tim
>
> On Sat, Nov 2, 2019, 1:56 PM Jonathan Roby <[hidden email]>
> wrote:
>
>>
>> No, its not fixed. I wiped out the data folder and the persistant folder.
>> both were regenerated but still is failing
>>
>>
>> its weird as we have other instances set up exactly the same as this one
>> that dont have this error. same versions, same everything
>>
>>
>> so it sounds like you might be correct about it being the filesystem. i
>> hope not as that can be tricky to debug
>>
>>
>> Thank you for your help
>>
>> Jon
>>
>> ________________________________
>> From: Jean-Baptiste Onofré <[hidden email]>
>> Sent: Saturday, November 2, 2019 12:48:29 PM
>> To: [hidden email]
>> Subject: Re: org.apache.activemq.kaha.impl.index.?
>>
>> I guess it's fixed with you complete remove the kahadb folder right ?
>>
>> I remember couple of users mentioned similar issue on virtual machine.
>>
>> Did you try ActiveMQ 5.15.10 ? We did a bunch of fixes and improvements
>> on this version around kahadb.
>>
>> Regards
>> JB
>>
>> On 02/11/2019 18:46, Jean-Baptiste Onofré wrote:
>>> Hi,
>>>
>>> It's definitely something weird while reading file on the filesystem.
>>>
>>> Is it a local filesystem or shared one (NFS, CIFS, ...) ?
>>>
>>> Regards
>>> JB
>>>
>>> On 02/11/2019 18:18, Jonathan Roby wrote:
>>>> Hi
>>>>
>>>>
>>>> as far as I know, the version of ActiveMQ hasn't been updated and i've
>> now deleted the kahadb directory a few times (it regenerated) and even
>> moved the broker path in case it was a drive error
>>>>
>>>>
>>>> Jon
>>>>
>>>> ________________________________
>>>> From: Jean-Baptiste Onofré <[hidden email]>
>>>> Sent: Saturday, November 2, 2019 12:16:31 PM
>>>> To: [hidden email]
>>>> Subject: Re: org.apache.activemq.kaha.impl.index.?
>>>>
>>>> Hi,
>>>>
>>>> Does the kahadb folder has been created with the same ActiveMQ version
>>>> (and you updated ActiveMQ version later) ?
>>>>
>>>> It seems the kahadb index is not consistent.
>>>>
>>>> Regards
>>>> JB
>>>>
>>>> On 02/11/2019 18:14, Jonathan Roby wrote:
>>>>> Hi, i've just joined the list so apologies if i've not this correctly.
>>>>>
>>>>>
>>>>> We run a number of islandora instances that use ActiveMQ 5.7.0 for
>> message passing between components. Yesterday, one of our instances  seemed
>> to stop working. after several restarts of components and the system
>> itself, I noticed that when the components restarted, it always produced a
>> nested exception and the very last error was
>> org.apache.activemq.kaha.impl.index.BadMagicException.
>>>>>
>>>>>
>>>>> I've googled this and taken the advice given, which is usually to
>> empty/delete the data directory or the kahadb directory and it still isnt
>> working. All the other components seem to be working fine.
>>>>>
>>>>>
>>>>> Thank you for any help anyone can give me.
>>>>>
>>>>> Jon
>>>>>
>>>>>
>>>>> Here is the last part of the stack trace.
>>>>>
>>>>>
>>>>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException: null
>>>>>     at
>> org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.BrokerService.start(BrokerService.java:578)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at
>> org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539)
>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> ~[na:1.8.0_162]
>>>>>     at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> ~[na:1.8.0_162]
>>>>>     at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> ~[na:1.8.0_162]
>>>>>     at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_162]
>>>>>     at
>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340)
>> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>>>>     at
>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293)
>> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>>>>     at
>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130)
>> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>>>>
>>>>
>>>> --
>>>> Jean-Baptiste Onofré
>>>> [hidden email]
>>>> http://blog.nanthrax.net
>>>> Talend - http://www.talend.com
>> Talend - A Cloud Data Integration Leader (modern ETL)<
>> http://www.talend.com/>
>> www.talend.com<http://www.talend.com>
>> Talend Data Fabric offers a single suite of cloud apps for data
>> integration and data integrity to help enterprises collect, govern,
>> transform, and share data.
>>
>>
>>>>
>>>
>>
>> --
>> Jean-Baptiste Onofré
>> [hidden email]
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>> Talend - A Cloud Data Integration Leader (modern ETL)<
>> http://www.talend.com/>
>> www.talend.com<http://www.talend.com>
>> Talend Data Fabric offers a single suite of cloud apps for data
>> integration and data integrity to help enterprises collect, govern,
>> transform, and share data.
>>
>>
>>
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: org.apache.activemq.kaha.impl.index.?

Tim Bain
Is there any chance that a JAR from a newer version of ActiveMQ ended up on
the broker's classpath? If you delete the contents of the lib directory and
replace it with the contents from a freshly installed copy of 5.7.0, does
the behavior change?

Also, can you share your activemq.xml with us? And does this behavior
change if you use the copy of activemq.xml from a fresh install?

Tim

On Sun, Nov 3, 2019, 2:35 PM Jonathan Roby <[hidden email]>
wrote:

> Hi
>
>
> I tried putting the kahadb directory on the local directory, it
> regenerated on startup but still gave me a BadMagicException
>
> ________________________________
> From: Jean-Baptiste Onofré <[hidden email]>
> Sent: Sunday, November 3, 2019 11:07:22 AM
> To: [hidden email]
> Subject: Re: org.apache.activemq.kaha.impl.index.?
>
> Hi Jonathan,
>
> user mailing list is fine for this kind of question.
>
> IMHO, I'm pretty sure it's related to the filesystem.
>
> As Tim suggested, maybe you can try to put the kahadb directory on local
> filesystem instead of the SAN. It could be an interesting test.
>
> Regards
> JB
>
> On 03/11/2019 16:52, Jonathan Roby wrote:
> > Hi
> >
> >
> > Also, i'm sorry for the subject heading of my original email, I'm not
> sure what happened but its somewhat nonsensical.
> >
> >
> > Also, would anyone recommend I post this question to the developer
> mailing list, in case they might have other ideas?
> >
> >
> > I've asked our IT infrastructure team if they can check the VMs
> storage/machine layout for recent changes that could have caused this.
> hopefully t hey find something or I get a lucky break
> >
> >
> > Thank you everyone for your help so far
> >
> > Jon
> >
> > ________________________________
> > From: Jonathan Roby
> > Sent: Sunday, November 3, 2019 9:02:56 AM
> > To: ActiveMQ Users
> > Subject: Re: org.apache.activemq.kaha.impl.index.?
> >
> >
> > Hi
> >
> >
> > I just checked and the directory is created exactly when the process
> tries to start. I've also checked and the ports used by the broker are not
> visible in the output of the netstat command.
> >
> >
> > I also tried moving the kahadb directory onto the local drive and still
> get the BadMagicException at the end of nested exception stacktrace
> >
> >
> > Thank you
> >
> > Jon
> >
> > ________________________________
> > From: Tim Bain <[hidden email]>
> > Sent: Sunday, November 3, 2019 8:06:29 AM
> > To: ActiveMQ Users
> > Subject: Re: org.apache.activemq.kaha.impl.index.?
> >
> > Following up on JB's line of questioning, is the directory only recreated
> > when (exactly when, to the second, with a matching timestamp) this broker
> > process is started? Might it be possible that a different broker process,
> > running elsewhere, might be creating the directory and its files?
> >
> > If you reconfigure this broker to store the KahaDB data file on local
> disk
> > instead of your SAN (just temporarily), does the process start
> > successfully?
> >
> > Tim
> >
> > On Sat, Nov 2, 2019, 1:56 PM Jonathan Roby <[hidden email]>
> > wrote:
> >
> >>
> >> No, its not fixed. I wiped out the data folder and the persistant
> folder.
> >> both were regenerated but still is failing
> >>
> >>
> >> its weird as we have other instances set up exactly the same as this one
> >> that dont have this error. same versions, same everything
> >>
> >>
> >> so it sounds like you might be correct about it being the filesystem. i
> >> hope not as that can be tricky to debug
> >>
> >>
> >> Thank you for your help
> >>
> >> Jon
> >>
> >> ________________________________
> >> From: Jean-Baptiste Onofré <[hidden email]>
> >> Sent: Saturday, November 2, 2019 12:48:29 PM
> >> To: [hidden email]
> >> Subject: Re: org.apache.activemq.kaha.impl.index.?
> >>
> >> I guess it's fixed with you complete remove the kahadb folder right ?
> >>
> >> I remember couple of users mentioned similar issue on virtual machine.
> >>
> >> Did you try ActiveMQ 5.15.10 ? We did a bunch of fixes and improvements
> >> on this version around kahadb.
> >>
> >> Regards
> >> JB
> >>
> >> On 02/11/2019 18:46, Jean-Baptiste Onofré wrote:
> >>> Hi,
> >>>
> >>> It's definitely something weird while reading file on the filesystem.
> >>>
> >>> Is it a local filesystem or shared one (NFS, CIFS, ...) ?
> >>>
> >>> Regards
> >>> JB
> >>>
> >>> On 02/11/2019 18:18, Jonathan Roby wrote:
> >>>> Hi
> >>>>
> >>>>
> >>>> as far as I know, the version of ActiveMQ hasn't been updated and i've
> >> now deleted the kahadb directory a few times (it regenerated) and even
> >> moved the broker path in case it was a drive error
> >>>>
> >>>>
> >>>> Jon
> >>>>
> >>>> ________________________________
> >>>> From: Jean-Baptiste Onofré <[hidden email]>
> >>>> Sent: Saturday, November 2, 2019 12:16:31 PM
> >>>> To: [hidden email]
> >>>> Subject: Re: org.apache.activemq.kaha.impl.index.?
> >>>>
> >>>> Hi,
> >>>>
> >>>> Does the kahadb folder has been created with the same ActiveMQ version
> >>>> (and you updated ActiveMQ version later) ?
> >>>>
> >>>> It seems the kahadb index is not consistent.
> >>>>
> >>>> Regards
> >>>> JB
> >>>>
> >>>> On 02/11/2019 18:14, Jonathan Roby wrote:
> >>>>> Hi, i've just joined the list so apologies if i've not this
> correctly.
> >>>>>
> >>>>>
> >>>>> We run a number of islandora instances that use ActiveMQ 5.7.0 for
> >> message passing between components. Yesterday, one of our instances
> seemed
> >> to stop working. after several restarts of components and the system
> >> itself, I noticed that when the components restarted, it always
> produced a
> >> nested exception and the very last error was
> >> org.apache.activemq.kaha.impl.index.BadMagicException.
> >>>>>
> >>>>>
> >>>>> I've googled this and taken the advice given, which is usually to
> >> empty/delete the data directory or the kahadb directory and it still
> isnt
> >> working. All the other components seem to be working fine.
> >>>>>
> >>>>>
> >>>>> Thank you for any help anyone can give me.
> >>>>>
> >>>>> Jon
> >>>>>
> >>>>>
> >>>>> Here is the last part of the stack trace.
> >>>>>
> >>>>>
> >>>>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException:
> null
> >>>>>     at
> >> org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >> org.apache.activemq.broker.BrokerService.start(BrokerService.java:578)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> ~[na:1.8.0_162]
> >>>>>     at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >> ~[na:1.8.0_162]
> >>>>>     at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> ~[na:1.8.0_162]
> >>>>>     at java.lang.reflect.Method.invoke(Method.java:498)
> ~[na:1.8.0_162]
> >>>>>     at
> >>
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340)
> >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>>>>     at
> >>
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293)
> >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>>>>     at
> >>
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130)
> >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>>>>
> >>>>
> >>>> --
> >>>> Jean-Baptiste Onofré
> >>>> [hidden email]
> >>>> http://blog.nanthrax.net
> >>>> Talend - http://www.talend.com
> >> Talend - A Cloud Data Integration Leader (modern ETL)<
> >> http://www.talend.com/>
> >> www.talend.com<http://www.talend.com>
> >> Talend Data Fabric offers a single suite of cloud apps for data
> >> integration and data integrity to help enterprises collect, govern,
> >> transform, and share data.
> >>
> >>
> >>>>
> >>>
> >>
> >> --
> >> Jean-Baptiste Onofré
> >> [hidden email]
> >> http://blog.nanthrax.net
> >> Talend - http://www.talend.com
> >> Talend - A Cloud Data Integration Leader (modern ETL)<
> >> http://www.talend.com/>
> >> www.talend.com<http://www.talend.com>
> >> Talend Data Fabric offers a single suite of cloud apps for data
> >> integration and data integrity to help enterprises collect, govern,
> >> transform, and share data.
> >>
> >>
> >>
> >
>
> --
> Jean-Baptiste Onofré
> [hidden email]
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Reply | Threaded
Open this post in threaded view
|

Re: org.apache.activemq.kaha.impl.index.?

Jonathan Roby
Hi All


just an update: I figured out the issue. It was due to the  software we use (fedora-commons/tomcat) using its own set of configuration files and data directories. So I was trying to debug the wrong set of config files and data directories, thats why nothing seemed to work. And in the end, it was just deleting the data directory.


Thanks Jean-Baptiste and Tim for your help

Jon

________________________________
From: Tim Bain <[hidden email]>
Sent: Tuesday, November 5, 2019 7:34:10 AM
To: ActiveMQ Users
Subject: Re: org.apache.activemq.kaha.impl.index.?

Is there any chance that a JAR from a newer version of ActiveMQ ended up on
the broker's classpath? If you delete the contents of the lib directory and
replace it with the contents from a freshly installed copy of 5.7.0, does
the behavior change?

Also, can you share your activemq.xml with us? And does this behavior
change if you use the copy of activemq.xml from a fresh install?

Tim

On Sun, Nov 3, 2019, 2:35 PM Jonathan Roby <[hidden email]>
wrote:

> Hi
>
>
> I tried putting the kahadb directory on the local directory, it
> regenerated on startup but still gave me a BadMagicException
>
> ________________________________
> From: Jean-Baptiste Onofré <[hidden email]>
> Sent: Sunday, November 3, 2019 11:07:22 AM
> To: [hidden email]
> Subject: Re: org.apache.activemq.kaha.impl.index.?
>
> Hi Jonathan,
>
> user mailing list is fine for this kind of question.
>
> IMHO, I'm pretty sure it's related to the filesystem.
>
> As Tim suggested, maybe you can try to put the kahadb directory on local
> filesystem instead of the SAN. It could be an interesting test.
>
> Regards
> JB
>
> On 03/11/2019 16:52, Jonathan Roby wrote:
> > Hi
> >
> >
> > Also, i'm sorry for the subject heading of my original email, I'm not
> sure what happened but its somewhat nonsensical.
> >
> >
> > Also, would anyone recommend I post this question to the developer
> mailing list, in case they might have other ideas?
> >
> >
> > I've asked our IT infrastructure team if they can check the VMs
> storage/machine layout for recent changes that could have caused this.
> hopefully t hey find something or I get a lucky break
> >
> >
> > Thank you everyone for your help so far
> >
> > Jon
> >
> > ________________________________
> > From: Jonathan Roby
> > Sent: Sunday, November 3, 2019 9:02:56 AM
> > To: ActiveMQ Users
> > Subject: Re: org.apache.activemq.kaha.impl.index.?
> >
> >
> > Hi
> >
> >
> > I just checked and the directory is created exactly when the process
> tries to start. I've also checked and the ports used by the broker are not
> visible in the output of the netstat command.
> >
> >
> > I also tried moving the kahadb directory onto the local drive and still
> get the BadMagicException at the end of nested exception stacktrace
> >
> >
> > Thank you
> >
> > Jon
> >
> > ________________________________
> > From: Tim Bain <[hidden email]>
> > Sent: Sunday, November 3, 2019 8:06:29 AM
> > To: ActiveMQ Users
> > Subject: Re: org.apache.activemq.kaha.impl.index.?
> >
> > Following up on JB's line of questioning, is the directory only recreated
> > when (exactly when, to the second, with a matching timestamp) this broker
> > process is started? Might it be possible that a different broker process,
> > running elsewhere, might be creating the directory and its files?
> >
> > If you reconfigure this broker to store the KahaDB data file on local
> disk
> > instead of your SAN (just temporarily), does the process start
> > successfully?
> >
> > Tim
> >
> > On Sat, Nov 2, 2019, 1:56 PM Jonathan Roby <[hidden email]>
> > wrote:
> >
> >>
> >> No, its not fixed. I wiped out the data folder and the persistant
> folder.
> >> both were regenerated but still is failing
> >>
> >>
> >> its weird as we have other instances set up exactly the same as this one
> >> that dont have this error. same versions, same everything
> >>
> >>
> >> so it sounds like you might be correct about it being the filesystem. i
> >> hope not as that can be tricky to debug
> >>
> >>
> >> Thank you for your help
> >>
> >> Jon
> >>
> >> ________________________________
> >> From: Jean-Baptiste Onofré <[hidden email]>
> >> Sent: Saturday, November 2, 2019 12:48:29 PM
> >> To: [hidden email]
> >> Subject: Re: org.apache.activemq.kaha.impl.index.?
> >>
> >> I guess it's fixed with you complete remove the kahadb folder right ?
> >>
> >> I remember couple of users mentioned similar issue on virtual machine.
> >>
> >> Did you try ActiveMQ 5.15.10 ? We did a bunch of fixes and improvements
> >> on this version around kahadb.
> >>
> >> Regards
> >> JB
> >>
> >> On 02/11/2019 18:46, Jean-Baptiste Onofré wrote:
> >>> Hi,
> >>>
> >>> It's definitely something weird while reading file on the filesystem.
> >>>
> >>> Is it a local filesystem or shared one (NFS, CIFS, ...) ?
> >>>
> >>> Regards
> >>> JB
> >>>
> >>> On 02/11/2019 18:18, Jonathan Roby wrote:
> >>>> Hi
> >>>>
> >>>>
> >>>> as far as I know, the version of ActiveMQ hasn't been updated and i've
> >> now deleted the kahadb directory a few times (it regenerated) and even
> >> moved the broker path in case it was a drive error
> >>>>
> >>>>
> >>>> Jon
> >>>>
> >>>> ________________________________
> >>>> From: Jean-Baptiste Onofré <[hidden email]>
> >>>> Sent: Saturday, November 2, 2019 12:16:31 PM
> >>>> To: [hidden email]
> >>>> Subject: Re: org.apache.activemq.kaha.impl.index.?
> >>>>
> >>>> Hi,
> >>>>
> >>>> Does the kahadb folder has been created with the same ActiveMQ version
> >>>> (and you updated ActiveMQ version later) ?
> >>>>
> >>>> It seems the kahadb index is not consistent.
> >>>>
> >>>> Regards
> >>>> JB
> >>>>
> >>>> On 02/11/2019 18:14, Jonathan Roby wrote:
> >>>>> Hi, i've just joined the list so apologies if i've not this
> correctly.
> >>>>>
> >>>>>
> >>>>> We run a number of islandora instances that use ActiveMQ 5.7.0 for
> >> message passing between components. Yesterday, one of our instances
> seemed
> >> to stop working. after several restarts of components and the system
> >> itself, I noticed that when the components restarted, it always
> produced a
> >> nested exception and the very last error was
> >> org.apache.activemq.kaha.impl.index.BadMagicException.
> >>>>>
> >>>>>
> >>>>> I've googled this and taken the advice given, which is usually to
> >> empty/delete the data directory or the kahadb directory and it still
> isnt
> >> working. All the other components seem to be working fine.
> >>>>>
> >>>>>
> >>>>> Thank you for any help anyone can give me.
> >>>>>
> >>>>> Jon
> >>>>>
> >>>>>
> >>>>> Here is the last part of the stack trace.
> >>>>>
> >>>>>
> >>>>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException:
> null
> >>>>>     at
> >> org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >> org.apache.activemq.broker.BrokerService.start(BrokerService.java:578)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at
> >>
> org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539)
> >> ~[activemq-core-5.7.0.jar:5.7.0]
> >>>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >> ~[na:1.8.0_162]
> >>>>>     at
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> >> ~[na:1.8.0_162]
> >>>>>     at
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> >> ~[na:1.8.0_162]
> >>>>>     at java.lang.reflect.Method.invoke(Method.java:498)
> ~[na:1.8.0_162]
> >>>>>     at
> >>
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340)
> >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>>>>     at
> >>
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293)
> >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>>>>     at
> >>
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130)
> >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> >>>>>
> >>>>
> >>>> --
> >>>> Jean-Baptiste Onofré
> >>>> [hidden email]
> >>>> http://blog.nanthrax.net
> >>>> Talend - http://www.talend.com
> >> Talend - A Cloud Data Integration Leader (modern ETL)<
> >> http://www.talend.com/>
> >> www.talend.com<http://www.talend.com>
> >> Talend Data Fabric offers a single suite of cloud apps for data
> >> integration and data integrity to help enterprises collect, govern,
> >> transform, and share data.
> >>
> >>
> >>>>
> >>>
> >>
> >> --
> >> Jean-Baptiste Onofré
> >> [hidden email]
> >> http://blog.nanthrax.net
> >> Talend - http://www.talend.com
> >> Talend - A Cloud Data Integration Leader (modern ETL)<
> >> http://www.talend.com/>
> >> www.talend.com<http://www.talend.com>
> >> Talend Data Fabric offers a single suite of cloud apps for data
> >> integration and data integrity to help enterprises collect, govern,
> >> transform, and share data.
> >>
> >>
> >>
> >
>
> --
> Jean-Baptiste Onofré
> [hidden email]
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Reply | Threaded
Open this post in threaded view
|

Re: org.apache.activemq.kaha.impl.index.?

Tim Bain
Great, I'm glad you got it sorted out. Thanks for closing the loop and
letting us know.

Tim

On Tue, Nov 5, 2019, 8:27 AM Jonathan Roby <[hidden email]>
wrote:

> Hi All
>
>
> just an update: I figured out the issue. It was due to the  software we
> use (fedora-commons/tomcat) using its own set of configuration files and
> data directories. So I was trying to debug the wrong set of config files
> and data directories, thats why nothing seemed to work. And in the end, it
> was just deleting the data directory.
>
>
> Thanks Jean-Baptiste and Tim for your help
>
> Jon
>
> ________________________________
> From: Tim Bain <[hidden email]>
> Sent: Tuesday, November 5, 2019 7:34:10 AM
> To: ActiveMQ Users
> Subject: Re: org.apache.activemq.kaha.impl.index.?
>
> Is there any chance that a JAR from a newer version of ActiveMQ ended up on
> the broker's classpath? If you delete the contents of the lib directory and
> replace it with the contents from a freshly installed copy of 5.7.0, does
> the behavior change?
>
> Also, can you share your activemq.xml with us? And does this behavior
> change if you use the copy of activemq.xml from a fresh install?
>
> Tim
>
> On Sun, Nov 3, 2019, 2:35 PM Jonathan Roby <[hidden email]>
> wrote:
>
> > Hi
> >
> >
> > I tried putting the kahadb directory on the local directory, it
> > regenerated on startup but still gave me a BadMagicException
> >
> > ________________________________
> > From: Jean-Baptiste Onofré <[hidden email]>
> > Sent: Sunday, November 3, 2019 11:07:22 AM
> > To: [hidden email]
> > Subject: Re: org.apache.activemq.kaha.impl.index.?
> >
> > Hi Jonathan,
> >
> > user mailing list is fine for this kind of question.
> >
> > IMHO, I'm pretty sure it's related to the filesystem.
> >
> > As Tim suggested, maybe you can try to put the kahadb directory on local
> > filesystem instead of the SAN. It could be an interesting test.
> >
> > Regards
> > JB
> >
> > On 03/11/2019 16:52, Jonathan Roby wrote:
> > > Hi
> > >
> > >
> > > Also, i'm sorry for the subject heading of my original email, I'm not
> > sure what happened but its somewhat nonsensical.
> > >
> > >
> > > Also, would anyone recommend I post this question to the developer
> > mailing list, in case they might have other ideas?
> > >
> > >
> > > I've asked our IT infrastructure team if they can check the VMs
> > storage/machine layout for recent changes that could have caused this.
> > hopefully t hey find something or I get a lucky break
> > >
> > >
> > > Thank you everyone for your help so far
> > >
> > > Jon
> > >
> > > ________________________________
> > > From: Jonathan Roby
> > > Sent: Sunday, November 3, 2019 9:02:56 AM
> > > To: ActiveMQ Users
> > > Subject: Re: org.apache.activemq.kaha.impl.index.?
> > >
> > >
> > > Hi
> > >
> > >
> > > I just checked and the directory is created exactly when the process
> > tries to start. I've also checked and the ports used by the broker are
> not
> > visible in the output of the netstat command.
> > >
> > >
> > > I also tried moving the kahadb directory onto the local drive and still
> > get the BadMagicException at the end of nested exception stacktrace
> > >
> > >
> > > Thank you
> > >
> > > Jon
> > >
> > > ________________________________
> > > From: Tim Bain <[hidden email]>
> > > Sent: Sunday, November 3, 2019 8:06:29 AM
> > > To: ActiveMQ Users
> > > Subject: Re: org.apache.activemq.kaha.impl.index.?
> > >
> > > Following up on JB's line of questioning, is the directory only
> recreated
> > > when (exactly when, to the second, with a matching timestamp) this
> broker
> > > process is started? Might it be possible that a different broker
> process,
> > > running elsewhere, might be creating the directory and its files?
> > >
> > > If you reconfigure this broker to store the KahaDB data file on local
> > disk
> > > instead of your SAN (just temporarily), does the process start
> > > successfully?
> > >
> > > Tim
> > >
> > > On Sat, Nov 2, 2019, 1:56 PM Jonathan Roby <[hidden email]
> >
> > > wrote:
> > >
> > >>
> > >> No, its not fixed. I wiped out the data folder and the persistant
> > folder.
> > >> both were regenerated but still is failing
> > >>
> > >>
> > >> its weird as we have other instances set up exactly the same as this
> one
> > >> that dont have this error. same versions, same everything
> > >>
> > >>
> > >> so it sounds like you might be correct about it being the filesystem.
> i
> > >> hope not as that can be tricky to debug
> > >>
> > >>
> > >> Thank you for your help
> > >>
> > >> Jon
> > >>
> > >> ________________________________
> > >> From: Jean-Baptiste Onofré <[hidden email]>
> > >> Sent: Saturday, November 2, 2019 12:48:29 PM
> > >> To: [hidden email]
> > >> Subject: Re: org.apache.activemq.kaha.impl.index.?
> > >>
> > >> I guess it's fixed with you complete remove the kahadb folder right ?
> > >>
> > >> I remember couple of users mentioned similar issue on virtual machine.
> > >>
> > >> Did you try ActiveMQ 5.15.10 ? We did a bunch of fixes and
> improvements
> > >> on this version around kahadb.
> > >>
> > >> Regards
> > >> JB
> > >>
> > >> On 02/11/2019 18:46, Jean-Baptiste Onofré wrote:
> > >>> Hi,
> > >>>
> > >>> It's definitely something weird while reading file on the filesystem.
> > >>>
> > >>> Is it a local filesystem or shared one (NFS, CIFS, ...) ?
> > >>>
> > >>> Regards
> > >>> JB
> > >>>
> > >>> On 02/11/2019 18:18, Jonathan Roby wrote:
> > >>>> Hi
> > >>>>
> > >>>>
> > >>>> as far as I know, the version of ActiveMQ hasn't been updated and
> i've
> > >> now deleted the kahadb directory a few times (it regenerated) and even
> > >> moved the broker path in case it was a drive error
> > >>>>
> > >>>>
> > >>>> Jon
> > >>>>
> > >>>> ________________________________
> > >>>> From: Jean-Baptiste Onofré <[hidden email]>
> > >>>> Sent: Saturday, November 2, 2019 12:16:31 PM
> > >>>> To: [hidden email]
> > >>>> Subject: Re: org.apache.activemq.kaha.impl.index.?
> > >>>>
> > >>>> Hi,
> > >>>>
> > >>>> Does the kahadb folder has been created with the same ActiveMQ
> version
> > >>>> (and you updated ActiveMQ version later) ?
> > >>>>
> > >>>> It seems the kahadb index is not consistent.
> > >>>>
> > >>>> Regards
> > >>>> JB
> > >>>>
> > >>>> On 02/11/2019 18:14, Jonathan Roby wrote:
> > >>>>> Hi, i've just joined the list so apologies if i've not this
> > correctly.
> > >>>>>
> > >>>>>
> > >>>>> We run a number of islandora instances that use ActiveMQ 5.7.0 for
> > >> message passing between components. Yesterday, one of our instances
> > seemed
> > >> to stop working. after several restarts of components and the system
> > >> itself, I noticed that when the components restarted, it always
> > produced a
> > >> nested exception and the very last error was
> > >> org.apache.activemq.kaha.impl.index.BadMagicException.
> > >>>>>
> > >>>>>
> > >>>>> I've googled this and taken the advice given, which is usually to
> > >> empty/delete the data directory or the kahadb directory and it still
> > isnt
> > >> working. All the other components seem to be working fine.
> > >>>>>
> > >>>>>
> > >>>>> Thank you for any help anyone can give me.
> > >>>>>
> > >>>>> Jon
> > >>>>>
> > >>>>>
> > >>>>> Here is the last part of the stack trace.
> > >>>>>
> > >>>>>
> > >>>>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException:
> > null
> > >>>>>     at
> > >> org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >>
> >
> org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >>
> >
> org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >>
> >
> org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >>
> >
> org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >>
> >
> org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >>
> >
> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >>
> >
> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >>
> >
> org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >>
> >
> org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >>
> >
> org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >>
> >
> org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >>
> >
> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >>
> >
> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >>
> >
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >>
> >
> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >>
> >
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >>
> >
> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >>
> >
> org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >>
> >
> org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >>
> >
> org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >>
> >
> org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >>
> >
> org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >>
> > org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >>
> >
> org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >>
> >
> org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >> org.apache.activemq.broker.BrokerService.start(BrokerService.java:578)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at
> > >>
> >
> org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539)
> > >> ~[activemq-core-5.7.0.jar:5.7.0]
> > >>>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > >> ~[na:1.8.0_162]
> > >>>>>     at
> > >>
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > >> ~[na:1.8.0_162]
> > >>>>>     at
> > >>
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > >> ~[na:1.8.0_162]
> > >>>>>     at java.lang.reflect.Method.invoke(Method.java:498)
> > ~[na:1.8.0_162]
> > >>>>>     at
> > >>
> >
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340)
> > >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> > >>>>>     at
> > >>
> >
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293)
> > >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> > >>>>>     at
> > >>
> >
> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130)
> > >> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
> > >>>>>
> > >>>>
> > >>>> --
> > >>>> Jean-Baptiste Onofré
> > >>>> [hidden email]
> > >>>> http://blog.nanthrax.net
> > >>>> Talend - http://www.talend.com
> > >> Talend - A Cloud Data Integration Leader (modern ETL)<
> > >> http://www.talend.com/>
> > >> www.talend.com<http://www.talend.com>
> > >> Talend Data Fabric offers a single suite of cloud apps for data
> > >> integration and data integrity to help enterprises collect, govern,
> > >> transform, and share data.
> > >>
> > >>
> > >>>>
> > >>>
> > >>
> > >> --
> > >> Jean-Baptiste Onofré
> > >> [hidden email]
> > >> http://blog.nanthrax.net
> > >> Talend - http://www.talend.com
> > >> Talend - A Cloud Data Integration Leader (modern ETL)<
> > >> http://www.talend.com/>
> > >> www.talend.com<http://www.talend.com>
> > >> Talend Data Fabric offers a single suite of cloud apps for data
> > >> integration and data integrity to help enterprises collect, govern,
> > >> transform, and share data.
> > >>
> > >>
> > >>
> > >
> >
> > --
> > Jean-Baptiste Onofré
> > [hidden email]
> > http://blog.nanthrax.net
> > Talend - http://www.talend.com
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: org.apache.activemq.kaha.impl.index.?

jbonofre
In reply to this post by Jonathan Roby
Thanks for the update !

Great you figured it out and always happy to help.

Regards
JB

On 05/11/2019 16:27, Jonathan Roby wrote:

> Hi All
>
>
> just an update: I figured out the issue. It was due to the  software we use (fedora-commons/tomcat) using its own set of configuration files and data directories. So I was trying to debug the wrong set of config files and data directories, thats why nothing seemed to work. And in the end, it was just deleting the data directory.
>
>
> Thanks Jean-Baptiste and Tim for your help
>
> Jon
>
> ________________________________
> From: Tim Bain <[hidden email]>
> Sent: Tuesday, November 5, 2019 7:34:10 AM
> To: ActiveMQ Users
> Subject: Re: org.apache.activemq.kaha.impl.index.?
>
> Is there any chance that a JAR from a newer version of ActiveMQ ended up on
> the broker's classpath? If you delete the contents of the lib directory and
> replace it with the contents from a freshly installed copy of 5.7.0, does
> the behavior change?
>
> Also, can you share your activemq.xml with us? And does this behavior
> change if you use the copy of activemq.xml from a fresh install?
>
> Tim
>
> On Sun, Nov 3, 2019, 2:35 PM Jonathan Roby <[hidden email]>
> wrote:
>
>> Hi
>>
>>
>> I tried putting the kahadb directory on the local directory, it
>> regenerated on startup but still gave me a BadMagicException
>>
>> ________________________________
>> From: Jean-Baptiste Onofré <[hidden email]>
>> Sent: Sunday, November 3, 2019 11:07:22 AM
>> To: [hidden email]
>> Subject: Re: org.apache.activemq.kaha.impl.index.?
>>
>> Hi Jonathan,
>>
>> user mailing list is fine for this kind of question.
>>
>> IMHO, I'm pretty sure it's related to the filesystem.
>>
>> As Tim suggested, maybe you can try to put the kahadb directory on local
>> filesystem instead of the SAN. It could be an interesting test.
>>
>> Regards
>> JB
>>
>> On 03/11/2019 16:52, Jonathan Roby wrote:
>>> Hi
>>>
>>>
>>> Also, i'm sorry for the subject heading of my original email, I'm not
>> sure what happened but its somewhat nonsensical.
>>>
>>>
>>> Also, would anyone recommend I post this question to the developer
>> mailing list, in case they might have other ideas?
>>>
>>>
>>> I've asked our IT infrastructure team if they can check the VMs
>> storage/machine layout for recent changes that could have caused this.
>> hopefully t hey find something or I get a lucky break
>>>
>>>
>>> Thank you everyone for your help so far
>>>
>>> Jon
>>>
>>> ________________________________
>>> From: Jonathan Roby
>>> Sent: Sunday, November 3, 2019 9:02:56 AM
>>> To: ActiveMQ Users
>>> Subject: Re: org.apache.activemq.kaha.impl.index.?
>>>
>>>
>>> Hi
>>>
>>>
>>> I just checked and the directory is created exactly when the process
>> tries to start. I've also checked and the ports used by the broker are not
>> visible in the output of the netstat command.
>>>
>>>
>>> I also tried moving the kahadb directory onto the local drive and still
>> get the BadMagicException at the end of nested exception stacktrace
>>>
>>>
>>> Thank you
>>>
>>> Jon
>>>
>>> ________________________________
>>> From: Tim Bain <[hidden email]>
>>> Sent: Sunday, November 3, 2019 8:06:29 AM
>>> To: ActiveMQ Users
>>> Subject: Re: org.apache.activemq.kaha.impl.index.?
>>>
>>> Following up on JB's line of questioning, is the directory only recreated
>>> when (exactly when, to the second, with a matching timestamp) this broker
>>> process is started? Might it be possible that a different broker process,
>>> running elsewhere, might be creating the directory and its files?
>>>
>>> If you reconfigure this broker to store the KahaDB data file on local
>> disk
>>> instead of your SAN (just temporarily), does the process start
>>> successfully?
>>>
>>> Tim
>>>
>>> On Sat, Nov 2, 2019, 1:56 PM Jonathan Roby <[hidden email]>
>>> wrote:
>>>
>>>>
>>>> No, its not fixed. I wiped out the data folder and the persistant
>> folder.
>>>> both were regenerated but still is failing
>>>>
>>>>
>>>> its weird as we have other instances set up exactly the same as this one
>>>> that dont have this error. same versions, same everything
>>>>
>>>>
>>>> so it sounds like you might be correct about it being the filesystem. i
>>>> hope not as that can be tricky to debug
>>>>
>>>>
>>>> Thank you for your help
>>>>
>>>> Jon
>>>>
>>>> ________________________________
>>>> From: Jean-Baptiste Onofré <[hidden email]>
>>>> Sent: Saturday, November 2, 2019 12:48:29 PM
>>>> To: [hidden email]
>>>> Subject: Re: org.apache.activemq.kaha.impl.index.?
>>>>
>>>> I guess it's fixed with you complete remove the kahadb folder right ?
>>>>
>>>> I remember couple of users mentioned similar issue on virtual machine.
>>>>
>>>> Did you try ActiveMQ 5.15.10 ? We did a bunch of fixes and improvements
>>>> on this version around kahadb.
>>>>
>>>> Regards
>>>> JB
>>>>
>>>> On 02/11/2019 18:46, Jean-Baptiste Onofré wrote:
>>>>> Hi,
>>>>>
>>>>> It's definitely something weird while reading file on the filesystem.
>>>>>
>>>>> Is it a local filesystem or shared one (NFS, CIFS, ...) ?
>>>>>
>>>>> Regards
>>>>> JB
>>>>>
>>>>> On 02/11/2019 18:18, Jonathan Roby wrote:
>>>>>> Hi
>>>>>>
>>>>>>
>>>>>> as far as I know, the version of ActiveMQ hasn't been updated and i've
>>>> now deleted the kahadb directory a few times (it regenerated) and even
>>>> moved the broker path in case it was a drive error
>>>>>>
>>>>>>
>>>>>> Jon
>>>>>>
>>>>>> ________________________________
>>>>>> From: Jean-Baptiste Onofré <[hidden email]>
>>>>>> Sent: Saturday, November 2, 2019 12:16:31 PM
>>>>>> To: [hidden email]
>>>>>> Subject: Re: org.apache.activemq.kaha.impl.index.?
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Does the kahadb folder has been created with the same ActiveMQ version
>>>>>> (and you updated ActiveMQ version later) ?
>>>>>>
>>>>>> It seems the kahadb index is not consistent.
>>>>>>
>>>>>> Regards
>>>>>> JB
>>>>>>
>>>>>> On 02/11/2019 18:14, Jonathan Roby wrote:
>>>>>>> Hi, i've just joined the list so apologies if i've not this
>> correctly.
>>>>>>>
>>>>>>>
>>>>>>> We run a number of islandora instances that use ActiveMQ 5.7.0 for
>>>> message passing between components. Yesterday, one of our instances
>> seemed
>>>> to stop working. after several restarts of components and the system
>>>> itself, I noticed that when the components restarted, it always
>> produced a
>>>> nested exception and the very last error was
>>>> org.apache.activemq.kaha.impl.index.BadMagicException.
>>>>>>>
>>>>>>>
>>>>>>> I've googled this and taken the advice given, which is usually to
>>>> empty/delete the data directory or the kahadb directory and it still
>> isnt
>>>> working. All the other components seem to be working fine.
>>>>>>>
>>>>>>>
>>>>>>> Thank you for any help anyone can give me.
>>>>>>>
>>>>>>> Jon
>>>>>>>
>>>>>>>
>>>>>>> Here is the last part of the stack trace.
>>>>>>>
>>>>>>>
>>>>>>> Caused by: org.apache.activemq.kaha.impl.index.BadMagicException:
>> null
>>>>>>>     at
>>>> org.apache.activemq.kaha.impl.index.IndexItem.read(IndexItem.java:141)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>>
>> org.apache.activemq.kaha.impl.index.StoreIndexReader.readItem(StoreIndexReader.java:50)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>>
>> org.apache.activemq.kaha.impl.index.IndexManager.initialize(IndexManager.java:207)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>>
>> org.apache.activemq.kaha.impl.index.IndexManager.<init>(IndexManager.java:60)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>>
>> org.apache.activemq.kaha.impl.KahaStore.getIndexManager(KahaStore.java:366)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>>
>> org.apache.activemq.kaha.impl.KahaStore.getMapContainer(KahaStore.java:219)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>>
>> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.getMapReferenceContainer(KahaReferenceStoreAdapter.java:219)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>>
>> org.apache.activemq.store.kahadaptor.KahaReferenceStoreAdapter.createQueueReferenceStore(KahaReferenceStoreAdapter.java:155)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>>
>> org.apache.activemq.store.amq.AMQPersistenceAdapter.createQueueMessageStore(AMQPersistenceAdapter.java:482)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>>
>> org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:83)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>>
>> org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:532)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>>
>> org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>>
>> org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:137)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>>
>> org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:283)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>>
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>>
>> org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:175)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>>
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>>
>> org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>>
>> org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>>
>> org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:99)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>>
>> org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:186)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>>
>> org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:121)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>> org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>>
>> org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:122)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>>
>> org.apache.activemq.broker.BrokerService$5.start(BrokerService.java:2110)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>>
>> org.apache.activemq.broker.BrokerService.doStartBroker(BrokerService.java:662)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>>
>> org.apache.activemq.broker.BrokerService.startBroker(BrokerService.java:642)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>> org.apache.activemq.broker.BrokerService.start(BrokerService.java:578)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at
>>>>
>> org.apache.activemq.broker.BrokerService.autoStart(BrokerService.java:539)
>>>> ~[activemq-core-5.7.0.jar:5.7.0]
>>>>>>>     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> ~[na:1.8.0_162]
>>>>>>>     at
>>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>>> ~[na:1.8.0_162]
>>>>>>>     at
>>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>>> ~[na:1.8.0_162]
>>>>>>>     at java.lang.reflect.Method.invoke(Method.java:498)
>> ~[na:1.8.0_162]
>>>>>>>     at
>>>>
>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:340)
>>>> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>>>>>>     at
>>>>
>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:293)
>>>> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>>>>>>     at
>>>>
>> org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:130)
>>>> ~[spring-beans-3.0.7.RELEASE.jar:3.0.7.RELEASE]
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Jean-Baptiste Onofré
>>>>>> [hidden email]
>>>>>> http://blog.nanthrax.net
>>>>>> Talend - http://www.talend.com
>>>> Talend - A Cloud Data Integration Leader (modern ETL)<
>>>> http://www.talend.com/>
>>>> www.talend.com<http://www.talend.com>
>>>> Talend Data Fabric offers a single suite of cloud apps for data
>>>> integration and data integrity to help enterprises collect, govern,
>>>> transform, and share data.
>>>>
>>>>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Jean-Baptiste Onofré
>>>> [hidden email]
>>>> http://blog.nanthrax.net
>>>> Talend - http://www.talend.com
>>>> Talend - A Cloud Data Integration Leader (modern ETL)<
>>>> http://www.talend.com/>
>>>> www.talend.com<http://www.talend.com>
>>>> Talend Data Fabric offers a single suite of cloud apps for data
>>>> integration and data integrity to help enterprises collect, govern,
>>>> transform, and share data.
>>>>
>>>>
>>>>
>>>
>>
>> --
>> Jean-Baptiste Onofré
>> [hidden email]
>> http://blog.nanthrax.net
>> Talend - http://www.talend.com
>>
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com