KahaDb issue (Detected missing journal files) when upgrading from 5.12.2 to 5.14.0

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

KahaDb issue (Detected missing journal files) when upgrading from 5.12.2 to 5.14.0

Martin Lichtin
When trying to upgrade from 5.12.2 to 5.14.0 we encounter this exception at startup:


Caused by: java.io.IOException: Detected missing journal files. [3]
at org.apache.activemq.store.kahadb.MessageDatabase.recoverIndex(MessageDatabase.java:978)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.store.kahadb.MessageDatabase$5.execute(MessageDatabase.java:714)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:779)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.store.kahadb.MessageDatabase.recover(MessageDatabase.java:711)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:464)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:483)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.store.kahadb.MessageDatabase.doStart(MessageDatabase.java:297)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.store.kahadb.KahaDBStore.doStart(KahaDBStore.java:216)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.doStart(KahaDBPersistenceAdapter.java:223)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.store.kahadb.MultiKahaDBPersistenceAdapter.doStart(MultiKahaDBPersistenceAdapter.java:355)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:674)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:658)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.broker.BrokerService.start(BrokerService.java:622)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.osgi.ActiveMQServiceFactory.updated(ActiveMQServiceFactory.java:140)[102:org.apache.activemq.activemq-osgi:5.14.0]
... 7 more

So can't upgrade... Downgrading to 5.12.2, it starts just fine with the same Kahadb files.
Has perhaps an incompatibility been introduced?
Reply | Threaded
Open this post in threaded view
|

Re: KahaDb issue (Detected missing journal files) when upgrading from 5.12.2 to 5.14.0

Martin Herrman
Hi Martin,


When updating from 5.11 to 5.14 I encountered a different issue (which might have existed during earlier upgrades as well). I am in the situation that I can remove the kahadb contents before starting 5.14. Maybe that can be a solution for you as well.


Also see the thread "Updating KahaDB" : http://mail-archives.apache.org/mod_mbox/activemq-users/201608.mbox/browser


Regards,


Martin

________________________________
Van: Martin Lichtin <[hidden email]>
Verzonden: donderdag 1 september 2016 10:26:49
Aan: ActiveMQ Users
Onderwerp: KahaDb issue (Detected missing journal files) when upgrading from 5.12.2 to 5.14.0

When trying to upgrade from 5.12.2 to 5.14.0 we encounter this exception at startup:


Caused by: java.io.IOException: Detected missing journal files. [3]
at org.apache.activemq.store.kahadb.MessageDatabase.recoverIndex(MessageDatabase.java:978)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.store.kahadb.MessageDatabase$5.execute(MessageDatabase.java:714)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:779)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.store.kahadb.MessageDatabase.recover(MessageDatabase.java:711)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:464)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:483)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.store.kahadb.MessageDatabase.doStart(MessageDatabase.java:297)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.store.kahadb.KahaDBStore.doStart(KahaDBStore.java:216)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.doStart(KahaDBPersistenceAdapter.java:223)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.store.kahadb.MultiKahaDBPersistenceAdapter.doStart(MultiKahaDBPersistenceAdapter.java:355)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:674)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:658)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.broker.BrokerService.start(BrokerService.java:622)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.osgi.ActiveMQServiceFactory.updated(ActiveMQServiceFactory.java:140)[102:org.apache.activemq.activemq-osgi:5.14.0]
... 7 more

So can't upgrade... Downgrading to 5.12.2, it starts just fine with the same Kahadb files.
Has perhaps an incompatibility been introduced?
Met vriendelijke groet/Kind regards,

Martin Herrman
Manager DevOps

[YMOR]<http://www.ymor.com>

Ptolemaeuslaan 70
3528 BP Utrecht
Mobiel: +31 6 10761936
Kantoor:        +31(0)30 6021149

Email:  [hidden email]

[ITOA]
Van reactief naar preventief IT-beheer?
Download de whitepaper:
'Van achteruitkijkspiegel naar glazen bol met IT Operations Analytics'<http://www.ymor.com/whitepapers/>


This e-mail message and its attachments are subject to the disclaimer published on our website<https://www.ymor.com/files/2016/04/2016-Disclaimer-and-Security-EN.pdf>
Reply | Threaded
Open this post in threaded view
|

Re: KahaDb issue (Detected missing journal files) when upgrading from 5.12.2 to 5.14.0

lichtin
In reply to this post by Martin Lichtin
Reply | Threaded
Open this post in threaded view
|

Re: KahaDb issue (Detected missing journal files) when upgrading from 5.12.2 to 5.14.0

lichtin
In reply to this post by Martin Lichtin
I read through all release notes 5.13.x and 5.14.0 and cannot find a KahaDb incompatibility issue.
This is quite an unexpected blocker for anyone trying to upgrade to 5.14.0.

Will try temporarily setting ignoreMissingJournalfiles=true and see it if comes up, then later try changing it back.

Martin Lichtin wrote
When trying to upgrade from 5.12.2 to 5.14.0 we encounter this exception at startup:

Caused by: java.io.IOException: Detected missing journal files. [3]
at org.apache.activemq.store.kahadb.MessageDatabase.recoverIndex(MessageDatabase.java:978)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.store.kahadb.MessageDatabase$5.execute(MessageDatabase.java:714)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.store.kahadb.disk.page.Transaction.execute(Transaction.java:779)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.store.kahadb.MessageDatabase.recover(MessageDatabase.java:711)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.store.kahadb.MessageDatabase.open(MessageDatabase.java:464)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:483)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.store.kahadb.MessageDatabase.doStart(MessageDatabase.java:297)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.store.kahadb.KahaDBStore.doStart(KahaDBStore.java:216)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.doStart(KahaDBPersistenceAdapter.java:223)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.store.kahadb.MultiKahaDBPersistenceAdapter.doStart(MultiKahaDBPersistenceAdapter.java:355)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.util.ServiceSupport.start(ServiceSupport.java:55)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(BrokerService.java:674)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.broker.BrokerService.startPersistenceAdapter(BrokerService.java:658)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.broker.BrokerService.start(BrokerService.java:622)[102:org.apache.activemq.activemq-osgi:5.14.0]
at org.apache.activemq.osgi.ActiveMQServiceFactory.updated(ActiveMQServiceFactory.java:140)[102:org.apache.activemq.activemq-osgi:5.14.0]
... 7 more

So can't upgrade... Downgrading to 5.12.2, it starts just fine with the same Kahadb files.
Has perhaps an incompatibility been introduced?
Reply | Threaded
Open this post in threaded view
|

Re: KahaDb issue (Detected missing journal files) when upgrading from 5.12.2 to 5.14.0

Tim Bain
Martin Lichtin, did you ever resolve what was causing your problem, or how
to work around it?

Tim

On Sep 2, 2016 3:11 AM, "lichtin" <[hidden email]> wrote:

> I read through all release notes 5.13.x and 5.14.0 and cannot find a KahaDb
> incompatibility issue.
> This is quite an unexpected blocker for anyone trying to upgrade to 5.14.0.
>
> Will try temporarily setting ignoreMissingJournalfiles=true and see it if
> comes up, then later try changing it back.
>
>
> Martin Lichtin wrote
> > When trying to upgrade from 5.12.2 to 5.14.0 we encounter this exception
> > at startup:
> >
> > Caused by: java.io.IOException: Detected missing journal files. [3]
> > at
> > org.apache.activemq.store.kahadb.MessageDatabase.
> recoverIndex(MessageDatabase.java:978)[102:org.apache.
> activemq.activemq-osgi:5.14.0]
> > at
> > org.apache.activemq.store.kahadb.MessageDatabase$5.
> execute(MessageDatabase.java:714)[102:org.apache.activemq.
> activemq-osgi:5.14.0]
> > at
> > org.apache.activemq.store.kahadb.disk.page.Transaction.
> execute(Transaction.java:779)[102:org.apache.activemq.
> activemq-osgi:5.14.0]
> > at
> > org.apache.activemq.store.kahadb.MessageDatabase.
> recover(MessageDatabase.java:711)[102:org.apache.activemq.
> activemq-osgi:5.14.0]
> > at
> > org.apache.activemq.store.kahadb.MessageDatabase.open(
> MessageDatabase.java:464)[102:org.apache.activemq.activemq-osgi:5.14.0]
> > at
> > org.apache.activemq.store.kahadb.MessageDatabase.load(
> MessageDatabase.java:483)[102:org.apache.activemq.activemq-osgi:5.14.0]
> > at
> > org.apache.activemq.store.kahadb.MessageDatabase.
> doStart(MessageDatabase.java:297)[102:org.apache.activemq.
> activemq-osgi:5.14.0]
> > at
> > org.apache.activemq.store.kahadb.KahaDBStore.doStart(
> KahaDBStore.java:216)[102:org.apache.activemq.activemq-osgi:5.14.0]
> > at
> > org.apache.activemq.util.ServiceSupport.start(
> ServiceSupport.java:55)[102:org.apache.activemq.activemq-osgi:5.14.0]
> > at
> > org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.doStart(
> KahaDBPersistenceAdapter.java:223)[102:org.apache.activemq.
> activemq-osgi:5.14.0]
> > at
> > org.apache.activemq.util.ServiceSupport.start(
> ServiceSupport.java:55)[102:org.apache.activemq.activemq-osgi:5.14.0]
> > at
> > org.apache.activemq.store.kahadb.MultiKahaDBPersistenceAdapter.doStart(
> MultiKahaDBPersistenceAdapter.java:355)[102:org.apache.
> activemq.activemq-osgi:5.14.0]
> > at
> > org.apache.activemq.util.ServiceSupport.start(
> ServiceSupport.java:55)[102:org.apache.activemq.activemq-osgi:5.14.0]
> > at
> > org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(
> BrokerService.java:674)[102:org.apache.activemq.activemq-osgi:5.14.0]
> > at
> > org.apache.activemq.broker.BrokerService.startPersistenceAdapter(
> BrokerService.java:658)[102:org.apache.activemq.activemq-osgi:5.14.0]
> > at
> > org.apache.activemq.broker.BrokerService.start(
> BrokerService.java:622)[102:org.apache.activemq.activemq-osgi:5.14.0]
> > at
> > org.apache.activemq.osgi.ActiveMQServiceFactory.updated(
> ActiveMQServiceFactory.java:140)[102:org.apache.activemq.
> activemq-osgi:5.14.0]
> > ... 7 more
> >
> > So can't upgrade... Downgrading to 5.12.2, it starts just fine with the
> > same Kahadb files.
> > Has perhaps an incompatibility been introduced?
>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/KahaDb-issue-Detected-missing-journal-
> files-when-upgrading-from-5-12-2-to-5-14-0-tp4716052p4716100.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Re: KahaDb issue (Detected missing journal files) when upgrading from 5.12.2 to 5.14.0

Martin Lichtin
The solution/workaround for us was to move away the 3 db.* files and let Kahadb start with only the journal files.
It recreated the 'index' files and since then we're ok.


On 05.10.2016 14:48, Tim Bain wrote:

> Martin Lichtin, did you ever resolve what was causing your problem, or how
> to work around it?
>
> Tim
>
> On Sep 2, 2016 3:11 AM, "lichtin" <[hidden email]> wrote:
>
>> I read through all release notes 5.13.x and 5.14.0 and cannot find a KahaDb
>> incompatibility issue.
>> This is quite an unexpected blocker for anyone trying to upgrade to 5.14.0.
>>
>> Will try temporarily setting ignoreMissingJournalfiles=true and see it if
>> comes up, then later try changing it back.
>>
>>
>> Martin Lichtin wrote
>>> When trying to upgrade from 5.12.2 to 5.14.0 we encounter this exception
>>> at startup:
>>>
>>> Caused by: java.io.IOException: Detected missing journal files. [3]
>>> at
>>> org.apache.activemq.store.kahadb.MessageDatabase.
>> recoverIndex(MessageDatabase.java:978)[102:org.apache.
>> activemq.activemq-osgi:5.14.0]
>>> at
>>> org.apache.activemq.store.kahadb.MessageDatabase$5.
>> execute(MessageDatabase.java:714)[102:org.apache.activemq.
>> activemq-osgi:5.14.0]
>>> at
>>> org.apache.activemq.store.kahadb.disk.page.Transaction.
>> execute(Transaction.java:779)[102:org.apache.activemq.
>> activemq-osgi:5.14.0]
>>> at
>>> org.apache.activemq.store.kahadb.MessageDatabase.
>> recover(MessageDatabase.java:711)[102:org.apache.activemq.
>> activemq-osgi:5.14.0]
>>> at
>>> org.apache.activemq.store.kahadb.MessageDatabase.open(
>> MessageDatabase.java:464)[102:org.apache.activemq.activemq-osgi:5.14.0]
>>> at
>>> org.apache.activemq.store.kahadb.MessageDatabase.load(
>> MessageDatabase.java:483)[102:org.apache.activemq.activemq-osgi:5.14.0]
>>> at
>>> org.apache.activemq.store.kahadb.MessageDatabase.
>> doStart(MessageDatabase.java:297)[102:org.apache.activemq.
>> activemq-osgi:5.14.0]
>>> at
>>> org.apache.activemq.store.kahadb.KahaDBStore.doStart(
>> KahaDBStore.java:216)[102:org.apache.activemq.activemq-osgi:5.14.0]
>>> at
>>> org.apache.activemq.util.ServiceSupport.start(
>> ServiceSupport.java:55)[102:org.apache.activemq.activemq-osgi:5.14.0]
>>> at
>>> org.apache.activemq.store.kahadb.KahaDBPersistenceAdapter.doStart(
>> KahaDBPersistenceAdapter.java:223)[102:org.apache.activemq.
>> activemq-osgi:5.14.0]
>>> at
>>> org.apache.activemq.util.ServiceSupport.start(
>> ServiceSupport.java:55)[102:org.apache.activemq.activemq-osgi:5.14.0]
>>> at
>>> org.apache.activemq.store.kahadb.MultiKahaDBPersistenceAdapter.doStart(
>> MultiKahaDBPersistenceAdapter.java:355)[102:org.apache.
>> activemq.activemq-osgi:5.14.0]
>>> at
>>> org.apache.activemq.util.ServiceSupport.start(
>> ServiceSupport.java:55)[102:org.apache.activemq.activemq-osgi:5.14.0]
>>> at
>>> org.apache.activemq.broker.BrokerService.doStartPersistenceAdapter(
>> BrokerService.java:674)[102:org.apache.activemq.activemq-osgi:5.14.0]
>>> at
>>> org.apache.activemq.broker.BrokerService.startPersistenceAdapter(
>> BrokerService.java:658)[102:org.apache.activemq.activemq-osgi:5.14.0]
>>> at
>>> org.apache.activemq.broker.BrokerService.start(
>> BrokerService.java:622)[102:org.apache.activemq.activemq-osgi:5.14.0]
>>> at
>>> org.apache.activemq.osgi.ActiveMQServiceFactory.updated(
>> ActiveMQServiceFactory.java:140)[102:org.apache.activemq.
>> activemq-osgi:5.14.0]
>>> ... 7 more
>>>
>>> So can't upgrade... Downgrading to 5.12.2, it starts just fine with the
>>> same Kahadb files.
>>> Has perhaps an incompatibility been introduced?
>>
>>
>>
>>
>> --
>> View this message in context: http://activemq.2283324.n4.
>> nabble.com/KahaDb-issue-Detected-missing-journal-
>> files-when-upgrading-from-5-12-2-to-5-14-0-tp4716052p4716100.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>