Leveldb not recovering from failover

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Leveldb not recovering from failover

mtod
I have a 3 node cluster on Redhat servers running ActiveMQ version 5.13.2 and Zookeeper 3.4.8

The system runs without error for days then when I fail it over it will sometimes will not recover see log segment below.
The strange thing is I’m getting a Connection refused even from the master connecting to itself.

Does anyone have any idea what might cause this?

Server1
ActiveMQ.log
2016-03-15 11:57:27,756 | WARN  | Store update waiting on 1 replica(s) to catch up to log position 1268728231.  | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-24
2016-03-15 11:57:28,757 | WARN  | Store update waiting on 1 replica(s) to catch up to log position 1268728231.  | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-24
2016-03-15 11:57:29,758 | WARN  | Store update waiting on 1 replica(s) to catch up to log position 1268728231.  | org.apache.activemq.leveldb.replicated.MasterLevelDBStore | ActiveMQ BrokerService[localhost] Task-24

Server2
ActiveMQ.log
2016-03-15 11:58:27,575 | INFO  | Using the pure java LevelDB implementation. | org.apache.activemq.leveldb.LevelDBClient | hawtdispatch-DEFAULT-1
2016-03-15 11:58:27,575 | INFO  | Attaching to master: tcp://Server2.corp.local:61619 | org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | hawtdispatch-DEFAULT-1
2016-03-15 11:58:27,577 | WARN  | Unexpected session error: java.net.ConnectException: Connection refused | org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | hawtdispatch-DEFAULT-1

Server3
ActiveMQ.log
2016-03-15 11:58:27,484 | INFO  | Using the pure java LevelDB implementation. | org.apache.activemq.leveldb.LevelDBClient | hawtdispatch-DEFAULT-1
2016-03-15 11:58:27,485 | INFO  | Attaching to master: tcp://Server2.corp.local:61619 | org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | hawtdispatch-DEFAULT-1
2016-03-15 11:58:27,487 | WARN  | Unexpected session error: java.net.ConnectException: Connection refused | org.apache.activemq.leveldb.replicated.SlaveLevelDBStore | hawtdispatch-DEFAULT-1

Thanks
Mike
Reply | Threaded
Open this post in threaded view
|

Re: Leveldb not recovering from failover

mtod
I think I found the issue it was a typo in the activemq.xml leveldb config.

I setup a new test I'll report back.

Thanks
Mike
Reply | Threaded
Open this post in threaded view
|

Re: Leveldb not recovering from failover

mtod
I'm coming to the conclusion that LevelDB is not ready for prime time.

I have run this scenario several times and it just keeps failing over time.

I think I'm going to have to look for another solution.

Thanks

Mike
 
Reply | Threaded
Open this post in threaded view
|

Re: Leveldb not recovering from failover

sandeepatmca
In reply to this post by mtod
Hi Team,
Did youfigured out the solution for below issue. I am facing similar issue
in our solution .



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Leveldb not recovering from failover

Tim Bain
LevelDB is no longer supported, because of a lack of knowledgeable people
to answer questions and fix bugs such as this one. If using the source code
and a debugger to troubleshoot your own problems doesn't sound appealing,
we recommend you switch to one of the supported persistence stores, or to
Artemis.

Tim

On Thu, Aug 16, 2018, 9:43 AM sandeepatmca <[hidden email]> wrote:

> Hi Team,
> Did youfigured out the solution for below issue. I am facing similar issue
> in our solution .
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Leveldb not recovering from failover

sandeepatmca
I am not sure how we can change the implementation from levelDB to other
persistence stores. Would that require just the configuration changes in
activemq.xml or more than that.
If someone can help to undrstand why below error continously comes in the
karaf and fuse log.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Leveldb not recovering from failover

Tim Bain
Changing the persistence store type is done by changing activemq.xml, as
you mentioned.

Questions about Fuse and Karaf should be directed to their respective user
communities or support channels (I assume they have those things), or to
StackOverflow. This mailing list is for questions about ActiveMQ.

Tim

On Tue, Nov 6, 2018, 12:30 PM sandeepatmca <[hidden email] wrote:

> I am not sure how we can change the implementation from levelDB to other
> persistence stores. Would that require just the configuration changes in
> activemq.xml or more than that.
> If someone can help to undrstand why below error continously comes in the
> karaf and fuse log.
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>