DB lease lock - broker should stop-start to retry acquiring the lock

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

DB lease lock - broker should stop-start to retry acquiring the lock

Martin Lichtin
Hi

Can you help me understand this situation?
I've ActiveMQ configured with a lease lock on a DB. Spontaneously it decided it can no longer keep the lock, and starts shutting down:

2019-02-27T16:47:45,491 | INFO  | Starting Thread | BrokerService          | he.activemq.broker.BrokerService  773 | 73 - activemq-osgi - 5.15.6 | Apache ActiveMQ 5.15.6 (broker, amq2) started

2019-02-27T16:51:05,907 | ERROR | KeepAlive Timer | LockableServiceSupport | mq.broker.LockableServiceSupport  143 | 73 - activemq-osgi - 5.15.6 | broker, no longer able to keep the exclusive lock so giving up being a master
2019-02-27T16:51:05,909 | INFO  | KeepAlive Timer | BrokerService          | he.activemq.broker.BrokerService  836 | 73 - activemq-osgi - 5.15.6 | Apache ActiveMQ 5.15.6 (broker, amq2) is shutting down

It doesn't say why it could no longer keep the lock, but anyways, the bigger issue is that after stopping, it does not restart to retry acquiring the lock.

Is there a configuration I'm missing? The code in LockableServiceSupport actually requests a restart, but where is that re-start done?

Thanks for any insight
- Martin

Reply | Threaded
Open this post in threaded view
|

Re: DB lease lock - broker should stop-start to retry acquiring the lock

Tim Bain
It's my understanding that restarts are handled by the Tanuki service
wrapper, when ActiveMQ is installed as a service, using the instructions at
http://activemq.apache.org/java-service-wrapper.html. Is that how you're
using ActiveMQ? Or are you running the broker directly from the executable
in the bin directory, which wouldn't use the wrapper (and so wouldn't
restart the process)?

Tim

On Thu, Feb 28, 2019, 1:57 PM Martin Lichtin <[hidden email]>
wrote:

> Hi
>
> Can you help me understand this situation?
> I've ActiveMQ configured with a lease lock on a DB. Spontaneously it
> decided it can no longer keep the lock, and starts shutting down:
>
> 2019-02-27T16:47:45,491 | INFO  | Starting Thread | BrokerService
> | he.activemq.broker.BrokerService  773 | 73 - activemq-osgi - 5.15.6 |
> Apache ActiveMQ 5.15.6 (broker, amq2) started
>
> 2019-02-27T16:51:05,907 | ERROR | KeepAlive Timer | LockableServiceSupport
> | mq.broker.LockableServiceSupport  143 | 73 - activemq-osgi - 5.15.6 |
> broker, no longer able to keep the exclusive lock so giving up being a
> master
> 2019-02-27T16:51:05,909 | INFO  | KeepAlive Timer | BrokerService
> | he.activemq.broker.BrokerService  836 | 73 - activemq-osgi - 5.15.6 |
> Apache ActiveMQ 5.15.6 (broker, amq2) is shutting down
>
> It doesn't say why it could no longer keep the lock, but anyways, the
> bigger issue is that after stopping, it does not restart to retry acquiring
> the lock.
>
> Is there a configuration I'm missing? The code in LockableServiceSupport
> actually requests a restart, but where is that re-start done?
>
> Thanks for any insight
> - Martin
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Re: DB lease lock - broker should stop-start to retry acquiring the lock

Martin Lichtin
I'm running the broker in a Karaf container.

Had to resort to installing a shutdown hook, and when called, move the corresponding .cfg file away and then back into place (to trigger Felix fileinstall) so ActiveMQServiceFactory.updated() would get called. Very ugly, but the desired effect is that the broker gets restarted and will try to acquire the lease lock again.

- Martin

On 09.03.2019 03:05, Tim Bain wrote:

> It's my understanding that restarts are handled by the Tanuki service
> wrapper, when ActiveMQ is installed as a service, using the instructions at
> http://activemq.apache.org/java-service-wrapper.html. Is that how you're
> using ActiveMQ? Or are you running the broker directly from the executable
> in the bin directory, which wouldn't use the wrapper (and so wouldn't
> restart the process)?
>
> Tim
>
> On Thu, Feb 28, 2019, 1:57 PM Martin Lichtin <[hidden email]>
> wrote:
>
>> Hi
>>
>> Can you help me understand this situation?
>> I've ActiveMQ configured with a lease lock on a DB. Spontaneously it
>> decided it can no longer keep the lock, and starts shutting down:
>>
>> 2019-02-27T16:47:45,491 | INFO  | Starting Thread | BrokerService
>> | he.activemq.broker.BrokerService  773 | 73 - activemq-osgi - 5.15.6 |
>> Apache ActiveMQ 5.15.6 (broker, amq2) started
>>
>> 2019-02-27T16:51:05,907 | ERROR | KeepAlive Timer | LockableServiceSupport
>> | mq.broker.LockableServiceSupport  143 | 73 - activemq-osgi - 5.15.6 |
>> broker, no longer able to keep the exclusive lock so giving up being a
>> master
>> 2019-02-27T16:51:05,909 | INFO  | KeepAlive Timer | BrokerService
>> | he.activemq.broker.BrokerService  836 | 73 - activemq-osgi - 5.15.6 |
>> Apache ActiveMQ 5.15.6 (broker, amq2) is shutting down
>>
>> It doesn't say why it could no longer keep the lock, but anyways, the
>> bigger issue is that after stopping, it does not restart to retry acquiring
>> the lock.
>>
>> Is there a configuration I'm missing? The code in LockableServiceSupport
>> actually requests a restart, but where is that re-start done?
>>
>> Thanks for any insight
>> - Martin
>>
>>
Reply | Threaded
Open this post in threaded view
|

Re: Re: DB lease lock - broker should stop-start to retry acquiring the lock

Tim Bain
Martin,

Thanks for sharing your solution (unclean though it is) to help whoever
might hit this later, and the fact that you're running within Karaf. I
don't have any experience working with Karaf so I can't suggest anything
cleaner (though you'd be more likely to get suggestions from the Karaf
community, since this is specific to the Karaf environment).

ActiveMQ's philosophy is that the process exits when the broker encounters
a fatal error, but it's the responsibility of some wrapper process (e.g.
the Tanuki service wrapper referenced from the page I sent earlier) to
restart the JVM. I'm not sure if it's possible to leverage the Tanuki
wrapper within Karaf (I assume not, since you said you had to use this
workaround), but without something like that, the behavior you saw is the
expected behavior.

Tim

On Sun, Mar 10, 2019, 7:44 AM Martin Lichtin <[hidden email]>
wrote:

> I'm running the broker in a Karaf container.
>
> Had to resort to installing a shutdown hook, and when called, move the
> corresponding .cfg file away and then back into place (to trigger Felix
> fileinstall) so ActiveMQServiceFactory.updated() would get called. Very
> ugly, but the desired effect is that the broker gets restarted and will try
> to acquire the lease lock again.
>
> - Martin
>
> On 09.03.2019 03:05, Tim Bain wrote:
> > It's my understanding that restarts are handled by the Tanuki service
> > wrapper, when ActiveMQ is installed as a service, using the instructions
> at
> > http://activemq.apache.org/java-service-wrapper.html. Is that how you're
> > using ActiveMQ? Or are you running the broker directly from the
> executable
> > in the bin directory, which wouldn't use the wrapper (and so wouldn't
> > restart the process)?
> >
> > Tim
> >
> > On Thu, Feb 28, 2019, 1:57 PM Martin Lichtin <[hidden email]>
> > wrote:
> >
> >> Hi
> >>
> >> Can you help me understand this situation?
> >> I've ActiveMQ configured with a lease lock on a DB. Spontaneously it
> >> decided it can no longer keep the lock, and starts shutting down:
> >>
> >> 2019-02-27T16:47:45,491 | INFO  | Starting Thread | BrokerService
> >> | he.activemq.broker.BrokerService  773 | 73 - activemq-osgi - 5.15.6 |
> >> Apache ActiveMQ 5.15.6 (broker, amq2) started
> >>
> >> 2019-02-27T16:51:05,907 | ERROR | KeepAlive Timer |
> LockableServiceSupport
> >> | mq.broker.LockableServiceSupport  143 | 73 - activemq-osgi - 5.15.6 |
> >> broker, no longer able to keep the exclusive lock so giving up being a
> >> master
> >> 2019-02-27T16:51:05,909 | INFO  | KeepAlive Timer | BrokerService
> >> | he.activemq.broker.BrokerService  836 | 73 - activemq-osgi - 5.15.6 |
> >> Apache ActiveMQ 5.15.6 (broker, amq2) is shutting down
> >>
> >> It doesn't say why it could no longer keep the lock, but anyways, the
> >> bigger issue is that after stopping, it does not restart to retry
> acquiring
> >> the lock.
> >>
> >> Is there a configuration I'm missing? The code in LockableServiceSupport
> >> actually requests a restart, but where is that re-start done?
> >>
> >> Thanks for any insight
> >> - Martin
> >>
> >>
>