Master-Slave being DB locked both

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Master-Slave being DB locked both

Jose Manuel Vega Monroy
Hi there,

With broker 5.15.0-1.el7.centos, looking like master and slave both brokers keeping DB locked with NFS remote share.

2020-05-06 15:08:52,372 [main] INFO  (AbstractApplicationContext.java:583) - Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@5025a98f: startup date [Wed May 06 15:08:52 BST 2020]; root of context hierarchy
2020-05-06 15:08:52,444 [main] INFO  (XmlBeanDefinitionReader.java:317) - Loading XML bean definitions from class path resource [activemq.xml]
2020-05-06 15:08:52,909 [main] INFO  (XmlBeanDefinitionReader.java:317) - Loading XML bean definitions from class path resource [jetty.xml]
2020-05-06 15:08:53,357 [main] INFO  (BrokerService.java:683) - Using Persistence Adapter: KahaDBPersistenceAdapter[/mnt/sbm]
2020-05-06 15:08:59,463 [main] INFO  (SharedFileLocker.java:70) - Database /mnt/sbm/lock is locked by another server. This broker is now in slave mode waiting a lock to be acquired

Any idea why this could be happening?

Having in /etc/fstab:

<NFS_SERVER>:/<NFS_SHARE_NAME>/mnt/sbmnfstcp,rsize=32768,wsize=32768,intr,noatime,actimeo=3,vers=300

And config:

  <persistenceAdapter>
            <kahaDB directory="/mnt/sbm"
                    enableIndexWriteAsync="true"
                    ignoreMissingJournalfiles="true"
                    checkForCorruptJournalFiles="true"
                    checksumJournalFiles="true"
                    preallocationStrategy="zeros"
            />
        </persistenceAdapter>

Thanks

Confidentiality: The contents of this e-mail and any attachments transmitted with it are intended to be confidential to the intended recipient; and may be privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. This e-mail is sent by a William Hill PLC group company. The William Hill group companies include, among others, William Hill PLC (registered number 4212563), William Hill Organization Limited (registered number 278208), William Hill US HoldCo Inc, WHG (International) Limited (registered number 99191) and Mr Green Limited (registered number C43260). Each of William Hill PLC and William Hill Organization Limited is registered in England and Wales and has its registered office at 1 Bedford Avenue, London, WC1B 3AU, UK. William Hill U.S. HoldCo, Inc. is registered in Delaware and has its registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle County DE 19801 Delaware, United States of America. WHG (International) Limited is registered in Gibraltar and has its registered office at 6/1 Waterport Place, Gibraltar. Mr Green Limited is registered in Malta and has its registered office at Tagliaferro Business Centre, Level 7, 14 High Street, Sliema SLM 1549, Malta. Unless specifically indicated otherwise, the contents of this e-mail are subject to contract; and are not an official statement, and do not necessarily represent the views, of William Hill PLC, its subsidiaries or affiliated companies. Please note that neither William Hill PLC, nor its subsidiaries and affiliated companies can accept any responsibility for any viruses contained within this e-mail and it is your responsibility to scan any emails and their attachments. William Hill PLC, its subsidiaries and affiliated companies may monitor e-mail traffic data and also the content of e-mails for effective operation of the e-mail system, or for security, purposes.
Reply | Threaded
Open this post in threaded view
|

Re: Master-Slave being DB locked both

jbonofre
Hi,

If it’s on NFS, what’s your rsize/wsize and sync mode enabled ?

Regards
JB

> Le 6 mai 2020 à 16:29, Jose Manuel Vega Monroy <[hidden email]> a écrit :
>
> Hi there,
>
> With broker 5.15.0-1.el7.centos, looking like master and slave both brokers keeping DB locked with NFS remote share.
>
> 2020-05-06 15:08:52,372 [main] INFO  (AbstractApplicationContext.java:583) - Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@5025a98f: startup date [Wed May 06 15:08:52 BST 2020]; root of context hierarchy
> 2020-05-06 15:08:52,444 [main] INFO  (XmlBeanDefinitionReader.java:317) - Loading XML bean definitions from class path resource [activemq.xml]
> 2020-05-06 15:08:52,909 [main] INFO  (XmlBeanDefinitionReader.java:317) - Loading XML bean definitions from class path resource [jetty.xml]
> 2020-05-06 15:08:53,357 [main] INFO  (BrokerService.java:683) - Using Persistence Adapter: KahaDBPersistenceAdapter[/mnt/sbm]
> 2020-05-06 15:08:59,463 [main] INFO  (SharedFileLocker.java:70) - Database /mnt/sbm/lock is locked by another server. This broker is now in slave mode waiting a lock to be acquired
>
> Any idea why this could be happening?
>
> Having in /etc/fstab:
>
> <NFS_SERVER>:/<NFS_SHARE_NAME>/mnt/sbmnfstcp,rsize=32768,wsize=32768,intr,noatime,actimeo=3,vers=300
>
> And config:
>
>  <persistenceAdapter>
>            <kahaDB directory="/mnt/sbm"
>                    enableIndexWriteAsync="true"
>                    ignoreMissingJournalfiles="true"
>                    checkForCorruptJournalFiles="true"
>                    checksumJournalFiles="true"
>                    preallocationStrategy="zeros"
>            />
>        </persistenceAdapter>
>
> Thanks
>
> Confidentiality: The contents of this e-mail and any attachments transmitted with it are intended to be confidential to the intended recipient; and may be privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. This e-mail is sent by a William Hill PLC group company. The William Hill group companies include, among others, William Hill PLC (registered number 4212563), William Hill Organization Limited (registered number 278208), William Hill US HoldCo Inc, WHG (International) Limited (registered number 99191) and Mr Green Limited (registered number C43260). Each of William Hill PLC and William Hill Organization Limited is registered in England and Wales and has its registered office at 1 Bedford Avenue, London, WC1B 3AU, UK. William Hill U.S. HoldCo, Inc. is registered in Delaware and has its registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle County DE 19801 Delaware, United States of America. WHG (International) Limited is registered in Gibraltar and has its registered office at 6/1 Waterport Place, Gibraltar. Mr Green Limited is registered in Malta and has its registered office at Tagliaferro Business Centre, Level 7, 14 High Street, Sliema SLM 1549, Malta. Unless specifically indicated otherwise, the contents of this e-mail are subject to contract; and are not an official statement, and do not necessarily represent the views, of William Hill PLC, its subsidiaries or affiliated companies. Please note that neither William Hill PLC, nor its subsidiaries and affiliated companies can accept any responsibility for any viruses contained within this e-mail and it is your responsibility to scan any emails and their attachments. William Hill PLC, its subsidiaries and affiliated companies may monitor e-mail traffic data and also the content of e-mails for effective operation of the e-mail system, or for security, purposes.

Reply | Threaded
Open this post in threaded view
|

Re: Master-Slave being DB locked both

Tim Bain
I'm not an NFS expert, but is vers=300 really correct? I don't think
there's an NFSv300.

Also, note that ActiveMQ offers the best support for shared-filesystem
master-slave on NFSv4, whereas NFSv3 has undesirable behavior (described in
https://activemq.apache.org/shared-file-system-master-slave) that might be
what you're describing. (Though your description sounds like it might be
occurring even after terminating all brokers, so maybe it's similar but
different.) Either way, if you're trying to run with NFSv3, you might want
to switch to v4 if that's an option.

Tim

On Wed, May 6, 2020, 8:35 AM Jean-Baptiste Onofre <[hidden email]> wrote:

> Hi,
>
> If it’s on NFS, what’s your rsize/wsize and sync mode enabled ?
>
> Regards
> JB
>
> > Le 6 mai 2020 à 16:29, Jose Manuel Vega Monroy <
> [hidden email]> a écrit :
> >
> > Hi there,
> >
> > With broker 5.15.0-1.el7.centos, looking like master and slave both
> brokers keeping DB locked with NFS remote share.
> >
> > 2020-05-06 15:08:52,372 [main] INFO
> (AbstractApplicationContext.java:583) - Refreshing
> org.apache.activemq.xbean.XBeanBrokerFactory$1@5025a98f: startup date
> [Wed May 06 15:08:52 BST 2020]; root of context hierarchy
> > 2020-05-06 15:08:52,444 [main] INFO  (XmlBeanDefinitionReader.java:317)
> - Loading XML bean definitions from class path resource [activemq.xml]
> > 2020-05-06 15:08:52,909 [main] INFO  (XmlBeanDefinitionReader.java:317)
> - Loading XML bean definitions from class path resource [jetty.xml]
> > 2020-05-06 15:08:53,357 [main] INFO  (BrokerService.java:683) - Using
> Persistence Adapter: KahaDBPersistenceAdapter[/mnt/sbm]
> > 2020-05-06 15:08:59,463 [main] INFO  (SharedFileLocker.java:70) -
> Database /mnt/sbm/lock is locked by another server. This broker is now in
> slave mode waiting a lock to be acquired
> >
> > Any idea why this could be happening?
> >
> > Having in /etc/fstab:
> >
> >
> <NFS_SERVER>:/<NFS_SHARE_NAME>/mnt/sbmnfstcp,rsize=32768,wsize=32768,intr,noatime,actimeo=3,vers=300
> >
> > And config:
> >
> >  <persistenceAdapter>
> >            <kahaDB directory="/mnt/sbm"
> >                    enableIndexWriteAsync="true"
> >                    ignoreMissingJournalfiles="true"
> >                    checkForCorruptJournalFiles="true"
> >                    checksumJournalFiles="true"
> >                    preallocationStrategy="zeros"
> >            />
> >        </persistenceAdapter>
> >
> > Thanks
> >
> > Confidentiality: The contents of this e-mail and any attachments
> transmitted with it are intended to be confidential to the intended
> recipient; and may be privileged or otherwise protected from disclosure. If
> you are not an intended recipient of this e-mail, do not duplicate or
> redistribute it by any means. Please delete it and any attachments and
> notify the sender that you have received it in error. This e-mail is sent
> by a William Hill PLC group company. The William Hill group companies
> include, among others, William Hill PLC (registered number 4212563),
> William Hill Organization Limited (registered number 278208), William Hill
> US HoldCo Inc, WHG (International) Limited (registered number 99191) and Mr
> Green Limited (registered number C43260). Each of William Hill PLC and
> William Hill Organization Limited is registered in England and Wales and
> has its registered office at 1 Bedford Avenue, London, WC1B 3AU, UK.
> William Hill U.S. HoldCo, Inc. is registered in Delaware and has its
> registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle
> County DE 19801 Delaware, United States of America. WHG (International)
> Limited is registered in Gibraltar and has its registered office at 6/1
> Waterport Place, Gibraltar. Mr Green Limited is registered in Malta and has
> its registered office at Tagliaferro Business Centre, Level 7, 14 High
> Street, Sliema SLM 1549, Malta. Unless specifically indicated otherwise,
> the contents of this e-mail are subject to contract; and are not an
> official statement, and do not necessarily represent the views, of William
> Hill PLC, its subsidiaries or affiliated companies. Please note that
> neither William Hill PLC, nor its subsidiaries and affiliated companies can
> accept any responsibility for any viruses contained within this e-mail and
> it is your responsibility to scan any emails and their attachments. William
> Hill PLC, its subsidiaries and affiliated companies may monitor e-mail
> traffic data and also the content of e-mails for effective operation of the
> e-mail system, or for security, purposes.
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: Master-Slave being DB locked both

Jose Manuel Vega Monroy
@Tim @Jean

Problem was in /etc/fstab with some errors (example, fstype 'nfs4' being deprecated) + probably app exiting wrongly.

After fixing, and rebooting some nodes, since blocked seeing the same lock, everything working fine.

Thanks

 <http://www.williamhill.com/>
 <http://www.whenthefunstops.co.uk/>
Jose Manuel Vega Monroy
Java Developer / Software Developer Engineer in Test
Direct: +0035 0 2008038 (Ext. 8038)
Email: [hidden email]
William Hill | 6/1 Waterport Place | Gibraltar | GX11 1AA




On 07/05/2020, 04:27, "Tim Bain" <[hidden email]> wrote:

    I'm not an NFS expert, but is vers=300 really correct? I don't think
    there's an NFSv300.

    Also, note that ActiveMQ offers the best support for shared-filesystem
    master-slave on NFSv4, whereas NFSv3 has undesirable behavior (described in
    https://urldefense.proofpoint.com/v2/url?u=https-3A__activemq.apache.org_shared-2Dfile-2Dsystem-2Dmaster-2Dslave&d=DwIFaQ&c=pWn2jKJ-j-AhxLuiRFe-Qw&r=i5Pk4pirVCmwsmddZqplM1jyQtVWeoOOb-vkuqku5P8&m=8tL802CIya_rK8mQBPHeKphxZsUXfdRmgC8Mfr3BJEU&s=1xyOO7nCJvrfhLxtmpyCa6ZhgeR3EmEvSPp-nnClI54&e= ) that might be
    what you're describing. (Though your description sounds like it might be
    occurring even after terminating all brokers, so maybe it's similar but
    different.) Either way, if you're trying to run with NFSv3, you might want
    to switch to v4 if that's an option.

    Tim

    On Wed, May 6, 2020, 8:35 AM Jean-Baptiste Onofre <[hidden email]> wrote:

    > Hi,
    >
    > If it’s on NFS, what’s your rsize/wsize and sync mode enabled ?
    >
    > Regards
    > JB
    >
    > > Le 6 mai 2020 à 16:29, Jose Manuel Vega Monroy <
    > [hidden email]> a écrit :
    > >
    > > Hi there,
    > >
    > > With broker 5.15.0-1.el7.centos, looking like master and slave both
    > brokers keeping DB locked with NFS remote share.
    > >
    > > 2020-05-06 15:08:52,372 [main] INFO
    > (AbstractApplicationContext.java:583) - Refreshing
    > org.apache.activemq.xbean.XBeanBrokerFactory$1@5025a98f: startup date
    > [Wed May 06 15:08:52 BST 2020]; root of context hierarchy
    > > 2020-05-06 15:08:52,444 [main] INFO  (XmlBeanDefinitionReader.java:317)
    > - Loading XML bean definitions from class path resource [activemq.xml]
    > > 2020-05-06 15:08:52,909 [main] INFO  (XmlBeanDefinitionReader.java:317)
    > - Loading XML bean definitions from class path resource [jetty.xml]
    > > 2020-05-06 15:08:53,357 [main] INFO  (BrokerService.java:683) - Using
    > Persistence Adapter: KahaDBPersistenceAdapter[/mnt/sbm]
    > > 2020-05-06 15:08:59,463 [main] INFO  (SharedFileLocker.java:70) -
    > Database /mnt/sbm/lock is locked by another server. This broker is now in
    > slave mode waiting a lock to be acquired
    > >
    > > Any idea why this could be happening?
    > >
    > > Having in /etc/fstab:
    > >
    > >
    > <NFS_SERVER>:/<NFS_SHARE_NAME>/mnt/sbmnfstcp,rsize=32768,wsize=32768,intr,noatime,actimeo=3,vers=300
    > >
    > > And config:
    > >
    > >  <persistenceAdapter>
    > >            <kahaDB directory="/mnt/sbm"
    > >                    enableIndexWriteAsync="true"
    > >                    ignoreMissingJournalfiles="true"
    > >                    checkForCorruptJournalFiles="true"
    > >                    checksumJournalFiles="true"
    > >                    preallocationStrategy="zeros"
    > >            />
    > >        </persistenceAdapter>
    > >
    > > Thanks
    > >
    > > Confidentiality: The contents of this e-mail and any attachments
    > transmitted with it are intended to be confidential to the intended
    > recipient; and may be privileged or otherwise protected from disclosure. If
    > you are not an intended recipient of this e-mail, do not duplicate or
    > redistribute it by any means. Please delete it and any attachments and
    > notify the sender that you have received it in error. This e-mail is sent
    > by a William Hill PLC group company. The William Hill group companies
    > include, among others, William Hill PLC (registered number 4212563),
    > William Hill Organization Limited (registered number 278208), William Hill
    > US HoldCo Inc, WHG (International) Limited (registered number 99191) and Mr
    > Green Limited (registered number C43260). Each of William Hill PLC and
    > William Hill Organization Limited is registered in England and Wales and
    > has its registered office at 1 Bedford Avenue, London, WC1B 3AU, UK.
    > William Hill U.S. HoldCo, Inc. is registered in Delaware and has its
    > registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle
    > County DE 19801 Delaware, United States of America. WHG (International)
    > Limited is registered in Gibraltar and has its registered office at 6/1
    > Waterport Place, Gibraltar. Mr Green Limited is registered in Malta and has
    > its registered office at Tagliaferro Business Centre, Level 7, 14 High
    > Street, Sliema SLM 1549, Malta. Unless specifically indicated otherwise,
    > the contents of this e-mail are subject to contract; and are not an
    > official statement, and do not necessarily represent the views, of William
    > Hill PLC, its subsidiaries or affiliated companies. Please note that
    > neither William Hill PLC, nor its subsidiaries and affiliated companies can
    > accept any responsibility for any viruses contained within this e-mail and
    > it is your responsibility to scan any emails and their attachments. William
    > Hill PLC, its subsidiaries and affiliated companies may monitor e-mail
    > traffic data and also the content of e-mails for effective operation of the
    > e-mail system, or for security, purposes.
    >
    >


Confidentiality: The contents of this e-mail and any attachments transmitted with it are intended to be confidential to the intended recipient; and may be privileged or otherwise protected from disclosure. If you are not an intended recipient of this e-mail, do not duplicate or redistribute it by any means. Please delete it and any attachments and notify the sender that you have received it in error. This e-mail is sent by a William Hill PLC group company. The William Hill group companies include, among others, William Hill PLC (registered number 4212563), William Hill Organization Limited (registered number 278208), William Hill US HoldCo Inc, WHG (International) Limited (registered number 99191) and Mr Green Limited (registered number C43260). Each of William Hill PLC and William Hill Organization Limited is registered in England and Wales and has its registered office at 1 Bedford Avenue, London, WC1B 3AU, UK. William Hill U.S. HoldCo, Inc. is registered in Delaware and has its registered office at 1007 N. Orange Street, 9 Floor, Wilmington, New Castle County DE 19801 Delaware, United States of America. WHG (International) Limited is registered in Gibraltar and has its registered office at 6/1 Waterport Place, Gibraltar. Mr Green Limited is registered in Malta and has its registered office at Tagliaferro Business Centre, Level 7, 14 High Street, Sliema SLM 1549, Malta. Unless specifically indicated otherwise, the contents of this e-mail are subject to contract; and are not an official statement, and do not necessarily represent the views, of William Hill PLC, its subsidiaries or affiliated companies. Please note that neither William Hill PLC, nor its subsidiaries and affiliated companies can accept any responsibility for any viruses contained within this e-mail and it is your responsibility to scan any emails and their attachments. William Hill PLC, its subsidiaries and affiliated companies may monitor e-mail traffic data and also the content of e-mails for effective operation of the e-mail system, or for security, purposes.