How to setup 3-node Artemis (2.5.0) HA cluster (1 master and 2 slaves) with replication, automatic failover/failback and static connectors

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

How to setup 3-node Artemis (2.5.0) HA cluster (1 master and 2 slaves) with replication, automatic failover/failback and static connectors

Jarek Przygódzki
Hi,
I'm trying to setup 3-node Artemis (2.5.0) HA cluster (1 master and 2
slaves) with replication, automatic failover/failback and static connectors.
It works fine with just one slave, but with 2 slaves strange thing happens
- failback doesn't work, when original master comes back online both servers
claim to be live
- sometimes one of the servers gets stuck - only "Server is stopped"
messages appear in the log

Each broker configuration is created with

  /opt/apache-artemis/bin/artemis create artemis \
    --home /opt/apache-artemis \
    --user artemis \
    --password simetraehcapa \
    --role amq \
    --require-login \
    --cluster-user artemisCluster \
    --cluster-password simetraehcaparetsulc

with some changes to setup cluster

*node artemis-1*

    <name>artemis-1</name>
    <connectors>
      <connector name="artemis-1">tcp://artemis-1:61616</connector>
      <connector name="artemis-2">tcp://artemis-2:61616</connector>
      <connector name="artemis-3">tcp://artemis-3:61616</connector>
    </connectors>

  <cluster-connections>
      <cluster-connection name="replication-cluster">
        <connector-ref>artemis-1</connector-ref>
        <retry-interval>1000</retry-interval>
        <retry-interval-multiplier>2</retry-interval-multiplier>
        <max-retry-interval>8000</max-retry-interval>
        <use-duplicate-detection>true</use-duplicate-detection>
        <message-load-balancing>OFF</message-load-balancing>
        <max-hops>1</max-hops>
        <static-connectors allow-direct-connections-only="true">
          <connector-ref>artemis-2</connector-ref>
          <connector-ref>artemis-3</connector-ref>
        </static-connectors>
      </cluster-connection>
    </cluster-connections>
    <ha-policy>
      <replication>
        <master>
          <check-for-live-server>true</check-for-live-server>
        </master>
      </replication>
    </ha-policy>
       
*node artemis-2*

  <connectors>
      <connector name="artemis-2">tcp://artemis-2:61616</connector>
      <connector name="artemis-1">tcp://artemis-1:61616</connector>
      <connector name="artemis-3">tcp://artemis-3:61616</connector>
    </connectors>
    <cluster-connections>
      <cluster-connection name="replication-cluster">
        <connector-ref>artemis-2</connector-ref>
        <retry-interval>1000</retry-interval>
        <retry-interval-multiplier>2</retry-interval-multiplier>
        <max-retry-interval>8000</max-retry-interval>
        <use-duplicate-detection>true</use-duplicate-detection>
        <message-load-balancing>OFF</message-load-balancing>
        <max-hops>1</max-hops>
        <static-connectors allow-direct-connections-only="true">
          <connector-ref>artemis-1</connector-ref>
          <connector-ref>artemis-3</connector-ref>
        </static-connectors>
      </cluster-connection>
    </cluster-connections>
    <ha-policy>
      <replication>
        <slave>
          <allow-failback>true</allow-failback>
        </slave>
      </replication>
    </ha-policy>


*node artemis-3*

    <name>artemis-3</name>
    <connectors>
      <connector name="artemis-3">tcp://artemis-3:61616</connector>
      <connector name="artemis-1">tcp://artemis-1:61616</connector>
      <connector name="artemis-2">tcp://artemis-2:61616</connector>
    </connectors>
    <cluster-connections>
      <cluster-connection name="replication-cluster">
        <connector-ref>artemis-3</connector-ref>
        <retry-interval>1000</retry-interval>
        <retry-interval-multiplier>2</retry-interval-multiplier>
        <max-retry-interval>8000</max-retry-interval>
        <use-duplicate-detection>true</use-duplicate-detection>
        <message-load-balancing>OFF</message-load-balancing>
        <max-hops>1</max-hops>
        <static-connectors allow-direct-connections-only="true">
          <connector-ref>artemis-1</connector-ref>
          <connector-ref>artemis-2</connector-ref>
        </static-connectors>
      </cluster-connection>
    </cluster-connections>
    <ha-policy>
      <replication>
        <slave>
          <allow-failback>true</allow-failback>
        </slave>
      </replication>
    </ha-policy>
       
I am happy to provide further information if required.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: How to setup 3-node Artemis (2.5.0) HA cluster (1 master and 2 slaves) with replication, automatic failover/failback and static connectors

jbertram
Fail-back only works for a live/backup pair.  It doesn't work for a
live/backup/backup triplet.

In your use-case there are 3 broker instances configured as live, backup,
backup.  However, a backup server is owned by only one live server.  This
means that when the 3 broker instances are started there will be 1
live-backup pair and a "left-over" backup which will be in a kind of idle
state waiting to attach to a live broker without a backup.  Then if the
live broker fails the live broker's backup will take over and become live
and the other backup will now become the backup of the server which just
became live.  Once the broker instance which failed is restarted it will
attempt to register itself as a backup of the now-live broker and initiate
fail-back.  However, since the now-live broker already has a backup it will
reject the registration message from the original live because it already
has a backup, and therefore fail-back will not occur.

To be clear, a live can only replicate to a *single* backup at a time. It's
not really clear to me what benefit multiple backups provide given the fact
that they prevent fail-back from working.


Justin

On Tue, Mar 20, 2018 at 8:36 AM, jarek.przygodzki <
[hidden email]> wrote:

> Hi,
> I'm trying to setup 3-node Artemis (2.5.0) HA cluster (1 master and 2
> slaves) with replication, automatic failover/failback and static
> connectors.
> It works fine with just one slave, but with 2 slaves strange thing happens
> - failback doesn't work, when original master comes back online both
> servers
> claim to be live
> - sometimes one of the servers gets stuck - only "Server is stopped"
> messages appear in the log
>
> Each broker configuration is created with
>
>   /opt/apache-artemis/bin/artemis create artemis \
>     --home /opt/apache-artemis \
>     --user artemis \
>     --password simetraehcapa \
>     --role amq \
>     --require-login \
>     --cluster-user artemisCluster \
>     --cluster-password simetraehcaparetsulc
>
> with some changes to setup cluster
>
> *node artemis-1*
>
>     <name>artemis-1</name>
>     <connectors>
>       <connector name="artemis-1">tcp://artemis-1:61616</connector>
>       <connector name="artemis-2">tcp://artemis-2:61616</connector>
>       <connector name="artemis-3">tcp://artemis-3:61616</connector>
>     </connectors>
>
>   <cluster-connections>
>       <cluster-connection name="replication-cluster">
>         <connector-ref>artemis-1</connector-ref>
>         <retry-interval>1000</retry-interval>
>         <retry-interval-multiplier>2</retry-interval-multiplier>
>         <max-retry-interval>8000</max-retry-interval>
>         <use-duplicate-detection>true</use-duplicate-detection>
>         <message-load-balancing>OFF</message-load-balancing>
>         <max-hops>1</max-hops>
>         <static-connectors allow-direct-connections-only="true">
>           <connector-ref>artemis-2</connector-ref>
>           <connector-ref>artemis-3</connector-ref>
>         </static-connectors>
>       </cluster-connection>
>     </cluster-connections>
>     <ha-policy>
>       <replication>
>         <master>
>           <check-for-live-server>true</check-for-live-server>
>         </master>
>       </replication>
>     </ha-policy>
>
> *node artemis-2*
>
>   <connectors>
>       <connector name="artemis-2">tcp://artemis-2:61616</connector>
>       <connector name="artemis-1">tcp://artemis-1:61616</connector>
>       <connector name="artemis-3">tcp://artemis-3:61616</connector>
>     </connectors>
>     <cluster-connections>
>       <cluster-connection name="replication-cluster">
>         <connector-ref>artemis-2</connector-ref>
>         <retry-interval>1000</retry-interval>
>         <retry-interval-multiplier>2</retry-interval-multiplier>
>         <max-retry-interval>8000</max-retry-interval>
>         <use-duplicate-detection>true</use-duplicate-detection>
>         <message-load-balancing>OFF</message-load-balancing>
>         <max-hops>1</max-hops>
>         <static-connectors allow-direct-connections-only="true">
>           <connector-ref>artemis-1</connector-ref>
>           <connector-ref>artemis-3</connector-ref>
>         </static-connectors>
>       </cluster-connection>
>     </cluster-connections>
>     <ha-policy>
>       <replication>
>         <slave>
>           <allow-failback>true</allow-failback>
>         </slave>
>       </replication>
>     </ha-policy>
>
>
> *node artemis-3*
>
>     <name>artemis-3</name>
>     <connectors>
>       <connector name="artemis-3">tcp://artemis-3:61616</connector>
>       <connector name="artemis-1">tcp://artemis-1:61616</connector>
>       <connector name="artemis-2">tcp://artemis-2:61616</connector>
>     </connectors>
>     <cluster-connections>
>       <cluster-connection name="replication-cluster">
>         <connector-ref>artemis-3</connector-ref>
>         <retry-interval>1000</retry-interval>
>         <retry-interval-multiplier>2</retry-interval-multiplier>
>         <max-retry-interval>8000</max-retry-interval>
>         <use-duplicate-detection>true</use-duplicate-detection>
>         <message-load-balancing>OFF</message-load-balancing>
>         <max-hops>1</max-hops>
>         <static-connectors allow-direct-connections-only="true">
>           <connector-ref>artemis-1</connector-ref>
>           <connector-ref>artemis-2</connector-ref>
>         </static-connectors>
>       </cluster-connection>
>     </cluster-connections>
>     <ha-policy>
>       <replication>
>         <slave>
>           <allow-failback>true</allow-failback>
>         </slave>
>       </replication>
>     </ha-policy>
>
> I am happy to provide further information if required.
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>
Reply | Threaded
Open this post in threaded view
|

Re: How to setup 3-node Artemis (2.5.0) HA cluster (1 master and 2 slaves) with replication, automatic failover/failback and static connectors

Jarek Przygódzki
Thank you for detailed explanation. The idea was to provide some sort of idle
spare backup to still have live/backup pair even after master node failure.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: How to setup 3-node Artemis (2.5.0) HA cluster (1 master and 2 slaves) with replication, automatic failover/failback and static connectors

jbertram
I guess you'll have to weigh whether or not having an "idle spare backup"
is worth preventing fail-back for your application/use-case.


Justin

On Tue, Mar 20, 2018 at 2:18 PM, jarek.przygodzki <
[hidden email]> wrote:

> Thank you for detailed explanation. The idea was to provide some sort of
> idle
> spare backup to still have live/backup pair even after master node failure.
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>
Reply | Threaded
Open this post in threaded view
|

Re: How to setup 3-node Artemis (2.5.0) HA cluster (1 master and 2 slaves) with replication, automatic failover/failback and static connectors

schalmers
In reply to this post by jbertram
Hi Justin,

Is it possible to have a 3 node cluster with no failover? i.e
live-live-live?



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: How to setup 3-node Artemis (2.5.0) HA cluster (1 master and 2 slaves) with replication, automatic failover/failback and static connectors

Justin Bertram
> Is it possible to have a 3 node cluster with no failover? i.e
live-live-live?

Yes, it is.


Justin


On Sun, Jun 10, 2018 at 10:37 PM, schalmers <
[hidden email]> wrote:

> Hi Justin,
>
> Is it possible to have a 3 node cluster with no failover? i.e
> live-live-live?
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>