[DISCUSSION] New Quorum vote pluggable implementation

classic Classic list List threaded Threaded
29 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[DISCUSSION] New Quorum vote pluggable implementation

nigro_franz
Hi folks,

especially due to the requirements of the current Artemis quorum vote
algorithm, we've thought to re-implementing it with a different focus in
mind:
1) to make it pluggable (eg by using the many Raft implementations,
ZooKeeper or others)
2) to cleanly separate the election phase and cluster member states (ie it
should be the Topology shared between them)
3) to simplify most common setups in both amount of configuration and
requirements (eg "witness" nodes could be implemented to get a minimum 2*n
+1 quorum of nodes instead of forcing 2*n + 1 master-backup pairs)
4) [OPTIONALLY] to reduce/eliminate implicit "good practices" in term of
order of actions to be performed on nodes in "special states" eg proper
restart sequence after failover or similar cases
5) [OPTIONALLY] to make shared-store and replication behaviour more similar:
journal's presence should be the only difference between the 2s

A proposal of steps to be followed to get this:
1) abstract away the current quorum vote: it requires extra-care because the
logic is melted together with the replication/clustering behaviour
2) refactor it in order to separate election phase and cluster member states
3) implement a RI version of such APIs

Post-actions to help people adopt it, but need to be thought upfront:
1) a clean upgrade path for current HA replication users
2) deprecate or integrate the current HA replication into the new version

I've opened this here because many of the HA replication users are devs too
and given that this isn't yet implemented: we're stull in the
design/proposal phase, so anyone that want to express their
ideas/opinions/POC on this, is invited to talk here ;)

Cheers,
Franz





--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSSION] New Quorum vote pluggable implementation

Martyn Taylor-2
I think this is a great idea Franz.  The HA and replication components have
been a source of issues over the years.  Two problems I see are that 1)
there isn't a clean separation between the consensus mechanism and the
replication, and 2) the consensus algorithm used in Artemis isn't based on
any standard algorithm or a research paper.  Hence, all the issues that
were caught over the years due to various edge cases.  Integration with
ZooKeeper seems like the obvious solution (i.e. push the consensus logic
off to a third party lib).  I suspect though, this will be a considerable
amount of work and is likely to introduce new issues, so I'd proceed with
caution.

Cheers



On Mon, Mar 2, 2020 at 8:28 AM nigro_franz <[hidden email]> wrote:

> Hi folks,
>
> especially due to the requirements of the current Artemis quorum vote
> algorithm, we've thought to re-implementing it with a different focus in
> mind:
> 1) to make it pluggable (eg by using the many Raft implementations,
> ZooKeeper or others)
> 2) to cleanly separate the election phase and cluster member states (ie it
> should be the Topology shared between them)
> 3) to simplify most common setups in both amount of configuration and
> requirements (eg "witness" nodes could be implemented to get a minimum 2*n
> +1 quorum of nodes instead of forcing 2*n + 1 master-backup pairs)
> 4) [OPTIONALLY] to reduce/eliminate implicit "good practices" in term of
> order of actions to be performed on nodes in "special states" eg proper
> restart sequence after failover or similar cases
> 5) [OPTIONALLY] to make shared-store and replication behaviour more
> similar:
> journal's presence should be the only difference between the 2s
>
> A proposal of steps to be followed to get this:
> 1) abstract away the current quorum vote: it requires extra-care because
> the
> logic is melted together with the replication/clustering behaviour
> 2) refactor it in order to separate election phase and cluster member
> states
> 3) implement a RI version of such APIs
>
> Post-actions to help people adopt it, but need to be thought upfront:
> 1) a clean upgrade path for current HA replication users
> 2) deprecate or integrate the current HA replication into the new version
>
> I've opened this here because many of the HA replication users are devs too
> and given that this isn't yet implemented: we're stull in the
> design/proposal phase, so anyone that want to express their
> ideas/opinions/POC on this, is invited to talk here ;)
>
> Cheers,
> Franz
>
>
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html
>
Reply | Threaded
Open this post in threaded view
|

Re:Re: [DISCUSSION] New Quorum vote pluggable implementation

KimmKing-2
Can't agree any more.
The HA&Replication is more and more important for a modern messaging system.
Other apache opensource mq, kafka/rocketmq/pulsar maybe referred to.


And In my experience this also bring some extra-complexity about performance/brainsplitting issues when rebalance data.

At 2020-03-02 20:33:31, "Martyn Taylor" <[hidden email]> wrote:

>I think this is a great idea Franz.  The HA and replication components have
>been a source of issues over the years.  Two problems I see are that 1)
>there isn't a clean separation between the consensus mechanism and the
>replication, and 2) the consensus algorithm used in Artemis isn't based on
>any standard algorithm or a research paper.  Hence, all the issues that
>were caught over the years due to various edge cases.  Integration with
>ZooKeeper seems like the obvious solution (i.e. push the consensus logic
>off to a third party lib).  I suspect though, this will be a considerable
>amount of work and is likely to introduce new issues, so I'd proceed with
>caution.
>
>Cheers
>
>
>
>On Mon, Mar 2, 2020 at 8:28 AM nigro_franz <[hidden email]> wrote:
>
>> Hi folks,
>>
>> especially due to the requirements of the current Artemis quorum vote
>> algorithm, we've thought to re-implementing it with a different focus in
>> mind:
>> 1) to make it pluggable (eg by using the many Raft implementations,
>> ZooKeeper or others)
>> 2) to cleanly separate the election phase and cluster member states (ie it
>> should be the Topology shared between them)
>> 3) to simplify most common setups in both amount of configuration and
>> requirements (eg "witness" nodes could be implemented to get a minimum 2*n
>> +1 quorum of nodes instead of forcing 2*n + 1 master-backup pairs)
>> 4) [OPTIONALLY] to reduce/eliminate implicit "good practices" in term of
>> order of actions to be performed on nodes in "special states" eg proper
>> restart sequence after failover or similar cases
>> 5) [OPTIONALLY] to make shared-store and replication behaviour more
>> similar:
>> journal's presence should be the only difference between the 2s
>>
>> A proposal of steps to be followed to get this:
>> 1) abstract away the current quorum vote: it requires extra-care because
>> the
>> logic is melted together with the replication/clustering behaviour
>> 2) refactor it in order to separate election phase and cluster member
>> states
>> 3) implement a RI version of such APIs
>>
>> Post-actions to help people adopt it, but need to be thought upfront:
>> 1) a clean upgrade path for current HA replication users
>> 2) deprecate or integrate the current HA replication into the new version
>>
>> I've opened this here because many of the HA replication users are devs too
>> and given that this isn't yet implemented: we're stull in the
>> design/proposal phase, so anyone that want to express their
>> ideas/opinions/POC on this, is invited to talk here ;)
>>
>> Cheers,
>> Franz
>>
>>
>>
>>
>>
>> --
>> Sent from:
>> http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html
>>
Reply | Threaded
Open this post in threaded view
|

Re: Re: [DISCUSSION] New Quorum vote pluggable implementation

andytaylor
Personally I wouldn't use Zookeeper, I think there are better options. Also
looks like Kafka are replacing it as well. Saying that, it doesn't really
matter what is used, the main thing is we need to remove the burden of
providing consensus away from the broker.

It would make sense to make it pluggable.

A

On Mon, 2 Mar 2020 at 19:19, KimmKing <[hidden email]> wrote:

> Can't agree any more.
> The HA&Replication is more and more important for a modern messaging
> system.
> Other apache opensource mq, kafka/rocketmq/pulsar maybe referred to.
>
>
> And In my experience this also bring some extra-complexity about
> performance/brainsplitting issues when rebalance data.
>
> At 2020-03-02 20:33:31, "Martyn Taylor" <[hidden email]> wrote:
> >I think this is a great idea Franz.  The HA and replication components
> have
> >been a source of issues over the years.  Two problems I see are that 1)
> >there isn't a clean separation between the consensus mechanism and the
> >replication, and 2) the consensus algorithm used in Artemis isn't based on
> >any standard algorithm or a research paper.  Hence, all the issues that
> >were caught over the years due to various edge cases.  Integration with
> >ZooKeeper seems like the obvious solution (i.e. push the consensus logic
> >off to a third party lib).  I suspect though, this will be a considerable
> >amount of work and is likely to introduce new issues, so I'd proceed with
> >caution.
> >
> >Cheers
> >
> >
> >
> >On Mon, Mar 2, 2020 at 8:28 AM nigro_franz <[hidden email]> wrote:
> >
> >> Hi folks,
> >>
> >> especially due to the requirements of the current Artemis quorum vote
> >> algorithm, we've thought to re-implementing it with a different focus in
> >> mind:
> >> 1) to make it pluggable (eg by using the many Raft implementations,
> >> ZooKeeper or others)
> >> 2) to cleanly separate the election phase and cluster member states (ie
> it
> >> should be the Topology shared between them)
> >> 3) to simplify most common setups in both amount of configuration and
> >> requirements (eg "witness" nodes could be implemented to get a minimum
> 2*n
> >> +1 quorum of nodes instead of forcing 2*n + 1 master-backup pairs)
> >> 4) [OPTIONALLY] to reduce/eliminate implicit "good practices" in term of
> >> order of actions to be performed on nodes in "special states" eg proper
> >> restart sequence after failover or similar cases
> >> 5) [OPTIONALLY] to make shared-store and replication behaviour more
> >> similar:
> >> journal's presence should be the only difference between the 2s
> >>
> >> A proposal of steps to be followed to get this:
> >> 1) abstract away the current quorum vote: it requires extra-care because
> >> the
> >> logic is melted together with the replication/clustering behaviour
> >> 2) refactor it in order to separate election phase and cluster member
> >> states
> >> 3) implement a RI version of such APIs
> >>
> >> Post-actions to help people adopt it, but need to be thought upfront:
> >> 1) a clean upgrade path for current HA replication users
> >> 2) deprecate or integrate the current HA replication into the new
> version
> >>
> >> I've opened this here because many of the HA replication users are devs
> too
> >> and given that this isn't yet implemented: we're stull in the
> >> design/proposal phase, so anyone that want to express their
> >> ideas/opinions/POC on this, is invited to talk here ;)
> >>
> >> Cheers,
> >> Franz
> >>
> >>
> >>
> >>
> >>
> >> --
> >> Sent from:
> >> http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html
> >>
>
Reply | Threaded
Open this post in threaded view
|

Re: Re: [DISCUSSION] New Quorum vote pluggable implementation

clebertsuconic
I think we should have a management component, that runs outside of
the broker and would manage quorum.

That way you can have the quorum running outside of the broker itself.
which would improve the need of multiple brokers to manage the quorum.
You just need Quorum Managers in distinct places.

I had recently worked with a software called Ceph... And Ceph has the
concept of managers working away from their "broker" (it's not a
broker.. it's a DB, but in a sense it's the same concept here). I
think we should do the same.

I have talked to Franz and other guys in private.. and it seems
everybody these days mention raft consensus algorithm.  Perhaps we
should look into it.. and make it pluggable. I believe there's JRaft
working on top of JGroups already.

If we make this pluggable and make it a manager a separate process,
this will be a big win IMO.

On Tue, Mar 3, 2020 at 9:55 AM Andy Taylor <[hidden email]> wrote:

>
> Personally I wouldn't use Zookeeper, I think there are better options. Also
> looks like Kafka are replacing it as well. Saying that, it doesn't really
> matter what is used, the main thing is we need to remove the burden of
> providing consensus away from the broker.
>
> It would make sense to make it pluggable.
>
> A
>
> On Mon, 2 Mar 2020 at 19:19, KimmKing <[hidden email]> wrote:
>
> > Can't agree any more.
> > The HA&Replication is more and more important for a modern messaging
> > system.
> > Other apache opensource mq, kafka/rocketmq/pulsar maybe referred to.
> >
> >
> > And In my experience this also bring some extra-complexity about
> > performance/brainsplitting issues when rebalance data.
> >
> > At 2020-03-02 20:33:31, "Martyn Taylor" <[hidden email]> wrote:
> > >I think this is a great idea Franz.  The HA and replication components
> > have
> > >been a source of issues over the years.  Two problems I see are that 1)
> > >there isn't a clean separation between the consensus mechanism and the
> > >replication, and 2) the consensus algorithm used in Artemis isn't based on
> > >any standard algorithm or a research paper.  Hence, all the issues that
> > >were caught over the years due to various edge cases.  Integration with
> > >ZooKeeper seems like the obvious solution (i.e. push the consensus logic
> > >off to a third party lib).  I suspect though, this will be a considerable
> > >amount of work and is likely to introduce new issues, so I'd proceed with
> > >caution.
> > >
> > >Cheers
> > >
> > >
> > >
> > >On Mon, Mar 2, 2020 at 8:28 AM nigro_franz <[hidden email]> wrote:
> > >
> > >> Hi folks,
> > >>
> > >> especially due to the requirements of the current Artemis quorum vote
> > >> algorithm, we've thought to re-implementing it with a different focus in
> > >> mind:
> > >> 1) to make it pluggable (eg by using the many Raft implementations,
> > >> ZooKeeper or others)
> > >> 2) to cleanly separate the election phase and cluster member states (ie
> > it
> > >> should be the Topology shared between them)
> > >> 3) to simplify most common setups in both amount of configuration and
> > >> requirements (eg "witness" nodes could be implemented to get a minimum
> > 2*n
> > >> +1 quorum of nodes instead of forcing 2*n + 1 master-backup pairs)
> > >> 4) [OPTIONALLY] to reduce/eliminate implicit "good practices" in term of
> > >> order of actions to be performed on nodes in "special states" eg proper
> > >> restart sequence after failover or similar cases
> > >> 5) [OPTIONALLY] to make shared-store and replication behaviour more
> > >> similar:
> > >> journal's presence should be the only difference between the 2s
> > >>
> > >> A proposal of steps to be followed to get this:
> > >> 1) abstract away the current quorum vote: it requires extra-care because
> > >> the
> > >> logic is melted together with the replication/clustering behaviour
> > >> 2) refactor it in order to separate election phase and cluster member
> > >> states
> > >> 3) implement a RI version of such APIs
> > >>
> > >> Post-actions to help people adopt it, but need to be thought upfront:
> > >> 1) a clean upgrade path for current HA replication users
> > >> 2) deprecate or integrate the current HA replication into the new
> > version
> > >>
> > >> I've opened this here because many of the HA replication users are devs
> > too
> > >> and given that this isn't yet implemented: we're stull in the
> > >> design/proposal phase, so anyone that want to express their
> > >> ideas/opinions/POC on this, is invited to talk here ;)
> > >>
> > >> Cheers,
> > >> Franz
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> --
> > >> Sent from:
> > >> http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html
> > >>
> >



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: Re: [DISCUSSION] New Quorum vote pluggable implementation

Atri Sharma
Happy to contribute to this effort.

On Mon, 9 Mar 2020 at 20:25, Clebert Suconic <[hidden email]>
wrote:

> I think we should have a management component, that runs outside of
> the broker and would manage quorum.
>
> That way you can have the quorum running outside of the broker itself.
> which would improve the need of multiple brokers to manage the quorum.
> You just need Quorum Managers in distinct places.
>
> I had recently worked with a software called Ceph... And Ceph has the
> concept of managers working away from their "broker" (it's not a
> broker.. it's a DB, but in a sense it's the same concept here). I
> think we should do the same.
>
> I have talked to Franz and other guys in private.. and it seems
> everybody these days mention raft consensus algorithm.  Perhaps we
> should look into it.. and make it pluggable. I believe there's JRaft
> working on top of JGroups already.
>
> If we make this pluggable and make it a manager a separate process,
> this will be a big win IMO.
>
> On Tue, Mar 3, 2020 at 9:55 AM Andy Taylor <[hidden email]> wrote:
> >
> > Personally I wouldn't use Zookeeper, I think there are better options.
> Also
> > looks like Kafka are replacing it as well. Saying that, it doesn't really
> > matter what is used, the main thing is we need to remove the burden of
> > providing consensus away from the broker.
> >
> > It would make sense to make it pluggable.
> >
> > A
> >
> > On Mon, 2 Mar 2020 at 19:19, KimmKing <[hidden email]> wrote:
> >
> > > Can't agree any more.
> > > The HA&Replication is more and more important for a modern messaging
> > > system.
> > > Other apache opensource mq, kafka/rocketmq/pulsar maybe referred to.
> > >
> > >
> > > And In my experience this also bring some extra-complexity about
> > > performance/brainsplitting issues when rebalance data.
> > >
> > > At 2020-03-02 20:33:31, "Martyn Taylor" <[hidden email]> wrote:
> > > >I think this is a great idea Franz.  The HA and replication components
> > > have
> > > >been a source of issues over the years.  Two problems I see are that
> 1)
> > > >there isn't a clean separation between the consensus mechanism and the
> > > >replication, and 2) the consensus algorithm used in Artemis isn't
> based on
> > > >any standard algorithm or a research paper.  Hence, all the issues
> that
> > > >were caught over the years due to various edge cases.  Integration
> with
> > > >ZooKeeper seems like the obvious solution (i.e. push the consensus
> logic
> > > >off to a third party lib).  I suspect though, this will be a
> considerable
> > > >amount of work and is likely to introduce new issues, so I'd proceed
> with
> > > >caution.
> > > >
> > > >Cheers
> > > >
> > > >
> > > >
> > > >On Mon, Mar 2, 2020 at 8:28 AM nigro_franz <[hidden email]>
> wrote:
> > > >
> > > >> Hi folks,
> > > >>
> > > >> especially due to the requirements of the current Artemis quorum
> vote
> > > >> algorithm, we've thought to re-implementing it with a different
> focus in
> > > >> mind:
> > > >> 1) to make it pluggable (eg by using the many Raft implementations,
> > > >> ZooKeeper or others)
> > > >> 2) to cleanly separate the election phase and cluster member states
> (ie
> > > it
> > > >> should be the Topology shared between them)
> > > >> 3) to simplify most common setups in both amount of configuration
> and
> > > >> requirements (eg "witness" nodes could be implemented to get a
> minimum
> > > 2*n
> > > >> +1 quorum of nodes instead of forcing 2*n + 1 master-backup pairs)
> > > >> 4) [OPTIONALLY] to reduce/eliminate implicit "good practices" in
> term of
> > > >> order of actions to be performed on nodes in "special states" eg
> proper
> > > >> restart sequence after failover or similar cases
> > > >> 5) [OPTIONALLY] to make shared-store and replication behaviour more
> > > >> similar:
> > > >> journal's presence should be the only difference between the 2s
> > > >>
> > > >> A proposal of steps to be followed to get this:
> > > >> 1) abstract away the current quorum vote: it requires extra-care
> because
> > > >> the
> > > >> logic is melted together with the replication/clustering behaviour
> > > >> 2) refactor it in order to separate election phase and cluster
> member
> > > >> states
> > > >> 3) implement a RI version of such APIs
> > > >>
> > > >> Post-actions to help people adopt it, but need to be thought
> upfront:
> > > >> 1) a clean upgrade path for current HA replication users
> > > >> 2) deprecate or integrate the current HA replication into the new
> > > version
> > > >>
> > > >> I've opened this here because many of the HA replication users are
> devs
> > > too
> > > >> and given that this isn't yet implemented: we're stull in the
> > > >> design/proposal phase, so anyone that want to express their
> > > >> ideas/opinions/POC on this, is invited to talk here ;)
> > > >>
> > > >> Cheers,
> > > >> Franz
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Sent from:
> > > >> http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html
> > > >>
> > >
>
>
>
> --
> Clebert Suconic
>
--
Regards,

Atri
Apache Concerted
Reply | Threaded
Open this post in threaded view
|

Re: Re: [DISCUSSION] New Quorum vote pluggable implementation

jdanek
In reply to this post by clebertsuconic
On Tue, Mar 3, 2020 at 4:02 PM Andy Taylor <[hidden email]> wrote:

> Personally I wouldn't use Zookeeper, I think there are better options. Also
> looks like Kafka are replacing it as well. Saying that, it doesn't really
> matter what is used, the main thing is we need to remove the burden of
> providing consensus away from the broker.


https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum

There is a blog from person who prototyped replacing Zookeeper with
Atomix.io.
https://medium.com/@lukasz.antoniak/apache-kafka-leaves-the-zoo-bef529ba82b7.
It is afaik independent work to the KIP-500, as the proposal does not say
Atomix.io is to be used. The blog links to other reimplementations/forks;
one that replaces Zookeeper with etcd, and another one that does something
with Go and Raft.

Rabbitmq has implemented Raft in Erlang and they now have "mirrored queues"
(legacy, the pre-raft version) and Quorum queues, the new thing (
https://www.rabbitmq.com/quorum-queues.html) There is a presentation that
tries to explain it to people who don't know RabbitMQ
https://www.slideshare.net/Pivotal/implementing-raft-in-rabbitmq. For some
reason, they wanted to vote up a leader for each queue separately, and then
they had to make it perform well.

On Mon, Mar 9, 2020 at 4:02 PM Clebert Suconic <[hidden email]>
wrote:

> I think we should have a management component, that runs outside of
> the broker and would manage quorum.
>
> That way you can have the quorum running outside of the broker itself.
> which would improve the need of multiple brokers to manage the quorum.
> You just need Quorum Managers in distinct places.
>
> I had recently worked with a software called Ceph... And Ceph has the
> concept of managers working away from their "broker" (it's not a
> broker.. it's a DB, but in a sense it's the same concept here). I
> think we should do the same.
>

One of the reasons for KIP-500 is simplification of deployment. From this
point of view, taking inspiration for ActiveMQ from Ceph, which is
notoriously hard to install, at least in folk memory, could be a step back.
Also see section Rejected Alternatives -> Pluggable Consensus in the KIP.

Btw, Kafka has controller brokers and ordinary brokers. Sounds a bit like
Ceph manager.... Also, a Kafka broker is a DB of sorts (of events).
--
Mit freundlichen Grüßen / Kind regards
Jiri Daněk
Reply | Threaded
Open this post in threaded view
|

Re: Re: [DISCUSSION] New Quorum vote pluggable implementation

clebertsuconic
Ceph is difficult to install I agree.  But it has nothing to do with the
notion of a thing manager instead of a broker.


All you need is an option on CLI such as ./artemis node-manager



You are replacing a broker by s manager.  Nothing different.

On Tue, Mar 10, 2020 at 4:28 AM Jiri Daněk <[hidden email]> wrote:

> On Tue, Mar 3, 2020 at 4:02 PM Andy Taylor <[hidden email]> wrote:
>
> > Personally I wouldn't use Zookeeper, I think there are better options.
> Also
> > looks like Kafka are replacing it as well. Saying that, it doesn't really
> > matter what is used, the main thing is we need to remove the burden of
> > providing consensus away from the broker.
>
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
>
> There is a blog from person who prototyped replacing Zookeeper with
> Atomix.io.
>
> https://medium.com/@lukasz.antoniak/apache-kafka-leaves-the-zoo-bef529ba82b7
> .
> It is afaik independent work to the KIP-500, as the proposal does not say
> Atomix.io is to be used. The blog links to other reimplementations/forks;
> one that replaces Zookeeper with etcd, and another one that does something
> with Go and Raft.
>
> Rabbitmq has implemented Raft in Erlang and they now have "mirrored queues"
> (legacy, the pre-raft version) and Quorum queues, the new thing (
> https://www.rabbitmq.com/quorum-queues.html) There is a presentation that
> tries to explain it to people who don't know RabbitMQ
> https://www.slideshare.net/Pivotal/implementing-raft-in-rabbitmq. For some
> reason, they wanted to vote up a leader for each queue separately, and then
> they had to make it perform well.
>
> On Mon, Mar 9, 2020 at 4:02 PM Clebert Suconic <[hidden email]>
> wrote:
>
> > I think we should have a management component, that runs outside of
> > the broker and would manage quorum.
> >
> > That way you can have the quorum running outside of the broker itself.
> > which would improve the need of multiple brokers to manage the quorum.
> > You just need Quorum Managers in distinct places.
> >
> > I had recently worked with a software called Ceph... And Ceph has the
> > concept of managers working away from their "broker" (it's not a
> > broker.. it's a DB, but in a sense it's the same concept here). I
> > think we should do the same.
> >
>
> One of the reasons for KIP-500 is simplification of deployment. From this
> point of view, taking inspiration for ActiveMQ from Ceph, which is
> notoriously hard to install, at least in folk memory, could be a step back.
> Also see section Rejected Alternatives -> Pluggable Consensus in the KIP.
>
> Btw, Kafka has controller brokers and ordinary brokers. Sounds a bit like
> Ceph manager.... Also, a Kafka broker is a DB of sorts (of events).
> --
> Mit freundlichen Grüßen / Kind regards
> Jiri Daněk
>
--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: Re: [DISCUSSION] New Quorum vote pluggable implementation

clebertsuconic
It will probably be a good idea to create sub-tasks to a JIRA... and
have this being developed as part of a different branch before we
merge into master.


What do you think?


That way we can have more people participating on discussions /
contributing to the code in parallel.

On Tue, Mar 10, 2020 at 8:32 AM Clebert Suconic
<[hidden email]> wrote:

>
> Ceph is difficult to install I agree.  But it has nothing to do with the notion of a thing manager instead of a broker.
>
>
> All you need is an option on CLI such as ./artemis node-manager
>
>
>
> You are replacing a broker by s manager.  Nothing different.
>
> On Tue, Mar 10, 2020 at 4:28 AM Jiri Daněk <[hidden email]> wrote:
>>
>> On Tue, Mar 3, 2020 at 4:02 PM Andy Taylor <[hidden email]> wrote:
>>
>> > Personally I wouldn't use Zookeeper, I think there are better options. Also
>> > looks like Kafka are replacing it as well. Saying that, it doesn't really
>> > matter what is used, the main thing is we need to remove the burden of
>> > providing consensus away from the broker.
>>
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
>>
>> There is a blog from person who prototyped replacing Zookeeper with
>> Atomix.io.
>> https://medium.com/@lukasz.antoniak/apache-kafka-leaves-the-zoo-bef529ba82b7.
>> It is afaik independent work to the KIP-500, as the proposal does not say
>> Atomix.io is to be used. The blog links to other reimplementations/forks;
>> one that replaces Zookeeper with etcd, and another one that does something
>> with Go and Raft.
>>
>> Rabbitmq has implemented Raft in Erlang and they now have "mirrored queues"
>> (legacy, the pre-raft version) and Quorum queues, the new thing (
>> https://www.rabbitmq.com/quorum-queues.html) There is a presentation that
>> tries to explain it to people who don't know RabbitMQ
>> https://www.slideshare.net/Pivotal/implementing-raft-in-rabbitmq. For some
>> reason, they wanted to vote up a leader for each queue separately, and then
>> they had to make it perform well.
>>
>> On Mon, Mar 9, 2020 at 4:02 PM Clebert Suconic <[hidden email]>
>> wrote:
>>
>> > I think we should have a management component, that runs outside of
>> > the broker and would manage quorum.
>> >
>> > That way you can have the quorum running outside of the broker itself.
>> > which would improve the need of multiple brokers to manage the quorum.
>> > You just need Quorum Managers in distinct places.
>> >
>> > I had recently worked with a software called Ceph... And Ceph has the
>> > concept of managers working away from their "broker" (it's not a
>> > broker.. it's a DB, but in a sense it's the same concept here). I
>> > think we should do the same.
>> >
>>
>> One of the reasons for KIP-500 is simplification of deployment. From this
>> point of view, taking inspiration for ActiveMQ from Ceph, which is
>> notoriously hard to install, at least in folk memory, could be a step back.
>> Also see section Rejected Alternatives -> Pluggable Consensus in the KIP.
>>
>> Btw, Kafka has controller brokers and ordinary brokers. Sounds a bit like
>> Ceph manager.... Also, a Kafka broker is a DB of sorts (of events).
>> --
>> Mit freundlichen Grüßen / Kind regards
>> Jiri Daněk
>
> --
> Clebert Suconic



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: Re: [DISCUSSION] New Quorum vote pluggable implementation

Atri Sharma
+1

On Tue, 10 Mar 2020 at 20:24, Clebert Suconic <[hidden email]>
wrote:

> It will probably be a good idea to create sub-tasks to a JIRA... and
> have this being developed as part of a different branch before we
> merge into master.
>
>
> What do you think?
>
>
> That way we can have more people participating on discussions /
> contributing to the code in parallel.
>
> On Tue, Mar 10, 2020 at 8:32 AM Clebert Suconic
> <[hidden email]> wrote:
> >
> > Ceph is difficult to install I agree.  But it has nothing to do with the
> notion of a thing manager instead of a broker.
> >
> >
> > All you need is an option on CLI such as ./artemis node-manager
> >
> >
> >
> > You are replacing a broker by s manager.  Nothing different.
> >
> > On Tue, Mar 10, 2020 at 4:28 AM Jiri Daněk <[hidden email]> wrote:
> >>
> >> On Tue, Mar 3, 2020 at 4:02 PM Andy Taylor <[hidden email]>
> wrote:
> >>
> >> > Personally I wouldn't use Zookeeper, I think there are better
> options. Also
> >> > looks like Kafka are replacing it as well. Saying that, it doesn't
> really
> >> > matter what is used, the main thing is we need to remove the burden of
> >> > providing consensus away from the broker.
> >>
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
> >>
> >> There is a blog from person who prototyped replacing Zookeeper with
> >> Atomix.io.
> >>
> https://medium.com/@lukasz.antoniak/apache-kafka-leaves-the-zoo-bef529ba82b7
> .
> >> It is afaik independent work to the KIP-500, as the proposal does not
> say
> >> Atomix.io is to be used. The blog links to other
> reimplementations/forks;
> >> one that replaces Zookeeper with etcd, and another one that does
> something
> >> with Go and Raft.
> >>
> >> Rabbitmq has implemented Raft in Erlang and they now have "mirrored
> queues"
> >> (legacy, the pre-raft version) and Quorum queues, the new thing (
> >> https://www.rabbitmq.com/quorum-queues.html) There is a presentation
> that
> >> tries to explain it to people who don't know RabbitMQ
> >> https://www.slideshare.net/Pivotal/implementing-raft-in-rabbitmq. For
> some
> >> reason, they wanted to vote up a leader for each queue separately, and
> then
> >> they had to make it perform well.
> >>
> >> On Mon, Mar 9, 2020 at 4:02 PM Clebert Suconic <
> [hidden email]>
> >> wrote:
> >>
> >> > I think we should have a management component, that runs outside of
> >> > the broker and would manage quorum.
> >> >
> >> > That way you can have the quorum running outside of the broker itself.
> >> > which would improve the need of multiple brokers to manage the quorum.
> >> > You just need Quorum Managers in distinct places.
> >> >
> >> > I had recently worked with a software called Ceph... And Ceph has the
> >> > concept of managers working away from their "broker" (it's not a
> >> > broker.. it's a DB, but in a sense it's the same concept here). I
> >> > think we should do the same.
> >> >
> >>
> >> One of the reasons for KIP-500 is simplification of deployment. From
> this
> >> point of view, taking inspiration for ActiveMQ from Ceph, which is
> >> notoriously hard to install, at least in folk memory, could be a step
> back.
> >> Also see section Rejected Alternatives -> Pluggable Consensus in the
> KIP.
> >>
> >> Btw, Kafka has controller brokers and ordinary brokers. Sounds a bit
> like
> >> Ceph manager.... Also, a Kafka broker is a DB of sorts (of events).
> >> --
> >> Mit freundlichen Grüßen / Kind regards
> >> Jiri Daněk
> >
> > --
> > Clebert Suconic
>
>
>
> --
> Clebert Suconic
>
--
Regards,

Atri
Apache Concerted
Reply | Threaded
Open this post in threaded view
|

Re: Re: [DISCUSSION] New Quorum vote pluggable implementation

michael.andre.pearce
In reply to this post by clebertsuconic
I think a main thing here is a pluggable api.If someone already has etcd, zookeeper, consul etc. In their infrastructure they should be able to reuse.Like wise if you have multiple bulkheaded broker sets it be useful to be able to reuse the same quorum compontent e.g. in ZK that would be a different base root on the tree path per broker set.Likewise that said we should have some out the box support plugins from the community. Obviously what ever we do whilst it maybe a break change e.g. a 3.0 release there must be a migration solution (accepted that it may need outage) for existing brokers that works via the pluggable api not to a specific solution.Also anything from a management api pov should also be visible in the web management console. Sent from my Samsung Galaxy smartphone.
-------- Original message --------From: Clebert Suconic <[hidden email]> Date: 09/03/2020  14:55  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation I think we should have a management component, that runs outside ofthe broker and would manage quorum.That way you can have the quorum running outside of the broker itself.which would improve the need of multiple brokers to manage the quorum.You just need Quorum Managers in distinct places.I had recently worked with a software called Ceph... And Ceph has theconcept of managers working away from their "broker" (it's not abroker.. it's a DB, but in a sense it's the same concept here). Ithink we should do the same.I have talked to Franz and other guys in private.. and it seemseverybody these days mention raft consensus algorithm.  Perhaps weshould look into it.. and make it pluggable. I believe there's JRaftworking on top of JGroups already.If we make this pluggable and make it a manager a separate process,this will be a big win IMO.On Tue, Mar 3, 2020 at 9:55 AM Andy Taylor <[hidden email]> wrote:>> Personally I wouldn't use Zookeeper, I think there are better options. Also> looks like Kafka are replacing it as well. Saying that, it doesn't really> matter what is used, the main thing is we need to remove the burden of> providing consensus away from the broker.>> It would make sense to make it pluggable.>> A>> On Mon, 2 Mar 2020 at 19:19, KimmKing <[hidden email]> wrote:>> > Can't agree any more.> > The HA&Replication is more and more important for a modern messaging> > system.> > Other apache opensource mq, kafka/rocketmq/pulsar maybe referred to.> >> >> > And In my experience this also bring some extra-complexity about> > performance/brainsplitting issues when rebalance data.> >> > At 2020-03-02 20:33:31, "Martyn Taylor" <[hidden email]> wrote:> > >I think this is a great idea Franz.  The HA and replication components> > have> > >been a source of issues over the years.  Two problems I see are that 1)> > >there isn't a clean separation between the consensus mechanism and the> > >replication, and 2) the consensus algorithm used in Artemis isn't based on> > >any standard algorithm or a research paper.  Hence, all the issues that> > >were caught over the years due to various edge cases.  Integration with> > >ZooKeeper seems like the obvious solution (i.e. push the consensus logic> > >off to a third party lib).  I suspect though, this will be a considerable> > >amount of work and is likely to introduce new issues, so I'd proceed with> > >caution.> > >> > >Cheers> > >> > >> > >> > >On Mon, Mar 2, 2020 at 8:28 AM nigro_franz <[hidden email]> wrote:> > >> > >> Hi folks,> > >>> > >> especially due to the requirements of the current Artemis quorum vote> > >> algorithm, we've thought to re-implementing it with a different focus in> > >> mind:> > >> 1) to make it pluggable (eg by using the many Raft implementations,> > >> ZooKeeper or others)> > >> 2) to cleanly separate the election phase and cluster member states (ie> > it> > >> should be the Topology shared between them)> > >> 3) to simplify most common setups in both amount of configuration and> > >> requirements (eg "witness" nodes could be implemented to get a minimum> > 2*n> > >> +1 quorum of nodes instead of forcing 2*n + 1 master-backup pairs)> > >> 4) [OPTIONALLY] to reduce/eliminate implicit "good practices" in term of> > >> order of actions to be performed on nodes in "special states" eg proper> > >> restart sequence after failover or similar cases> > >> 5) [OPTIONALLY] to make shared-store and replication behaviour more> > >> similar:> > >> journal's presence should be the only difference between the 2s> > >>> > >> A proposal of steps to be followed to get this:> > >> 1) abstract away the current quorum vote: it requires extra-care because> > >> the> > >> logic is melted together with the replication/clustering behaviour> > >> 2) refactor it in order to separate election phase and cluster member> > >> states> > >> 3) implement a RI version of such APIs> > >>> > >> Post-actions to help people adopt it, but need to be thought upfront:> > >> 1) a clean upgrade path for current HA replication users> > >> 2) deprecate or integrate the current HA replication into the new> > version> > >>> > >> I've opened this here because many of the HA replication users are devs> > too> > >> and given that this isn't yet implemented: we're stull in the> > >> design/proposal phase, so anyone that want to express their> > >> ideas/opinions/POC on this, is invited to talk here ;)> > >>> > >> Cheers,> > >> Franz> > >>> > >>> > >>> > >>> > >>> > >> --> > >> Sent from:> > >> http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html> > >>> >-- Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: Re: [DISCUSSION] New Quorum vote pluggable implementation

clebertsuconic
In reply to this post by clebertsuconic
We can make pluggable as much as you want, but going forward I would
like us to change how we have been treating plugins.

Because of other discussions, plugins are not really implemented as
part of the community.

We should elect one implementation we can make it and implement it
well.... and bring it as part of the community. People can add more
implementations as part of the open source project.

This is probably the same with Micrometer. Justin implemented
Micrometer and the implementation is not part of activemq. I think we
should change that going forward.

This will be a 3.0, so we should definitely improve how we do plugins
going forward.

On Fri, Mar 13, 2020 at 3:42 AM michael.andre.pearce
<[hidden email]> wrote:
>
> I think a main thing here is a pluggable api.If someone already has etcd, zookeeper, consul etc. In their infrastructure they should be able to reuse.Like wise if you have multiple bulkheaded broker sets it be useful to be able to reuse the same quorum compontent e.g. in ZK that would be a different base root on the tree path per broker set.Likewise that said we should have some out the box support plugins from the community. Obviously what ever we do whilst it maybe a break change e.g. a 3.0 release there must be a migration solution (accepted that it may need outage) for existing brokers that works via the pluggable api not to a specific solution.Also anything from a management api pov should also be visible in the web management console. Sent from my Samsung Galaxy smartphone.
> -------- Original message --------From: Clebert Suconic <[hidden email]> Date: 09/03/2020  14:55  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation I think we should have a management component, that runs outside ofthe broker and would manage quorum.That way you can have the quorum running outside of the broker itself.which would improve the need of multiple brokers to manage the quorum.You just need Quorum Managers in distinct places.I had recently worked with a software called Ceph... And Ceph has theconcept of managers working away from their "broker" (it's not abroker.. it's a DB, but in a sense it's the same concept here). Ithink we should do the same.I have talked to Franz and other guys in private.. and it seemseverybody these days mention raft consensus algorithm.  Perhaps weshould look into it.. and make it pluggable. I believe there's JRaftworking on top of JGroups already.If we make this pluggable and make it a manager a separate process,this will be a big win IMO.On Tue, Mar 3, 2020 at 9:55 AM Andy Taylor <[hidden email]> wrote:>> Personally I wouldn't use Zookeeper, I think there are better options. Also> looks like Kafka are replacing it as well. Saying that, it doesn't really> matter what is used, the main thing is we need to remove the burden of> providing consensus away from the broker.>> It would make sense to make it pluggable.>> A>> On Mon, 2 Mar 2020 at 19:19, KimmKing <[hidden email]> wrote:>> > Can't agree any more.> > The HA&Replication is more and more important for a modern messaging> > system.> > Other apache opensource mq, kafka/rocketmq/pulsar maybe referred to.> >> >> > And In my experience this also bring some extra-complexity about> > performance/brainsplitting issues when rebalance data.> >> > At 2020-03-02 20:33:31, "Martyn Taylor" <[hidden email]> wrote:> > >I think this is a great idea Franz.  The HA and replication components> > have> > >been a source of issues over the years.  Two problems I see are that 1)> > >there isn't a clean separation between the consensus mechanism and the> > >replication, and 2) the consensus algorithm used in Artemis isn't based on> > >any standard algorithm or a research paper.  Hence, all the issues that> > >were caught over the years due to various edge cases.  Integration with> > >ZooKeeper seems like the obvious solution (i.e. push the consensus logic> > >off to a third party lib).  I suspect though, this will be a considerable> > >amount of work and is likely to introduce new issues, so I'd proceed with> > >caution.> > >> > >Cheers> > >> > >> > >> > >On Mon, Mar 2, 2020 at 8:28 AM nigro_franz <[hidden email]> wrote:> > >> > >> Hi folks,> > >>> > >> especially due to the requirements of the current Artemis quorum vote> > >> algorithm, we've thought to re-implementing it with a different focus in> > >> mind:> > >> 1) to make it pluggable (eg by using the many Raft implementations,> > >> ZooKeeper or others)> > >> 2) to cleanly separate the election phase and cluster member states (ie> > it> > >> should be the Topology shared between them)> > >> 3) to simplify most common setups in both amount of configuration and> > >> requirements (eg "witness" nodes could be implemented to get a minimum> > 2*n> > >> +1 quorum of nodes instead of forcing 2*n + 1 master-backup pairs)> > >> 4) [OPTIONALLY] to reduce/eliminate implicit "good practices" in term of> > >> order of actions to be performed on nodes in "special states" eg proper> > >> restart sequence after failover or similar cases> > >> 5) [OPTIONALLY] to make shared-store and replication behaviour more> > >> similar:> > >> journal's presence should be the only difference between the 2s> > >>> > >> A proposal of steps to be followed to get this:> > >> 1) abstract away the current quorum vote: it requires extra-care because> > >> the> > >> logic is melted together with the replication/clustering behaviour> > >> 2) refactor it in order to separate election phase and cluster member> > >> states> > >> 3) implement a RI version of such APIs> > >>> > >> Post-actions to help people adopt it, but need to be thought upfront:> > >> 1) a clean upgrade path for current HA replication users> > >> 2) deprecate or integrate the current HA replication into the new> > version> > >>> > >> I've opened this here because many of the HA replication users are devs> > too> > >> and given that this isn't yet implemented: we're stull in the> > >> design/proposal phase, so anyone that want to express their> > >> ideas/opinions/POC on this, is invited to talk here ;)> > >>> > >> Cheers,> > >> Franz> > >>> > >>> > >>> > >>> > >>> > >> --> > >> Sent from:> > >> http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html> > >>> >-- Clebert Suconic



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: Re: [DISCUSSION] New Quorum vote pluggable implementation

michael.andre.pearce
I think the issue with plugins is that whats the rule to include one or not.You know exactly where im coming from there... re kafka plugin.Im totally for a number of plugins being added. Including the kafka one.But we have to have a clear rule of what will be allowed or not and a place to maintain.Sent from my Samsung Galaxy smartphone.
-------- Original message --------From: Clebert Suconic <[hidden email]> Date: 16/03/2020  01:20  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation We can make pluggable as much as you want, but going forward I wouldlike us to change how we have been treating plugins.Because of other discussions, plugins are not really implemented aspart of the community.We should elect one implementation we can make it and implement itwell.... and bring it as part of the community. People can add moreimplementations as part of the open source project.This is probably the same with Micrometer. Justin implementedMicrometer and the implementation is not part of activemq. I think weshould change that going forward.This will be a 3.0, so we should definitely improve how we do pluginsgoing forward.On Fri, Mar 13, 2020 at 3:42 AM michael.andre.pearce<[hidden email]> wrote:>> I think a main thing here is a pluggable api.If someone already has etcd, zookeeper, consul etc. In their infrastructure they should be able to reuse.Like wise if you have multiple bulkheaded broker sets it be useful to be able to reuse the same quorum compontent e.g. in ZK that would be a different base root on the tree path per broker set.Likewise that said we should have some out the box support plugins from the community. Obviously what ever we do whilst it maybe a break change e.g. a 3.0 release there must be a migration solution (accepted that it may need outage) for existing brokers that works via the pluggable api not to a specific solution.Also anything from a management api pov should also be visible in the web management console. Sent from my Samsung Galaxy smartphone.> -------- Original message --------From: Clebert Suconic <[hidden email]> Date: 09/03/2020  14:55  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation I think we should have a management component, that runs outside ofthe broker and would manage quorum.That way you can have the quorum running outside of the broker itself.which would improve the need of multiple brokers to manage the quorum.You just need Quorum Managers in distinct places.I had recently worked with a software called Ceph... And Ceph has theconcept of managers working away from their "broker" (it's not abroker.. it's a DB, but in a sense it's the same concept here). Ithink we should do the same.I have talked to Franz and other guys in private.. and it seemseverybody these days mention raft consensus algorithm.  Perhaps weshould look into it.. and make it pluggable. I believe there's JRaftworking on top of JGroups already.If we make this pluggable and make it a manager a separate process,this will be a big win IMO.On Tue, Mar 3, 2020 at 9:55 AM Andy Taylor <[hidden email]> wrote:>> Personally I wouldn't use Zookeeper, I think there are better options. Also> looks like Kafka are replacing it as well. Saying that, it doesn't really> matter what is used, the main thing is we need to remove the burden of> providing consensus away from the broker.>> It would make sense to make it pluggable.>> A>> On Mon, 2 Mar 2020 at 19:19, KimmKing <[hidden email]> wrote:>> > Can't agree any more.> > The HA&Replication is more and more important for a modern messaging> > system.> > Other apache opensource mq, kafka/rocketmq/pulsar maybe referred to.> >> >> > And In my experience this also bring some extra-complexity about> > performance/brainsplitting issues when rebalance data.> >> > At 2020-03-02 20:33:31, "Martyn Taylor" <[hidden email]> wrote:> > >I think this is a great idea Franz.  The HA and replication components> > have> > >been a source of issues over the years.  Two problems I see are that 1)> > >there isn't a clean separation between the consensus mechanism and the> > >replication, and 2) the consensus algorithm used in Artemis isn't based on> > >any standard algorithm or a research paper.  Hence, all the issues that> > >were caught over the years due to various edge cases.  Integration with> > >ZooKeeper seems like the obvious solution (i.e. push the consensus logic> > >off to a third party lib).  I suspect though, this will be a considerable> > >amount of work and is likely to introduce new issues, so I'd proceed with> > >caution.> > >> > >Cheers> > >> > >> > >> > >On Mon, Mar 2, 2020 at 8:28 AM nigro_franz <[hidden email]> wrote:> > >> > >> Hi folks,> > >>> > >> especially due to the requirements of the current Artemis quorum vote> > >> algorithm, we've thought to re-implementing it with a different focus in> > >> mind:> > >> 1) to make it pluggable (eg by using the many Raft implementations,> > >> ZooKeeper or others)> > >> 2) to cleanly separate the election phase and cluster member states (ie> > it> > >> should be the Topology shared between them)> > >> 3) to simplify most common setups in both amount of configuration and> > >> requirements (eg "witness" nodes could be implemented to get a minimum> > 2*n> > >> +1 quorum of nodes instead of forcing 2*n + 1 master-backup pairs)> > >> 4) [OPTIONALLY] to reduce/eliminate implicit "good practices" in term of> > >> order of actions to be performed on nodes in "special states" eg proper> > >> restart sequence after failover or similar cases> > >> 5) [OPTIONALLY] to make shared-store and replication behaviour more> > >> similar:> > >> journal's presence should be the only difference between the 2s> > >>> > >> A proposal of steps to be followed to get this:> > >> 1) abstract away the current quorum vote: it requires extra-care because> > >> the> > >> logic is melted together with the replication/clustering behaviour> > >> 2) refactor it in order to separate election phase and cluster member> > >> states> > >> 3) implement a RI version of such APIs> > >>> > >> Post-actions to help people adopt it, but need to be thought upfront:> > >> 1) a clean upgrade path for current HA replication users> > >> 2) deprecate or integrate the current HA replication into the new> > version> > >>> > >> I've opened this here because many of the HA replication users are devs> > too> > >> and given that this isn't yet implemented: we're stull in the> > >> design/proposal phase, so anyone that want to express their> > >> ideas/opinions/POC on this, is invited to talk here ;)> > >>> > >> Cheers,> > >> Franz> > >>> > >>> > >>> > >>> > >>> > >> --> > >> Sent from:> > >> http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html> > >>> >-- Clebert Suconic-- Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: Re: [DISCUSSION] New Quorum vote pluggable implementation

clebertsuconic
In reply to this post by clebertsuconic
It's on a case by case basis... What we can't do is transform every
new core functionality (such as monitoring. .it's pretty standard /
core functionality) into a plug-in, and then block its implementation
based on a past discussion.

I say, if there's a plugin, there should be at least one
implementation on the broker.. that goes for Monitoring... we can
support more.. but this has to have an implementation.


You are a committer, if you want to have your kafka plugin again, just
raise a discussion.. but we can't block every new functionality based
on that baggage.




On Tue, Mar 17, 2020 at 7:35 PM michael.andre.pearce
<[hidden email]> wrote:
>
> I think the issue with plugins is that whats the rule to include one or not.You know exactly where im coming from there... re kafka plugin.Im totally for a number of plugins being added. Including the kafka one.But we have to have a clear rule of what will be allowed or not and a place to maintain.Sent from my Samsung Galaxy smartphone.
> -------- Original message --------From: Clebert Suconic <[hidden email]> Date: 16/03/2020  01:20  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation We can make pluggable as much as you want, but going forward I wouldlike us to change how we have been treating plugins.Because of other discussions, plugins are not really implemented aspart of the community.We should elect one implementation we can make it and implement itwell.... and bring it as part of the community. People can add moreimplementations as part of the open source project.This is probably the same with Micrometer. Justin implementedMicrometer and the implementation is not part of activemq. I think weshould change that going forward.This will be a 3.0, so we should definitely improve how we do pluginsgoing forward.On Fri, Mar 13, 2020 at 3:42 AM michael.andre.pearce<[hidden email]> wrote:>> I think a main thing here is a pluggable api.If someone already has etcd, zookeeper, consul etc. In their infrastructure they should be able to reuse.Like wise if you have multiple bulkheaded broker sets it be useful to be able to reuse the same quorum compontent e.g. in ZK that would be a different base root on the tree path per broker set.Likewise that said we should have some out the box support plugins from the community. Obviously what ever we do whilst it maybe a break change e.g. a 3.0 release there must be a migration solution (accepted that it may need outage) for existing brokers that works via the pluggable api not to a specific solution.Also anything from a management api pov should also be visible in the web management console. Sent from my Samsung Galaxy smartphone.> -------- Original message --------From: Clebert Suconic <[hidden email]> Date: 09/03/2020  14:55  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation I think we should have a management component, that runs outside ofthe broker and would manage quorum.That way you can have the quorum running outside of the broker itself.which would improve the need of multiple brokers to manage the quorum.You just need Quorum Managers in distinct places.I had recently worked with a software called Ceph... And Ceph has theconcept of managers working away from their "broker" (it's not abroker.. it's a DB, but in a sense it's the same concept here). Ithink we should do the same.I have talked to Franz and other guys in private.. and it seemseverybody these days mention raft consensus algorithm.  Perhaps weshould look into it.. and make it pluggable. I believe there's JRaftworking on top of JGroups already.If we make this pluggable and make it a manager a separate process,this will be a big win IMO.On Tue, Mar 3, 2020 at 9:55 AM Andy Taylor <[hidden email]> wrote:>> Personally I wouldn't use Zookeeper, I think there are better options. Also> looks like Kafka are replacing it as well. Saying that, it doesn't really> matter what is used, the main thing is we need to remove the burden of> providing consensus away from the broker.>> It would make sense to make it pluggable.>> A>> On Mon, 2 Mar 2020 at 19:19, KimmKing <[hidden email]> wrote:>> > Can't agree any more.> > The HA&Replication is more and more important for a modern messaging> > system.> > Other apache opensource mq, kafka/rocketmq/pulsar maybe referred to.> >> >> > And In my experience this also bring some extra-complexity about> > performance/brainsplitting issues when rebalance data.> >> > At 2020-03-02 20:33:31, "Martyn Taylor" <[hidden email]> wrote:> > >I think this is a great idea Franz.  The HA and replication components> > have> > >been a source of issues over the years.  Two problems I see are that 1)> > >there isn't a clean separation between the consensus mechanism and the> > >replication, and 2) the consensus algorithm used in Artemis isn't based on> > >any standard algorithm or a research paper.  Hence, all the issues that> > >were caught over the years due to various edge cases.  Integration with> > >ZooKeeper seems like the obvious solution (i.e. push the consensus logic> > >off to a third party lib).  I suspect though, this will be a considerable> > >amount of work and is likely to introduce new issues, so I'd proceed with> > >caution.> > >> > >Cheers> > >> > >> > >> > >On Mon, Mar 2, 2020 at 8:28 AM nigro_franz <[hidden email]> wrote:> > >> > >> Hi folks,> > >>> > >> especially due to the requirements of the current Artemis quorum vote> > >> algorithm, we've thought to re-implementing it with a different focus in> > >> mind:> > >> 1) to make it pluggable (eg by using the many Raft implementations,> > >> ZooKeeper or others)> > >> 2) to cleanly separate the election phase and cluster member states (ie> > it> > >> should be the Topology shared between them)> > >> 3) to simplify most common setups in both amount of configuration and> > >> requirements (eg "witness" nodes could be implemented to get a minimum> > 2*n> > >> +1 quorum of nodes instead of forcing 2*n + 1 master-backup pairs)> > >> 4) [OPTIONALLY] to reduce/eliminate implicit "good practices" in term of> > >> order of actions to be performed on nodes in "special states" eg proper> > >> restart sequence after failover or similar cases> > >> 5) [OPTIONALLY] to make shared-store and replication behaviour more> > >> similar:> > >> journal's presence should be the only difference between the 2s> > >>> > >> A proposal of steps to be followed to get this:> > >> 1) abstract away the current quorum vote: it requires extra-care because> > >> the> > >> logic is melted together with the replication/clustering behaviour> > >> 2) refactor it in order to separate election phase and cluster member> > >> states> > >> 3) implement a RI version of such APIs> > >>> > >> Post-actions to help people adopt it, but need to be thought upfront:> > >> 1) a clean upgrade path for current HA replication users> > >> 2) deprecate or integrate the current HA replication into the new> > version> > >>> > >> I've opened this here because many of the HA replication users are devs> > too> > >> and given that this isn't yet implemented: we're stull in the> > >> design/proposal phase, so anyone that want to express their> > >> ideas/opinions/POC on this, is invited to talk here ;)> > >>> > >> Cheers,> > >> Franz> > >>> > >>> > >>> > >>> > >>> > >> --> > >> Sent from:> > >> http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html> > >>> >-- Clebert Suconic-- Clebert Suconic



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: Re: [DISCUSSION] New Quorum vote pluggable implementation

michael.andre.pearce
Well the point is without us having sorted the issue, then this will keep occuring.I am for one in favour we relax a little and if someone wants to add a plugin we simply allow it. Why should there be any filter.Sent from my Samsung Galaxy smartphone.
-------- Original message --------From: Clebert Suconic <[hidden email]> Date: 18/03/2020  00:38  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation It's on a case by case basis... What we can't do is transform everynew core functionality (such as monitoring. .it's pretty standard /core functionality) into a plug-in, and then block its implementationbased on a past discussion.I say, if there's a plugin, there should be at least oneimplementation on the broker.. that goes for Monitoring... we cansupport more.. but this has to have an implementation.You are a committer, if you want to have your kafka plugin again, justraise a discussion.. but we can't block every new functionality basedon that baggage.On Tue, Mar 17, 2020 at 7:35 PM michael.andre.pearce<[hidden email]> wrote:>> I think the issue with plugins is that whats the rule to include one or not.You know exactly where im coming from there... re kafka plugin.Im totally for a number of plugins being added. Including the kafka one.But we have to have a clear rule of what will be allowed or not and a place to maintain.Sent from my Samsung Galaxy smartphone.> -------- Original message --------From: Clebert Suconic <[hidden email]> Date: 16/03/2020  01:20  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation We can make pluggable as much as you want, but going forward I wouldlike us to change how we have been treating plugins.Because of other discussions, plugins are not really implemented aspart of the community.We should elect one implementation we can make it and implement itwell.... and bring it as part of the community. People can add moreimplementations as part of the open source project.This is probably the same with Micrometer. Justin implementedMicrometer and the implementation is not part of activemq. I think weshould change that going forward.This will be a 3.0, so we should definitely improve how we do pluginsgoing forward.On Fri, Mar 13, 2020 at 3:42 AM michael.andre.pearce<[hidden email]> wrote:>> I think a main thing here is a pluggable api.If someone already has etcd, zookeeper, consul etc. In their infrastructure they should be able to reuse.Like wise if you have multiple bulkheaded broker sets it be useful to be able to reuse the same quorum compontent e.g. in ZK that would be a different base root on the tree path per broker set.Likewise that said we should have some out the box support plugins from the community. Obviously what ever we do whilst it maybe a break change e.g. a 3.0 release there must be a migration solution (accepted that it may need outage) for existing brokers that works via the pluggable api not to a specific solution.Also anything from a management api pov should also be visible in the web management console. Sent from my Samsung Galaxy smartphone.> -------- Original message --------From: Clebert Suconic <[hidden email]> Date: 09/03/2020  14:55  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation I think we should have a management component, that runs outside ofthe broker and would manage quorum.That way you can have the quorum running outside of the broker itself.which would improve the need of multiple brokers to manage the quorum.You just need Quorum Managers in distinct places.I had recently worked with a software called Ceph... And Ceph has theconcept of managers working away from their "broker" (it's not abroker.. it's a DB, but in a sense it's the same concept here). Ithink we should do the same.I have talked to Franz and other guys in private.. and it seemseverybody these days mention raft consensus algorithm.  Perhaps weshould look into it.. and make it pluggable. I believe there's JRaftworking on top of JGroups already.If we make this pluggable and make it a manager a separate process,this will be a big win IMO.On Tue, Mar 3, 2020 at 9:55 AM Andy Taylor <[hidden email]> wrote:>> Personally I wouldn't use Zookeeper, I think there are better options. Also> looks like Kafka are replacing it as well. Saying that, it doesn't really> matter what is used, the main thing is we need to remove the burden of> providing consensus away from the broker.>> It would make sense to make it pluggable.>> A>> On Mon, 2 Mar 2020 at 19:19, KimmKing <[hidden email]> wrote:>> > Can't agree any more.> > The HA&Replication is more and more important for a modern messaging> > system.> > Other apache opensource mq, kafka/rocketmq/pulsar maybe referred to.> >> >> > And In my experience this also bring some extra-complexity about> > performance/brainsplitting issues when rebalance data.> >> > At 2020-03-02 20:33:31, "Martyn Taylor" <[hidden email]> wrote:> > >I think this is a great idea Franz.  The HA and replication components> > have> > >been a source of issues over the years.  Two problems I see are that 1)> > >there isn't a clean separation between the consensus mechanism and the> > >replication, and 2) the consensus algorithm used in Artemis isn't based on> > >any standard algorithm or a research paper.  Hence, all the issues that> > >were caught over the years due to various edge cases.  Integration with> > >ZooKeeper seems like the obvious solution (i.e. push the consensus logic> > >off to a third party lib).  I suspect though, this will be a considerable> > >amount of work and is likely to introduce new issues, so I'd proceed with> > >caution.> > >> > >Cheers> > >> > >> > >> > >On Mon, Mar 2, 2020 at 8:28 AM nigro_franz <[hidden email]> wrote:> > >> > >> Hi folks,> > >>> > >> especially due to the requirements of the current Artemis quorum vote> > >> algorithm, we've thought to re-implementing it with a different focus in> > >> mind:> > >> 1) to make it pluggable (eg by using the many Raft implementations,> > >> ZooKeeper or others)> > >> 2) to cleanly separate the election phase and cluster member states (ie> > it> > >> should be the Topology shared between them)> > >> 3) to simplify most common setups in both amount of configuration and> > >> requirements (eg "witness" nodes could be implemented to get a minimum> > 2*n> > >> +1 quorum of nodes instead of forcing 2*n + 1 master-backup pairs)> > >> 4) [OPTIONALLY] to reduce/eliminate implicit "good practices" in term of> > >> order of actions to be performed on nodes in "special states" eg proper> > >> restart sequence after failover or similar cases> > >> 5) [OPTIONALLY] to make shared-store and replication behaviour more> > >> similar:> > >> journal's presence should be the only difference between the 2s> > >>> > >> A proposal of steps to be followed to get this:> > >> 1) abstract away the current quorum vote: it requires extra-care because> > >> the> > >> logic is melted together with the replication/clustering behaviour> > >> 2) refactor it in order to separate election phase and cluster member> > >> states> > >> 3) implement a RI version of such APIs> > >>> > >> Post-actions to help people adopt it, but need to be thought upfront:> > >> 1) a clean upgrade path for current HA replication users> > >> 2) deprecate or integrate the current HA replication into the new> > version> > >>> > >> I've opened this here because many of the HA replication users are devs> > too> > >> and given that this isn't yet implemented: we're stull in the> > >> design/proposal phase, so anyone that want to express their> > >> ideas/opinions/POC on this, is invited to talk here ;)> > >>> > >> Cheers,> > >> Franz> > >>> > >>> > >>> > >>> > >>> > >> --> > >> Sent from:> > >> http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html> > >>> >-- Clebert Suconic-- Clebert Suconic-- Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: Re: [DISCUSSION] New Quorum vote pluggable implementation

clebertsuconic
In reply to this post by clebertsuconic
Just like anything else in apache... committers/community can ask or
vote features out. it doesn't even need to be a plugin.

I don't mean to hijack this discussion away from Quorum.. although it
would affect the implementation.. .we can keep a plugin's area in the
broker... and activate them based on the CLI / config.

I just want to make sure we can unlock this going forward.


On Tue, Mar 17, 2020 at 8:53 PM michael.andre.pearce
<[hidden email]> wrote:
>
> Well the point is without us having sorted the issue, then this will keep occuring.I am for one in favour we relax a little and if someone wants to add a plugin we simply allow it. Why should there be any filter.Sent from my Samsung Galaxy smartphone.
> -------- Original message --------From: Clebert Suconic <[hidden email]> Date: 18/03/2020  00:38  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation It's on a case by case basis... What we can't do is transform everynew core functionality (such as monitoring. .it's pretty standard /core functionality) into a plug-in, and then block its implementationbased on a past discussion.I say, if there's a plugin, there should be at least oneimplementation on the broker.. that goes for Monitoring... we cansupport more.. but this has to have an implementation.You are a committer, if you want to have your kafka plugin again, justraise a discussion.. but we can't block every new functionality basedon that baggage.On Tue, Mar 17, 2020 at 7:35 PM michael.andre.pearce<[hidden email]> wrote:>> I think the issue with plugins is that whats the rule to include one or not.You know exactly where im coming from there... re kafka plugin.Im totally for a number of plugins being added. Including the kafka one.But we have to have a clear rule of what will be allowed or not and a place to maintain.Sent from my Samsung Galaxy smartphone.> -------- Original message --------From: Clebert Suconic <[hidden email]> Date: 16/03/2020  01:20  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation We can make pluggable as much as you want, but going forward I wouldlike us to change how we have been treating plugins.Because of other discussions, plugins are not really implemented aspart of the community.We should elect one implementation we can make it and implement itwell.... and bring it as part of the community. People can add moreimplementations as part of the open source project.This is probably the same with Micrometer. Justin implementedMicrometer and the implementation is not part of activemq. I think weshould change that going forward.This will be a 3.0, so we should definitely improve how we do pluginsgoing forward.On Fri, Mar 13, 2020 at 3:42 AM michael.andre.pearce<[hidden email]> wrote:>> I think a main thing here is a pluggable api.If someone already has etcd, zookeeper, consul etc. In their infrastructure they should be able to reuse.Like wise if you have multiple bulkheaded broker sets it be useful to be able to reuse the same quorum compontent e.g. in ZK that would be a different base root on the tree path per broker set.Likewise that said we should have some out the box support plugins from the community. Obviously what ever we do whilst it maybe a break change e.g. a 3.0 release there must be a migration solution (accepted that it may need outage) for existing brokers that works via the pluggable api not to a specific solution.Also anything from a management api pov should also be visible in the web management console. Sent from my Samsung Galaxy smartphone.> -------- Original message --------From: Clebert Suconic <[hidden email]> Date: 09/03/2020  14:55  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation I think we should have a management component, that runs outside ofthe broker and would manage quorum.That way you can have the quorum running outside of the broker itself.which would improve the need of multiple brokers to manage the quorum.You just need Quorum Managers in distinct places.I had recently worked with a software called Ceph... And Ceph has theconcept of managers working away from their "broker" (it's not abroker.. it's a DB, but in a sense it's the same concept here). Ithink we should do the same.I have talked to Franz and other guys in private.. and it seemseverybody these days mention raft consensus algorithm.  Perhaps weshould look into it.. and make it pluggable. I believe there's JRaftworking on top of JGroups already.If we make this pluggable and make it a manager a separate process,this will be a big win IMO.On Tue, Mar 3, 2020 at 9:55 AM Andy Taylor <[hidden email]> wrote:>> Personally I wouldn't use Zookeeper, I think there are better options. Also> looks like Kafka are replacing it as well. Saying that, it doesn't really> matter what is used, the main thing is we need to remove the burden of> providing consensus away from the broker.>> It would make sense to make it pluggable.>> A>> On Mon, 2 Mar 2020 at 19:19, KimmKing <[hidden email]> wrote:>> > Can't agree any more.> > The HA&Replication is more and more important for a modern messaging> > system.> > Other apache opensource mq, kafka/rocketmq/pulsar maybe referred to.> >> >> > And In my experience this also bring some extra-complexity about> > performance/brainsplitting issues when rebalance data.> >> > At 2020-03-02 20:33:31, "Martyn Taylor" <[hidden email]> wrote:> > >I think this is a great idea Franz.  The HA and replication components> > have> > >been a source of issues over the years.  Two problems I see are that 1)> > >there isn't a clean separation between the consensus mechanism and the> > >replication, and 2) the consensus algorithm used in Artemis isn't based on> > >any standard algorithm or a research paper.  Hence, all the issues that> > >were caught over the years due to various edge cases.  Integration with> > >ZooKeeper seems like the obvious solution (i.e. push the consensus logic> > >off to a third party lib).  I suspect though, this will be a considerable> > >amount of work and is likely to introduce new issues, so I'd proceed with> > >caution.> > >> > >Cheers> > >> > >> > >> > >On Mon, Mar 2, 2020 at 8:28 AM nigro_franz <[hidden email]> wrote:> > >> > >> Hi folks,> > >>> > >> especially due to the requirements of the current Artemis quorum vote> > >> algorithm, we've thought to re-implementing it with a different focus in> > >> mind:> > >> 1) to make it pluggable (eg by using the many Raft implementations,> > >> ZooKeeper or others)> > >> 2) to cleanly separate the election phase and cluster member states (ie> > it> > >> should be the Topology shared between them)> > >> 3) to simplify most common setups in both amount of configuration and> > >> requirements (eg "witness" nodes could be implemented to get a minimum> > 2*n> > >> +1 quorum of nodes instead of forcing 2*n + 1 master-backup pairs)> > >> 4) [OPTIONALLY] to reduce/eliminate implicit "good practices" in term of> > >> order of actions to be performed on nodes in "special states" eg proper> > >> restart sequence after failover or similar cases> > >> 5) [OPTIONALLY] to make shared-store and replication behaviour more> > >> similar:> > >> journal's presence should be the only difference between the 2s> > >>> > >> A proposal of steps to be followed to get this:> > >> 1) abstract away the current quorum vote: it requires extra-care because> > >> the> > >> logic is melted together with the replication/clustering behaviour> > >> 2) refactor it in order to separate election phase and cluster member> > >> states> > >> 3) implement a RI version of such APIs> > >>> > >> Post-actions to help people adopt it, but need to be thought upfront:> > >> 1) a clean upgrade path for current HA replication users> > >> 2) deprecate or integrate the current HA replication into the new> > version> > >>> > >> I've opened this here because many of the HA replication users are devs> > too> > >> and given that this isn't yet implemented: we're stull in the> > >> design/proposal phase, so anyone that want to express their> > >> ideas/opinions/POC on this, is invited to talk here ;)> > >>> > >> Cheers,> > >> Franz> > >>> > >>> > >>> > >>> > >>> > >> --> > >> Sent from:> > >> http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html> > >>> >-- Clebert Suconic-- Clebert Suconic-- Clebert Suconic



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: Re: [DISCUSSION] New Quorum vote pluggable implementation

michael.andre.pearce
I totally get it.As said im for the camp that we have plugin modules in the broker. And if someone wants to commit one, we allow it without need to votes etc and all this discussion every time.(e.g. Justin and I shouldnt need to make discussion thread, it should just be accepted and added)If something gets un-maintained (some peoples concerns) we simply remove it / attic it. The fact its un-maintained means no longer no one supports it But its for everyone to agree to this more open way of working.....not just me and you.An alternative approach if people are concerned these are in broker is that we have a separate plugin repo that we stick them all, and release...but this then makes it harder for end users to enable or use...Sent from my Samsung Galaxy smartphone.
-------- Original message --------From: Clebert Suconic <[hidden email]> Date: 18/03/2020  00:57  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation Just like anything else in apache... committers/community can ask orvote features out. it doesn't even need to be a plugin.I don't mean to hijack this discussion away from Quorum.. although itwould affect the implementation.. .we can keep a plugin's area in thebroker... and activate them based on the CLI / config.I just want to make sure we can unlock this going forward.On Tue, Mar 17, 2020 at 8:53 PM michael.andre.pearce<[hidden email]> wrote:>> Well the point is without us having sorted the issue, then this will keep occuring.I am for one in favour we relax a little and if someone wants to add a plugin we simply allow it. Why should there be any filter.Sent from my Samsung Galaxy smartphone.> -------- Original message --------From: Clebert Suconic <[hidden email]> Date: 18/03/2020  00:38  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation It's on a case by case basis... What we can't do is transform everynew core functionality (such as monitoring. .it's pretty standard /core functionality) into a plug-in, and then block its implementationbased on a past discussion.I say, if there's a plugin, there should be at least oneimplementation on the broker.. that goes for Monitoring... we cansupport more.. but this has to have an implementation.You are a committer, if you want to have your kafka plugin again, justraise a discussion.. but we can't block every new functionality basedon that baggage.On Tue, Mar 17, 2020 at 7:35 PM michael.andre.pearce<[hidden email]> wrote:>> I think the issue with plugins is that whats the rule to include one or not.You know exactly where im coming from there... re kafka plugin.Im totally for a number of plugins being added. Including the kafka one.But we have to have a clear rule of what will be allowed or not and a place to maintain.Sent from my Samsung Galaxy smartphone.> -------- Original message --------From: Clebert Suconic <[hidden email]> Date: 16/03/2020  01:20  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation We can make pluggable as much as you want, but going forward I wouldlike us to change how we have been treating plugins.Because of other discussions, plugins are not really implemented aspart of the community.We should elect one implementation we can make it and implement itwell.... and bring it as part of the community. People can add moreimplementations as part of the open source project.This is probably the same with Micrometer. Justin implementedMicrometer and the implementation is not part of activemq. I think weshould change that going forward.This will be a 3.0, so we should definitely improve how we do pluginsgoing forward.On Fri, Mar 13, 2020 at 3:42 AM michael.andre.pearce<[hidden email]> wrote:>> I think a main thing here is a pluggable api.If someone already has etcd, zookeeper, consul etc. In their infrastructure they should be able to reuse.Like wise if you have multiple bulkheaded broker sets it be useful to be able to reuse the same quorum compontent e.g. in ZK that would be a different base root on the tree path per broker set.Likewise that said we should have some out the box support plugins from the community. Obviously what ever we do whilst it maybe a break change e.g. a 3.0 release there must be a migration solution (accepted that it may need outage) for existing brokers that works via the pluggable api not to a specific solution.Also anything from a management api pov should also be visible in the web management console. Sent from my Samsung Galaxy smartphone.> -------- Original message --------From: Clebert Suconic <[hidden email]> Date: 09/03/2020  14:55  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation I think we should have a management component, that runs outside ofthe broker and would manage quorum.That way you can have the quorum running outside of the broker itself.which would improve the need of multiple brokers to manage the quorum.You just need Quorum Managers in distinct places.I had recently worked with a software called Ceph... And Ceph has theconcept of managers working away from their "broker" (it's not abroker.. it's a DB, but in a sense it's the same concept here). Ithink we should do the same.I have talked to Franz and other guys in private.. and it seemseverybody these days mention raft consensus algorithm.  Perhaps weshould look into it.. and make it pluggable. I believe there's JRaftworking on top of JGroups already.If we make this pluggable and make it a manager a separate process,this will be a big win IMO.On Tue, Mar 3, 2020 at 9:55 AM Andy Taylor <[hidden email]> wrote:>> Personally I wouldn't use Zookeeper, I think there are better options. Also> looks like Kafka are replacing it as well. Saying that, it doesn't really> matter what is used, the main thing is we need to remove the burden of> providing consensus away from the broker.>> It would make sense to make it pluggable.>> A>> On Mon, 2 Mar 2020 at 19:19, KimmKing <[hidden email]> wrote:>> > Can't agree any more.> > The HA&Replication is more and more important for a modern messaging> > system.> > Other apache opensource mq, kafka/rocketmq/pulsar maybe referred to.> >> >> > And In my experience this also bring some extra-complexity about> > performance/brainsplitting issues when rebalance data.> >> > At 2020-03-02 20:33:31, "Martyn Taylor" <[hidden email]> wrote:> > >I think this is a great idea Franz.  The HA and replication components> > have> > >been a source of issues over the years.  Two problems I see are that 1)> > >there isn't a clean separation between the consensus mechanism and the> > >replication, and 2) the consensus algorithm used in Artemis isn't based on> > >any standard algorithm or a research paper.  Hence, all the issues that> > >were caught over the years due to various edge cases.  Integration with> > >ZooKeeper seems like the obvious solution (i.e. push the consensus logic> > >off to a third party lib).  I suspect though, this will be a considerable> > >amount of work and is likely to introduce new issues, so I'd proceed with> > >caution.> > >> > >Cheers> > >> > >> > >> > >On Mon, Mar 2, 2020 at 8:28 AM nigro_franz <[hidden email]> wrote:> > >> > >> Hi folks,> > >>> > >> especially due to the requirements of the current Artemis quorum vote> > >> algorithm, we've thought to re-implementing it with a different focus in> > >> mind:> > >> 1) to make it pluggable (eg by using the many Raft implementations,> > >> ZooKeeper or others)> > >> 2) to cleanly separate the election phase and cluster member states (ie> > it> > >> should be the Topology shared between them)> > >> 3) to simplify most common setups in both amount of configuration and> > >> requirements (eg "witness" nodes could be implemented to get a minimum> > 2*n> > >> +1 quorum of nodes instead of forcing 2*n + 1 master-backup pairs)> > >> 4) [OPTIONALLY] to reduce/eliminate implicit "good practices" in term of> > >> order of actions to be performed on nodes in "special states" eg proper> > >> restart sequence after failover or similar cases> > >> 5) [OPTIONALLY] to make shared-store and replication behaviour more> > >> similar:> > >> journal's presence should be the only difference between the 2s> > >>> > >> A proposal of steps to be followed to get this:> > >> 1) abstract away the current quorum vote: it requires extra-care because> > >> the> > >> logic is melted together with the replication/clustering behaviour> > >> 2) refactor it in order to separate election phase and cluster member> > >> states> > >> 3) implement a RI version of such APIs> > >>> > >> Post-actions to help people adopt it, but need to be thought upfront:> > >> 1) a clean upgrade path for current HA replication users> > >> 2) deprecate or integrate the current HA replication into the new> > version> > >>> > >> I've opened this here because many of the HA replication users are devs> > too> > >> and given that this isn't yet implemented: we're stull in the> > >> design/proposal phase, so anyone that want to express their> > >> ideas/opinions/POC on this, is invited to talk here ;)> > >>> > >> Cheers,> > >> Franz> > >>> > >>> > >>> > >>> > >>> > >> --> > >> Sent from:> > >> http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html> > >>> >-- Clebert Suconic-- Clebert Suconic-- Clebert Suconic-- Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: Re: [DISCUSSION] New Quorum vote pluggable implementation

clebertsuconic
In reply to this post by clebertsuconic
>> no longer no one supports it  <<

I think this is just apache license here.

As long as you have community around it.. code is maintained.. it's
all fine. I mean... in apache terms you guarantee things as in apache
license. I try to do good work on the open source community, fixing
bugs.. everything... but as you know the limits of the apache license
terms.

The term support here I think applies to third parties.. companies
listed here, (Including the company I work for... Red Hat / IBM).:
https://activemq.apache.org/support.html

It will be up to them what they consider supported or not.. and that
goes both for plugins or core functionalities. Company X may decide to
not support certain functionality as in providing you patches, help..
etc. and you would be on your own on such functionality like any other
apache license.

that's pretty standard... not just for ActiveMQ.. I think that's
common for any other project.


On Tue, Mar 17, 2020 at 9:05 PM michael.andre.pearce
<[hidden email]> wrote:
>
> I totally get it.As said im for the camp that we have plugin modules in the broker. And if someone wants to commit one, we allow it without need to votes etc and all this discussion every time.(e.g. Justin and I shouldnt need to make discussion thread, it should just be accepted and added)If something gets un-maintained (some peoples concerns) we simply remove it / attic it. The fact its un-maintained means no longer no one supports it But its for everyone to agree to this more open way of working.....not just me and you.An alternative approach if people are concerned these are in broker is that we have a separate plugin repo that we stick them all, and release...but this then makes it harder for end users to enable or use...Sent from my Samsung Galaxy smartphone.
> -------- Original message --------From: Clebert Suconic <[hidden email]> Date: 18/03/2020  00:57  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation Just like anything else in apache... committers/community can ask orvote features out. it doesn't even need to be a plugin.I don't mean to hijack this discussion away from Quorum.. although itwould affect the implementation.. .we can keep a plugin's area in thebroker... and activate them based on the CLI / config.I just want to make sure we can unlock this going forward.On Tue, Mar 17, 2020 at 8:53 PM michael.andre.pearce<[hidden email]> wrote:>> Well the point is without us having sorted the issue, then this will keep occuring.I am for one in favour we relax a little and if someone wants to add a plugin we simply allow it. Why should there be any filter.Sent from my Samsung Galaxy smartphone.> -------- Original message --------From: Clebert Suconic <[hidden email]> Date: 18/03/2020  00:38  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation It's on a case by case basis... What we can't do is transform everynew core functionality (such as monitoring. .it's pretty standard /core functionality) into a plug-in, and then block its implementationbased on a past discussion.I say, if there's a plugin, there should be at least oneimplementation on the broker.. that goes for Monitoring... we cansupport more.. but this has to have an implementation.You are a committer, if you want to have your kafka plugin again, justraise a discussion.. but we can't block every new functionality basedon that baggage.On Tue, Mar 17, 2020 at 7:35 PM michael.andre.pearce<[hidden email]> wrote:>> I think the issue with plugins is that whats the rule to include one or not.You know exactly where im coming from there... re kafka plugin.Im totally for a number of plugins being added. Including the kafka one.But we have to have a clear rule of what will be allowed or not and a place to maintain.Sent from my Samsung Galaxy smartphone.> -------- Original message --------From: Clebert Suconic <[hidden email]> Date: 16/03/2020  01:20  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation We can make pluggable as much as you want, but going forward I wouldlike us to change how we have been treating plugins.Because of other discussions, plugins are not really implemented aspart of the community.We should elect one implementation we can make it and implement itwell.... and bring it as part of the community. People can add moreimplementations as part of the open source project.This is probably the same with Micrometer. Justin implementedMicrometer and the implementation is not part of activemq. I think weshould change that going forward.This will be a 3.0, so we should definitely improve how we do pluginsgoing forward.On Fri, Mar 13, 2020 at 3:42 AM michael.andre.pearce<[hidden email]> wrote:>> I think a main thing here is a pluggable api.If someone already has etcd, zookeeper, consul etc. In their infrastructure they should be able to reuse.Like wise if you have multiple bulkheaded broker sets it be useful to be able to reuse the same quorum compontent e.g. in ZK that would be a different base root on the tree path per broker set.Likewise that said we should have some out the box support plugins from the community. Obviously what ever we do whilst it maybe a break change e.g. a 3.0 release there must be a migration solution (accepted that it may need outage) for existing brokers that works via the pluggable api not to a specific solution.Also anything from a management api pov should also be visible in the web management console. Sent from my Samsung Galaxy smartphone.> -------- Original message --------From: Clebert Suconic <[hidden email]> Date: 09/03/2020  14:55  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation I think we should have a management component, that runs outside ofthe broker and would manage quorum.That way you can have the quorum running outside of the broker itself.which would improve the need of multiple brokers to manage the quorum.You just need Quorum Managers in distinct places.I had recently worked with a software called Ceph... And Ceph has theconcept of managers working away from their "broker" (it's not abroker.. it's a DB, but in a sense it's the same concept here). Ithink we should do the same.I have talked to Franz and other guys in private.. and it seemseverybody these days mention raft consensus algorithm.  Perhaps weshould look into it.. and make it pluggable. I believe there's JRaftworking on top of JGroups already.If we make this pluggable and make it a manager a separate process,this will be a big win IMO.On Tue, Mar 3, 2020 at 9:55 AM Andy Taylor <[hidden email]> wrote:>> Personally I wouldn't use Zookeeper, I think there are better options. Also> looks like Kafka are replacing it as well. Saying that, it doesn't really> matter what is used, the main thing is we need to remove the burden of> providing consensus away from the broker.>> It would make sense to make it pluggable.>> A>> On Mon, 2 Mar 2020 at 19:19, KimmKing <[hidden email]> wrote:>> > Can't agree any more.> > The HA&Replication is more and more important for a modern messaging> > system.> > Other apache opensource mq, kafka/rocketmq/pulsar maybe referred to.> >> >> > And In my experience this also bring some extra-complexity about> > performance/brainsplitting issues when rebalance data.> >> > At 2020-03-02 20:33:31, "Martyn Taylor" <[hidden email]> wrote:> > >I think this is a great idea Franz.  The HA and replication components> > have> > >been a source of issues over the years.  Two problems I see are that 1)> > >there isn't a clean separation between the consensus mechanism and the> > >replication, and 2) the consensus algorithm used in Artemis isn't based on> > >any standard algorithm or a research paper.  Hence, all the issues that> > >were caught over the years due to various edge cases.  Integration with> > >ZooKeeper seems like the obvious solution (i.e. push the consensus logic> > >off to a third party lib).  I suspect though, this will be a considerable> > >amount of work and is likely to introduce new issues, so I'd proceed with> > >caution.> > >> > >Cheers> > >> > >> > >> > >On Mon, Mar 2, 2020 at 8:28 AM nigro_franz <[hidden email]> wrote:> > >> > >> Hi folks,> > >>> > >> especially due to the requirements of the current Artemis quorum vote> > >> algorithm, we've thought to re-implementing it with a different focus in> > >> mind:> > >> 1) to make it pluggable (eg by using the many Raft implementations,> > >> ZooKeeper or others)> > >> 2) to cleanly separate the election phase and cluster member states (ie> > it> > >> should be the Topology shared between them)> > >> 3) to simplify most common setups in both amount of configuration and> > >> requirements (eg "witness" nodes could be implemented to get a minimum> > 2*n> > >> +1 quorum of nodes instead of forcing 2*n + 1 master-backup pairs)> > >> 4) [OPTIONALLY] to reduce/eliminate implicit "good practices" in term of> > >> order of actions to be performed on nodes in "special states" eg proper> > >> restart sequence after failover or similar cases> > >> 5) [OPTIONALLY] to make shared-store and replication behaviour more> > >> similar:> > >> journal's presence should be the only difference between the 2s> > >>> > >> A proposal of steps to be followed to get this:> > >> 1) abstract away the current quorum vote: it requires extra-care because> > >> the> > >> logic is melted together with the replication/clustering behaviour> > >> 2) refactor it in order to separate election phase and cluster member> > >> states> > >> 3) implement a RI version of such APIs> > >>> > >> Post-actions to help people adopt it, but need to be thought upfront:> > >> 1) a clean upgrade path for current HA replication users> > >> 2) deprecate or integrate the current HA replication into the new> > version> > >>> > >> I've opened this here because many of the HA replication users are devs> > too> > >> and given that this isn't yet implemented: we're stull in the> > >> design/proposal phase, so anyone that want to express their> > >> ideas/opinions/POC on this, is invited to talk here ;)> > >>> > >> Cheers,> > >> Franz> > >>> > >>> > >>> > >>> > >>> > >> --> > >> Sent from:> > >> http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html> > >>> >-- Clebert Suconic-- Clebert Suconic-- Clebert Suconic-- Clebert Suconic



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: Re: [DISCUSSION] New Quorum vote pluggable implementation

michael.andre.pearce
I was more refereeing to supporting the plugin.Concerns raised were:Other systems client dependencies e.g. kafka client lib or Prometheus lib in core broker, getting out of dateExposure of cves in core broker that every plugin with its dependencies could bring.Plugin code becoming un-maintained e.g. the older vertex ones that were removed. People seemed to be against having plugins being in broker and then lapsing in maintenance.All I care for is we have a solution that we agree on and stop having to revisit this, and its a simply process to contribute new ones. I disagree strongly that it should be case by case, we should have an agreed place and process.Both the above can be sorted by either being simply accepted risk, or by being separately maintained in a seperate plugin repo.My personal is take the risk as will be also easier to deliver.But I understand the concerns of others, it might be easier to go the new git repo for plugins (i see this alot in other projects) Sent from my Samsung Galaxy smartphone.
-------- Original message --------From: Clebert Suconic <[hidden email]> Date: 18/03/2020  01:25  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation >> no longer no one supports it  <<I think this is just apache license here.As long as you have community around it.. code is maintained.. it'sall fine. I mean... in apache terms you guarantee things as in apachelicense. I try to do good work on the open source community, fixingbugs.. everything... but as you know the limits of the apache licenseterms.The term support here I think applies to third parties.. companieslisted here, (Including the company I work for... Red Hat / IBM).:https://activemq.apache.org/support.htmlIt will be up to them what they consider supported or not.. and thatgoes both for plugins or core functionalities. Company X may decide tonot support certain functionality as in providing you patches, help..etc. and you would be on your own on such functionality like any otherapache license.that's pretty standard... not just for ActiveMQ.. I think that'scommon for any other project.On Tue, Mar 17, 2020 at 9:05 PM michael.andre.pearce<[hidden email]> wrote:>> I totally get it.As said im for the camp that we have plugin modules in the broker. And if someone wants to commit one, we allow it without need to votes etc and all this discussion every time.(e.g. Justin and I shouldnt need to make discussion thread, it should just be accepted and added)If something gets un-maintained (some peoples concerns) we simply remove it / attic it. The fact its un-maintained means no longer no one supports it But its for everyone to agree to this more open way of working.....not just me and you.An alternative approach if people are concerned these are in broker is that we have a separate plugin repo that we stick them all, and release...but this then makes it harder for end users to enable or use...Sent from my Samsung Galaxy smartphone.> -------- Original message --------From: Clebert Suconic <[hidden email]> Date: 18/03/2020  00:57  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation Just like anything else in apache... committers/community can ask orvote features out. it doesn't even need to be a plugin.I don't mean to hijack this discussion away from Quorum.. although itwould affect the implementation.. .we can keep a plugin's area in thebroker... and activate them based on the CLI / config.I just want to make sure we can unlock this going forward.On Tue, Mar 17, 2020 at 8:53 PM michael.andre.pearce<[hidden email]> wrote:>> Well the point is without us having sorted the issue, then this will keep occuring.I am for one in favour we relax a little and if someone wants to add a plugin we simply allow it. Why should there be any filter.Sent from my Samsung Galaxy smartphone.> -------- Original message --------From: Clebert Suconic <[hidden email]> Date: 18/03/2020  00:38  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation It's on a case by case basis... What we can't do is transform everynew core functionality (such as monitoring. .it's pretty standard /core functionality) into a plug-in, and then block its implementationbased on a past discussion.I say, if there's a plugin, there should be at least oneimplementation on the broker.. that goes for Monitoring... we cansupport more.. but this has to have an implementation.You are a committer, if you want to have your kafka plugin again, justraise a discussion.. but we can't block every new functionality basedon that baggage.On Tue, Mar 17, 2020 at 7:35 PM michael.andre.pearce<[hidden email]> wrote:>> I think the issue with plugins is that whats the rule to include one or not.You know exactly where im coming from there... re kafka plugin.Im totally for a number of plugins being added. Including the kafka one.But we have to have a clear rule of what will be allowed or not and a place to maintain.Sent from my Samsung Galaxy smartphone.> -------- Original message --------From: Clebert Suconic <[hidden email]> Date: 16/03/2020  01:20  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation We can make pluggable as much as you want, but going forward I wouldlike us to change how we have been treating plugins.Because of other discussions, plugins are not really implemented aspart of the community.We should elect one implementation we can make it and implement itwell.... and bring it as part of the community. People can add moreimplementations as part of the open source project.This is probably the same with Micrometer. Justin implementedMicrometer and the implementation is not part of activemq. I think weshould change that going forward.This will be a 3.0, so we should definitely improve how we do pluginsgoing forward.On Fri, Mar 13, 2020 at 3:42 AM michael.andre.pearce<[hidden email]> wrote:>> I think a main thing here is a pluggable api.If someone already has etcd, zookeeper, consul etc. In their infrastructure they should be able to reuse.Like wise if you have multiple bulkheaded broker sets it be useful to be able to reuse the same quorum compontent e.g. in ZK that would be a different base root on the tree path per broker set.Likewise that said we should have some out the box support plugins from the community. Obviously what ever we do whilst it maybe a break change e.g. a 3.0 release there must be a migration solution (accepted that it may need outage) for existing brokers that works via the pluggable api not to a specific solution.Also anything from a management api pov should also be visible in the web management console. Sent from my Samsung Galaxy smartphone.> -------- Original message --------From: Clebert Suconic <[hidden email]> Date: 09/03/2020  14:55  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation I think we should have a management component, that runs outside ofthe broker and would manage quorum.That way you can have the quorum running outside of the broker itself.which would improve the need of multiple brokers to manage the quorum.You just need Quorum Managers in distinct places.I had recently worked with a software called Ceph... And Ceph has theconcept of managers working away from their "broker" (it's not abroker.. it's a DB, but in a sense it's the same concept here). Ithink we should do the same.I have talked to Franz and other guys in private.. and it seemseverybody these days mention raft consensus algorithm.  Perhaps weshould look into it.. and make it pluggable. I believe there's JRaftworking on top of JGroups already.If we make this pluggable and make it a manager a separate process,this will be a big win IMO.On Tue, Mar 3, 2020 at 9:55 AM Andy Taylor <[hidden email]> wrote:>> Personally I wouldn't use Zookeeper, I think there are better options. Also> looks like Kafka are replacing it as well. Saying that, it doesn't really> matter what is used, the main thing is we need to remove the burden of> providing consensus away from the broker.>> It would make sense to make it pluggable.>> A>> On Mon, 2 Mar 2020 at 19:19, KimmKing <[hidden email]> wrote:>> > Can't agree any more.> > The HA&Replication is more and more important for a modern messaging> > system.> > Other apache opensource mq, kafka/rocketmq/pulsar maybe referred to.> >> >> > And In my experience this also bring some extra-complexity about> > performance/brainsplitting issues when rebalance data.> >> > At 2020-03-02 20:33:31, "Martyn Taylor" <[hidden email]> wrote:> > >I think this is a great idea Franz.  The HA and replication components> > have> > >been a source of issues over the years.  Two problems I see are that 1)> > >there isn't a clean separation between the consensus mechanism and the> > >replication, and 2) the consensus algorithm used in Artemis isn't based on> > >any standard algorithm or a research paper.  Hence, all the issues that> > >were caught over the years due to various edge cases.  Integration with> > >ZooKeeper seems like the obvious solution (i.e. push the consensus logic> > >off to a third party lib).  I suspect though, this will be a considerable> > >amount of work and is likely to introduce new issues, so I'd proceed with> > >caution.> > >> > >Cheers> > >> > >> > >> > >On Mon, Mar 2, 2020 at 8:28 AM nigro_franz <[hidden email]> wrote:> > >> > >> Hi folks,> > >>> > >> especially due to the requirements of the current Artemis quorum vote> > >> algorithm, we've thought to re-implementing it with a different focus in> > >> mind:> > >> 1) to make it pluggable (eg by using the many Raft implementations,> > >> ZooKeeper or others)> > >> 2) to cleanly separate the election phase and cluster member states (ie> > it> > >> should be the Topology shared between them)> > >> 3) to simplify most common setups in both amount of configuration and> > >> requirements (eg "witness" nodes could be implemented to get a minimum> > 2*n> > >> +1 quorum of nodes instead of forcing 2*n + 1 master-backup pairs)> > >> 4) [OPTIONALLY] to reduce/eliminate implicit "good practices" in term of> > >> order of actions to be performed on nodes in "special states" eg proper> > >> restart sequence after failover or similar cases> > >> 5) [OPTIONALLY] to make shared-store and replication behaviour more> > >> similar:> > >> journal's presence should be the only difference between the 2s> > >>> > >> A proposal of steps to be followed to get this:> > >> 1) abstract away the current quorum vote: it requires extra-care because> > >> the> > >> logic is melted together with the replication/clustering behaviour> > >> 2) refactor it in order to separate election phase and cluster member> > >> states> > >> 3) implement a RI version of such APIs> > >>> > >> Post-actions to help people adopt it, but need to be thought upfront:> > >> 1) a clean upgrade path for current HA replication users> > >> 2) deprecate or integrate the current HA replication into the new> > version> > >>> > >> I've opened this here because many of the HA replication users are devs> > too> > >> and given that this isn't yet implemented: we're stull in the> > >> design/proposal phase, so anyone that want to express their> > >> ideas/opinions/POC on this, is invited to talk here ;)> > >>> > >> Cheers,> > >> Franz> > >>> > >>> > >>> > >>> > >>> > >> --> > >> Sent from:> > >> http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html> > >>> >-- Clebert Suconic-- Clebert Suconic-- Clebert Suconic-- Clebert Suconic-- Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: Re: [DISCUSSION] New Quorum vote pluggable implementation

clebertsuconic
In reply to this post by clebertsuconic
Fair enough with separate github plugin. I think we have consensus
here from previous discussions to go ahead and create such repo.

We should probably create a separate discussion now on a plugin repo,
and move back to the quorum discussion here. But I believe we have a
solution in place?

On Tue, Mar 17, 2020 at 9:46 PM michael.andre.pearce
<[hidden email]> wrote:
>
> I was more refereeing to supporting the plugin.Concerns raised were:Other systems client dependencies e.g. kafka client lib or Prometheus lib in core broker, getting out of dateExposure of cves in core broker that every plugin with its dependencies could bring.Plugin code becoming un-maintained e.g. the older vertex ones that were removed. People seemed to be against having plugins being in broker and then lapsing in maintenance.All I care for is we have a solution that we agree on and stop having to revisit this, and its a simply process to contribute new ones. I disagree strongly that it should be case by case, we should have an agreed place and process.Both the above can be sorted by either being simply accepted risk, or by being separately maintained in a seperate plugin repo.My personal is take the risk as will be also easier to deliver.But I understand the concerns of others, it might be easier to go the new git repo for plugins (i see this alot in other projects) Sent from my Samsung Galaxy smartphone.
> -------- Original message --------From: Clebert Suconic <[hidden email]> Date: 18/03/2020  01:25  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation >> no longer no one supports it  <<I think this is just apache license here.As long as you have community around it.. code is maintained.. it'sall fine. I mean... in apache terms you guarantee things as in apachelicense. I try to do good work on the open source community, fixingbugs.. everything... but as you know the limits of the apache licenseterms.The term support here I think applies to third parties.. companieslisted here, (Including the company I work for... Red Hat / IBM).:https://activemq.apache.org/support.htmlIt will be up to them what they consider supported or not.. and thatgoes both for plugins or core functionalities. Company X may decide tonot support certain functionality as in providing you patches, help..etc. and you would be on your own on such functionality like any otherapache license.that's pretty standard... not just for ActiveMQ.. I think that'scommon for any other project.On Tue, Mar 17, 2020 at 9:05 PM michael.andre.pearce<[hidden email]> wrote:>> I totally get it.As said im for the camp that we have plugin modules in the broker. And if someone wants to commit one, we allow it without need to votes etc and all this discussion every time.(e.g. Justin and I shouldnt need to make discussion thread, it should just be accepted and added)If something gets un-maintained (some peoples concerns) we simply remove it / attic it. The fact its un-maintained means no longer no one supports it But its for everyone to agree to this more open way of working.....not just me and you.An alternative approach if people are concerned these are in broker is that we have a separate plugin repo that we stick them all, and release...but this then makes it harder for end users to enable or use...Sent from my Samsung Galaxy smartphone.> -------- Original message --------From: Clebert Suconic <[hidden email]> Date: 18/03/2020  00:57  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation Just like anything else in apache... committers/community can ask orvote features out. it doesn't even need to be a plugin.I don't mean to hijack this discussion away from Quorum.. although itwould affect the implementation.. .we can keep a plugin's area in thebroker... and activate them based on the CLI / config.I just want to make sure we can unlock this going forward.On Tue, Mar 17, 2020 at 8:53 PM michael.andre.pearce<[hidden email]> wrote:>> Well the point is without us having sorted the issue, then this will keep occuring.I am for one in favour we relax a little and if someone wants to add a plugin we simply allow it. Why should there be any filter.Sent from my Samsung Galaxy smartphone.> -------- Original message --------From: Clebert Suconic <[hidden email]> Date: 18/03/2020  00:38  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation It's on a case by case basis... What we can't do is transform everynew core functionality (such as monitoring. .it's pretty standard /core functionality) into a plug-in, and then block its implementationbased on a past discussion.I say, if there's a plugin, there should be at least oneimplementation on the broker.. that goes for Monitoring... we cansupport more.. but this has to have an implementation.You are a committer, if you want to have your kafka plugin again, justraise a discussion.. but we can't block every new functionality basedon that baggage.On Tue, Mar 17, 2020 at 7:35 PM michael.andre.pearce<[hidden email]> wrote:>> I think the issue with plugins is that whats the rule to include one or not.You know exactly where im coming from there... re kafka plugin.Im totally for a number of plugins being added. Including the kafka one.But we have to have a clear rule of what will be allowed or not and a place to maintain.Sent from my Samsung Galaxy smartphone.> -------- Original message --------From: Clebert Suconic <[hidden email]> Date: 16/03/2020  01:20  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation We can make pluggable as much as you want, but going forward I wouldlike us to change how we have been treating plugins.Because of other discussions, plugins are not really implemented aspart of the community.We should elect one implementation we can make it and implement itwell.... and bring it as part of the community. People can add moreimplementations as part of the open source project.This is probably the same with Micrometer. Justin implementedMicrometer and the implementation is not part of activemq. I think weshould change that going forward.This will be a 3.0, so we should definitely improve how we do pluginsgoing forward.On Fri, Mar 13, 2020 at 3:42 AM michael.andre.pearce<[hidden email]> wrote:>> I think a main thing here is a pluggable api.If someone already has etcd, zookeeper, consul etc. In their infrastructure they should be able to reuse.Like wise if you have multiple bulkheaded broker sets it be useful to be able to reuse the same quorum compontent e.g. in ZK that would be a different base root on the tree path per broker set.Likewise that said we should have some out the box support plugins from the community. Obviously what ever we do whilst it maybe a break change e.g. a 3.0 release there must be a migration solution (accepted that it may need outage) for existing brokers that works via the pluggable api not to a specific solution.Also anything from a management api pov should also be visible in the web management console. Sent from my Samsung Galaxy smartphone.> -------- Original message --------From: Clebert Suconic <[hidden email]> Date: 09/03/2020  14:55  (GMT+00:00) To: [hidden email] Subject: Re: Re: [DISCUSSION] New Quorum vote pluggable implementation I think we should have a management component, that runs outside ofthe broker and would manage quorum.That way you can have the quorum running outside of the broker itself.which would improve the need of multiple brokers to manage the quorum.You just need Quorum Managers in distinct places.I had recently worked with a software called Ceph... And Ceph has theconcept of managers working away from their "broker" (it's not abroker.. it's a DB, but in a sense it's the same concept here). Ithink we should do the same.I have talked to Franz and other guys in private.. and it seemseverybody these days mention raft consensus algorithm.  Perhaps weshould look into it.. and make it pluggable. I believe there's JRaftworking on top of JGroups already.If we make this pluggable and make it a manager a separate process,this will be a big win IMO.On Tue, Mar 3, 2020 at 9:55 AM Andy Taylor <[hidden email]> wrote:>> Personally I wouldn't use Zookeeper, I think there are better options. Also> looks like Kafka are replacing it as well. Saying that, it doesn't really> matter what is used, the main thing is we need to remove the burden of> providing consensus away from the broker.>> It would make sense to make it pluggable.>> A>> On Mon, 2 Mar 2020 at 19:19, KimmKing <[hidden email]> wrote:>> > Can't agree any more.> > The HA&Replication is more and more important for a modern messaging> > system.> > Other apache opensource mq, kafka/rocketmq/pulsar maybe referred to.> >> >> > And In my experience this also bring some extra-complexity about> > performance/brainsplitting issues when rebalance data.> >> > At 2020-03-02 20:33:31, "Martyn Taylor" <[hidden email]> wrote:> > >I think this is a great idea Franz.  The HA and replication components> > have> > >been a source of issues over the years.  Two problems I see are that 1)> > >there isn't a clean separation between the consensus mechanism and the> > >replication, and 2) the consensus algorithm used in Artemis isn't based on> > >any standard algorithm or a research paper.  Hence, all the issues that> > >were caught over the years due to various edge cases.  Integration with> > >ZooKeeper seems like the obvious solution (i.e. push the consensus logic> > >off to a third party lib).  I suspect though, this will be a considerable> > >amount of work and is likely to introduce new issues, so I'd proceed with> > >caution.> > >> > >Cheers> > >> > >> > >> > >On Mon, Mar 2, 2020 at 8:28 AM nigro_franz <[hidden email]> wrote:> > >> > >> Hi folks,> > >>> > >> especially due to the requirements of the current Artemis quorum vote> > >> algorithm, we've thought to re-implementing it with a different focus in> > >> mind:> > >> 1) to make it pluggable (eg by using the many Raft implementations,> > >> ZooKeeper or others)> > >> 2) to cleanly separate the election phase and cluster member states (ie> > it> > >> should be the Topology shared between them)> > >> 3) to simplify most common setups in both amount of configuration and> > >> requirements (eg "witness" nodes could be implemented to get a minimum> > 2*n> > >> +1 quorum of nodes instead of forcing 2*n + 1 master-backup pairs)> > >> 4) [OPTIONALLY] to reduce/eliminate implicit "good practices" in term of> > >> order of actions to be performed on nodes in "special states" eg proper> > >> restart sequence after failover or similar cases> > >> 5) [OPTIONALLY] to make shared-store and replication behaviour more> > >> similar:> > >> journal's presence should be the only difference between the 2s> > >>> > >> A proposal of steps to be followed to get this:> > >> 1) abstract away the current quorum vote: it requires extra-care because> > >> the> > >> logic is melted together with the replication/clustering behaviour> > >> 2) refactor it in order to separate election phase and cluster member> > >> states> > >> 3) implement a RI version of such APIs> > >>> > >> Post-actions to help people adopt it, but need to be thought upfront:> > >> 1) a clean upgrade path for current HA replication users> > >> 2) deprecate or integrate the current HA replication into the new> > version> > >>> > >> I've opened this here because many of the HA replication users are devs> > too> > >> and given that this isn't yet implemented: we're stull in the> > >> design/proposal phase, so anyone that want to express their> > >> ideas/opinions/POC on this, is invited to talk here ;)> > >>> > >> Cheers,> > >> Franz> > >>> > >>> > >>> > >>> > >>> > >> --> > >> Sent from:> > >> http://activemq.2283324.n4.nabble.com/ActiveMQ-Dev-f2368404.html> > >>> >-- Clebert Suconic-- Clebert Suconic-- Clebert Suconic-- Clebert Suconic-- Clebert Suconic



--
Clebert Suconic
12