Message Group rebalancing

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Message Group rebalancing

sspind
Hi there,

a question related to message groups: is it possible to rebalance message groups when a new consumer registers with the broker?
In my setup I have two consumers that consume from the same queue where all messages have a message group ID. If one consumer loses connection all messages go to the other. If the consumer reconnects messages will still go to the other one. At least if the broker saw the respective group ID already. I know that new messages will again be given to either of the consumers. However, in my scenario there is only a relatively small set of "long lived" group IDs. So it can happen that all messages will be routed to one consumer while the other one is idle.

Looking forward to your replies!

sspind
Reply | Threaded
Open this post in threaded view
|

Re: Message Group rebalancing

Tim Bain
I've never heard of such a feature (but maybe someone else here has), but I
think it would be a great addition if it doesn't exist.  So if you don't
get a response within a couple days, I encourage you to submit an
enhancement request in JIRA.

Tim
On Nov 10, 2015 2:20 PM, "sspind" <[hidden email]> wrote:

> Hi there,
>
> a question related to message groups: is it possible to rebalance message
> groups when a new consumer registers with the broker?
> In my setup I have two consumers that consume from the same queue where all
> messages have a message group ID. If one consumer loses connection all
> messages go to the other. If the consumer reconnects messages will still go
> to the other one. At least if the broker saw the respective group ID
> already. I know that new messages will again be given to either of the
> consumers. However, in my scenario there is only a relatively small set of
> "long lived" group IDs. So it can happen that all messages will be routed
> to
> one consumer while the other one is idle.
>
> Looking forward to your replies!
>
> sspind
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Message-Group-rebalancing-tp4703823.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Message Group rebalancing

rajdavies
Nice feature - submit a JIRA!

> On 11 Nov 2015, at 14:28, Tim Bain <[hidden email]> wrote:
>
> I've never heard of such a feature (but maybe someone else here has), but I
> think it would be a great addition if it doesn't exist.  So if you don't
> get a response within a couple days, I encourage you to submit an
> enhancement request in JIRA.
>
> Tim
> On Nov 10, 2015 2:20 PM, "sspind" <[hidden email]> wrote:
>
>> Hi there,
>>
>> a question related to message groups: is it possible to rebalance message
>> groups when a new consumer registers with the broker?
>> In my setup I have two consumers that consume from the same queue where all
>> messages have a message group ID. If one consumer loses connection all
>> messages go to the other. If the consumer reconnects messages will still go
>> to the other one. At least if the broker saw the respective group ID
>> already. I know that new messages will again be given to either of the
>> consumers. However, in my scenario there is only a relatively small set of
>> "long lived" group IDs. So it can happen that all messages will be routed
>> to
>> one consumer while the other one is idle.
>>
>> Looking forward to your replies!
>>
>> sspind
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Message-Group-rebalancing-tp4703823.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>

Reply | Threaded
Open this post in threaded view
|

Re: Message Group rebalancing

Takawale, Pankaj
You could achieve it from your application logic. You could have your
producer or consumers break the JMSXGroupID stickiness by sending
JMSXGroupSeq=-1
message after producing or receiving N messages from the same groupID.
After stickiness break up, new consumers would get a chance to acquire the
groups.



On Wed, Nov 11, 2015 at 9:49 AM, Rob Davies <[hidden email]> wrote:

> Nice feature - submit a JIRA!
>
> > On 11 Nov 2015, at 14:28, Tim Bain <[hidden email]> wrote:
> >
> > I've never heard of such a feature (but maybe someone else here has),
> but I
> > think it would be a great addition if it doesn't exist.  So if you don't
> > get a response within a couple days, I encourage you to submit an
> > enhancement request in JIRA.
> >
> > Tim
> > On Nov 10, 2015 2:20 PM, "sspind" <[hidden email]> wrote:
> >
> >> Hi there,
> >>
> >> a question related to message groups: is it possible to rebalance
> message
> >> groups when a new consumer registers with the broker?
> >> In my setup I have two consumers that consume from the same queue where
> all
> >> messages have a message group ID. If one consumer loses connection all
> >> messages go to the other. If the consumer reconnects messages will
> still go
> >> to the other one. At least if the broker saw the respective group ID
> >> already. I know that new messages will again be given to either of the
> >> consumers. However, in my scenario there is only a relatively small set
> of
> >> "long lived" group IDs. So it can happen that all messages will be
> routed
> >> to
> >> one consumer while the other one is idle.
> >>
> >> Looking forward to your replies!
> >>
> >> sspind
> >>
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://activemq.2283324.n4.nabble.com/Message-Group-rebalancing-tp4703823.html
> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Message Group rebalancing

Tim Bain
https://issues.apache.org/jira/browse/AMQ-2106 already covers this request,
and in the comments there, Gary's opinion was that the only safe way to do
this was to have the producer close the group, as Pankaj suggests.
Unfortunately, the current implementation appears to begin dispatching
messages from before the close request to the new consumer instead of
letting the old consumer finish them:
https://issues.apache.org/jira/browse/AMQ-3027.  If you want that issue
fixed, add your vote to it in JIRA.

Tim

On Mon, Nov 16, 2015 at 10:38 AM, Takawale, Pankaj <
[hidden email]> wrote:

> You could achieve it from your application logic. You could have your
> producer or consumers break the JMSXGroupID stickiness by sending
> JMSXGroupSeq=-1
> message after producing or receiving N messages from the same groupID.
> After stickiness break up, new consumers would get a chance to acquire the
> groups.
>
>
>
> On Wed, Nov 11, 2015 at 9:49 AM, Rob Davies <[hidden email]> wrote:
>
> > Nice feature - submit a JIRA!
> >
> > > On 11 Nov 2015, at 14:28, Tim Bain <[hidden email]> wrote:
> > >
> > > I've never heard of such a feature (but maybe someone else here has),
> > but I
> > > think it would be a great addition if it doesn't exist.  So if you
> don't
> > > get a response within a couple days, I encourage you to submit an
> > > enhancement request in JIRA.
> > >
> > > Tim
> > > On Nov 10, 2015 2:20 PM, "sspind" <[hidden email]> wrote:
> > >
> > >> Hi there,
> > >>
> > >> a question related to message groups: is it possible to rebalance
> > message
> > >> groups when a new consumer registers with the broker?
> > >> In my setup I have two consumers that consume from the same queue
> where
> > all
> > >> messages have a message group ID. If one consumer loses connection all
> > >> messages go to the other. If the consumer reconnects messages will
> > still go
> > >> to the other one. At least if the broker saw the respective group ID
> > >> already. I know that new messages will again be given to either of the
> > >> consumers. However, in my scenario there is only a relatively small
> set
> > of
> > >> "long lived" group IDs. So it can happen that all messages will be
> > routed
> > >> to
> > >> one consumer while the other one is idle.
> > >>
> > >> Looking forward to your replies!
> > >>
> > >> sspind
> > >>
> > >>
> > >>
> > >> --
> > >> View this message in context:
> > >>
> >
> http://activemq.2283324.n4.nabble.com/Message-Group-rebalancing-tp4703823.html
> > >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> > >>
> >
> >
>