How to optimize performance of multiple producers and consumers

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

How to optimize performance of multiple producers and consumers

nhrdls
Greetings,

We are on version 5.10 - recently migrated, but don't think my problem
is version specific.

In short, it boils down to can I evaluate the selector information on
producer side? I need to make decision whether this is a good time to
place the job or not.

I have multiple producers and consumers using same queue. Producers can
be broadly categorized as live and batch consumers.

Between the time producer sends the message and consumer responds back
with results, the entity in question gets in "locked" state. Since the
consumers have finite capacity as message processing takes lot of time
(more than 5 minutes usually), I am trying to avoid locking of entities.

Our consumers already publish heartbeat on a topic which has lot of
useful data including number of jobs they are running and capacity they
are configured with. So I have some idea whether I can place the job or not.

The part I can not get around is selectors on the consumers. Consumers
prioritize messages from live users as long as number of concurrently
processed messages don't exceed the capacity defined. Batch jobs which
produce many more messages, however get lower priority.

Given the selector, can I some how evaluate the  selector on producer
side to check if consumer will process the the job quickly enough?

The consumer capacity is dynamic as operations team can disable some
consumers or the traffic pattern changes. So asking consumers via hear
beat seems to be best solution. I don't want consumers to sit idle, nor
I want to load up the messages and lock my entities for long time. I am
open to other ideas also

Regards,

Niranjan
Reply | Threaded
Open this post in threaded view
|

Re: How to optimize performance of multiple producers and consumers

Noel OConnor
I'd look at using camel routes embedded in activemq to control this as
you can retrieve queue depths to determine the processing approach.

Take a look at http://rajdavies.blogspot.com.au/2013/09/apache-camel-broker-component-for.html

On Wed, Jul 16, 2014 at 11:27 AM, Niranjan Rao <[hidden email]> wrote:

> Greetings,
>
> We are on version 5.10 - recently migrated, but don't think my problem is
> version specific.
>
> In short, it boils down to can I evaluate the selector information on
> producer side? I need to make decision whether this is a good time to place
> the job or not.
>
> I have multiple producers and consumers using same queue. Producers can be
> broadly categorized as live and batch consumers.
>
> Between the time producer sends the message and consumer responds back with
> results, the entity in question gets in "locked" state. Since the consumers
> have finite capacity as message processing takes lot of time (more than 5
> minutes usually), I am trying to avoid locking of entities.
>
> Our consumers already publish heartbeat on a topic which has lot of useful
> data including number of jobs they are running and capacity they are
> configured with. So I have some idea whether I can place the job or not.
>
> The part I can not get around is selectors on the consumers. Consumers
> prioritize messages from live users as long as number of concurrently
> processed messages don't exceed the capacity defined. Batch jobs which
> produce many more messages, however get lower priority.
>
> Given the selector, can I some how evaluate the  selector on producer side
> to check if consumer will process the the job quickly enough?
>
> The consumer capacity is dynamic as operations team can disable some
> consumers or the traffic pattern changes. So asking consumers via hear beat
> seems to be best solution. I don't want consumers to sit idle, nor I want to
> load up the messages and lock my entities for long time. I am open to other
> ideas also
>
> Regards,
>
> Niranjan