Consumer queues of virtual topic and the web console

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Consumer queues of virtual topic and the web console

krishy
We are using the stock ActiveMQ 5.10.0 configuration (with minor modifications for JMX). With this configuration we have run into a very strange issue every now and then with messages not being dispatched from the consumer queues of a virtual topic until we click 'browse' on the web console of that queue! On clicking browse, a flood of message is received by the consumers and then nothing. This problem, seemingly, pops up randomly and is not rectified until we restart ActiveMQ. What could explain this behaviour?

On restarting the ActiveMQ service, the message dispatch returns to normal and stays that way for some time.

The destinationPolicy configuration is (from the vanilla 5.10 setup):

<destinationPolicy>
            <policyMap>
              <policyEntries>
                <policyEntry topic=">" >
                   
                  <pendingMessageLimitStrategy>
                    <constantPendingMessageLimitStrategy limit="1000"/>
                  </pendingMessageLimitStrategy>
                </policyEntry>
              </policyEntries>
            </policyMap>
        </destinationPolicy>

We use Apache Camel for routing with the ActiveMQComponent (and a org.apache.activemq.pool.PooledConnectionFactory). The activemq client version is 5.5 (is this a problem?).

Thanks for any and all pointers!
Reply | Threaded
Open this post in threaded view
|

Re: Consumer queues of virtual topic and the web console

artnaseef
Does this happen without the JMX modifications?

That does sound odd.  One question comes to mind - does the queue actually exist in the broker at the time of the message dispatch?  Check the consumers - make sure they are active at the time the message is dispatched, and that the broker sees the consumer and the queue.

The reason the browser can change this operation is that clicking on the link to browse the queue will create it.  That's one "oddity" of virtual topics - that the messages are put in the queues based on the presence of the queue at the time of message dispatch to the Topic.

If that is the cause, it is possible to force creation of the consumer queues at broker startup through the activemq.xml file.
Reply | Threaded
Open this post in threaded view
|

Re: Consumer queues of virtual topic and the web console

krishy
I am able to reproduce this consistently. Messages don't arrive at the consumer of the virtual topic queue.
I have a simple test case in this maven project:

https://bitbucket.org/calvinkrishy/camel-jms-test

One should be able to run it with mvn camel:run. The test uses 5.10.0 as an embedded broker. There are two Camel routes. One sends messages to the virtual topic and another consumes from the queue of the virtual topic based on a selector.

These are the patterns I observe:

- Initially message show up at the consumer almost immediately.
- By the 350-400th message there is a marked slow-down. Messages arrive in batches at the consumer.
- After around the 700th message being sent, nothing comes to the consumer.

I have also tried it with stock ActiveMQ 5.10.0 installation (i.e. stand alone broker, default install with no changes) and can still see the problem. Even if I completely stop the program and start it again nothing arrives at the consumer. Only after I restart the broker does the behavior return to normal (and that too for a short time).

Any pointers?
Reply | Threaded
Open this post in threaded view
|

Re: Consumer queues of virtual topic and the web console

krishy
I made the virtual topic selectorAware and the problem went away. All messages are received by the consumer without much delays.

I was able to reproduce this only with queues as well. That is the messages were sent to and consumed from a queue based on a selector. After a while the consumer seemed to stop receiving messages. I

Though my problem is solved, I am curious what is controlling this behaviour. Why don't messages turn up from the queue after a while?

 
Reply | Threaded
Open this post in threaded view
|

Re: Consumer queues of virtual topic and the web console

Tim Bain
Since it sounds like you were able to create a simple setup to cause the
issue,  can you package it up, create a JIRA bug, and attach your test
scenario to the JIRA?  That way someone can step through it with a debugger
and figure out what's going on.
On Jan 23, 2015 10:40 AM, "krishy" <[hidden email]> wrote:

> I made the virtual topic selectorAware and the problem went away. All
> messages are received by the consumer without much delays.
>
> I was able to reproduce this only with queues as well. That is the messages
> were sent to and consumed from a queue based on a selector. After a while
> the consumer seemed to stop receiving messages. I
>
> Though my problem is solved, I am curious what is controlling this
> behaviour. Why don't messages turn up from the queue after a while?
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Consumer-queues-of-virtual-topic-and-the-web-console-tp4689995p4690349.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Consumer queues of virtual topic and the web console

Tim Bain
OK, so I finally got around to looking at your producer and consumer code
in BitBucket.  So it looks like you're intentionally only consuming half
the messages you produce; is that right?  If so, to what end?  You never
mentioned this particular characteristic of your test; any reason you
didn't?
On Jan 27, 2015 7:01 AM, "Tim Bain" <[hidden email]> wrote:

> Since it sounds like you were able to create a simple setup to cause the
> issue,  can you package it up, create a JIRA bug, and attach your test
> scenario to the JIRA?  That way someone can step through it with a debugger
> and figure out what's going on.
> On Jan 23, 2015 10:40 AM, "krishy" <[hidden email]> wrote:
>
>> I made the virtual topic selectorAware and the problem went away. All
>> messages are received by the consumer without much delays.
>>
>> I was able to reproduce this only with queues as well. That is the
>> messages
>> were sent to and consumed from a queue based on a selector. After a while
>> the consumer seemed to stop receiving messages. I
>>
>> Though my problem is solved, I am curious what is controlling this
>> behaviour. Why don't messages turn up from the queue after a while?
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Consumer-queues-of-virtual-topic-and-the-web-console-tp4689995p4690349.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Consumer queues of virtual topic and the web console

krishy
tbain98 wrote
OK, so I finally got around to looking at your producer and consumer code
in BitBucket.  So it looks like you're intentionally only consuming half
the messages you produce; is that right?  If so, to what end?  You never
mentioned this particular characteristic of your test; any reason you
didn't?
I am sorry, I should have been more clear about that. As I mentioned - "One sends messages to the virtual topic and another consumes from the queue of the virtual topic based on a selector." So, yes, the consumer does not care about all the messages sent to the virtualtopic/queue but is rather interested in only certain messages.

The simplification of a scenario we use this is to pick messages from a stream of events. The events are pumped into a virtual topic and having multiple consumers subscribed to the corresponding queue allows us to run multiple instances of the consumer application (and thereby load balance the downstream processing).
Reply | Threaded
Open this post in threaded view
|

Re: Consumer queues of virtual topic and the web console

Tim Bain
And what's your expect behavior for the messages that don't match the
filter for your consumer?  What do you anticipate happening to them with
this configuration?

On Tue, Jan 27, 2015 at 3:15 PM, krishy <[hidden email]> wrote:

> tbain98 wrote
> > OK, so I finally got around to looking at your producer and consumer code
> > in BitBucket.  So it looks like you're intentionally only consuming half
> > the messages you produce; is that right?  If so, to what end?  You never
> > mentioned this particular characteristic of your test; any reason you
> > didn't?
>
> I am sorry, I should have been more clear about that. As I mentioned - "One
> sends messages to the virtual topic and another consumes from the queue of
> the virtual topic based on a selector." So, yes, the consumer does not care
> about all the messages sent to the virtualtopic/queue but is rather
> interested in only certain messages.
>
> The simplification of a scenario we use this is to pick messages from a
> stream of events. The events are pumped into a virtual topic and having
> multiple consumers subscribed to the corresponding queue allows us to run
> multiple instances of the consumer application (and thereby load balance
> the
> downstream processing).
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Consumer-queues-of-virtual-topic-and-the-web-console-tp4689995p4690491.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Consumer queues of virtual topic and the web console

krishy
tbain98 wrote
And what's your expect behavior for the messages that don't match the
filter for your consumer?  What do you anticipate happening to them with
this configuration?
Since the messages I send are non-persistent I would expect the
messages that do not match the selector to stay in the queue until
ActiveMQ is restarted. But messages that match the selector being
delayed is what confuses me.
Reply | Threaded
Open this post in threaded view
|

Re: Consumer queues of virtual topic and the web console

krishy
I went ahead and created a ticket as well:

https://issues.apache.org/jira/browse/AMQ-5582
Reply | Threaded
Open this post in threaded view
|

Re: Consumer queues of virtual topic and the web console

Tim Bain
I see that Tim closed your JIRA as Not A Problem, but didn't explain why.
In case you weren't sure why this is expected behavior, here's an
explanation:

Messages get read from the broker's persistent store via a cursor (see
http://activemq.apache.org/message-cursors.html), which is assumed to
always have the next message any consumer might want.  (This is sometimes a
bad assumption, but it's the design.)  A cursor has a fixed size; once it's
full, a new message cannot be read into the cursor (and therefore delivered
to a consumer) until enough messages are read from the cursor to free up
space for the new message.  If none of the existing messages in the cursor
are read, nothing happens.

In your case, you're intentionally creating messages that won't be
consumed.  These messages build up in the cursor, and once it's full, none
of the new messages can get into the cursor so none of them can get
dispatched so none of them get consumed.  And for that reason all messages
stop flowing.

Tim's suggestion to tweak the cursor size wouldn't solve this problem, it
would just delay the onset, in the same way that increasing your JVM heap
size when you have a memory leak only delays the problem rather than
solving it.  The underlying issue is that you're doing something wrong:
you're publishing messages to a queue without ensuing that all of them will
be processed (consumed or discarded) in a timely manner.  So either make
sure your consumers have selectors that match all messages or configure
something to discard messages if they aren't consumed in a timely manner
(JMSExpiration plus the DiscardingDLQPlugin is what we use).

Tim
On Feb 11, 2015 10:22 PM, "krishy" <[hidden email]> wrote:

> I went ahead and created a ticket as well:
>
> https://issues.apache.org/jira/browse/AMQ-5582
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Consumer-queues-of-virtual-topic-and-the-web-console-tp4689995p4691427.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>