JMS QueueBrowser does not return actual number of messages in the queue

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

JMS QueueBrowser does not return actual number of messages in the queue

cooshal
Hi:

I am using ActiveMQ version 5.15.8

I have the following code fragment, which is basically there to count the
number of messages in a particular queue (here, it is tst.errors)

brokerConnection = connectionFactory.createConnection();
session = this.createSession(brokerConnection);
brokerConnection.start();

if (selector == null) {
    selector = "";
}

Queue queue = session.createQueue("tst.errors");
queueBrowser = session.createBrowser(queue, selector);

Enumeration e = queueBrowser.getEnumeration();
while (e.hasMoreElements()) {
    e.nextElement();
    queueEntriesCount++;
}

But, this does not retrieve messages more than 400. I knew that this was a
problem with the Jolokia API of ActiveMQ. But, I did not realize that there
was similar limitation with JMS API as well. Is there a way to retrieve all
the messages via JMS API?

Regards,
Cooshal.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: JMS QueueBrowser does not return actual number of messages in the queue

tabish121@gmail.com
On 6/19/19 12:13 PM, cooshal wrote:

> Hi:
>
> I am using ActiveMQ version 5.15.8
>
> I have the following code fragment, which is basically there to count the
> number of messages in a particular queue (here, it is tst.errors)
>
> brokerConnection = connectionFactory.createConnection();
> session = this.createSession(brokerConnection);
> brokerConnection.start();
>
> if (selector == null) {
>      selector = "";
> }
>
> Queue queue = session.createQueue("tst.errors");
> queueBrowser = session.createBrowser(queue, selector);
>
> Enumeration e = queueBrowser.getEnumeration();
> while (e.hasMoreElements()) {
>      e.nextElement();
>      queueEntriesCount++;
> }
>
> But, this does not retrieve messages more than 400.

This is expected as the default maxBrowsePage size configuration is
400.  See:

http://activemq.apache.org/per-destination-policies.html


> I knew that this was a
> problem with the Jolokia API of ActiveMQ. But, I did not realize that there
> was similar limitation with JMS API as well. Is there a way to retrieve all
> the messages via JMS API?
>
> Regards,
> Cooshal.
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>

--
Tim Bish

Reply | Threaded
Open this post in threaded view
|

Re: JMS QueueBrowser does not return actual number of messages in the queue

cooshal
Hi:

thank you for the clarification.

Is there a way to retrieve the queue stats? I wanted to retrieve the number
of messages, pending messages, etc. similar to those shown in activemq
console.

Regards,
Cooshal.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: JMS QueueBrowser does not return actual number of messages in the queue

tabish121@gmail.com
On 6/19/19 1:18 PM, cooshal wrote:
> Hi:
>
> thank you for the clarification.
>
> Is there a way to retrieve the queue stats? I wanted to retrieve the number
> of messages, pending messages, etc. similar to those shown in activemq
> console.

The only really reliable means would be to use JMX and ask the MBeans
for the queue count


>
> Regards,
> Cooshal.
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>

--
Tim Bish

Reply | Threaded
Open this post in threaded view
|

Re: JMS QueueBrowser does not return actual number of messages in the queue

jbonofre
In reply to this post by cooshal
Hi,

Just use JMX on the destination, you have lot of details (QueueSize,
cursor, memory, etc).

You can use Decanter to collect the JMX metrics.

Regards
JB

On 19/06/2019 19:18, cooshal wrote:

> Hi:
>
> thank you for the clarification.
>
> Is there a way to retrieve the queue stats? I wanted to retrieve the number
> of messages, pending messages, etc. similar to those shown in activemq
> console.
>
> Regards,
> Cooshal.
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: JMS QueueBrowser does not return actual number of messages in the queue

cooshal
In reply to this post by tabish121@gmail.com
Hi Tim and JB:

thank you.

I will check this out. Before using JMS API, I had tried Jolokia, and I
faced similar issues, so I had posted a question on StackOverflow about it
(https://stackoverflow.com/questions/55708890/is-it-possible-to-retrieve-more-than-400-messages-from-activemq-queue-via-joloki).
There I got the suggestion to switch to JMS API.

I had even tried JMX, but I struggled to connect to ActiveMQ JMX via
JConsole. This probably happened because I had multiple karaf instances
running on the same machine and could be that the JMX ports collided (this
is just my shallow speculation).

Also, I was not able to connect the JMX from Java. I will try this again.
Thank you for pointing this out.

Regards,
Cooshal.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: JMS QueueBrowser does not return actual number of messages in the queue

cooshal
Hi again:

I have one weird update to this issue.

I am using 5.15.8 locally, and 5.15.4 in production.

Strangely, I can get the actual messages count in the production
environment, which was definitely more than 400 (e.g. 3500).

Does any broker specific config parameter lets us retrieve messages more
than 400 via JMS API? My broker config looks normal to me:

<beans
  xmlns="http://www.springframework.org/schema/beans"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans.xsd
  http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd">

   
    <bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
        <property name="locations">
            <value>file:${activemq.conf}/credentials.properties</value>
        </property>
    </bean>

   
    <bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
          lazy-init="false" scope="singleton"
          init-method="start" destroy-method="stop">
    </bean>

   
    <broker xmlns="http://activemq.apache.org/schema/core"
brokerName="localhost" dataDirectory="${activemq.data}">

        <destinationPolicy>
            <policyMap>
              <policyEntries>
                <policyEntry topic=">" >
                   
                  <pendingMessageLimitStrategy>
                    <constantPendingMessageLimitStrategy limit="1000"/>
                  </pendingMessageLimitStrategy>
                </policyEntry>
              </policyEntries>
            </policyMap>
        </destinationPolicy>


       
        <managementContext>
            <managementContext createConnector="false"/>
        </managementContext>

       
        <persistenceAdapter>
            <kahaDB directory="${activemq.data}/kahadb"/>
        </persistenceAdapter>


         
          <systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage percentOfJvmHeap="70" />
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="100 gb"/>
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="50 gb"/>
                </tempUsage>
            </systemUsage>
        </systemUsage>

       
        <transportConnectors>
           
            <transportConnector name="openwire"
uri="tcp://0.0.0.0:61616?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600"/>
            <transportConnector name="amqp"
uri="amqp://0.0.0.0:5672?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600"/>
            <transportConnector name="stomp"
uri="stomp://0.0.0.0:61613?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600"/>
            <transportConnector name="mqtt"
uri="mqtt://0.0.0.0:1883?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600"/>
            <transportConnector name="ws"
uri="ws://0.0.0.0:61614?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600"/>
        </transportConnectors>

       
        <shutdownHooks>
            <bean xmlns="http://www.springframework.org/schema/beans"
class="org.apache.activemq.hooks.SpringContextHook" />
        </shutdownHooks>

    </broker>

   
    <import resource="jetty.xml"/>

</beans>

Regards,
Cooshal.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: JMS QueueBrowser does not return actual number of messages in the queue

cooshal
Hi again:

I checked further stuffs on this issue, and I think I am not able to fetch
JMS Messages > 400, because of optimizedDispatch="true" in the policyEntry
configuration of the broker. I had it as below:

`<policyEntry queue=">" optimizedDispatch='true'/>`

This was a recommended setting for scaling broker service.

Could it be because of this config, that this issue came up?

Regards,
Cooshal.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: JMS QueueBrowser does not return actual number of messages in the queue

gtully
there is a policyEntry maxBrowsePageSize with default value 400,
increase that value to see a different result.

<policyEntry queue=">" optimizedDispatch='true' maxBrowsePageSize="4000" />`

On Mon, 24 Jun 2019 at 16:11, cooshal <[hidden email]> wrote:

>
> Hi again:
>
> I checked further stuffs on this issue, and I think I am not able to fetch
> JMS Messages > 400, because of optimizedDispatch="true" in the policyEntry
> configuration of the broker. I had it as below:
>
> `<policyEntry queue=">" optimizedDispatch='true'/>`
>
> This was a recommended setting for scaling broker service.
>
> Could it be because of this config, that this issue came up?
>
> Regards,
> Cooshal.
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: JMS QueueBrowser does not return actual number of messages in the queue

cooshal
Hi:

ahh ok. then that was caused by optimizedDispatch, right?

Thanks for the clarification.

Regards,
Cooshal.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: JMS QueueBrowser does not return actual number of messages in the queue

cooshal
In reply to this post by gtully
Hi again:

so, does that also mean that my broker will be able to fetch upto 4000
messages at a time? If so, I guess I will run out of connections, right?

Or, will that be managed by ActiveMQ? I will try to play with these settings
and get back.

Regards,
Cooshal.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: JMS QueueBrowser does not return actual number of messages in the queue

cooshal
In reply to this post by gtully
Hi again @gtully,

I have a query again.

What if I do not use optimizedDispatch? How does optimizedDispatch help in
scaling. I guess, it does by limiting the message window size. But,
alternately, does it affect the broker's performance?

Any clarification on this would be very helpful.

Regards,
Cooshal.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: JMS QueueBrowser does not return actual number of messages in the queue

gtully
with optimizedDispatch=true, the queue dispatch thread is redundant
(the thread that reads a message does the dispatch) and does not get
to service a browse repeatedly.
it turns a browse into a one shot operation.
The benefits of optimizedDispatch=true are really only apparent with
straight through processing when thread context switching is the
bottleneck. If you have really large number of destinations, then it
may help but you would have to ask your computer to be sure.

On Tue, 25 Jun 2019 at 13:33, cooshal <[hidden email]> wrote:

>
> Hi again @gtully,
>
> I have a query again.
>
> What if I do not use optimizedDispatch? How does optimizedDispatch help in
> scaling. I guess, it does by limiting the message window size. But,
> alternately, does it affect the broker's performance?
>
> Any clarification on this would be very helpful.
>
> Regards,
> Cooshal.
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: JMS QueueBrowser does not return actual number of messages in the queue

cooshal
thank you for the explanation.

well, right now I have around 500+ destinations (queues and topics). This is
about to grow further, as my number of karaf instances (with camel routes)
accessing the broker grows. Alongside scaling, as many karaf instances, I
also need to ensure that my messages are not stuck in the queue due to slow
processing or they are fetched as quickly as possible.

Currently, I have around 200-300 messages per second.

But, I guess with optimizedDispatch="true", the CPU load will also be a bit
less, right? I am exploring the performance numbers manually right now.

Regards,
Cooshal.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html