activemq 3000 messages can not be received

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

activemq 3000 messages can not be received

softwbc
I'm using ActiveMQ5.13.1 ,spring-jms-3.0.5 After running for some time, I found that there are 3000 message pendding. But other messages can be consumed. That the queue is not blocked is there anyone that knows why these messages are in pending message list in broker? and i cannot consume?
Reply | Threaded
Open this post in threaded view
|

Re: activemq 3000 messages can not be received

Tim Bain
How many consumers do you have on the queue, and how many messages have
been dispatched (but not consumed) to each?  And what prefetch size are
they using?  The default prefetch size is 1000; any chance you have three
slow consumers that are each sitting on a full prefetch buffer?

Are you seeing the pending count on the web console, or somewhere else?

And what version of ActiveMQ are you running?  That's important information
anytime you ask a question on this mailing list.

Tim

On Aug 10, 2016 9:50 PM, "softwbc" <[hidden email]> wrote:

> I'm using ActiveMQ5.13.1 ,spring-jms-3.0.5After running for some time, I
> found that there are 3000 message pendding.But other messages can be
> consumed. That the queue is not blockedis there anyone that knows why these
> messages are in pending message list in broker? and i cannot consume?
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/activemq-3000-messages-can-not-be-received-tp4715396.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: activemq 3000 messages can not be received

softwbc
I have two consumers and Every consumer has 4 threads.
I'm use two broker to from cluster:
<networkConnectors>
            <networkConnector uri="masterslave:(tcp://192.168.83.167:61616,tcp://192.168.83.168:61617)?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600" userName="admin" password="xinhua" duplex="true" conduitSubscriptions="false" />
        </networkConnectors>

There are 3000 message have dispatched on server168 and 2000 message on server167

Im using the default prefetch size

I adjusted the parameters being observed.

Thanks
Reply | Threaded
Open this post in threaded view
|

Re: activemq 3000 messages can not be received

Tim Bain
The fact that your number of dispatched messages is an exact multiple of
your prefetch size tells me that some of your threads (which sound like
they're each acting as a JMS consumer, for a total of 8 consumers; you can
confirm that via the brokers' web consoles) are probably slow or blocked,
so I'd look at your message-consuming logic to see if you've got something
that could go wrong there.  If you can take a thread dump, that might
provide more information.

Tim

On Fri, Aug 12, 2016 at 12:21 AM, softwbc <[hidden email]> wrote:

> I have two consumers and Every consumer has 4 threads.
> I'm use two broker to from cluster:
> <networkConnectors>
>             <networkConnector
> uri="masterslave:(tcp://192.168.83.167:61616,tcp://192.168.83.168:61617)?
> maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600"
> userName="admin" password="xinhua" duplex="true"
> conduitSubscriptions="false" />
>         </networkConnectors>
>
> There are 3000 message have dispatched on server168 and 2000 message on
> server167
>
> Im using the default prefetch size
>
> I adjusted the parameters being observed.
>
> Thanks
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/activemq-3000-messages-can-not-be-received-
> tp4715396p4715440.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: activemq 3000 messages can not be received

softwbc
This problem is only found 2 times, I have no dump file.
This is the configuration of the producer
mq.brokerURL=failover:(tcp://192.168.83.64:61616?wireFormat.maxInactivityDuration=0)&maxReconnectDelay=1000
mq.userName=admin
mq.password=123456
mq.maxConnections=1
mq.maximumActiveSessionPerConnection=500

<context:property-placeholder location="classpath*:/activemq.properties" />
        <bean id="connectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory" destroy-method="stop">
                <property name="connectionFactory">
                        <bean class="org.apache.activemq.ActiveMQConnectionFactory">
                                <property name="brokerURL" value="${mq.brokerURL}" />
                                <property name="userName" value="${mq.userName}" />
                                <property name="password" value="${mq.password}" />
                        </bean>
                </property>
                <property name="maxConnections" value="${mq.maxConnections}" />
                <property name="maximumActiveSessionPerConnection" value="${mq.maximumActiveSessionPerConnection}" />
        </bean>

        <bean id="queueDestination" class="org.apache.activemq.command.ActiveMQQueue">
                <constructor-arg index="0" value="xhGeek.demo" />
        </bean>
        <bean id="jmsQueueTemplate" class="org.springframework.jms.core.JmsTemplate">
                <property name="connectionFactory" ref="connectionFactory" />
                <property name="defaultDestination" ref="queueDestination" />
                <property name="receiveTimeout" value="10000"></property>
        </bean>

        <bean id="senderService" class="com.xinhuanet.mq.service.impl.SenderServiceImpl">
                <property name="jmsTemplate" ref="jmsQueueTemplate"></property>
                <property name="destinations">
                        <map>
                                <entry key="xhGeek.demo">
                                        <bean class="org.apache.activemq.command.ActiveMQQueue">
                                                <constructor-arg index="0" value="xhGeek.demo" />
                                        </bean>
                                </entry>
                                <entry key="xhGeek.test">
                                        <bean class="org.apache.activemq.command.ActiveMQQueue">
                                                <constructor-arg index="0" value="xhGeek.test" />
                                        </bean>
                                </entry>
                                <entry key="xhGeek.test2">
                                        <bean class="org.apache.activemq.command.ActiveMQQueue">
                                                <constructor-arg index="0" value="xhGeek.test2" />
                                        </bean>
                                </entry>
                                <entry key="xhGeek.msg-center.message">
                                        <bean class="org.apache.activemq.command.ActiveMQQueue">
                                                <constructor-arg index="0" value="xhGeek.msg-center.message" />
                                        </bean>
                                </entry>
                                <entry key="xhGeek.msg-center.fqremove">
                                        <bean class="org.apache.activemq.command.ActiveMQQueue">
                                                <constructor-arg index="0" value="xhGeek.msg-center.fqremove" />
                                        </bean>
                                </entry>
                                <entry key="xhGeek.relationship-center.changepoint">
                                        <bean class="org.apache.activemq.command.ActiveMQQueue">
                                                <constructor-arg index="0" value="xhGeek.relationship-center.changepoint" />
                                        </bean>
                                </entry>

                                .....   (About 30)

                                <entry key="xhMobile.credit">
                                        <bean class="org.apache.activemq.command.ActiveMQQueue">
                                                <constructor-arg index="0" value="xhMobile.credit" />
                                        </bean>
                                </entry>
                        </map>
                </property>
        </bean>

        <bean id="topicDestination" class="org.apache.activemq.command.ActiveMQTopic">
                <constructor-arg index="0" value="xhGeek-topic" />
        </bean>
        <bean id="jmsTopicTemplate" class="org.springframework.jms.core.JmsTemplate">
                <property name="connectionFactory" ref="connectionFactory" />
                <property name="defaultDestination" ref="topicDestination" />
                <property name="pubSubDomain" value="true" />
                <property name="receiveTimeout" value="10000" />
        </bean>
        <bean id="publisherService" class="com.xinhuanet.mq.service.impl.PublisherServiceImpl">
                <property name="jmsTemplate" ref="jmsTopicTemplate"></property>
                <property name="destinations">
                        <map>
                                <entry key="VirtualTopic.xhGeek.authSystem">
                                        <bean class="org.apache.activemq.command.ActiveMQTopic">
                                                <constructor-arg index="0" value="VirtualTopic.xhGeek.authSystem" />
                                        </bean>
                                </entry>
                                <entry key="VirtualTopic.xhInteract.picture">
                                        <bean class="org.apache.activemq.command.ActiveMQTopic">
                                                <constructor-arg index="0" value="VirtualTopic.xhInteract.picture" />
                                        </bean>
                                </entry>
                                <entry key="VirtualTopic.xhInteract.video">
                                        <bean class="org.apache.activemq.command.ActiveMQTopic">
                                                <constructor-arg index="0" value="VirtualTopic.xhInteract.video" />
                                        </bean>
                                </entry>
                                <entry key="VirtualTopic.xhGeek.test">
                                        <bean class="org.apache.activemq.command.ActiveMQTopic">
                                                <constructor-arg index="0" value="VirtualTopic.xhGeek.test" />
                                        </bean>
                                </entry>
                                <entry key="VirtualTopic.xhGeek.liveUserMGT">
                                        <bean class="org.apache.activemq.command.ActiveMQTopic">
                                                <constructor-arg index="0" value="VirtualTopic.xhGeek.liveUserMGT" />
                                        </bean>
                                </entry>
                                <entry key="xhGeek.testtopic">
                                        <bean class="org.apache.activemq.command.ActiveMQTopic">
                                                <constructor-arg index="0" value="xhGeek.testtopic" />
                                        </bean>
                                </entry>
                                <entry key="xhShop.staticHtml">
                                        <bean class="org.apache.activemq.command.ActiveMQTopic">
                                                <constructor-arg index="0" value="xhShop.staticHtml" />
                                        </bean>
                                </entry>
                                <entry key="VirtualTopic.xhGeek.notice">
                                        <bean class="org.apache.activemq.command.ActiveMQTopic">
                                                <constructor-arg index="0" value="VirtualTopic.xhGeek.notice" />
                                        </bean>
                                </entry>
                        </map>
                </property>
        </bean>
Reply | Threaded
Open this post in threaded view
|

Re: activemq 3000 messages can not be received

softwbc
In reply to this post by Tim Bain
and the consoumer configuration
    <bean id="connectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
                <property name="brokerURL" value="${mq.brokerURL}" />
                <property name="userName" value="${mq.userName}" />
                <property name="password" value="${mq.password}" />
    </bean>
        <bean id="receiveQueueDestination" class="org.apache.activemq.command.ActiveMQQueue">
                <constructor-arg index="0" value="${mq.queues}" />
        </bean>
        <bean id="receiveListenerService" class="com.xinhua.activemq.listener.ReceiveListenerService">
        </bean>
       
        <bean id="typeConverter" class="com.xinhua.activemq.converter.TypeConverter">
        </bean>
        <bean id="queueListener" class="com.xinhua.activemq.adapter.MessageListenerAdapter">
                <constructor-arg ref="receiveListenerService"></constructor-arg>
                <property name="defaultListenerMethod" value="receive" />
                <property name="messageConverter" ref="typeConverter" />
        </bean>
        <bean id="queueListenerContainer" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
                <property name="connectionFactory" ref="connectionFactory" />
                <property name="destination" ref="receiveQueueDestination" />
                <property name="messageListener" ref="queueListener" />
                <property name="sessionAcknowledgeModeName" value="CLIENT_ACKNOWLEDGE"/>
                <property name="concurrentConsumers" value="4" />
        </bean>
Reply | Threaded
Open this post in threaded view
|

Re: activemq 3000 messages can not be received

softwbc
In reply to this post by Tim Bain
This is no problem to get the dump log (jstack.log )

I see there is a queue does not consume message
Reply | Threaded
Open this post in threaded view
|

Re: activemq 3000 messages can not be received

RuralHunter
In reply to this post by Tim Bain
I also have the similar case. Is there any way to avoid this on server side? such as time out and close the client connection if it hangs.
Reply | Threaded
Open this post in threaded view
|

Re: activemq 3000 messages can not be received

Raul Kripalani-2
In reply to this post by softwbc
Could you please take a thread dump of the consumer?

---
Raúl Kripalani
linkedin.com/in/raulkripalani | evosent.com
<http://evosent.com/?utm_source=email&utm_medium=email&utm_campaign=evosent_raul>
| blog: raul.io
<http://raul.io?utm_source=email&utm_medium=email&utm_campaign=evosent_raul> |
skype: raul.fuse

On Fri, Aug 19, 2016 at 4:06 AM, softwbc <[hidden email]> wrote:

> This is no problem to get the dump log ( jstack.log
> <http://activemq.2283324.n4.nabble.com/file/n4715645/jstack.log>   )
>
> I see there is a queue does not consume message
> <http://activemq.2283324.n4.nabble.com/file/n4715645/1.jpg>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/activemq-3000-messages-can-not-be-received-
> tp4715396p4715645.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: activemq 3000 messages can not be received

Tim Bain
This looks strongly like a bug in your consumer code, not a problem with
ActiveMQ.  You're using CLIENT_ACKNOWLEDGE mode (<property
name="sessionAcknowledgeModeName" value="CLIENT_ACKNOWLEDGE"/>), which
means that if you have a bug that causes you to not ack your messages (due
to logic errors, exceptions, infinite loops, thread death, long-running
calls to external services, etc.), this behavior is exactly what you'd
see.  Based on what you've shared so far, that seems like the most likely
explanation.

In addition to the thread dump (as Raul said, from the consumer, not the
broker), look at your consumer code for any of the errors I listed above.

Tim

On Aug 19, 2016 2:45 AM, "Raul Kripalani" <[hidden email]> wrote:

> Could you please take a thread dump of the consumer?
>
> ---
> Raúl Kripalani
> linkedin.com/in/raulkripalani | evosent.com
> <http://evosent.com/?utm_source=email&utm_medium=email&
> utm_campaign=evosent_raul>
> | blog: raul.io
> <http://raul.io?utm_source=email&utm_medium=email&utm_
> campaign=evosent_raul> |
> skype: raul.fuse
>
> On Fri, Aug 19, 2016 at 4:06 AM, softwbc <[hidden email]> wrote:
>
> > This is no problem to get the dump log ( jstack.log
> > <http://activemq.2283324.n4.nabble.com/file/n4715645/jstack.log>   )
> >
> > I see there is a queue does not consume message
> > <http://activemq.2283324.n4.nabble.com/file/n4715645/1.jpg>
> >
> >
> >
> > --
> > View this message in context: http://activemq.2283324.n4.
> > nabble.com/activemq-3000-messages-can-not-be-received-
> > tp4715396p4715645.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: activemq 3000 messages can not be received

softwbc
I have checked already.And another consumer that does not have CLIENT_ACKNOWLEDGE  also appears problem.
This is another consumer configuration:
<bean id="connectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
                <property name="brokerURL" value="${activeMQ.brokerURL}"/>
                <property name="userName" value="${activeMQ.username}"/>
                <property name="password" value="${activeMQ.passwd}"/>
            </bean>
    <bean id="receiveQueueDestinationA" class="org.apache.activemq.command.ActiveMQQueue">
        <constructor-arg index="0" value="${activeMQ.queueNameA}"/>
    </bean>

    <bean id="receiverListenerServiceA" class="com.xinhuanet.censor.data.service.impl.ReceiverListenerServiceImpl">
    </bean>
    <bean id="queueListenerA" class="com.xinhuanet.censor.data.adapter.MessageAdapter">
        <constructor-arg ref="receiverListenerServiceA"></constructor-arg>
        <property name="defaultListenerMethod" value="receiveA"/>
    </bean>
    <bean id="queueListenerContainerA" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
        <property name="connectionFactory" ref="connectionFactory"/>
        <property name="destination" ref="receiveQueueDestinationA"/>
        <property name="messageListener" ref="queueListenerA"/>
        <property name="concurrentConsumers" value="8"/>
    </bean>
    <bean id="receiveQueueDestinationB" class="org.apache.activemq.command.ActiveMQQueue">
        <constructor-arg index="0" value="${activeMQ.queueNameB}"/>
    </bean>
    <bean id="receiverListenerServiceB" class="com.xinhuanet.censor.data.service.impl.ReceiverListenerServiceImpl">
    </bean>
    <bean id="queueListenerB" class="com.xinhuanet.censor.data.adapter.MessageAdapter">
        <constructor-arg ref="receiverListenerServiceB"></constructor-arg>
        <property name="defaultListenerMethod" value="receiveB"/>
    </bean>
    <bean id="queueListenerContainerB" class="org.springframework.jms.listener.DefaultMessageListenerContainer">
        <property name="connectionFactory" ref="connectionFactory"/>
        <property name="destination" ref="receiveQueueDestinationB"/>
        <property name="messageListener" ref="queueListenerB"/>
        <property name="concurrentConsumers" value="8"/>
    </bean>

There is a backlog of 1000,today.
But the queue is not blocked.
Reply | Threaded
Open this post in threaded view
|

Re: activemq 3000 messages can not be received

softwbc
In reply to this post by Raul Kripalani-2
this is the dump file of consumer and grab at the time of the problem
jstack-239.log
jstack-240.log

Reply | Threaded
Open this post in threaded view
|

Re: activemq 3000 messages can not be received

softwbc
In reply to this post by softwbc
Add a little
Message is consumed after the consumer restart.