Message stuck on queue

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Message stuck on queue

watcher
We are currently experience a problem with activemq version 5.4.0
Every 2-4 days of testing we get 1 message stuck on the queue.
We have approx 10 queues were we serialize a POJO onto each queue.
We have no more than 1000 messages on each queue.

We have up to 5 consumers (slow consumers between 5secs and 1min) on each queue.

Each client will wait forever on the queue until the queue is empty.
 consumerTemplate.receive(MY_QUEUE);

Our client connect with a prefetch = 1, we also tried value of 0
failover:(tcp://127.0.0.1:61616)?jms.prefetchPolicy.queuePrefetch=1


<code>
   <broker xmlns="http://activemq.apache.org/schema/core" brokerName="mainBroker"
        dataDirectory="${activemq.base}/data" destroyApplicationContextOnStop="true" useJmx="true">
               
       
             
        <destinationPolicy>
            <policyMap>
              <policyEntries>
                <policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb">
                  <pendingSubscriberPolicy>
                    <vmCursor />
                  </pendingSubscriberPolicy>
                </policyEntry>
                <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb">
                 
                </policyEntry>
              </policyEntries>
            </policyMap>
        </destinationPolicy> 
 
       
       
        <managementContext>
            <managementContext connectorPort="2011" jmxDomainName="my-broker" />
        </managementContext>

       
        <persistenceAdapter>
            <kahaDB directory="${activemq.base}/data/kahadb"/>
        </persistenceAdapter>
       
          <plugins>
                        <statisticsBrokerPlugin/>
                </plugins>
           
         <systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage limit="256 mb"/>
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="1 gb" name="foo"/>
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="300 mb"/>
                </tempUsage>
            </systemUsage>
        </systemUsage>
               
                 
       
        <transportConnectors>
            <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
        </transportConnectors>
               

    </broker>

           
  <import resource="camel.xml"/>
 

   
    <import resource="jetty.xml"/>
   
</code>

two different setups
server windows 2008
client windows 7

and
server: windows server 2003
client : windows 7

When we restart activemq service the message appears.

Is there anything that we can do to prevent this from happening?
Thanks


Reply | Threaded
Open this post in threaded view
|

Re: Message stuck on queue

watcher
An update so far.

We stilll cant seem to find a resolution to this issue. We have noticed similar user having problems.
See here.
http://activemq.2283324.n4.nabble.com/Workaround-for-messages-stuck-in-a-queue-td2343453.html

We implemented a similar workaround. Whenever we get to the end on the queue and a message is stuck we purge the queue a republish the message. This is not a perfect solution as you can imagine.

We tried upgrading to 5.4.1 but the error still exists. Since then 5.4.2 has been released but we're not convinced that this will solve the issue but we will try soon.

Just another bit of info, we use the statisics plugin via camel to determine the queue size however I've not been able to implement a wildcard scan of queues so we ended up hardcoding a route for each queue that may or may not exist up to a max of 60 queues. The only errors we see in activemq logs are related to timeout when trying to get queue stats for queues that do not exist.


http://activemq.2283324.n4.nabble.com/statisticsBrokerPlugin-error-td2716123.html


Is there any logging we could enable to understand why a message will not get processed?

We getting to the stage of replacing activemq with another messaging provider but I like to think this could be solved. Any advise would be very helpful.

Thanks.
Reply | Threaded
Open this post in threaded view
|

Re: Message stuck on queue

rajdavies
For statistics - can you should be able to use wildcards - if you use '>' as the destination name - what do you get ?

On 10 Dec 2010, at 22:23, watcher wrote:

>
> An update so far.
>
> We stilll cant seem to find a resolution to this issue. We have noticed
> similar user having problems.
> See here.
> http://activemq.2283324.n4.nabble.com/Workaround-for-messages-stuck-in-a-queue-td2343453.html
> http://activemq.2283324.n4.nabble.com/Workaround-for-messages-stuck-in-a-queue-td2343453.html 
>
> We implemented a similar workaround. Whenever we get to the end on the queue
> and a message is stuck we purge the queue a republish the message. This is
> not a perfect solution as you can imagine.
>
> We tried upgrading to 5.4.1 but the error still exists. Since then 5.4.2 has
> been released but we're not convinced that this will solve the issue but we
> will try soon.
>
> Just another bit of info, we use the statisics plugin via camel to determine
> the queue size however I've not been able to implement a wildcard scan of
> queues so we ended up hardcoding a route for each queue that may or may not
> exist up to a max of 60 queues. The only errors we see in activemq logs are
> related to timeout when trying to get queue stats for queues that do not
> exist.
>
> http://activemq.2283324.n4.nabble.com/statisticsBrokerPlugin-error-td2716123.html 
> http://activemq.2283324.n4.nabble.com/statisticsBrokerPlugin-error-td2716123.html 
>
> Is there any logging we could enable to understand why a message will not
> get processed?
>
> We getting to the stage of replacing activemq with another messaging
> provider but I like to think this could be solved. Any advise would be very
> helpful.
>
> Thanks.
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Message-stuck-on-queue-tp3076607p3082780.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: Message stuck on queue

timi.blossom
This post has NOT been accepted by the mailing list yet.
Sight, did we resolve this issue?  We are burning so much now.
Reply | Threaded
Open this post in threaded view
|

Re: Message stuck on queue

lernen.2007
I have the same problem with message stuck on queue. I use as persistence a oracle database with jdbc. I think that is a bug in activemq. The version 5.3.1 hasn't any problem.
Reply | Threaded
Open this post in threaded view
|

Re: Message stuck on queue

watcher
We havnt been able to reproduce the bug after we turned off caching.

<policyEntry queue=">" producerFlowControl="false" memoryLimit="1mb"  useCache="false">

We were using version 5.2.0 in another application and never experience this problem before but I think that is because they changed the persistence storage to kahadb in 5.3 by default.
Reply | Threaded
Open this post in threaded view
|

Re: Message stuck on queue

gtully
I think you may be experiencing
https://issues.apache.org/jira/browse/AMQ-3167 - if you get a chance
you can validate a 5.5-SNAPSHOT with the cache enabled?

On 1 February 2011 12:00, watcher <[hidden email]> wrote:

>
> We havnt been able to reproduce the bug after we turned off caching.
>
> <policyEntry queue=">" producerFlowControl="false" memoryLimit="1mb"
> useCache="false">
>
> We were using version 5.2.0 in another application and never experience this
> problem before but I think that is because they changed the persistence
> storage to kahadb in 5.3 by default.
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Message-stuck-on-queue-tp3076607p3251012.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



--
http://blog.garytully.com
http://fusesource.com
Reply | Threaded
Open this post in threaded view
|

Re: Message stuck on queue

lernen.2007
I will test with the 5.5 snapshot and inform you about that. I will test it with the jdbc persistence adapter and will use oracle database 10.2.0.4
Reply | Threaded
Open this post in threaded view
|

Re: Message stuck on queue

anton
We are using ActiveMQ 5.4.2 on CentOS 5.4 and also run into a "message stuck in a queue" scenario. In our case these stuck messages had no content. Here's an outline of what was happening:

a) we were seeing a few empty messages submitted to the queue (our fault - there weren't really supposed to be empty, but because of a producer error they were)
b) our consumer application (using STOMP) would never get these empty messages and would wait in perpetuity
c) deleteing these empty messages from the queue and reconnecting the consumer would resume the flow of messages
d) In ActiveMQ log, we noticed this stack trace:

2011-01-30 17:12:34,811 | ERROR | Failed to page in more queue messages  | org.apache.activemq.broker.region.Queue | Queue:test66
java.lang.NullPointerException
        at org.apache.activemq.transport.stomp.LegacyFrameTranslator.convertMessage(LegacyFrameTranslator.java:90)
        at org.apache.activemq.transport.stomp.ProtocolConverter.convertMessage(ProtocolConverter.java:610)
        at org.apache.activemq.transport.stomp.StompSubscription.onMessageDispatch(StompSubscription.java:93)
        at org.apache.activemq.transport.stomp.ProtocolConverter.onActiveMQCommand(ProtocolConverter.java:592)
        at org.apache.activemq.transport.stomp.StompTransportFilter.oneway(StompTransportFilter.java:58)
        at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
        at org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1249)
        at org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:810)
        at org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:770)
        at org.apache.activemq.broker.region.PrefetchSubscription.dispatch(PrefetchSubscription.java:649)
        at org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:599)
        at org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:156)
        at org.apache.activemq.broker.region.Queue.doActualDispatch(Queue.java:1798)
        at org.apache.activemq.broker.region.Queue.doDispatch(Queue.java:1745)
        at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
        at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)


e) Putting a try/catch block around that line in LegacyFrameTranslator.java (activemq-core/src/main/java/org/apache/activemq/transport/stomp/LegacyFrameTranslator.java:90) causes the messages to not be stuck any more: they now show up successfully on the consumer side. Here's the change we did:


90c90,94
<             command.setContent(msg.getText().getBytes("UTF-8"));
---
>             try {
>                 command.setContent(msg.getText().getBytes("UTF-8"));
>             } catch (Throwable e) {
>                 command.setContent(new byte[0]);
>             }


Not sure if this is the "right" way to fix the problem.
Hoping someone on this list could shed some light.
Reply | Threaded
Open this post in threaded view
|

Re: Message stuck on queue

Norman Maurer-3
I think it would be better todo something like that for the fix:

    byte[] content;
    if (msg.getText== null) {
        content = new byte[0];
    } else {
        content = msg.getText().getBytes("UTF-8");
    }
    command.setContent(content);

But I'm not to familiar with stomp..

Bye,
Norman


2011/2/1 anton <[hidden email]>:

>
> We are using ActiveMQ 5.4.2 on CentOS 5.4 and also run into a "message stuck
> in a queue" scenario. In our case these stuck messages had no content.
> Here's an outline of what was happening:
>
> a) we were seeing a few empty messages submitted to the queue (our fault -
> there weren't really supposed to be empty, but because of a producer error
> they were)
> b) our consumer application (using STOMP) would never get these empty
> messages and would wait in perpetuity
> c) deleteing these empty messages from the queue and reconnecting the
> consumer would resume the flow of messages
> d) In ActiveMQ log, we noticed this stack trace:
>
> 2011-01-30 17:12:34,811 | ERROR | Failed to page in more queue messages  |
> org.apache.activemq.broker.region.Queue | Queue:test66
> java.lang.NullPointerException
>        at
> org.apache.activemq.transport.stomp.LegacyFrameTranslator.convertMessage(LegacyFrameTranslator.java:90)
>        at
> org.apache.activemq.transport.stomp.ProtocolConverter.convertMessage(ProtocolConverter.java:610)
>        at
> org.apache.activemq.transport.stomp.StompSubscription.onMessageDispatch(StompSubscription.java:93)
>        at
> org.apache.activemq.transport.stomp.ProtocolConverter.onActiveMQCommand(ProtocolConverter.java:592)
>        at
> org.apache.activemq.transport.stomp.StompTransportFilter.oneway(StompTransportFilter.java:58)
>        at
> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
>        at
> org.apache.activemq.broker.TransportConnection.dispatch(TransportConnection.java:1249)
>        at
> org.apache.activemq.broker.TransportConnection.processDispatch(TransportConnection.java:810)
>        at
> org.apache.activemq.broker.TransportConnection.dispatchSync(TransportConnection.java:770)
>        at
> org.apache.activemq.broker.region.PrefetchSubscription.dispatch(PrefetchSubscription.java:649)
>        at
> org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:599)
>        at
> org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:156)
>        at
> org.apache.activemq.broker.region.Queue.doActualDispatch(Queue.java:1798)
>        at org.apache.activemq.broker.region.Queue.doDispatch(Queue.java:1745)
>        at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
>        at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)
>        at
> org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
>        at
> org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)
>
>
> e) Putting a try/catch block around that line in LegacyFrameTranslator.java
> (activemq-core/src/main/java/org/apache/activemq/transport/stomp/LegacyFrameTranslator.java:90)
> causes the messages to not be stuck any more: they now show up successfully
> on the consumer side. Here's the change we did:
>
>
> 90c90,94
> <             command.setContent(msg.getText().getBytes("UTF-8"));
> ---
>>             try {
>>                 command.setContent(msg.getText().getBytes("UTF-8"));
>>             } catch (Throwable e) {
>>                 command.setContent(new byte[0]);
>>             }
>
>
> Not sure if this is the "right" way to fix the problem.
> Hoping someone on this list could shed some light.
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Message-stuck-on-queue-tp3076607p3252992.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Message stuck on queue

lernen.2007
I check it with the latest activemq snapshot 5.5.0 and it is the same attitude --> the messages stuck in queue. We want to go in production because of this bug we can't.