MDB stops consuming messages

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

MDB stops consuming messages

sbarlabanov
Hi,

this is the continuation of my forum entry http://activemq.2283324.n4.nabble.com/ActiveMQ-5-10-0-queue-slowed-down-restart-helped-td4690706.html#a4690723.
But since the discussion there went to another direction I start a new topic.
We got the issue again several times in the last two month.
More details: we have a standalone running ActiveMQ 5.10.0, Glassfish 3.1.2.2 with ActiveMQ RAR and an MDB consuming messages from an ActiveMQ queue.
Suddenly after weeks successful processing the MDB stops processing any message (I mentioned that already). In jconsole we see the corresponding JMS consumer on the queue. MessageCountAwaitingAcknowledge of the consumer is equal the prefetch buffer (10 in our case). From time to time a new consumer occurs on the queue (this is seen in jconsole), but the first one stays with MessageCountAwaitingAcknowledge=prefetch buffer.
When we stop Glassfish, the following interesting exception appears in activemq.log:

java.lang.IllegalStateException: Cannot remove session from connection that had not been registered: ID:NOEUD-SI2-55792-1427222776632-47:1
        at org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:677)[activemq-broker-5.10.0-patch4.jar:5.10.0-patch4]
        at org.apache.activemq.command.RemoveInfo.visit(RemoveInfo.java:74)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
        at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)[activemq-broker-5.10.0-patch4.jar:5.10.0-patch4]
        at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:148)[activemq-broker-5.10.0-patch4.jar:5.10.0-patch4]
        at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
        at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
        at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
        at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
        at org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:138)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
        at org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:69)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
        at org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:94)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
        at org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:119)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_45]
        at java.lang.Thread.run(Thread.java:744)[:1.7.0_45]

It seems that something goes wrong in ActiveMQ RAR. Seems that something hangs around in the RAR locking the messages. Because otherwise ActiveMQ will not complain about the session which is going to be removed from an unregistered session. So something went wrong before. And the broker did not get it and still thinks that the consumer/session/connection is open.
Reply | Threaded
Open this post in threaded view
|

Re: MDB stops consuming messages

Tim Bain
Did you ever solve this issue?

If not, can you please clarify a couple things in your description?

   1. Did you confirm (via a thread dump) that the MDB wasn't actually hung
   somewhere in your consumer code?  Clearly the broker believed it had
   dispatched a prefetch buffer worth of messages to the consumer, so the
   question is whether the consumer's ActiveMQ client/RAR code had attempted
   to dispatch the next message to your MDB or not.  And if the thread wasn't
   stuck in your MDB code, where was it stuck instead?
   2. When you shut down Glassfish, was your MDB consumer connected or
   disconnected?  The first half of your description sounded like it was
   connected, while the last paragraph sounded like it was disconnected.
   3. When you connected the additional consumer, did it process messages
   as expected?  You never said what happened when those additional consumers
   connected.
   4. Did that ID match one of the consumers (or one of their connections;
   I can never tell consumer IDs and connection IDs apart)?  If so, which one?
   5. If you can reproduce this reliably, does the exception happen if you
   shut down Glassfish without attempting to connect any additional consumers?

Tim

On Tue, Mar 24, 2015 at 3:24 PM, sbarlabanov <[hidden email]>
wrote:

> Hi,
>
> this is the continuation of my forum entry
>
> http://activemq.2283324.n4.nabble.com/ActiveMQ-5-10-0-queue-slowed-down-restart-helped-td4690706.html#a4690723
> .
> But since the discussion there went to another direction I start a new
> topic.
> We got the issue again several times in the last two month.
> More details: we have a standalone running ActiveMQ 5.10.0, Glassfish
> 3.1.2.2 with ActiveMQ RAR and an MDB consuming messages from an ActiveMQ
> queue.
> Suddenly after weeks successful processing the MDB stops processing any
> message (I mentioned that already). In jconsole we see the corresponding
> JMS
> consumer on the queue. MessageCountAwaitingAcknowledge of the consumer is
> equal the prefetch buffer (10 in our case). From time to time a new
> consumer
> occurs on the queue (this is seen in jconsole), but the first one stays
> with
> MessageCountAwaitingAcknowledge=prefetch buffer.
> When we stop Glassfish, the following interesting exception appears in
> activemq.log:
>
> java.lang.IllegalStateException: Cannot remove session from connection that
> had not been registered: ID:NOEUD-SI2-55792-1427222776632-47:1
>         at
>
> org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:677)[activemq-broker-5.10.0-patch4.jar:5.10.0-patch4]
>         at
>
> org.apache.activemq.command.RemoveInfo.visit(RemoveInfo.java:74)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
>         at
>
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)[activemq-broker-5.10.0-patch4.jar:5.10.0-patch4]
>         at
>
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:148)[activemq-broker-5.10.0-patch4.jar:5.10.0-patch4]
>         at
>
> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
>         at
>
> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
>         at
>
> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
>         at
>
> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
>         at
>
> org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:138)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
>         at
>
> org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:69)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
>         at
>
> org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:94)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
>         at
>
> org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:119)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
>         at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_45]
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_45]
>         at java.lang.Thread.run(Thread.java:744)[:1.7.0_45]
>
> It seems that something goes wrong in ActiveMQ RAR. Seems that something
> hangs around in the RAR locking the messages. Because otherwise ActiveMQ
> will not complain about the session which is going to be removed from an
> unregistered session. So something went wrong before. And the broker did
> not
> get it and still thinks that the consumer/session/connection is open.
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/MDB-stops-consuming-messages-tp4693769.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: MDB stops consuming messages

Tim Bain
Oh, I forgot: was there any indication of Producer Flow Control kicking in
on the broker prior to when the consumer stopped processing messages?

On Sun, Apr 26, 2015 at 9:06 AM, Tim Bain <[hidden email]> wrote:

> Did you ever solve this issue?
>
> If not, can you please clarify a couple things in your description?
>
>    1. Did you confirm (via a thread dump) that the MDB wasn't actually
>    hung somewhere in your consumer code?  Clearly the broker believed it had
>    dispatched a prefetch buffer worth of messages to the consumer, so the
>    question is whether the consumer's ActiveMQ client/RAR code had attempted
>    to dispatch the next message to your MDB or not.  And if the thread wasn't
>    stuck in your MDB code, where was it stuck instead?
>    2. When you shut down Glassfish, was your MDB consumer connected or
>    disconnected?  The first half of your description sounded like it was
>    connected, while the last paragraph sounded like it was disconnected.
>    3. When you connected the additional consumer, did it process messages
>    as expected?  You never said what happened when those additional consumers
>    connected.
>    4. Did that ID match one of the consumers (or one of their
>    connections; I can never tell consumer IDs and connection IDs apart)?  If
>    so, which one?
>    5. If you can reproduce this reliably, does the exception happen if
>    you shut down Glassfish without attempting to connect any additional
>    consumers?
>
> Tim
>
> On Tue, Mar 24, 2015 at 3:24 PM, sbarlabanov <[hidden email]
> > wrote:
>
>> Hi,
>>
>> this is the continuation of my forum entry
>>
>> http://activemq.2283324.n4.nabble.com/ActiveMQ-5-10-0-queue-slowed-down-restart-helped-td4690706.html#a4690723
>> .
>> But since the discussion there went to another direction I start a new
>> topic.
>> We got the issue again several times in the last two month.
>> More details: we have a standalone running ActiveMQ 5.10.0, Glassfish
>> 3.1.2.2 with ActiveMQ RAR and an MDB consuming messages from an ActiveMQ
>> queue.
>> Suddenly after weeks successful processing the MDB stops processing any
>> message (I mentioned that already). In jconsole we see the corresponding
>> JMS
>> consumer on the queue. MessageCountAwaitingAcknowledge of the consumer is
>> equal the prefetch buffer (10 in our case). From time to time a new
>> consumer
>> occurs on the queue (this is seen in jconsole), but the first one stays
>> with
>> MessageCountAwaitingAcknowledge=prefetch buffer.
>> When we stop Glassfish, the following interesting exception appears in
>> activemq.log:
>>
>> java.lang.IllegalStateException: Cannot remove session from connection
>> that
>> had not been registered: ID:NOEUD-SI2-55792-1427222776632-47:1
>>         at
>>
>> org.apache.activemq.broker.TransportConnection.processRemoveSession(TransportConnection.java:677)[activemq-broker-5.10.0-patch4.jar:5.10.0-patch4]
>>         at
>>
>> org.apache.activemq.command.RemoveInfo.visit(RemoveInfo.java:74)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
>>         at
>>
>> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)[activemq-broker-5.10.0-patch4.jar:5.10.0-patch4]
>>         at
>>
>> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:148)[activemq-broker-5.10.0-patch4.jar:5.10.0-patch4]
>>         at
>>
>> org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
>>         at
>>
>> org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
>>         at
>>
>> org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:270)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
>>         at
>>
>> org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
>>         at
>>
>> org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:138)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
>>         at
>>
>> org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:69)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
>>         at
>>
>> org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:94)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
>>         at
>>
>> org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:119)[activemq-client-5.10.0-patch2.jar:5.10.0-patch2]
>>         at
>>
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)[:1.7.0_45]
>>         at
>>
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)[:1.7.0_45]
>>         at java.lang.Thread.run(Thread.java:744)[:1.7.0_45]
>>
>> It seems that something goes wrong in ActiveMQ RAR. Seems that something
>> hangs around in the RAR locking the messages. Because otherwise ActiveMQ
>> will not complain about the session which is going to be removed from an
>> unregistered session. So something went wrong before. And the broker did
>> not
>> get it and still thinks that the consumer/session/connection is open.
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/MDB-stops-consuming-messages-tp4693769.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>