Lost messages - Abnormal Behaviour of activeMQ

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Lost messages - Abnormal Behaviour of activeMQ

Uttara
This post was updated on .
Hi,
I have been working on a project,where in activeMQ is our main messaging bus. We have  developed our product to be multitenent app,wherein we support many customers.

Now that we use ActiveMQ,we found that messages are being lost without any behaviors in only two queues . There are three customers for which we create three connection object and using which we create three receivers, now at times except for the intended customer , the other two customer`s receiver are reading it but not ACKNOWLEDGING the message ,then that particular message is  lost and not found in the broker. it is being dequeued with out ACKNOWLEDGING it as we have Individual acknowledge as acknowledgement mode. Thanks in advance.
Reply | Threaded
Open this post in threaded view
|

Re: Lost messages - Abnormal Behaviour of activeMQ

tabish121@gmail.com
On 07/14/2014 09:39 AM, Uttara wrote:

> Hi,
> I have been working on a project,where in activeMQ is our main messaging
> bus. We have  developed our product to be multitenent app,wherein we support
> many customers.
>
> Now that we use ActiveMQ,we found that messages are being lost without any
> behaviors in only two queues . There are three customers for which we create
> three connection object and using which we create three receivers, now at
> times except for the intended customer , the other two customer`s receiver
> are reading it but not ACKNOWLEDGING the message ,then that particular
> message is  lost and not found in the broker. it is being dequeued with out
> ACKNOWLEDGING it as we have client acknowledge as acknowledgement mode.
> Thanks in advance.
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Lost-messages-Abnormal-Behaviour-of-activeMQ-tp4683240.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Be careful with client Ack, it acknowledges all received messages for
the Session so far which can be confusing if you don't know it's
happening.  For more help try to provide some more concrete info or a
code sample.

--
Tim Bish
Sr Software Engineer | RedHat Inc.
[hidden email] | www.fusesource.com | www.redhat.com
skype: tabish121 | twitter: @tabish121
blog: http://timbish.blogspot.com/

Reply | Threaded
Open this post in threaded view
|

Re: Lost messages - Abnormal Behaviour of activeMQ

Uttara
Hi Tim,
Thanks for your reply,We are using Individual acknowledgement.
Actually three threads run in the background for three customers.And from the queue we read the message based on the customer id in the message . If the intended thread receives the message it acknowledges it. We are also logging once after the message is read before acknowledging it. In certain scenarios the message is being read in other two customer threads but not for the intended customer as we could see the log points before acknowledging it. And then to my surprise,the message gets lost. PFB the piece of code where we have applied acknowledging strategy.


        for(int count=0;count<10;count++){
                        Object receivedMessage = queueReceiver.receiveNoWait();
                        if(receivedMessage instanceof TextMessage)
                        {
                       
                                message = (TextMessage) receivedMessage;
                               
                                try {
                                        objMsgVO = util.convertMessageToVO(message.getText());
                                } catch (Throwable e) {
                                        logger.error("Invalid Json ticket data.Going to Acknowledge");
                                        message.acknowledge();
                                }
                               
                                if(objMsgVO.getCustomerId()!=null){
                                boolean flag=false;
                                for(int i=0,j=0;i<ToolsIntegratorPoller.customerName.length;i++,j++){
                                        if(objMsgVO.getCustomerId().equalsIgnoreCase(ToolsIntegratorPoller.customerName[i]))
                                        {
                                                flag=true;
                                        }
                                }
                                if(flag){
                                if(objMsgVO.getCustomerId().equalsIgnoreCase(mqInitContext.getProperty("CUSTOMER_NAME")))
                                {
                                        logger.info("received: " + message.getText());
                                       
                                        returnMassage = message.getText();
                                       
                                        messageList.add(returnMassage);
                                        message.acknowledge();
                                       
                                }
                               
                                }
                                else {
                                        message.acknowledge();
                                        logger.error("Message does not have a valid customer name or Customer name is not present in the ticket data");

                                }
                        }
                        }
                       
                       
               
                       
               
                }
Reply | Threaded
Open this post in threaded view
|

Re: Lost messages - Abnormal Behaviour of activeMQ

artnaseef
If I understand correctly, the code is doing the following:

* Competing consumers read from the same queue
* One of several consumers will be able to process a message while the other two will not
* When the process can handle the message, it does so and acknowledges it
* When the process cannot handle the message, it rejects (I don't see rejection in the code), so the message can be handled by another process

Is this correct?  If so, I recommend an architectural review.  This model of processing will lead to many problems with ActiveMQ.

For example, ActiveMQ never guarantees which consumer will receive a message.  So, the same message could be redelivered to a single consumer multiple times and never to any other consumer.  In addition, ActiveMQ has settings for redelivery; if a message fails to process too many times, it will be discarded (either sent to a DLQ or just dropped).

Let me know if I've misunderstood.
Reply | Threaded
Open this post in threaded view
|

Re: Lost messages - Abnormal Behaviour of activeMQ

Noel OConnor
+1
This approach is a derivative of the single purpose queue anti-pattern see

http://books.google.com.au/books?id=1G3Pa_LoIGQC&pg=PA236&lpg=PA236&dq=single+purpose+queue&source=bl&ots=O57wy-t5MZ&sig=ADfUPDpAUywSVNnrnanjnkus13g&hl=en&sa=X&ei=nNDFU6DYIcvn8AWPgoKYBw&ved=0CB4Q6AEwAA#v=onepage&q=single%20purpose%20queue&f=false

Consider using topics with selectors or camel with content based
routing to improve it.


On Wed, Jul 16, 2014 at 2:45 AM, artnaseef <[hidden email]> wrote:

> If I understand correctly, the code is doing the following:
>
> * Competing consumers read from the same queue
> * One of several consumers will be able to process a message while the other
> two will not
> * When the process can handle the message, it does so and acknowledges it
> * When the process cannot handle the message, it rejects (I don't see
> rejection in the code), so the message can be handled by another process
>
> Is this correct?  If so, I recommend an architectural review.  This model of
> processing will lead to many problems with ActiveMQ.
>
> For example, ActiveMQ never guarantees which consumer will receive a
> message.  So, the same message could be redelivered to a single consumer
> multiple times and never to any other consumer.  In addition, ActiveMQ has
> settings for redelivery; if a message fails to process too many times, it
> will be discarded (either sent to a DLQ or just dropped).
>
> Let me know if I've misunderstood.
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Lost-messages-Abnormal-Behaviour-of-activeMQ-tp4683240p4683287.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.