Lost messages.

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Lost messages.

dfco21
Hello,

We're experiencing a problem with our ActiveMQ queue. We have a producer
(Cloud solution) which sends messages to an ActiveMQ queue.

Our issue is the following:
Erratically some of the messages are not delivered to the ActiveMQ queue and
are lost. We checked the producer's log and it shows no error. The volume is
very low (<10 messages).

In the screenshot, we can see that no messages have been delivered although
3 have been sent by the producer.

<http://activemq.2283324.n4.nabble.com/file/t378791/Bidon.jpg>

Hopefully, someone can recommend how to fix the issue.

Thanks, AEB



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Lost messages.

Jean-Baptiste Onofré
Hi

Is the producer use sendAsync() or send() ?

Are the messages persistent ?

Regards
JB

On Nov 21, 2017, 22:38, at 22:38, dfco21 <[hidden email]> wrote:

>Hello,
>
>We're experiencing a problem with our ActiveMQ queue. We have a
>producer
>(Cloud solution) which sends messages to an ActiveMQ queue.
>
>Our issue is the following:
>Erratically some of the messages are not delivered to the ActiveMQ
>queue and
>are lost. We checked the producer's log and it shows no error. The
>volume is
>very low (<10 messages).
>
>In the screenshot, we can see that no messages have been delivered
>although
>3 have been sent by the producer.
>
><http://activemq.2283324.n4.nabble.com/file/t378791/Bidon.jpg>
>
>Hopefully, someone can recommend how to fix the issue.
>
>Thanks, AEB
>
>
>
>--
>Sent from:
>http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Lost messages.

dfco21
Hi,

The message is persistent. See the screenshot for more details on the
configuration we've implemented:

<http://activemq.2283324.n4.nabble.com/file/t378791/Picture2.png>

We're sending messages to the ActiveMQ queue through a Cloud service.

Thanks, AEB



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Lost messages.

dfco21
Hello,

Please find more details.

Actually, all the messages are sent to the Queue (starting with uuid_ in the
screenshot below) but 3 are automatically queued and only 2 (instead of 5)
are available to the consumer (not starting with uuid_ in the screenshot
below).

We're using the version 5.14.5. No consumer strategy is enabled nor
expiration time.


Properties screenshot (we are sending messages through a Service Cloud) :
<http://activemq.2283324.n4.nabble.com/file/t378791/Picture2.png>

Queues page of the web console (5 messages have been sent, all the messages
have been received in the queue but 3 are automatically dequeued and the
consumer has only 2 messages available instead of 5):
<http://activemq.2283324.n4.nabble.com/file/t378791/Picture3.png>

Please let me know if any question.

Thanks, AEB



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Lost messages.

Tim Bain
OK, so you've sent 5 messages to the queue, and they've been round-robined
across the two consumers, so one consumer got three messages and one got
two. Apparently the consumer you're looking at is the one that got two
messages, but the key here is that the other three messages weren't lost,
they were consumed by the other consumer.

So, why do you have a second consumer on that queue?

One possibility is that you're using a network of brokers and the second
consumer is a networked broker, but since you've told us very little about
how you've configured your broker(s) I can't guess whether that might be
the explanation. If not, it's something you're doing (or that a co-worker
is doing), because ActiveMQ doesn't create consumers on your destinations
except in the case of a network of brokers (and even then, messages aren't
lost, they just go to the other brokers and from there to other consumers).

Tim

On Nov 22, 2017 11:22 AM, "dfco21" <[hidden email]> wrote:

> Hello,
>
> Please find more details.
>
> Actually, all the messages are sent to the Queue (starting with uuid_ in
> the
> screenshot below) but 3 are automatically queued and only 2 (instead of 5)
> are available to the consumer (not starting with uuid_ in the screenshot
> below).
>
> We're using the version 5.14.5. No consumer strategy is enabled nor
> expiration time.
>
>
> Properties screenshot (we are sending messages through a Service Cloud) :
> <http://activemq.2283324.n4.nabble.com/file/t378791/Picture2.png>
>
> Queues page of the web console (5 messages have been sent, all the messages
> have been received in the queue but 3 are automatically dequeued and the
> consumer has only 2 messages available instead of 5):
> <http://activemq.2283324.n4.nabble.com/file/t378791/Picture3.png>
>
> Please let me know if any question.
>
> Thanks, AEB
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Lost messages.

dfco21
Exact. The consumer which had the 2 messages is the one which is supposed to
get the 5 messages. It's a consumer-broker created by another cloud service
supposed to consume the 5 messages sent.

It looks like the 2nd consumer with the 3 messages is created by the
producer. But 5 messages have been produced and no broker has been
configured on the producer side.

Maybe it's a networked broker as you mentioned but how can I confirm. As I
said no broker has been configured to get created by the producer and 3 of
the 5 messages are not consumed by the correct broker.

Explanations I see:
* the 2nd broker is created by the producer but why 3 messages are consumed
not 5?
* the 1st broker has not enough time to consume the message and a 2nd
default broker consumes it

They are only hypothesis, I am not sure how to confirm and fix the issue.

Below the producer properties:
<http://activemq.2283324.n4.nabble.com/file/t378791/Picture2.png>


Thanks for your time and help.

-AEB



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Lost messages.

Tim Bain
On Nov 22, 2017 10:43 PM, "dfco21" <[hidden email]> wrote:

Exact. The consumer which had the 2 messages is the one which is supposed to
get the 5 messages. It's a consumer-broker created by another cloud service
supposed to consume the 5 messages sent.


OK, so the consumer you're looking at is another broker, not an "end"
consumer? I don't feel like you've really given us a very good description
of what you're actually doing, which makes it hard to know what questions
to ask or suggestions to make.

It looks like the 2nd consumer with the 3 messages is created by the
producer. But 5 messages have been produced and no broker has been
configured on the producer side.


On what are you basing the guess that it's created by the producer? Does it
appear when the producer starts and disappear when the producer stops? Does
the connection come from the producer's IP address?

Maybe it's a networked broker as you mentioned but how can I confirm.


You're going to want to investigate further via a JMX viewer such as
JConsole. The web console is great for getting status of the broker, but
when you need to investigate something (as you do now), the more-detailed
information available through JMX is really what you want. The MBeans
exposed via JMX will let you see which IP and port that connection comes
from, which will let you tie it to the process that owns it.

As I
said no broker has been configured to get created by the producer and 3 of
the 5 messages are not consumed by the correct broker.


So there's only supposed to be these two brokers networked together? No
other brokers? Who's supposed to be consuming these messages (from the
other broker, by implication of what you wrote above)? Is that consumer
connected to the right broker?

Explanations I see:
* the 2nd broker is created by the producer but why 3 messages are consumed
not 5?


I'm not following this. If five messages are published to a queue with two
consumers, why would you expect either one to get all five?

* the 1st broker has not enough time to consume the message and a 2nd
default broker consumes it


Unless you've enabled slow consumer strategies, this wouldn't be the case.

Clearly you have two consumers and you expect only one, which fully
explains the behavior you're seeing, so use JMX to figure out what the
other consumer is and go from there.

They are only hypothesis, I am not sure how to confirm and fix the issue.

Below the producer properties:
<http://activemq.2283324.n4.nabble.com/file/t378791/Picture2.png>


Thanks for your time and help.

-AEB



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Lost messages.

dfco21
Thanks for the update.

I don't have a lot of information on the Sevice Cloud consumer, it's a
third-party application and the access is very limited.

I tried to mimic the process in a development environment and I saw an
ActiveMQ.DLQ queue consuming messages I sent to a Queue.

Could it be an explanation of my issue.

Thanks, AEB



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Lost messages.

Tim Bain
On Nov 24, 2017 2:53 PM, "dfco21" <[hidden email]> wrote:

Thanks for the update.

I don't have a lot of information on the Sevice Cloud consumer, it's a
third-party application and the access is very limited.


This sounds like it will make it difficult for you to troubleshoot the
behavior you're seeing. If that Service Cloud consumer turns out to be the
origin of the unexpected consumer on the topic, you'll probably need to
figure out a way to get answers. Maybe that takes a phone call to the right
person, maybe it takes your company buying a support contract with them,
maybe it takes something else. But first you need to use the approach I
described earlier to figure out who is the origin of the unexpected
consumer.

I tried to mimic the process in a development environment and I saw an
ActiveMQ.DLQ queue consuming messages I sent to a Queue.


No, DLQs don't consume messages, they receive them when a consumer tries to
consume the messages and fails (or under certain configurations, when
messages expire or consumers are slow, but you have to explicitly enable
that configuration and you've not said anything that indicates that you've
done that so I don't think that's what's going on).

If messages are going into your DLQ, you need to figure out who your
consumers are and why some or all of them are failing to consume your
messages.

So yes, it explains your issue, with the same explanation I gave you in my
previous response: your messages are being consumed by those two consumers,
and if you're only expecting one then you need to figure out what the
second one is and get rid of it.

Could it be an explanation of my issue.

Thanks, AEB



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html