OutOfMemory encountered in Artemis CoreAmqpConverter

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

OutOfMemory encountered in Artemis CoreAmqpConverter

Dirkjan Ochtman
Hi there,

I'm still trying to build an AMQP RPC message flow. I have the request path
working and am now debugging the response path. In doing so, it seems that
the response message is correctly being sent over the expected channel, but
the broker seems to die from an out of memory error. In trying to
understand what's going on, I've got some more details, but not sure yet
what's going on.

This is with Artemis 2.6.2.

In CoreAmqpConverter.convertBody(), my message hits the ServerJMSMessage
branch. The CoreMessage's buffer has ridx 0 and widx 9755, with
endOfBodyPosition = 9218. For some reason, the readerIndex from
message.getInnerMessage().getBodyBuffer().readerIndex() ends up being 4
(it's not clear to me exactly how/why this happens). This seems to be
skipping over a prefix that contains [0, 0, 36, 2]. That data is directly
followed by what I recognize as the actual message contents, with starts
with a 5-byte alphanumeric string. However, when the code then tries to
readNullableSimpleString() from the buffer it seems to read the length of
the buffer to be used from the alpha string, resulting in a buffer length
that ends up going out of bounds. SimpleString.readNullableSimpleString()
then reads another byte from the buffer to check whether it's null (it does
not appear to rewind the reader index if it isn't?). readSimpleString then
reads a length of 1.8 GB and the process dies.

Can someone explain to me what might be going on here?

Kind regards,

Dirkjan
Reply | Threaded
Open this post in threaded view
|

Re: OutOfMemory encountered in Artemis CoreAmqpConverter

tabish121@gmail.com
On 3/27/20 10:00 AM, Dirkjan Ochtman wrote:

> Hi there,
>
> I'm still trying to build an AMQP RPC message flow. I have the request path
> working and am now debugging the response path. In doing so, it seems that
> the response message is correctly being sent over the expected channel, but
> the broker seems to die from an out of memory error. In trying to
> understand what's going on, I've got some more details, but not sure yet
> what's going on.
>
> This is with Artemis 2.6.2.
>
> In CoreAmqpConverter.convertBody(), my message hits the ServerJMSMessage
> branch. The CoreMessage's buffer has ridx 0 and widx 9755, with
> endOfBodyPosition = 9218. For some reason, the readerIndex from
> message.getInnerMessage().getBodyBuffer().readerIndex() ends up being 4
> (it's not clear to me exactly how/why this happens). This seems to be
> skipping over a prefix that contains [0, 0, 36, 2]. That data is directly
> followed by what I recognize as the actual message contents, with starts
> with a 5-byte alphanumeric string. However, when the code then tries to
> readNullableSimpleString() from the buffer it seems to read the length of
> the buffer to be used from the alpha string, resulting in a buffer length
> that ends up going out of bounds. SimpleString.readNullableSimpleString()
> then reads another byte from the buffer to check whether it's null (it does
> not appear to rewind the reader index if it isn't?). readSimpleString then
> reads a length of 1.8 GB and the process dies.
>
> Can someone explain to me what might be going on here?
>
> Kind regards,
>
> Dirkjan
>
I'd suggest trying with Artemis 2.11.0 as the version you are on is
quite old and there have been many AMQP bug fixes since then that could
account for some issues you are seeing.

--
Tim Bish

Reply | Threaded
Open this post in threaded view
|

Re: OutOfMemory encountered in Artemis CoreAmqpConverter

Dirkjan Ochtman
On Fri, Mar 27, 2020 at 8:45 PM Timothy Bish <[hidden email]> wrote:

> I'd suggest trying with Artemis 2.11.0 as the version you are on is
> quite old and there have been many AMQP bug fixes since then that could
> account for some issues you are seeing.
>

That's fair. :) I forked the product that includes the server to upgrade
Artemis to 2.11.0 and was able to get a proper response now.

However, this took some time to debug because of a particular change that I
hadn't expected: it appears that the JMSReplyTo property now gets prefixed
with "queue://" for AMQP messages (but not for Core messages). This seems
fairly surprising -- the server product I'm working with failed to work
correctly because this reply to address doesn't match the BINDING_ADDED
notification's `name`. Is this intended behavior?

Kind regards,

Dirkjan
Reply | Threaded
Open this post in threaded view
|

Re: OutOfMemory encountered in Artemis CoreAmqpConverter

Dirkjan Ochtman
On Thu, Apr 2, 2020 at 1:27 PM Dirkjan Ochtman <[hidden email]> wrote:

> On Fri, Mar 27, 2020 at 8:45 PM Timothy Bish <[hidden email]> wrote:
>
>> I'd suggest trying with Artemis 2.11.0 as the version you are on is
>> quite old and there have been many AMQP bug fixes since then that could
>> account for some issues you are seeing.
>>
>
> That's fair. :) I forked the product that includes the server to upgrade
> Artemis to 2.11.0 and was able to get a proper response now.
>
> However, this took some time to debug because of a particular change that
> I hadn't expected: it appears that the JMSReplyTo property now gets
> prefixed with "queue://" for AMQP messages (but not for Core messages).
> This seems fairly surprising -- the server product I'm working with failed
> to work correctly because this reply to address doesn't match the
> BINDING_ADDED notification's `name`. Is this intended behavior?
>

Actually, I spoke too soon. After stripping the weird "queue://" prefix and
decoding the returned message, it turns out to contain the message
"Conversion to AMQP error!". The ignored exception that triggers this is an
IndexOutOfBoundsException with detailMessage "Error reading in
simpleString, length=1869767777 is greater than readableBytes=9200", so it
seems that there is still a similar problem (also, it might be nice to
splice the exception message in the AMQP response directly).

Kind regards,

Dirkjan
Reply | Threaded
Open this post in threaded view
|

Re: OutOfMemory encountered in Artemis CoreAmqpConverter

Dirkjan Ochtman
On Fri, Apr 3, 2020 at 1:49 PM Dirkjan Ochtman <[hidden email]> wrote:

> On Thu, Apr 2, 2020 at 1:27 PM Dirkjan Ochtman <[hidden email]> wrote:
>
>> However, this took some time to debug because of a particular change that
>> I hadn't expected: it appears that the JMSReplyTo property now gets
>> prefixed with "queue://" for AMQP messages (but not for Core messages).
>> This seems fairly surprising -- the server product I'm working with failed
>> to work correctly because this reply to address doesn't match the
>> BINDING_ADDED notification's `name`. Is this intended behavior?
>>
>
> Actually, I spoke too soon. After stripping the weird "queue://" prefix
> and decoding the returned message, it turns out to contain the message
> "Conversion to AMQP error!". The ignored exception that triggers this is an
> IndexOutOfBoundsException with detailMessage "Error reading in
> simpleString, length=1869767777 is greater than readableBytes=9200", so it
> seems that there is still a similar problem
>

It seems the issue here is that the CoreAmqpConverter wants to encode
messages as a string value body by default. This seems like a surprising
default to me, but it was easily worked around by setting the ClientMessage
type to BYTES_TYPE.


> (also, it might be nice to splice the exception message in the AMQP
> response directly).
>

Created https://github.com/apache/activemq-artemis/pull/3060 for this.

Kind regards,

Dirkjan