[ActiveMQ 5.11.1] Question around WireFormat byte array size

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

[ActiveMQ 5.11.1] Question around WireFormat byte array size

mmacpherson
Hello,

This was posted as a sub question to a thread in the Dev forum, but as I am really struggling to understand where this setting is coming from I thought I would ask on the more general User group. Maybe someone has run into this =)

Other form thread subject (MaxFrameSize not honored and allowing 64MB messages to go through)

Basic info, ActiveMQ 5.11 .1 (we have plans to test with 5.14), and we are getting some odd results while a node becomes the active broker.  

at org.apache.activemq.openwire.v10.BaseDataStreamMarshaller.tightUnmarshalByteSequence
at org.apache.activemq.openwire.v10.WireFormatInfoMarshaller.tightUnmarshal
at org.apache.activemq.openwire.OpenWireFormat.doUnmarshal
at org.apache.activemq.openwire.OpenWireFormat.unmarshal

The core issue for us appears to be the byte stream is getting parsed with tightEncodingEnabled set to TRUE, but the stream itself is not actually formatted in this manner. This takes things on a unexpected path.

My concern is that if we address this, simply but setting tightEncodingEnabled to false, I still see some issues with the Byte stream.

The Size value being provided before the ActiveMQ Magic bytes and the Datatype parameter appears to be a negative value.

byte|[21] |0
byte|[22] |0
byte|[23] |0
byte|[24] |-16

byte|[25] |1
byte|[26] |65 ->A
byte|[27] |99 ->c
byte|[28] |116 ->t
byte|[29] |105 ->i
byte|[30] |118 ->v
byte|[31] |101 ->e
byte|[32] |77 ->M
byte|[33] |81 ->Q

And then after this, if its parsed correctly, after the Magic bytes and then the Version bytes, the next section is used to control the size of a Byte[] that’s created. When you do the shifting that size should turn out to be  16777216:

ActiveMQ magic:
byte|[26] |65 ->A
byte|[27] |99 ->c
byte|[28] |116 ->t
byte|[29] |105 ->i
byte|[30] |118 ->v
byte|[31] |101 ->e
byte|[32] |77 ->M
byte|[33] |81 ->Q
Version (10)
byte|[34] |0
byte|[35] |0
byte|[36] |0
byte|[37] |10
Byte array size (16777216)
byte|[38] |1
byte|[39] |0
byte|[40] |0
byte|[41] |0

Is anyone aware of a setting we are missing? We use the ActiveMQSslConnectionFactory and pass the following params in currently:

initialReconnectDelay=10
maxReconnectDelay=30000
maxReconnectAttempts=10
startupMaxReconnectAttempts=1
timeout=60000
randomize=false
useExponentialBackOff=true
priorityBackup=true
jms.useAsyncSend=true
nested.soTimeout=60000
nested.soWriteTimeout=60000
nested.wireFormat.maxFrameSize=262144

Matt
Reply | Threaded
Open this post in threaded view
|

Re: [ActiveMQ 5.11.1] Question around WireFormat byte array size

christopher.l.shannon
What happens if you turn off caching for the wireFormat by setting
wireFormat.cacheEnabled=false ?

On Wed, Aug 24, 2016 at 12:58 PM, mmacpherson <[hidden email]>
wrote:

> Hello,
>
> This was posted as a sub question to a thread in the Dev forum, but as I am
> really struggling to understand where this setting is coming from I thought
> I would ask on the more general User group. Maybe someone has run into this
> =)
>
> Other form thread subject (MaxFrameSize not honored and allowing 64MB
> messages to go through)
>
> Basic info, ActiveMQ 5.11 .1 (we have plans to test with 5.14), and we are
> getting some odd results while a node becomes the active broker.
>
> at
> org.apache.activemq.openwire.v10.BaseDataStreamMarshaller.
> tightUnmarshalByteSequence
> at org.apache.activemq.openwire.v10.WireFormatInfoMarshaller.
> tightUnmarshal
> at org.apache.activemq.openwire.OpenWireFormat.doUnmarshal
> at org.apache.activemq.openwire.OpenWireFormat.unmarshal
>
> The core issue for us appears to be the byte stream is getting parsed with
> tightEncodingEnabled set to TRUE, but the stream itself is not actually
> formatted in this manner. This takes things on a unexpected path.
>
> My concern is that if we address this, simply but setting
> tightEncodingEnabled to false, I still see some issues with the Byte
> stream.
>
> The Size value being provided before the ActiveMQ Magic bytes and the
> Datatype parameter appears to be a negative value.
>
> byte|[21] |0
> byte|[22] |0
> byte|[23] |0
> byte|[24] |-16
>
> byte|[25] |1
> byte|[26] |65 ->A
> byte|[27] |99 ->c
> byte|[28] |116 ->t
> byte|[29] |105 ->i
> byte|[30] |118 ->v
> byte|[31] |101 ->e
> byte|[32] |77 ->M
> byte|[33] |81 ->Q
>
> And then after this, if its parsed correctly, after the Magic bytes and
> then
> the Version bytes, the next section is used to control the size of a Byte[]
> that’s created. When you do the shifting that size should turn out to be
> 16777216:
>
> ActiveMQ magic:
> byte|[26] |65 ->A
> byte|[27] |99 ->c
> byte|[28] |116 ->t
> byte|[29] |105 ->i
> byte|[30] |118 ->v
> byte|[31] |101 ->e
> byte|[32] |77 ->M
> byte|[33] |81 ->Q
> Version (10)
> byte|[34] |0
> byte|[35] |0
> byte|[36] |0
> byte|[37] |10
> Byte array size (16777216)
> byte|[38] |1
> byte|[39] |0
> byte|[40] |0
> byte|[41] |0
>
> Is anyone aware of a setting we are missing? We use the
> ActiveMQSslConnectionFactory and pass the following params in currently:
>
> initialReconnectDelay=10
> maxReconnectDelay=30000
> maxReconnectAttempts=10
> startupMaxReconnectAttempts=1
> timeout=60000
> randomize=false
> useExponentialBackOff=true
> priorityBackup=true
> jms.useAsyncSend=true
> nested.soTimeout=60000
> nested.soWriteTimeout=60000
> nested.wireFormat.maxFrameSize=262144
>
> Matt
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/ActiveMQ-5-11-1-Question-around-WireFormat-
> byte-array-size-tp4715794.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: [ActiveMQ 5.11.1] Question around WireFormat byte array size

mmacpherson
Thanks we can give that a try in one of our labs, but I am not seeing what impact that would have on the byte array that ends up getting allocated within tightUnmarshalByteSequence ( BaseDataStreamMarshaller.java). This allocation would still end up consuming 16mb, and if say when the node becomes the active broker it creates 70 threads for a IP, you’re going to end up consuming a good chunk of your memory.