ActiveMQ Stomp producer messages size limitation

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

ActiveMQ Stomp producer messages size limitation

khandelwalanuj
Hi,

I am not able to send message more than 100MB to ActiveMQ broker. Producer throws below exception:

[20150113 03:16:59:609 stomp_client.py:80 ERROR] Received an error: org.apache.activemq.transport.stomp.ProtocolException: The maximum data length was exceeded
        at org.apache.activemq.transport.stomp.StompWireFormat.unmarshal(StompWireFormat.java:128)
        at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
        at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
        at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
        at java.lang.Thread.run(Thread.java:745)


I checked the source code (http://activemq.apache.org/maven/apidocs/src-html/org/apache/activemq/transport/stomp/StompWireFormat.html) and found that "private static final int MAX_DATA_LENGTH = 1024 * 1024 * 100;" is causing the issue. How can I change it ? what should I do to send 140 size messages ?

Thanks,
Anuj
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ Stomp producer messages size limitation

tabish121@gmail.com
On 01/13/2015 09:05 AM, khandelwalanuj wrote:

> Hi,
>
> I am not able to send message more than 100MB to ActiveMQ broker. Producer
> throws below exception:
>
> /[20150113 03:16:59:609 stomp_client.py:80 ERROR] Received an error:
> org.apache.activemq.transport.stomp.ProtocolException: The maximum data
> length was exceeded
>          at
> org.apache.activemq.transport.stomp.StompWireFormat.unmarshal(StompWireFormat.java:128)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
>          at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
>          at java.lang.Thread.run(Thread.java:745)/
>
> I checked the source code
> (http://activemq.apache.org/maven/apidocs/src-html/org/apache/activemq/transport/stomp/StompWireFormat.html)
> and found that "private static final int MAX_DATA_LENGTH = 1024 * 1024 *
> 100;" is causing the issue. How can I change it ? what should I do to send
> 140 size messages ?
>
> Thanks,
> Anuj
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/ActiveMQ-Stomp-producer-messages-size-limitation-tp4689847.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
You need to configure a large maxFrameSize on the STOMP
transportConnector in your Broker configuration, see:
http://activemq.apache.org/xml-configuration.html

--
Tim Bish
Sr Software Engineer | RedHat Inc.
[hidden email] | www.redhat.com
skype: tabish121 | twitter: @tabish121
blog: http://timbish.blogspot.com/

Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ Stomp producer messages size limitation

Tim Bain
In reply to this post by khandelwalanuj
One option would be to download the 5.10.0 code, change the value of that
constant, build and deploy.  But you'll have to test to make sure that
limit wasn't protecting the broker from some problem that happens with
messages larger than 100MB.

Another would be to split the message in half and have your consumer only
ack both messages when the last one is received.  You have to make sure the
same consumer gets all the messages that comprise the original;
JMSCorrelationID might be useful here.

You could put the actual content somewhere both producer and consumer can
access and just pass a reference in the message.  This might or might not
work for your use case, but if it does it'll allow you to reduce the memory
the broker needs, which is a good thing.

You could reduce the amount of data you're sending.  What single message is
140MB in size, anyway?
On Jan 13, 2015 7:08 AM, "khandelwalanuj" <[hidden email]>
wrote:

> Hi,
>
> I am not able to send message more than 100MB to ActiveMQ broker. Producer
> throws below exception:
>
> /[20150113 03:16:59:609 stomp_client.py:80 ERROR] Received an error:
> org.apache.activemq.transport.stomp.ProtocolException: The maximum data
> length was exceeded
>         at
>
> org.apache.activemq.transport.stomp.StompWireFormat.unmarshal(StompWireFormat.java:128)
>         at
>
> org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
>         at
> org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
>         at
> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
>         at java.lang.Thread.run(Thread.java:745)/
>
> I checked the source code
> (
> http://activemq.apache.org/maven/apidocs/src-html/org/apache/activemq/transport/stomp/StompWireFormat.html
> )
> and found that "private static final int MAX_DATA_LENGTH = 1024 * 1024 *
> 100;" is causing the issue. How can I change it ? what should I do to send
> 140 size messages ?
>
> Thanks,
> Anuj
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-Stomp-producer-messages-size-limitation-tp4689847.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ Stomp producer messages size limitation

khandelwalanuj
Hi,

I am sending the complete stack trace, Please see below:
[I have added 'maxFrameSize' in broker URI, I didn't change anything on client size: I have tried with and without 'maxframesize' but result is always same. I am getting below error]:

[20150204 19:00:54:306 stomp_client.py:80 ERROR] Received an error: org.apache.activemq.transport.stomp.ProtocolException: The maximum data length was exceeded
        at org.apache.activemq.transport.stomp.StompWireFormat.unmarshal(StompWireFormat.java:128)
        at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
        at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
        at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
        at java.lang.Thread.run(Thread.java:745)

[20150204 19:00:54:306 stomp_client.py:80 ERROR] Received an error: org.apache.activemq.transport.stomp.ProtocolException: The maximum command length was exceeded
        at
org.apache.activemq.transport.stomp.StompWireFormat.readHeaderLine(StompWireFormat.java:158)
        at org.apache.activemq.transport.stomp.StompWireFormat.readLine(StompWireFormat.java:148)
        at org.apache.activemq.transport.stomp.StompWireFormat.parseAction(StompWireFormat.java:181)
        at org.apache.activemq.transport.stomp.StompWireFormat.unmarshal(StompWireFormat.java:98)
        at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
        at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
        at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
        at java.lang.Thread.run(Thread.java:745)


Thanks,
Anuj
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ Stomp producer messages size limitation

khandelwalanuj
Another concern here:

As Tim mentioned :
> You need to configure a large maxFrameSize on the STOMP transportConnector in your Broker
> configuration, see: http://activemq.apache.org/xml-configuration.html

I looked into the source code of ActiveMQ. I don't think changing the 'maxFrameSize' will work here. From the stack trace I posted I can clearly see that 'MAX_DATA_LENGTH' and 'MAX_COMMAND_LENGTH' is exceeded when I try to send more than 100MB of data. And these are constants defined in StompWireFormat.java. I didn't find any way to modified these parameters in entire ActiveMQ source code.

My Questions:

1. Is there any way I can send more than 100MB of data from STOMP (python and perl) producers?
2. What is the difference between 'MAX_DATA_LENGTH' and 'MAX_COMMAND_LENGTH' ?
3. These parameters should be configurable. Should I log a JIRA for this ?

Thanks,
Anuj

Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ Stomp producer messages size limitation

Tim Bain
1. As I said in my earlier email, you can download the code, change that
constant, and recompile, or you can split your messages and make your
clients handle that properly (which may be problematic since you need to
ensure that all parts of a given message are consumed, in order, by the
same consumer).  Maybe someone else has a way to do this without doing one
of those, but I think you'd have heard it by now if someone had an answer
for you.
2. I can't answer that, but maybe someone else can.
3. Sounds like a great idea, and it sounds like it'll be an easy fix for
someone.  (Maybe even you?  You've clearly looked at this code enough to
know it better than most people.)  Too bad it missed the cutoff for 5.11.0,
though.

Tim

On Wed, Feb 4, 2015 at 7:21 AM, khandelwalanuj <[hidden email]>
wrote:

> Another concern here:
>
> As Tim mentioned :
> > You need to configure a large maxFrameSize on the STOMP
> transportConnector
> > in your Broker
> > configuration, see: http://activemq.apache.org/xml-configuration.html
>
> I looked into the source code of ActiveMQ. I don't think changing the
> 'maxFrameSize' will work here. From the stack trace I posted I can clearly
> see that 'MAX_DATA_LENGTH' and 'MAX_COMMAND_LENGTH' is exceeded when I try
> to send more than 100MB of data. And these are constants defined in
> StompWireFormat.java. I didn't find any way to modified these parameters in
> entire ActiveMQ source code.
>
> *My Questions:
> *
> 1. Is there any way I can send more than 100MB of data from STOMP (python
> and perl) producers?
> 2. What is the difference between 'MAX_DATA_LENGTH' and
> 'MAX_COMMAND_LENGTH'
> ?
> 3. These parameters should be configurable. Should I log a JIRA for this ?
>
> Thanks,
> Anuj
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-Stomp-producer-messages-size-limitation-tp4689847p4691050.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>