nio buffer memory problem

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

nio buffer memory problem

RuralHunter
Hi,

We've been run activemq 5.10.2 with thousands of clients in our production for quite a long time. Several days ago, we upgraded to the newer version 5.13.3. After running for several days, we saw the OutOfMemoryError and activemq hung up in busy GC:
2016-06-06 02:01:54,238 | ERROR | Error in thread 'ActiveMQ BrokerService[localhost] Task-10229' | org.apache.activemq.thread.TaskRunnerFactory | ActiveMQ BrokerService[localhost] Task-10229
  java.lang.OutOfMemoryError: Direct buffer memory
        at java.nio.Bits.reserveMemory(Bits.java:658)[:1.7.0_60]
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)[:1.7.0_60]
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:306)[:1.7.0_60]
        at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:174)[:1.7.0_60]
        at sun.nio.ch.IOUtil.write(IOUtil.java:58)[:1.7.0_60]
        at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487)[:1.7.0_60]
        at org.apache.activemq.transport.nio.NIOOutputStream.write(NIOOutputStream.java:206)[activemq-client-5.13.3.jar:5.13.3]
        at org.apache.activemq.transport.nio.NIOOutputStream.write(NIOOutputStream.java:116)[activemq-client-5.13.3.jar:5.13.3]
        at java.io.DataOutputStream.write(DataOutputStream.java:107)[:1.7.0_60]
        at org.apache.activemq.openwire.v10.BaseDataStreamMarshaller.tightMarshalByteSequence2(BaseDataStreamMarshaller.java:431)[activemq-client-5.13.3.jar:5.13.3]
        at org.apache.activemq.openwire.v10.MessageMarshaller.tightMarshal2(MessageMarshaller.java:182)[activemq-client-5.13.3.jar:5.13.3]
        at org.apache.activemq.openwire.v10.ActiveMQMessageMarshaller.tightMarshal2(ActiveMQMessageMarshaller.java:89)[activemq-client-5.13.3.jar:5.13.3]
        at org.apache.activemq.openwire.v10.ActiveMQTextMessageMarshaller.tightMarshal2(ActiveMQTextMessageMarshaller.java:89)[activemq-client-5.13.3.jar:5.13.3]
        at org.apache.activemq.openwire.OpenWireFormat.tightMarshalNestedObject2(OpenWireFormat.java:426)[activemq-client-5.13.3.jar:5.13.3]
        at org.apache.activemq.openwire.v10.BaseDataStreamMarshaller.tightMarshalNestedObject2(BaseDataStreamMarshaller.java:135)[activemq-client-5.13.3.jar:5.13.3]
        at org.apache.activemq.openwire.v10.MessageDispatchMarshaller.tightMarshal2(MessageDispatchMarshaller.java:105)[activemq-client-5.13.3.jar:5.13.3]
        at org.apache.activemq.openwire.OpenWireFormat.marshal(OpenWireFormat.java:234)[activemq-client-5.13.3.jar:5.13.3]
        at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:193)[activemq-client-5.13.3.jar:5.13.3]

I searched the error on web and found it's nio direct buffer limit problem and we can increase the jvm setting -XX:MaxDirectMemorySize. The default value of this setting is pretty small, only 64m . We never set this value with 5.10.2. So I guessed the new version required more memory than the old version so I set this value to 4g. I thought it should be more than enough. We also started to monitor the direct buffer memory usage which we hadn't done before.

Unfortunately, after one day we saw the direct buffer usage reached 4g and activemq again was busy GC'ing. After a while we saw the same OutOfMemoryError again. I then thought there might be some memory leak in the new version. So I reverted back to 5.10.2, without the MaxDirectMemorySize setting. Interesting thing happened. The old version runs well but the actual direct buffer it uses is far more than 4g. It's around 10g all the time!

So I have 2 questions here:
1. How much direct buffer activemq actually needs, or based what should we set this memory setting?
2. How can activemq 5.10.2 ignores the -XX:MaxDirectMemorySize while the new version can not?
Reply | Threaded
Open this post in threaded view
|

Re: nio buffer memory problem

RuralHunter
I found this article: http://www.evanjones.ca/java-bytebuffer-leak.html
I guess 5.10.2 uses 'Heap ByteBuffers' which is backed with unlimited 'temporary Direct ByteBuffer', while 5.13.3 uses 'Direct ByteBuffer' directly?
Reply | Threaded
Open this post in threaded view
|

Re: nio buffer memory problem

christopher.l.shannon
Yes, the buffer was changed to a direct byte buffer for performance
reasons. This was done in https://issues.apache.org/jira/browse/AMQ-6184

 I think the best thing you can do for now is to keep increasing the size
and see if it stabilizes.  You mentioned that you could see it using 10g
under version 5.10 so maybe you can set it to that and see if it will run
without running out of memory.  My guess is your app just needs that much
memory to run and there isn't an actual memory leak going on but hard to
say until you increase the available memory to the same amount 5.10.2 uses.

On Tue, Jun 7, 2016 at 10:11 PM, RuralHunter <[hidden email]> wrote:

> I found this article: http://www.evanjones.ca/java-bytebuffer-leak.html
> I guess 5.10.2 uses 'Heap ByteBuffers' which is backed with unlimited
> 'temporary Direct ByteBuffer', while 5.13.3 uses 'Direct ByteBuffer'
> directly?
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/nio-buffer-memory-problem-tp4712706p4712766.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: nio buffer memory problem

RuralHunter
All right. I will try with that. Thanks for the info.