ArrayIndexOutOfBoundsException in ActiveMQ v5.11.1

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

ArrayIndexOutOfBoundsException in ActiveMQ v5.11.1

xabhi
This post was updated on .
Hi ActiveMQ devs,

I observed below ArrayIndexOutOfBounds exception in my production setup and it caused high non-persistent memory usage on the affected topic due to transaction rollbacks. It is not clear why this happened.

Could you please help in investigating this? What could be the possible causes for this?

[ 2016-08-08 06:37:34,321 [ActiveMQ Transport: tcp://mqbrokerprod/149.77.3.169:61616@52260]  failover.FailoverTransport.handleTransportFailure():260 WARN ]: Transport (tcp://mqbrokerprod/149.77.3.169:61616@52260) failed, reason:  , attempting to automatically reconnect
java.io.IOException: Unexpected error occured: java.lang.ArrayIndexOutOfBoundsException: 12851
        at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:203)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 12851
        at org.apache.activemq.openwire.OpenWireFormat.getFromUnmarshallCache(OpenWireFormat.java:534)
        at org.apache.activemq.openwire.v10.BaseDataStreamMarshaller.tightUnmarsalCachedObject(BaseDataStreamMarshaller.java:148)
        at org.apache.activemq.openwire.v10.MessageMarshaller.tightUnmarshal(MessageMarshaller.java:75)
        at org.apache.activemq.openwire.v10.ActiveMQMessageMarshaller.tightUnmarshal(ActiveMQMessageMarshaller.java:66)
        at org.apache.activemq.openwire.v10.ActiveMQTextMessageMarshaller.tightUnmarshal(ActiveMQTextMessageMarshaller.java:66)
        at org.apache.activemq.openwire.OpenWireFormat.tightUnmarshalNestedObject(OpenWireFormat.java:445)
        at org.apache.activemq.openwire.v10.BaseDataStreamMarshaller.tightUnmarsalNestedObject(BaseDataStreamMarshaller.java:125)
        at org.apache.activemq.openwire.v10.MessageDispatchMarshaller.tightUnmarshal(MessageDispatchMarshaller.java:71)
        at org.apache.activemq.openwire.OpenWireFormat.doUnmarshal(OpenWireFormat.java:354)
        at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268)
        at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:221)
        at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:213)
        at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:196)
        ... 1 more
[ 2016-08-08 06:37:34,443 [ActiveMQ Task-3]  failover.FailoverTransport.doReconnect():1057 INFO ]: Successfully reconnected to tcp://mqbrokerprod:61616?wireFormat.maxInactivityDuration=120000
[ 2016-08-08 06:37:34,444 [referenceDataSubscriberContainer-1]  activemq.TransactionContext.commit():332 INFO ]: commit failed for transaction TX:ID:helium42.nyc.deshaw.com-58718-1470637448000-3:1:1411
javax.jms.TransactionRolledBackException: Transaction completion in doubt due to failover. Forcing rollback of TX:ID:helium42.nyc.deshaw.com-58718-1470637448000-3:1:1411
        at org.apache.activemq.state.ConnectionStateTracker.restoreTransactions(ConnectionStateTracker.java:254)
        at org.apache.activemq.state.ConnectionStateTracker.restore(ConnectionStateTracker.java:191)
        at org.apache.activemq.transport.failover.FailoverTransport.restoreTransport(FailoverTransport.java:842)
        at org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:1020)
        at org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:148)
        at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:133)
        at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Thanks,
Abhi
Reply | Threaded
Open this post in threaded view
|

Re: ArrayIndexOutOfBoundsException in ActiveMQ v5.11.1

xabhi
Is this something I should worry about happening again? What scenarios can cause the ArrayIndexOutOfBoundsException exception (in the first post)?

Thanks,
Abhi

Reply | Threaded
Open this post in threaded view
|

Re: ArrayIndexOutOfBoundsException in ActiveMQ v5.11.1

xabhi
Hi,

What do these exceptions indicate? I saw below exceptions along with the ArrayIndexOutOfBounds exception but without the stack strace i an not able to figure out anything?

Has anyone else encountered these exceptions before?

Exceptions:
org.apache.activemq.broker.TransportConnection#serviceTransportException 245 WARN] - Transport Connection to: tcp://10.79.30.101:33184 failed: java.io.IOException: Direct buffer memory :

org.apache.activemq.broker.TransportConnection#serviceTransportException 245 WARN] - Transport Connection to: tcp://10.79.27.107:29377 failed: java.io.IOException: Java heap space :

Please help as I am encountering these exceptions almost daily in production? The log messages are not very informative about what has to be done as well.

Thanks,
Abhi
Reply | Threaded
Open this post in threaded view
|

Re: ArrayIndexOutOfBoundsException in ActiveMQ v5.11.1

Martin Herrman

Hi Xabhi,


I'm really a noob in ActiveMQ, but reading your mails, I would do the following:
1. create a script that reads the log file and automates the repair actions when the issue happens
2. upgrade to the latest release (5.11.1 is dated february 2015)
3. because I do see openwire in your error log: check if the indexes have the latest version. For more info see the archives for 'Updating KahaDB' at http://mail-archives.apache.org/mod_mbox/activemq-users/201608.mbox/browser

HTH,

Martin






Van: xabhi <[hidden email]>
Verzonden: woensdag 10 augustus 2016 8:45
Aan: [hidden email]
Onderwerp: Re: ArrayIndexOutOfBoundsException in ActiveMQ v5.11.1

Hi,

What do these exceptions indicate? I saw below exceptions along with the
ArrayIndexOutOfBounds exception but without the stack strace i an not able
to figure out anything?

Has anyone else encountered these exceptions before?

Exceptions:
org.apache.activemq.broker.TransportConnection#serviceTransportException 245
WARN] - Transport Connection to: tcp://10.79.30.101:33184 failed:
java.io.IOException: Direct buffer memory :

org.apache.activemq.broker.TransportConnection#serviceTransportException 245
WARN] - Transport Connection to: tcp://10.79.27.107:29377 failed:
java.io.IOException: Java heap space :

Please help as I am encountering these exceptions almost daily in
production? The log messages are not very informative about what has to be
done as well.

Thanks,
Abhi



--
View this message in context:  http://activemq.2283324.n4.nabble.com/ArrayIndexOutOfBoundsException-in-ActiveMQ-v5-11-1-tp4715139p4715293.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Met vriendelijke groet/Kind regards,

Martin Herrman
Manager DevOps

[YMOR]<http://www.ymor.com>

Ptolemaeuslaan 70
3528 BP Utrecht
Mobiel: +31 6 10761936
Kantoor:        +31(0)30 6021149

Email:  [hidden email]

[ITOA]
Van reactief naar preventief IT-beheer?
Download de whitepaper:
'Van achteruitkijkspiegel naar glazen bol met IT Operations Analytics’<http://www.ymor.com/whitepapers/>


This e-mail message and its attachments are subject to the disclaimer published on our website<https://www.ymor.com/files/2016/04/2016-Disclaimer-and-Security-EN.pdf>
Reply | Threaded
Open this post in threaded view
|

Re: ArrayIndexOutOfBoundsException in ActiveMQ v5.11.1

Tim Bain
In reply to this post by xabhi
RuralHunter had what appears to be the same issue not long ago:
http://activemq.2283324.n4.nabble.com/nio-buffer-memory-problem-td4712706.html

AMQ-6184 <https://issues.apache.org/jira/browse/AMQ-6184> appears to fix
the problem (rather than just work around it like the suggestions in the
mailing list thread), but it would require an upgrade to the just-released
5.14.0.

Tim

On Wed, Aug 10, 2016 at 12:45 AM, xabhi <[hidden email]> wrote:

> Hi,
>
> What do these exceptions indicate? I saw below exceptions along with the
> ArrayIndexOutOfBounds exception but without the stack strace i an not able
> to figure out anything?
>
> Has anyone else encountered these exceptions before?
>
> Exceptions:
> org.apache.activemq.broker.TransportConnection#serviceTransportException
> 245
> WARN] - Transport Connection to: tcp://10.79.30.101:33184 failed:
> java.io.IOException: Direct buffer memory :
>
> org.apache.activemq.broker.TransportConnection#serviceTransportException
> 245
> WARN] - Transport Connection to: tcp://10.79.27.107:29377 failed:
> java.io.IOException: Java heap space :
>
> Please help as I am encountering these exceptions almost daily in
> production? The log messages are not very informative about what has to be
> done as well.
>
> Thanks,
> Abhi
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/
> ArrayIndexOutOfBoundsException-in-ActiveMQ-v5-11-1-tp4715139p4715293.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: ArrayIndexOutOfBoundsException in ActiveMQ v5.11.1

xabhi
Hi Tim,

Thanks for taking a look at this. But rural hunter encountered this problem in ActiveMQ v5.13.3 and I am seeing these warnings on server in ActiveMQ v5.11.1

Also do you have any idea what can cause ArrayIndexOutOfBounds exception on client side in ActiveMQ v5.11.1 (see first message) ? Is this is a bug?

Could you give some pointers that can help identify the cause of these warnings so that I can debug this at my end.

Thanks,
Abhi
Reply | Threaded
Open this post in threaded view
|

Re: ArrayIndexOutOfBoundsException in ActiveMQ v5.11.1

Tim Bain
Right, what I'm saying is that if this bug is fixed in 5.14.0, upgrading
would be the least-work option to fix your problem.  And of course, just
because the description looks like it should fix the problem doesn't mean
it's guaranteed to.

If that's not an appealing option, you could attach a debugger and step
through the affected code to see exactly what it's doing.  You could also
grab the 5.11.1 code and attempt to apply to patch to it and recompile.
Also, reading the source code that was changed for that fix might provide
some insight into exactly what the root cause was.

Tim

On Aug 22, 2016 3:52 AM, "xabhi" <[hidden email]> wrote:

> Hi Tim,
>
> Thanks for taking a look at this. But rural hunter encountered this problem
> in ActiveMQ v5.13.3 and I am seeing these warnings on server in ActiveMQ
> v5.11.1
>
> Also do you have any idea what can cause ArrayIndexOutOfBounds exception on
> client side in ActiveMQ v5.11.1 ( see first message
> <http://activemq.2283324.n4.nabble.com/ArrayIndexOutOfBoundsException
> -in-ActiveMQ-v5-11-1-tp4715139.html>
> ) ? Is this is a bug?
>
> Could you give some pointers that can help identify the cause of these
> warnings so that I can debug this at my end.
>
> Thanks,
> Abhi
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/
> ArrayIndexOutOfBoundsException-in-ActiveMQ-v5-11-1-tp4715139p4715698.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>