I'm facing a strange memory leak on one of my production environment (others
environments are ok)
1) AIX 7.1
2) IBM JVM 1.7
3) ActiveMQ broker 5.12.2 embedded in our java monitoring application
3) Client are either java client (embedded inside a front office war
application) or C++ client based on CMS 3.8 (Back office standalone
the operating mode is straightforward : front office application invoke
business services (request - reply) to back office applications.
We are sure that all requests are correctly queued and de-queued. We also
didn't notice slow consumer effects.
Our memory analysis tools points 2 suspects and one hint:
suspect 1 : 113 206 instance of ActiveMQMessage
suspect 2 : 157 instances of
hint : all suspects have a common chain : o.a.a.broker.BrokerService =>
can that mean that the problem is related to jmx monitoring ?
From the side, it does look like pending messages accumulated in memory
because of a slow or blocked subscriber. Since you have heap dump, you
should look at this specific TopicSubscription object just to make sure you
don't have actually a consumer subscribed to some of your topics, but not
accepting messages (e.g., a consumer created but no listener attached).
Within TopicSubscription you'll have destination info, consumer info, and
some other useful data that could indicate the cause of your issue.
On a side note, it looks like you are using VM Cursor, which was the main
cursor type prior to version 5. It holds pending messages in memory, thus
can cause OOM. By default, 5.x would use Store cursor avoiding memory
issues. You can read more about it here:
http://activemq.apache.org/message-cursors.html .. and I would advise not using VM Cursor.