Help with memory leak

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Help with memory leak

mauromol
Hello,
I'm not an ActiveMQ expert and I have the following problem.

Running a web application in Tomcat 8.x, Java 8, Spring Framework 4.3.18.
My web application both sends and receives messages with ActiveMQ, using
org.apache.activemq:activemq-spring:5.11.0 dependency.

I'm setting up an ActiveMQ connection factory in this way:

        <amq:connectionFactory id="amqJmsFactory" brokerURL="${jms.broker.url}" />
        <bean id="jmsConnectionFactory"
                class="org.apache.activemq.pool.PooledConnectionFactory"
destroy-method="stop">
                <property name="connectionFactory" ref="amqJmsFactory" />
                <property name="maxConnections" value="2" />
                <property name="idleTimeout" value="60000" />
                <property name="timeBetweenExpirationCheckMillis" value="600000" />
                <property name="maximumActiveSessionPerConnection" value="10" />
        </bean>

The last property ('maximumActiveSessionPerConnection') has been set to try
to solve the following problem (the default seems to be 500, which is quite
high IMHO), but I'm not sure it really helped, because I'm still getting
OutOfMemory errors.

This connection factory is referenced by a listener container factory:

        <jms:listener-container factory-id="activationJmsListenerContainerFactory"
                container-type="default" connection-factory="jmsConnectionFactory"
                concurrency="1" transaction-manager="centralTransactionManager">
        </jms:listener-container>

by one Spring Integration 4.3.17 inbound adapter:

        <int-jms:message-driven-channel-adapter
id="invoiceEventJmsInboundChannelAdapter"
                channel="incomingInvoiceEventJmsChannel"
                connection-factory="jmsConnectionFactory"
                destination-name="incomingEvent"
                max-concurrent-consumers="2"
                transaction-manager="customerTransactionManager"
                error-channel="unexpectedErrorChannel" />

and by two outbound adapters:

        <int-jms:outbound-channel-adapter
id="invoiceEventJmsOutboundChannelAdapter"
                channel="outgoingInvoiceEventJmsChannel" destination-name="outgoingEvent"
                connection-factory="jmsConnectionFactory" explicit-qos-enabled="true"
delivery-persistent="true"
                session-transacted="true" />

        <int-jms:outbound-channel-adapter
                id="passwordResetTokenSubmitterJmsOutboundChannelAdapter"
                channel="passwordResetTokenSubmitterJmsChannel"
                destination-name="passwordReset"
                connection-factory="jmsConnectionFactory" explicit-qos-enabled="true"
                delivery-persistent="false" session-transacted="false" />

Things work well, but what I observe is that ActiveMQ, as a message producer
(for the invoiceEventJmsOutboundChannelAdapter adapter), is accumulating a
lot of objects in memory and it's causing OutOfMemory errors in my
application. My messages may be some KBs long, because their payloads are
XML files, but nevertheless I don't expect to hold so much memory for a long
time.

Here are my findings on a heap dump produced on the most recent OutOfMemory
error (using Eclipse MAT to investigate). Two leak suspects are found and
both lead to ConnectionStateTracker.

Here is one of the two accumulators:

Class Name                                                                                                
| Shallow Heap | Retained Heap
-------------------------------------------------------------------------------------------------------------------------------------------
java.util.concurrent.ConcurrentHashMap$HashEntry[4] @ 0xe295da78                                          
|           32 |    58.160.312
'- table java.util.concurrent.ConcurrentHashMap$Segment @ 0xe295da30                                      
|           40 |    58.160.384
   '- [15] java.util.concurrent.ConcurrentHashMap$Segment[16] @ 0xe295d9e0                                
|           80 |    68.573.384
      '- segments java.util.concurrent.ConcurrentHashMap @ 0xe295d9b0                                      
|           48 |    68.573.432
         '- sessions org.apache.activemq.state.ConnectionState @ 0xe295d7e0                                
|           40 |    68.575.312
            '- value java.util.concurrent.ConcurrentHashMap$HashEntry @
0xe295d728                          |           32 |    68.575.344
               '- [1] java.util.concurrent.ConcurrentHashMap$HashEntry[2] @
0xe295d710                      |           24 |    68.575.368
                  '- table java.util.concurrent.ConcurrentHashMap$Segment @
0xe295d6c8                      |           40 |    68.575.440
                     '- [12]
java.util.concurrent.ConcurrentHashMap$Segment[16] @ 0xe295d678              
|           80 |    68.575.616
                        '- segments java.util.concurrent.ConcurrentHashMap @
0xe295d648                     |           48 |    68.575.664
                           '- connectionStates
org.apache.activemq.state.ConnectionStateTracker @ 0xe295d620|           40
|    68.575.808
-------------------------------------------------------------------------------------------------------------------------------------------

As you can see, an instance of ConnectionStateTracker is retaining around 70
MB of heap space. There are two instances of ConnectionStateTracker (one for
each outbound adapter I guess) which are retaining a total of about 120 MB
of heap. They are accumulating it in two instances of ConnectionState, which
in turn have a map of "sessions" containing a cumulated total of 10
SessionState instances, which in turn have a ConcurrentHashMap of producers
holding a cumulated total of 1,258 ProducerState instances. These are
retaining those 120 MB of heap in their transactionState field, which is of
type TransactionState, which in turn has a commands ArrayList which seems to
be retaining the whole messages I'm sending out.

My question is: why ActiveMQ is keeping in memory the messages already sent
out? There are also some security concerns in keeping in memory all those
messages.

Thanks in advance for any help.
Mauro



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html