Properly closing transport threads in ActiveMQ with listeners in Spring-servlet

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Properly closing transport threads in ActiveMQ with listeners in Spring-servlet

Arjen van der Meijden
Hi list,

I know this question has occasionally been asked before, but I can't
seem to figure out the correct method with the most recent ActiveMQ.

Our application runs with an in-memory database which is kept in sync by
listening to about 30 different topics on ActiveMQ which are monitored
via Spring-JMS' listener-setup via ActiveMQ's JMS-client (version 5.15.6).
The Spring version is the latest stable 5.x (altough it started with
2.x), is deployed on Tomcat 8.5 which runs on Java 10.

All libraries, including ActiveMQ are integrated in the war-archive
(i.e. no shared libraries).

Whenever the application is undeployed, Tomcat warns about Threads not
being closed:

WARNING [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
The web application [appname] appears to have started a thread named
[ActiveMQ InactivityMonitor ReadCheckTimer] but has failed to stop it.
This is very likely to create a memory leak.

 17:05:20.267 WARNING
[ContainerBackgroundProcessor[StandardEngine[Catalina]]]
org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
The web application [appname] appears to have started a thread named
[ActiveMQ Transport: tcp://servername/10.10.10.10:61616@49831] but has
failed to stop it. This is very likely to create a memory leak.

And some more.

I know in other discussions this has been dismissed as a non-issue/false
positive. But the ClassLoader for the specific app cannot be garbage
collected, which fixes all class-objects to remain, which locks all
static-properties (including some static caches of beans and other
objects) in memory. And the threads never seem to close (at least not in
several minutes) after undeploy; the connection does seem to have been
disconnected (at least, there are no consumers anymore on the broker).
Whether the memory leak is really due to the ActiveMQ-threads is
somewhat harder to proof, but I'd like to be able to eliminate that
option regardless.

Currently the configuration is in Spring-xml:
    <bean id="activeMqConnectionFactory"
class="org.apache.activemq.ActiveMQConnectionFactory" />
    <bean id="jmsConnectionFactory"
class="org.apache.activemq.pool.PooledConnectionFactoryBean"
destroy-method="destroy">
        <property name="connectionFactory"
ref="activeMqConnectionFactory" />
    </bean>
  <jms:listener-container connection-factory="jmsConnectionFactory"
destination-type="topic">
         <jms:listener destination="va_cache_advertisements"
ref="advertisementListener" />
    </jms:listener-container>

My initial configuration didn't use the PooledConnectionFactoryBean. But
while this reduces the amount of concurrent connections (and thus
threads), it didn't actually close any of the threads. The
advertisementListener is a MessageListener-implementation and has no
access to the JMS-connection.

I already have a ServletContextListener which is called upon undeploy
(to stop a few other leaking threads). And I did add ActiveMQ's
DefaultThreadPools's shutdown. I also tried a 'hack' to access
AbstractInactivityMonitor.stopMonitorThreads to stop the statically
cached threads/timers there. But both options don't seem to matter.

I've tried to find a proper resource on how to fix this, but most search
results seem to be very old and/or reported not to work.
So, how do I actually make sure those Threads (both inactivity
monitoring and tcp transport) are properly closed and/or any other
object that needs to be closed?

Thanks in advance,

Arjen
Reply | Threaded
Open this post in threaded view
|

Re: Properly closing transport threads in ActiveMQ with listeners in Spring-servlet

Arjen van der Meijden
It looks like the below configuration for Spring is actually working.
Is it indeed the expected way to do this properly?

Best regards,

Arjen

On 25-10-2018 19:46, Arjen van der Meijden wrote:

> Hi list,
>
> I know this question has occasionally been asked before, but I can't
> seem to figure out the correct method with the most recent ActiveMQ.
>
> Our application runs with an in-memory database which is kept in sync by
> listening to about 30 different topics on ActiveMQ which are monitored
> via Spring-JMS' listener-setup via ActiveMQ's JMS-client (version 5.15.6).
> The Spring version is the latest stable 5.x (altough it started with
> 2.x), is deployed on Tomcat 8.5 which runs on Java 10.
>
> All libraries, including ActiveMQ are integrated in the war-archive
> (i.e. no shared libraries).
>
> Whenever the application is undeployed, Tomcat warns about Threads not
> being closed:
>
> WARNING [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
> The web application [appname] appears to have started a thread named
> [ActiveMQ InactivityMonitor ReadCheckTimer] but has failed to stop it.
> This is very likely to create a memory leak.
>
>  17:05:20.267 WARNING
> [ContainerBackgroundProcessor[StandardEngine[Catalina]]]
> org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads
> The web application [appname] appears to have started a thread named
> [ActiveMQ Transport: tcp://servername/10.10.10.10:61616@49831] but has
> failed to stop it. This is very likely to create a memory leak.
>
> And some more.
>
> I know in other discussions this has been dismissed as a non-issue/false
> positive. But the ClassLoader for the specific app cannot be garbage
> collected, which fixes all class-objects to remain, which locks all
> static-properties (including some static caches of beans and other
> objects) in memory. And the threads never seem to close (at least not in
> several minutes) after undeploy; the connection does seem to have been
> disconnected (at least, there are no consumers anymore on the broker).
> Whether the memory leak is really due to the ActiveMQ-threads is
> somewhat harder to proof, but I'd like to be able to eliminate that
> option regardless.
>
> Currently the configuration is in Spring-xml:
>     <bean id="activeMqConnectionFactory"
> class="org.apache.activemq.ActiveMQConnectionFactory" />
>     <bean id="jmsConnectionFactory"
> class="org.apache.activemq.pool.PooledConnectionFactoryBean"
> destroy-method="destroy">
>         <property name="connectionFactory"
> ref="activeMqConnectionFactory" />
>     </bean>
>   <jms:listener-container connection-factory="jmsConnectionFactory"
> destination-type="topic">
>          <jms:listener destination="va_cache_advertisements"
> ref="advertisementListener" />
>     </jms:listener-container>
>
> My initial configuration didn't use the PooledConnectionFactoryBean. But
> while this reduces the amount of concurrent connections (and thus
> threads), it didn't actually close any of the threads. The
> advertisementListener is a MessageListener-implementation and has no
> access to the JMS-connection.
>
> I already have a ServletContextListener which is called upon undeploy
> (to stop a few other leaking threads). And I did add ActiveMQ's
> DefaultThreadPools's shutdown. I also tried a 'hack' to access
> AbstractInactivityMonitor.stopMonitorThreads to stop the statically
> cached threads/timers there. But both options don't seem to matter.
>
> I've tried to find a proper resource on how to fix this, but most search
> results seem to be very old and/or reported not to work.
> So, how do I actually make sure those Threads (both inactivity
> monitoring and tcp transport) are properly closed and/or any other
> object that needs to be closed?
>
> Thanks in advance,
>
> Arjen
>