OutOfMemoryError due to TopicSubscription

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

OutOfMemoryError due to TopicSubscription

Ishitori
Hi,

Recently our live AMQ 5.7.0 server on Windows experienced OutOfMemory exception. I've read the article  http://blog.garytully.com/2009/07/apache-activemq-out-of-memory.html that states that it is usually a problem of configuration. I also read FAQ page http://activemq.apache.org/javalangoutofmemory.html to understand what kind of configuration parameters should I tune.

But what I can't understand is why TopicSubscription is mentioned in the log? I don't use topics at all, I use queues only. But the log message states that TopicSubscription was the reason I got the exception:

2013-02-22 00:27:09,330 | INFO  | TopicSubscription: consumer=ID:AMQ1-49531-1360656683385-98:2:1:1, destinations=7, dispatched=1, de
livered=16458, matched=51393, discarded=0: Pending message cursor [org.apache.activemq.broker.region.cursors.VMPendingMessageCursor@13e97dd]
is full, temp usage (59%) or memory usage (101%) limit reached, blocking message add() pending the release of resources. | org.apache.activ
emq.broker.region.TopicSubscription | ActiveMQ NIO Worker 168983

So, I can guess that cursor got to the limit of memory and stopped processing. But why is it TopicSubscription? If it is a queue then what is the name of that queue? And am I right that I should increase <memoryUsage limit="64 mb"/> to solve the problem in future?

Reply | Threaded
Open this post in threaded view
|

Re: OutOfMemoryError due to TopicSubscription

ceposta
Maybe you can post your complete config file? Are you using network of
brokers? Or using advisory topics?


On Mon, Feb 25, 2013 at 6:45 AM, Ishitori <[hidden email]> wrote:

> Hi,
>
> Recently our live AMQ 5.7.0 server on Windows experienced OutOfMemory
> exception. I've read the article
> http://blog.garytully.com/2009/07/apache-activemq-out-of-memory.html that
> states that it is usually a problem of configuration. I also read FAQ page
> http://activemq.apache.org/javalangoutofmemory.html to understand what
> kind
> of configuration parameters should I tune.
>
> But what I can't understand is why TopicSubscription is mentioned in the
> log? I don't use topics at all, I use queues only. But the log message
> states that TopicSubscription was the reason I got the exception:
>
> 2013-02-22 00:27:09,330 | INFO  | TopicSubscription:
> consumer=ID:AMQ1-49531-1360656683385-98:2:1:1, destinations=7,
> dispatched=1,
> de
> livered=16458, matched=51393, discarded=0: Pending message cursor
> [org.apache.activemq.broker.region.cursors.VMPendingMessageCursor@13e97dd]
> is full, temp usage (59%) or memory usage (101%) limit reached, blocking
> message add() pending the release of resources. | org.apache.activ
> emq.broker.region.TopicSubscription | ActiveMQ NIO Worker 168983
>
> So, I can guess that cursor got to the limit of memory and stopped
> processing. But why is it TopicSubscription? If it is a queue then what is
> the name of that queue? And am I right that I should increase <memoryUsage
> limit="64 mb"/> to solve the problem in future?
>
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/OutOfMemoryError-due-to-TopicSubscription-tp4664025.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Reply | Threaded
Open this post in threaded view
|

Re: OutOfMemoryError due to TopicSubscription

Ishitori
I don't use advisory topics, but I do use network of brokers. There are 2 brokers and they are connected by network connector per queue basis. Here is the example:

           <networkConnector
               name="TopicsOnly-duplex"
               uri="static:(nio://<ip>:62626)"
               networkTTL="2"
               prefetchSize="1"
               duplex="true">
               <dynamicallyIncludedDestinations>
                   <topic physicalName=">"/>
               </dynamicallyIncludedDestinations>
            </networkConnector>

            <networkConnector
               name="SpecificQueue-duplex"
               uri="static:(nio://<ip>:62626)"
               networkTTL="2"
               prefetchSize="1"
               duplex="true"
               decreaseNetworkConsumerPriority = "true"
               suppressDuplicateQueueSubscriptions = "true"
               conduitSubscriptions="false">
               <dynamicallyIncludedDestinations>
                   <queue physicalName="<queue_name_here>"/>
               </dynamicallyIncludedDestinations>
            </networkConnector>

And there are more networkConnector for each queue I have with same parameters. About 5 of them.
Reply | Threaded
Open this post in threaded view
|

Re: OutOfMemoryError due to TopicSubscription

ceposta
The network of brokers relies on advisory topics. Any good reason why you
have a separate bridge per destination?

My *guess* is you're using default config with default memory limits, and
since you have a lot of network bridges, thus separate advisory topic subs,
you're running out of memory.

If you post your complete config, we can see.


On Mon, Feb 25, 2013 at 9:45 AM, Ishitori <[hidden email]> wrote:

> I don't use advisory topics, but I do use network of brokers. There are 2
> brokers and they are connected by network connector per queue basis. Here
> is
> the example:
>
>            <networkConnector
>                name=&quot;TopicsOnly-duplex&quot;
>                uri=&quot;static:(nio://&lt;ip>:62626)"
>                networkTTL="2"
>                prefetchSize="1"
>                duplex="true">
>                <dynamicallyIncludedDestinations>
>                    <topic physicalName=">"/>
>                </dynamicallyIncludedDestinations>
>             </networkConnector>
>
>             <networkConnector
>                name=&quot;SpecificQueue-duplex&quot;
>                uri=&quot;static:(nio://&lt;ip>:62626)"
>                networkTTL="2"
>                prefetchSize="1"
>                duplex="true"
>                decreaseNetworkConsumerPriority = "true"
>                suppressDuplicateQueueSubscriptions = "true"
>                conduitSubscriptions="false">
>                <dynamicallyIncludedDestinations>
>                    <queue physicalName=&quot;&lt;queue_name_here>"/>
>                </dynamicallyIncludedDestinations>
>             </networkConnector>
>
> And there are more networkConnector for each queue I have with same
> parameters. About 5 of them.
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/OutOfMemoryError-due-to-TopicSubscription-tp4664025p4664043.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Reply | Threaded
Open this post in threaded view
|

Re: OutOfMemoryError due to TopicSubscription

Ishitori
I am not sure why it uses a bridge to destination feature. Is there another way to configure it? (sysadmin did it that way, not me)

But I wonder is 6 bridges is really a high number? How many resources does it take to support a separate advisory topic subs? Here is the full config:

<beans
  xmlns="http://www.springframework.org/schema/beans"
  xmlns:amq="http://activemq.apache.org/schema/core"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
  http://activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">

    <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
        <property name="locations">
            <value>credentials_path</value>
        </property>     
    </bean>

    <broker xmlns="http://activemq.apache.org/schema/core" brokerName="AMQ_Broker_1" useJmx="true" persistent="true" dataDirectory="<path>" useShutdownHook="true" schedulerSupport="true" schedulerDirectory="<scheduler_path>" >
             
        <destinationPolicy>
            <policyMap>
              <policyEntries>
                <policyEntry topic=">" producerFlowControl="false" memoryLimit="1mb">
                  <pendingSubscriberPolicy>
                    <vmCursor />
                  </pendingSubscriberPolicy>
                </policyEntry>
                <policyEntry queue=">" producerFlowControl="false" memoryLimit="10mb" enableAudit="false">
                  <networkBridgeFilterFactory>
                    <conditionalNetworkBridgeFilterFactory replayWhenNoConsumers="true" />
                  </networkBridgeFilterFactory>
                  <deadLetterStrategy>
                    <individualDeadLetterStrategy queuePrefix="DLQ." processExpired="false"/>
                  </deadLetterStrategy>
                  <pendingQueuePolicy>
                    <fileQueueCursor/>
                  </pendingQueuePolicy>
                </policyEntry>
              </policyEntries>
            </policyMap>
        </destinationPolicy> 
 
        <managementContext>
            <managementContext createConnector="true" connectorHost="host" connectorPort="port" rmiServerPort="rmiPort"/>
        </managementContext>

        <networkConnectors>
            <networkConnector
               name="TopicsOnly-duplex"
               uri="static:(nio://ip)"
               networkTTL="2"
               prefetchSize="1"
               duplex="true">
               <dynamicallyIncludedDestinations>
                   <topic physicalName=">"/>
               </dynamicallyIncludedDestinations>
            </networkConnector>

            <networkConnector
               name="queue1-duplex"
               uri="static:(nio://ip)"
               networkTTL="2"
               prefetchSize="1"
               duplex="true"
               decreaseNetworkConsumerPriority = "true"
               suppressDuplicateQueueSubscriptions = "true"
               conduitSubscriptions="false">
               <dynamicallyIncludedDestinations>
                   <queue physicalName="queue1"/>
               </dynamicallyIncludedDestinations>
            </networkConnector>

            <networkConnector
               name="queue2-duplex"
               uri="static:(nio://ip)"
               networkTTL="2"
               prefetchSize="1"
               duplex="true"
               decreaseNetworkConsumerPriority = "true"
               suppressDuplicateQueueSubscriptions = "true"
               conduitSubscriptions="false">
               <dynamicallyIncludedDestinations>
                   <queue physicalName="queue2"/>
               </dynamicallyIncludedDestinations>
            </networkConnector>

            <networkConnector
               name="queue3-duplex"
               uri="static:(nio://ip)"
               networkTTL="2"
               prefetchSize="1"
               duplex="true"
               decreaseNetworkConsumerPriority = "true"
               suppressDuplicateQueueSubscriptions = "true"
               conduitSubscriptions="false">
               <dynamicallyIncludedDestinations>
                   <queue physicalName="queue3"/>
               </dynamicallyIncludedDestinations>
            </networkConnector>

            <networkConnector
               name="queue4-duplex"
               uri="static:(nio://ip)"
               networkTTL="2"
               prefetchSize="1"
               duplex="true"
               decreaseNetworkConsumerPriority = "true"
               suppressDuplicateQueueSubscriptions = "true"
               conduitSubscriptions="false">
               <dynamicallyIncludedDestinations>
                   <queue physicalName="queue4"/>
               </dynamicallyIncludedDestinations>
            </networkConnector>

            <networkConnector
               name="queue5-duplex"
               uri="static:(nio://ip)"
               networkTTL="2"
               prefetchSize="1"
               duplex="true"
               decreaseNetworkConsumerPriority = "true"
               suppressDuplicateQueueSubscriptions = "true"
               conduitSubscriptions="false">
               <dynamicallyIncludedDestinations>
                   <queue physicalName="queue5"/>
               </dynamicallyIncludedDestinations>
            </networkConnector>
        </networkConnectors>

        <persistenceAdapter>
            <kahaDB directory="path"/>
        </persistenceAdapter>

        <systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage limit="64 mb"/>
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="45 gb"/>
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="2 gb"/>
                </tempUsage>
            </systemUsage>
        </systemUsage>
                 
        <transportConnectors>
            <transportConnector name="nio" uri="nio://ip" auditNetworkProducers="true"/>
        </transportConnectors>

    </broker>
    <import resource="jetty.xml"/>
   
</beans>
Reply | Threaded
Open this post in threaded view
|

Re: OutOfMemoryError due to TopicSubscription

ceposta
bridge per destination is unnecessary. especially if you're using the
default mem options which are much to low. also, why are you altering the
prefetch settings on the bridge and setting to 1? looks like someone wasn't
very sure about the config options that were chosen... you're using default
memory configs, unnecessarily limiting the bridge prefetch, producer flow
control is false, etc, etc, etc. this config is ripe for OOM.

It would be very worthwhile to understand all of the config settings from
the following wiki docs and make very calculated decisions about which ones
to change:

http://activemq.apache.org/message-cursors.html
http://activemq.apache.org/javalangoutofmemory.html
http://activemq.apache.org/producer-flow-control.html
http://activemq.apache.org/networks-of-brokers.html

hope that helps..


On Mon, Feb 25, 2013 at 1:34 PM, Ishitori <[hidden email]> wrote:

> I am not sure why it uses a bridge to destination feature. Is there another
> way to configure it? (sysadmin did it that way, not me)
>
> But I wonder is 6 bridges is really a high number? How many resources does
> it take to support a separate advisory topic subs? Here is the full config:
>
> <beans
>   xmlns="http://www.springframework.org/schema/beans"
>   xmlns:amq="http://activemq.apache.org/schema/core"
>   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>   xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>   http://activemq.apache.org/schema/core
> http://activemq.apache.org/schema/core/activemq-core.xsd">
>
>     <bean
>
> class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
>         <property name="locations">
>             <value>credentials_path</value>
>         </property>
>     </bean>
>
>     <broker xmlns=&quot;http://activemq.apache.org/schema/core&quot;
> brokerName=&quot;AMQ_Broker_1&quot; useJmx=&quot;true&quot;
> persistent=&quot;true&quot; dataDirectory=&quot;&lt;path>"
> useShutdownHook="true" schedulerSupport="true"
> schedulerDirectory="<scheduler_path>" >
>
>         <destinationPolicy>
>             <policyMap>
>               <policyEntries>
>                 <policyEntry topic=">" producerFlowControl="false"
> memoryLimit="1mb">
>                   <pendingSubscriberPolicy>
>                     <vmCursor />
>                   </pendingSubscriberPolicy>
>                 </policyEntry>
>                 <policyEntry queue=">" producerFlowControl="false"
> memoryLimit="10mb" enableAudit="false">
>                   <networkBridgeFilterFactory>
>                     <conditionalNetworkBridgeFilterFactory
> replayWhenNoConsumers="true" />
>                   </networkBridgeFilterFactory>
>                   <deadLetterStrategy>
>                     <individualDeadLetterStrategy queuePrefix="DLQ."
> processExpired="false"/>
>                   </deadLetterStrategy>
>                   <pendingQueuePolicy>
>                     <fileQueueCursor/>
>                   </pendingQueuePolicy>
>                 </policyEntry>
>               </policyEntries>
>             </policyMap>
>         </destinationPolicy>
>
>         <managementContext>
>             <managementContext createConnector="true" connectorHost="host"
> connectorPort="port" rmiServerPort="rmiPort"/>
>         </managementContext>
>
>         <networkConnectors>
>             <networkConnector
>                name="TopicsOnly-duplex"
>                uri="static:(nio://ip)"
>                networkTTL="2"
>                prefetchSize="1"
>                duplex="true">
>                <dynamicallyIncludedDestinations>
>                    <topic physicalName=">"/>
>                </dynamicallyIncludedDestinations>
>             </networkConnector>
>
>             <networkConnector
>                name="queue1-duplex"
>                uri="static:(nio://ip)"
>                networkTTL="2"
>                prefetchSize="1"
>                duplex="true"
>                decreaseNetworkConsumerPriority = "true"
>                suppressDuplicateQueueSubscriptions = "true"
>                conduitSubscriptions="false">
>                <dynamicallyIncludedDestinations>
>                    <queue physicalName="queue1"/>
>                </dynamicallyIncludedDestinations>
>             </networkConnector>
>
>             <networkConnector
>                name="queue2-duplex"
>                uri="static:(nio://ip)"
>                networkTTL="2"
>                prefetchSize="1"
>                duplex="true"
>                decreaseNetworkConsumerPriority = "true"
>                suppressDuplicateQueueSubscriptions = "true"
>                conduitSubscriptions="false">
>                <dynamicallyIncludedDestinations>
>                    <queue physicalName="queue2"/>
>                </dynamicallyIncludedDestinations>
>             </networkConnector>
>
>             <networkConnector
>                name="queue3-duplex"
>                uri="static:(nio://ip)"
>                networkTTL="2"
>                prefetchSize="1"
>                duplex="true"
>                decreaseNetworkConsumerPriority = "true"
>                suppressDuplicateQueueSubscriptions = "true"
>                conduitSubscriptions="false">
>                <dynamicallyIncludedDestinations>
>                    <queue physicalName="queue3"/>
>                </dynamicallyIncludedDestinations>
>             </networkConnector>
>
>             <networkConnector
>                name="queue4-duplex"
>                uri="static:(nio://ip)"
>                networkTTL="2"
>                prefetchSize="1"
>                duplex="true"
>                decreaseNetworkConsumerPriority = "true"
>                suppressDuplicateQueueSubscriptions = "true"
>                conduitSubscriptions="false">
>                <dynamicallyIncludedDestinations>
>                    <queue physicalName="queue4"/>
>                </dynamicallyIncludedDestinations>
>             </networkConnector>
>
>             <networkConnector
>                name="queue5-duplex"
>                uri="static:(nio://ip)"
>                networkTTL="2"
>                prefetchSize="1"
>                duplex="true"
>                decreaseNetworkConsumerPriority = "true"
>                suppressDuplicateQueueSubscriptions = "true"
>                conduitSubscriptions="false">
>                <dynamicallyIncludedDestinations>
>                    <queue physicalName="queue5"/>
>                </dynamicallyIncludedDestinations>
>             </networkConnector>
>         </networkConnectors>
>
>         <persistenceAdapter>
>             <kahaDB directory="path"/>
>         </persistenceAdapter>
>
>         <systemUsage>
>             <systemUsage>
>                 <memoryUsage>
>                     <memoryUsage limit="64 mb"/>
>                 </memoryUsage>
>                 <storeUsage>
>                     <storeUsage limit="45 gb"/>
>                 </storeUsage>
>                 <tempUsage>
>                     <tempUsage limit="2 gb"/>
>                 </tempUsage>
>             </systemUsage>
>         </systemUsage>
>
>         <transportConnectors>
>             <transportConnector name="nio" uri="nio://ip"
> auditNetworkProducers="true"/>
>         </transportConnectors>
>
>     </broker>
>     <import resource="jetty.xml"/>
>
> </beans>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/OutOfMemoryError-due-to-TopicSubscription-tp4664025p4664054.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



--
*Christian Posta*
http://www.christianposta.com/blog
twitter: @christianposta
Reply | Threaded
Open this post in threaded view
|

Re: OutOfMemoryError due to TopicSubscription

Ishitori
Thanks for the help! I would re-read these articles. But could you draft the worthwhile configuration for me? Is it enough to just increase <memoryUsage limit="> up to 128 mb? Or the whole configuration is wrong?