Out of memory with 14K large messages

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Out of memory with 14K large messages

Eichberger, German
Hi,

 

I am new to activeMQ and I have configured a network of brokers (two
servers) with two slow consumers and a durable queue. We are
approximately adding 13 messages/second and taking about one out per
second.

 

To deal with that asymmetry I have also added some of the performance
tips on the producer:

connection.setCopyMessageOnSend(false);

            connection.setOptimizeAcknowledge(true);

            connection.setUseAsyncSend(true);

 

 

We are sending 36,173 byte long messages to the system and after about
14,000 we get a java.lang.OutOfMemoryError - activeMQ is running in a
tomcat container which has only 512MB heap space.

 

I was wondering if there is any way (besides increasing the heap space)
to make activeMQ recover when it runs out of memory or if there is a
setting to enable it to run on a small memory footprint.

 

The stack trace looks like:

 

at
org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.
java:46)

        at
org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection
.java:1191)

        at
org.apache.activemq.AdvisoryConsumer.<init>(AdvisoryConsumer.java:46)

        at
org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQ
Connection.java:1281)

        at
org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:449
)

 

 

 

 

and the configuration file:

 

 

<!--

    Licensed to the Apache Software Foundation (ASF) under one or more

    contributor license agreements.  See the NOTICE file distributed
with

    this work for additional information regarding copyright ownership.

    The ASF licenses this file to You under the Apache License, Version
2.0

    (the "License"); you may not use this file except in compliance with

    the License.  You may obtain a copy of the License at

   

    http://www.apache.org/licenses/LICENSE-2.0

   

    Unless required by applicable law or agreed to in writing, software

    distributed under the License is distributed on an "AS IS" BASIS,

    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied.

    See the License for the specific language governing permissions and

    limitations under the License.

-->

<!-- START SNIPPET: example -->

<beans xmlns="http://activemq.org/config/1.0">

 

  <!-- Allows us to use system properties as variables in this
configuration file -->

  <bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfi
gurer"/>

 

  <broker useJmx="true">

 

    <!--  Use the following to set the broker memory limit (in bytes)
(test only has 512 MB)-->

    <memoryManager>  

        <usageManager id="memory-manager" limit="400000000"/>

    </memoryManager>

   

   

    <!-- Use the following to configure how ActiveMQ is exposed in JMX

    <managementContext>

       <managementContext connectorPort="1099"
jmxDomainName="org.apache.activemq"/>

    </managementContext>

    -->

 

    <!-- In ActiveMQ 4, you can setup destination policies -->  

    <!--<destinationPolicy>

      <policyMap><policyEntries>

       

          <policyEntry topic="FOO.>">

            <dispatchPolicy>

              <strictOrderDispatchPolicy />

            </dispatchPolicy>

            <subscriptionRecoveryPolicy>

              <lastImageSubscriptionRecoveryPolicy />

            </subscriptionRecoveryPolicy>

          </policyEntry>

 

      </policyEntries></policyMap>

    </destinationPolicy>-->

 

 

    <persistenceAdapter>

                        <!-- UNIX OPS was kind enough to create links
for each broker to the filer-->

        <!--<journaledJDBC journalLogFiles="5"
dataDirectory="/m2/tomcat5.5-temp/playersqueue/4209/q"/>-->

                        <!--<kahaPersistenceAdapter dir =
"/m2/tomcat5.5-temp/playersqueue/4209/q" maxDataFileLength =
"33554432"/>-->

                        <jdbcPersistenceAdapter
dataDirectory="/m2/tomcat5.5-temp/playersqueue/4209/q"/>

      <!-- To use a different datasource, use th following syntax : -->

      <!--

      <journaledJDBC journalLogFiles="5"
dataDirectory="../activemq-data" dataSource="#postgres-ds"/>

       -->

    </persistenceAdapter>

 

    <transportConnectors>

       <transportConnector name="default" uri="tcp://localhost:61616"/>

               <transportConnector name="local" uri="vm://localhost"/>

    </transportConnectors>

   

    <networkConnectors>

      <!-- by default just auto discover the other brokers -->

      <!--<networkConnector name="default"
uri="multicast://default"/>-->

      <networkConnector name="bridge1"
uri="static://(tcp://sdt-plattestt1.station.sony.com:61616)"
failover="true"/>

              <networkConnector name="bridge2"
uri="static://(tcp://sdt-plattestt2.station.sony.com:61616)"
failover="true"/>

    </networkConnectors>

   

  </broker>

 

  <!--  This xbean configuration file supports all the standard spring
xml configuration options -->

 

  <!-- Postgres DataSource Sample Setup -->

  <!--

  <bean id="postgres-ds" class="org.postgresql.ds.PGPoolingDataSource">

    <property name="serverName" value="localhost"/>

    <property name="databaseName" value="activemq"/>

    <property name="portNumber" value="0"/>

    <property name="user" value="activemq"/>

    <property name="password" value="activemq"/>

    <property name="dataSourceName" value="postgres"/>

    <property name="initialConnections" value="1"/>

    <property name="maxConnections" value="10"/>

  </bean>

  -->

 

  <!-- MySql DataSource Sample Setup -->

  <!--

  <bean id="mysql-ds" class="org.apache.commons.dbcp.BasicDataSource"
destroy-method="close">

    <property name="driverClassName" value="com.mysql.jdbc.Driver"/>

    <property name="url"
value="jdbc:mysql://localhost/activemq?relaxAutoCommit=true"/>

    <property name="username" value="activemq"/>

    <property name="password" value="activemq"/>

    <property name="poolPreparedStatements" value="true"/>

  </bean>

  -->  

   

  <!-- Embedded Derby DataSource Sample Setup -->

  <!--

  <bean id="derby-ds" class="org.apache.derby.jdbc.EmbeddedDataSource">

    <property name="databaseName" value="derbydb"/>

    <property name="createDatabase" value="create"/>

  </bean>

  -->  

 

</beans>

<!-- END SNIPPET: example -->

 

 

 

I know we are exploring some kind of edge case but we would like to
buffer a huge amount of messages during peak hours and catch up with our
consumers during non-peak...

 

Thanks,

German

 

---

German Eichberger - [hidden email]

 

Reply | Threaded
Open this post in threaded view
|

RE: Out of memory with 14K large messages

Chase Seibert

Isn't 36k * 14,000 msgs approx. 512MB? If that's the case, I'm not sure you can recover without losing messages. Maybe you could serialize the overflow to disk?

 

P.S. - Is this for Vanguard?

 
Chase Seibert

Application Developer

B U L L H O R N





Staffing and Recruiting Software, On Target, On Demand


33-41 Farnsworth Street, 5th Floor, Boston, MA 02210

(w) 617.478.9119

www.bullhorn.com




 

-----Original Message-----
From:Eichberger, German [hidden email]
To: "[hidden email]" <[hidden email]>;
Sent: Jan 2, 2007 07:06:44 PM
Subject: Out of memory with 14K large messages

Hi, I am new to activeMQ and I have configured a network of brokers (two servers) with two slow consumers and a durable queue. We are approximately adding 13 messages/second and taking about one out per second. To deal with that asymmetry I have also added some of the performance tips on the producer: connection.setCopyMessageOnSend(false); connection.setOptimizeAcknowledge(true); connection.setUseAsyncSend(true); We are sending 36,173 byte long messages to the system and after about 14,000 we get a java.lang.OutOfMemoryError - activeMQ is running in a tomcat container which has only 512MB heap space. I was wondering if there is any way (besides increasing the heap space) to make activeMQ recover when it runs out of memory or if there is a setting to enable it to run on a small memory footprint. The stack trace looks like: at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport. java:46) at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection .java:1191) at org.apache.activemq.AdvisoryConsumer.(AdvisoryConsumer.java:46) at org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQ Connection.java:1281) at org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:449 ) and the configuration file:

       

   

    -->

     

       

          ">

           

             

           

           

             

           

         

 

     

    -->-->-->

       -->-->

   

   

   

   

   

   

   

   

 

  -->

   

   

   

   

   

 

  -->

   

   

 

  -->I know we are exploring some kind of edge case but we would like to buffer a huge amount of messages during peak hours and catch up with our consumers during non-peak... Thanks, German --- German Eichberger - [hidden email]