Issues with jdbcPersistenceAdapter DB-Failover and ActiveMQ Bridges

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Issues with jdbcPersistenceAdapter DB-Failover and ActiveMQ Bridges

Marc Giger
Hi all,

Following setup (ActiveMQ 5.14.3):

ActiveMQ "bridge1" (Master/Slave) --(bridge)--> ActiveMQ "middle"
(Master/Slave) <--(bridge)-- ActiveMQ "bridge2" (Master/Slave)

whereby every Master/Slave pair is configured to use a MSSQL as
datasource for persistent messages. Every
broker pair has a separate database instance assigned.

Typically persistent messaging (queues) are used. Message sizes varies
up to ~12MB. For testing purposes I use 10MB messages.
Messages are produced and consumed crisscross e.g. sent to "bridge1" and
consumed on "bridge2" and the other way around
for other queues. Sometimes messages are also sent to the "middle"
queue.

A major issue we had in this setup was OutOfMemoryError (java heap
space) which I could solve mostly with the following
configuration (local setup for testing purposes with postgres and
kahadb):

Brokers "bridge1" and "bridge2":

-Xmx1G

   <policyEntry queue=">" producerFlowControl="true" memoryLimit="250mb"
     queuePrefetch="1" maxPageSize="10" lazyDispatch="true">
       <deadLetterStrategy>
           <individualDeadLetterStrategy queuePrefix=""
queueSuffix=".DLQ" useQueueForQueueMessages="true"/>
       </deadLetterStrategy>
   </policyEntry>

<persistenceAdapter>
     <kahaDB directory="${activemq.data}/kahadb"/>
</persistenceAdapter>

<systemUsage>
             <systemUsage>
                 <memoryUsage>
                     <memoryUsage percentOfJvmHeap="70" />
                 </memoryUsage>
                 <storeUsage>
                     <storeUsage limit="1 gb"/>
                 </storeUsage>
                 <tempUsage>
                     <tempUsage limit="1 gb"/>
                 </tempUsage>
             </systemUsage>
         </systemUsage>

<networkConnectors>
          <networkConnector
                 name="n1"
                 
uri="masterslave:(tcp://127.0.0.1:16162,tcp://127.0.0.1:16169)"
                 duplex="true"
                 decreaseNetworkConsumerPriority="true"
                 networkTTL="3"/>
         </networkConnectors>
<transportConnector name="openwire"
uri="tcp://0.0.0.0:16161?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600"/>

Broker "middle":

-Xmx3G

   <policyEntry queue=">" producerFlowControl="true" memoryLimit="250mb"
     queuePrefetch="1" maxPageSize="10" lazyDispatch="true">
       <deadLetterStrategy>
           <individualDeadLetterStrategy queuePrefix=""
queueSuffix=".DLQ" useQueueForQueueMessages="true"/>
       </deadLetterStrategy>
   </policyEntry>

<persistenceAdapter>
             <jdbcPersistenceAdapter dataDirectory="${activemq.data}"
dataSource="#mssql-ds" lockKeepAlivePeriod="5000">
                 <locker>
                   <lease-database-locker
lockAcquireSleepInterval="10000"/>
                 </locker>
             </jdbcPersistenceAdapter>
         </persistenceAdapter>

<systemUsage>
             <systemUsage>
                 <memoryUsage>
                     <memoryUsage percentOfJvmHeap="70" />
                 </memoryUsage>
                 <storeUsage>
                     <storeUsage limit="1 gb"/>
                 </storeUsage>
                 <tempUsage>
                     <tempUsage limit="1 gb"/>
                 </tempUsage>
             </systemUsage>
         </systemUsage>

<transportConnector name="openwire"
uri="tcp://0.0.0.0:16162?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600"/>

<bean id="mssql-ds" class="org.apache.commons.dbcp2.BasicDataSource"
destroy-method="close">
         <property name="driverClassName" value="org.postgresql.Driver"/>
         <property name="url"
value="jdbc:postgresql://localhost:5432/activemq"/>
     </bean>


If the db for the "middle" broker is shutdown heap-memory on "bridge1"
and "bridge2 " is getting used up to the 70% and
messages like the following appear in the log:
  INFO | Usage(default:store:queue://b:store) percentUsage=99%,
usage=1078519327, limit=1073741824,
percentUsageMinDelta=1%;Parent:Usage(default:store) percentUsage=100%,
usage=1078519327, limit=1073741824, percentUsageMinDelta=1%: Persistent
store is Full, 100% of 1073741824. Stopping producer
(ID:gir-VirtualBox-51223-1487171112604-1:1:1:1) to prevent flooding
queue://b. See http://activemq.apache.org/producer-flow-control.html for
more info (blocking for: 908s)

Producers getting throttled on bridge1 and bridge2 as desired. So far so
good.

Now, while writing this email I got again an OutOfMemoryError on
"bridge1":
INFO | Network connection between vm://bridge1#16 and
tcp://127.0.0.1:16162 shutdown due to a local error:
java.io.IOException: Connection refused. Check that the hostname and
port are correct and that the postmaster is accepting TCP/IP
connections.
  INFO | Connector vm://bridge1 stopped
  INFO | bridge1 bridge to middle stopped
Exception in thread "ActiveMQ Broker[bridge1] Scheduler"
java.lang.OutOfMemoryError: Java heap space
         at
org.apache.activemq.protobuf.BaseMessage.mergeFramed(BaseMessage.java:228)
         at
org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:1167)
...
  INFO | Establishing network connection from
vm://bridge1?async=false&create=false to
failover:(tcp://127.0.0.1:16162,tcp://127.0.0.1:16169)?randomize=false&maxReconnectAttempts=0
  INFO | Connector vm://bridge1 started
  INFO | Successfully connected to tcp://127.0.0.1:16162
  INFO | Network connection between vm://bridge1#20 and
tcp://127.0.0.1:16162 (middle) has been established.
  INFO | Usage(default:store:queue://b:store) percentUsage=99%,
usage=1080269357, limit=1073741824,
percentUsageMinDelta=1%;Parent:Usage(default:store) percentUsage=100%,
usage=1080269357, limit=1073741824, percentUsageMinDelta=1%: Persistent
store is Full, 100% of 1073741824. Stopping producer
(ID:gir-VirtualBox-53990-1487180814547-1:1:4:1) to prevent flooding
queue://b. See http://activemq.apache.org/producer-flow-control.html for
more info (blocking for: 32s)
)

And after restarting (kill -9) the bridge1 broker and starting the db
again I also got
and OutOfMemoryError on the "middle" broker:

INFO | Usage Manager Memory Limit (262144000) reached on queue://a, size
70. Producers will be throttled to the rate at which messages are
removed from this destination to prevent flooding it. See
http://activemq.apache.org/producer-flow-control.html for more info.
Exception in thread "ActiveMQ Broker[middle] Scheduler"
java.lang.OutOfMemoryError: Java heap space
         at
org.postgresql.util.PGbytea.toBytesHexEscaped(PGbytea.java:40)
         at org.postgresql.util.PGbytea.toBytes(PGbytea.java:35)
         at
org.postgresql.jdbc2.AbstractJdbc2ResultSet.getBytes(AbstractJdbc2ResultSet.java:2433)
         at
org.apache.commons.dbcp2.DelegatingResultSet.getBytes(DelegatingResultSet.java:237)
         at
org.apache.commons.dbcp2.DelegatingResultSet.getBytes(DelegatingResultSet.java:237)
         at
org.apache.activemq.store.jdbc.adapter.BytesJDBCAdapter.getBinaryData(BytesJDBCAdapter.java:39)
         at
org.apache.activemq.store.jdbc.adapter.DefaultJDBCAdapter.doRecoverNextMessages(DefaultJDBCAdapter.java:1094)
         at
org.apache.activemq.store.jdbc.JDBCMessageStore.recoverNextMessages(JDBCMessageStore.java:360)
         at
org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:110)
         at
org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:120)
         at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:420)
         at
org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:163)
         at
org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:169)
         at
org.apache.activemq.broker.region.Queue.doPageInForDispatch(Queue.java:1941)
         at
org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:2170)
         at
org.apache.activemq.broker.region.Queue.doBrowse(Queue.java:1134)
         at
org.apache.activemq.broker.region.Queue.expireMessages(Queue.java:914)
         at
org.apache.activemq.broker.region.Queue.access$100(Queue.java:105)
         at org.apache.activemq.broker.region.Queue$2.run(Queue.java:148)
         at
org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33)
         at java.util.TimerThread.mainLoop(Timer.java:555)
         at java.util.TimerThread.run(Timer.java:505)

It looks like my setup is still not working as expected...

Apart from the OutOfMemoryErrors I have other issues as well with this
setup:

Today while testing this setup I didn't got OutOfMemoryErrors in 2 test
runs but after restarting
the database all brokers where in a weird state. No messages were
exchanged anymore. I had to kill them with -9.
It didn't help to restart just one of them. All seemed to be affected.
Heap space seemed not be an problem from what jvisualvm told me.
Attached are the thread-dumps from the 3 brokers in this state.

In an other case where I got an OutOfMemoryError with this setup on the
"middle" broker the heap-dump
showed that most of the memory was allocated in two instances of
org.apache.activemq.broker.region.Queue.

As long as the connection to the database is reliable everything works
as expected. The issues
starts when a MSSQL DB Failover happens which takes up to minutes until
it the slave is online.
(Simulated in my setup with postgres stop/start).

It would be very nice and cool if some of you ActiveMQ gurus could give
me some advice
on how to make this setup reliable. Would a shared filesystem based db
be the better option for this usecase?
Does ActiveMQ expectd a 100% reliable jdbc store?

Many thanks,

Marc



thread-dump-bridge1.txt (58K) Download Attachment
thread-dump-bridge2.txt (60K) Download Attachment
thread-dump-middle.txt (57K) Download Attachment