Messages are getting struck in ActiveMQ v5.9.0 queue

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Messages are getting struck in ActiveMQ v5.9.0 queue

Raagu
Hi

I am facing problem of messages getting struck in activemq queue and consumer is not able to consume messages when I pump around ~100 messages to different queues.
I have configured ActiveMQ (v5.9.0) message broker in two separate linux instances. For persistenceAdapter conf on both the instances, I have mounted a NAS storage onto both the instances where activemq server is running. Hence same NAS storage is mounted on both activemq instances at mount point '/mnt/nas'. The size of the NAS storage is 20 GB.

So my persistenceAdapter conf looks like below
<persistenceAdapter>
            <kahaDB directory="/mnt/nas" ignoreMissingJournalfiles="true" checkForCorruptJournalFiles="true" checksumJournalFiles="true"/>
</persistenceAdapter>


The systemUsage configuration on both activemq server is like below

<systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage percentOfJvmHeap = "70"/>
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit = "15 gb"/>
                </storeUsage>
                <tempUsage>
                    <tempUsage limit = "7 gb"/>
                </tempUsage>
            </systemUsage>
 </systemUsage>

and I have enabled only 'tcp' transport connector
<transportConnector name = "openwire" uri = "tcp://0.0.0.0:61616?maximumConnections=1000&amp;wireFormat.maxFrameSize=104857600"/>

When I see the partition details of the mount point '/mnt/nas' on both activemq instances, I see following for the command df -k

Filesystem              1K-blocks        Used   Available Use% Mounted on
/dev/xvda2              102953264     5840280    91883264   6% /
tmpfs                      967652           0      967652   0% /dev/shm
/dev/xvda1                 253871       52511      188253  22% /boot
//nas151.service.softlayer.com/IBM278684-16
                     139328579072 56369051136 82959527936  41% /mnt/nas
                                         
Hence I see 41% of /mnt/nas is used

The problem is when I start the activemq server (on both instances), I see the following messages in the activemq.log

**************** START ***************
2014-06-05 12:48:40,350 | INFO  | PListStore:[/var/lib/apache-activemq-5.9.0/data/localhost/tmp_storage] started | org.apache.activemq.store.kahadb.plist.PListStoreImpl | main
2014-06-05 12:48:40,454 | INFO  | JMX consoles can connect to service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi | org.apache.activemq.broker.jmx.ManagementContext | JMX connector
2014-06-05 12:48:40,457 | INFO  | Using Persistence Adapter: KahaDBPersistenceAdapter[/mnt/nas] | org.apache.activemq.broker.BrokerService | main
2014-06-05 12:48:40,612 | INFO  | Corrupt journal records found in '/mnt/nas/db-1.log' between offsets: 8163..8209 | org.apache.activemq.store.kahadb.disk.journal.Journal | main
2014-06-05 12:48:40,613 | INFO  | Corrupt journal records found in '/mnt/nas/db-1.log' between offsets: 12256..12327 | org.apache.activemq.store.kahadb.disk.journal.Journal | main
2014-06-05 12:48:40,649 | INFO  | Corrupt journal records found in '/mnt/nas/db-1.log' between offsets: 20420..20585 | org.apache.activemq.store.kahadb.disk.journal.Journal | main
2014-06-05 12:48:40,650 | INFO  | Corrupt journal records found in '/mnt/nas/db-1.log' between offsets: 28559..28749 | org.apache.activemq.store.kahadb.disk.journal.Journal | main
2014-06-05 12:48:40,651 | INFO  | Corrupt journal records found in '/mnt/nas/db-1.log' between offsets: 32677..32842 | org.apache.activemq.store.kahadb.disk.journal.Journal | main
2014-06-05 12:48:40,652 | INFO  | Corrupt journal records found in '/mnt/nas/db-1.log' between offsets: 36770..36960 | org.apache.activemq.store.kahadb.disk.journal.Journal | main
2014-06-05 12:48:40,655 | INFO  | Corrupt journal records found in '/mnt/nas/db-1.log' between offsets: 49099..49264 | org.apache.activemq.store.kahadb.disk.journal.Journal | main
2014-06-05 12:48:40,657 | INFO  | Corrupt journal records found in '/mnt/nas/db-1.log' between offsets: 61403..61474 | org.apache.activemq.store.kahadb.disk.journal.Journal | main
2014-06-05 12:48:40,658 | INFO  | Corrupt journal records found in '/mnt/nas/db-1.log' between offsets: 65521..65567 | org.apache.activemq.store.kahadb.disk.journal.Journal | main
2014-06-05 12:48:40,659 | INFO  | Corrupt journal records found in '/mnt/nas/db-1.log' between offsets: 69614..69685 | org.apache.activemq.store.kahadb.disk.journal.Journal | main
2014-06-05 12:48:40,660 | INFO  | Corrupt journal records found in '/mnt/nas/db-1.log' between offsets: 77778..77824 | org.apache.activemq.store.kahadb.disk.journal.Journal | main
2014-06-05 12:48:41,543 | INFO  | KahaDB is version 5 | org.apache.activemq.store.kahadb.MessageDatabase | main
2014-06-05 12:48:41,592 | INFO  | Recovering from the journal ... | org.apache.activemq.store.kahadb.MessageDatabase | main
2014-06-05 12:48:41,604 | INFO  | Recovery replayed 66 operations from the journal in 0.028 seconds. | org.apache.activemq.store.kahadb.MessageDatabase | main
2014-06-05 12:48:41,772 | INFO  | Apache ActiveMQ 5.9.0 (localhost, ID:10.106.99.101-60576-1401972521638-0:1) is starting | org.apache.activemq.broker.BrokerService | main
2014-06-05 12:48:41,892 | INFO  | Listening for connections at: tcp://10.106.99.101:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600 | org.apache.activemq.transport.TransportServerThreadSupport | main
2014-06-05 12:48:41,893 | INFO  | Connector openwire started | org.apache.activemq.broker.TransportConnector | main
2014-06-05 12:48:41,893 | INFO  | Apache ActiveMQ 5.9.0 (localhost, ID:10.106.99.101-60576-1401972521638-0:1) started | org.apache.activemq.broker.BrokerService | main
2014-06-05 12:48:41,893 | INFO  | For help or more information please see: http://activemq.apache.org | org.apache.activemq.broker.BrokerService | main
2014-06-05 12:48:41,897 | WARN  | Store limit is 2048 mb, whilst the data directory: /mnt/nas only has 0 mb of usable space - resetting to maximum available disk space: 0 mb | org.apache.activemq.broker.BrokerService | main
2014-06-05 12:48:41,897 | ERROR | Store limit is 0 mb, whilst the max journal file size for the store is: 32 mb, the store will not accept any data when used. | org.apache.activemq.broker.BrokerService | main
******************** END **************

I see 'Corrupt journal records found in '/mnt/nas/db-1.log'. This comes for everytime restart even though if I delete this file and restart.
I had put the flag to recover but still this log entry comes for every restart.

Another problem is, even though my NAS storage is 20GB, it shows '/mnt/nas only has 0 mb of usable space'. This is really weired. I dont know hw it can see only 0 mb.

I request people here to give me some suggestions on why it is happening like this and suggest me any better configurations to avoid messages getting struck in queue.

Thanks
Raagu