Temp Store is Full - False Positive?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Temp Store is Full - False Positive?

dlee
I'm getting a strange error w/ ActiveMQ 5.10.0 as well as 5.11.1 (upgrading didn't help). The logs state that the temp store is full but at 0%:

javax.jms.ResourceAllocationException: Temp Store is Full (0% of 2147483648)

Restarting ActiveMQ always clears it up but eventually we get this error again. The admin console shows that the queue is well over 100% full though - usually reaching 200%-600%. However, on the physical machines our disk usage is normal w/ plenty of space and our queues and topics show all items dequeued. Our usage of ActiveMQ is frequent small messages and the behavior that we're seeing is the temp store will jump from 0% to >200% usage w/ one message of ~300 bytes and stay at that level until we restart that one instance.

Our setup is 3 AMQ instances clustered using ZooKeeper and the Replicated LevelDB. Here's the config for our usage limits:

       <systemUsage>
            <systemUsage sendFailIfNoSpaceAfterTimeout="10000">
                <memoryUsage>
                    <memoryUsage limit="4 gb"/>
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="20 gb"/>
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="2 gb"/>
                </tempUsage>
            </systemUsage>
        </systemUsage>

And we have turned producer flow control off for all queues and topics. We've been dealing with this for a few months now and am a bit stuck as to what to try next. Any help would be greatly appreciated!

Reply | Threaded
Open this post in threaded view
|

Re: Temp Store is Full - False Positive?

khandelwalanuj
Hi,

We have also faced the same issue with ActiveMQv5.14.5 recently couple of time and as you mentioned restart doesn't fix this. When I went to hard disk the size was normal.

Only way to resolve this is restarting broker by clearing the persistent store.

Any idea why this happened ? And it is actually harmful or just a false warning ?

Thank,
Anuj
Reply | Threaded
Open this post in threaded view
|

Re: Temp Store is Full - False Positive?

Tim Bain
When this happens, how much disk space is used for KahaDB data files for
the temp store?

And what temp store limit are you using?

Tim

On Aug 16, 2017 1:29 AM, "khandelwalanuj" <[hidden email]>
wrote:

> Hi,
>
> We have also faced the same issue with ActiveMQv5.14.5 recently couple of
> time and as you mentioned restart doesn't fix this. When I went to hard
> disk
> the size was normal.
>
> Only way to resolve this is restarting broker by clearing the persistent
> store.
>
> Any idea why this happened ? And it is actually harmful or just a false
> warning ?
>
> Thank,
> Anuj
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Temp-Store-is-Full-False-Positive-tp4695836p4729703.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Temp Store is Full - False Positive?

khandelwalanuj
My configuration is:

                <amq:memoryUsage> <amq:memoryUsage limit="1 gb" /> </amq:memoryUsage>
                <amq:storeUsage> <amq:storeUsage limit="5 gb" /> </amq:storeUsage>
                <amq:tempUsage> <amq:tempUsage limit="3 gb" /> </amq:tempUsage>

When this happened the size of persistent store was ~250 MB
Reply | Threaded
Open this post in threaded view
|

Re: Temp Store is Full - False Positive?

Tim Bain
By "persistent store," did you really mean "temp store"? If what you gave
is the persistent store (LevelDB, according to your other post), that
wasn't what I was asking for, and I meant the size of the KahaDB-backed
temp store.

On Aug 18, 2017 7:09 AM, "khandelwalanuj" <[hidden email]>
wrote:

> My configuration is:
>
>                 <amq:memoryUsage> <amq:memoryUsage limit="1 gb" />
> </amq:memoryUsage>
>                 <amq:storeUsage> <amq:storeUsage limit="5 gb" />
> </amq:storeUsage>
>                 <amq:tempUsage> <amq:tempUsage limit="3 gb" />
> </amq:tempUsage>
>
> When this happened the size of persistent store was ~250 MB
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Temp-Store-is-Full-False-Positive-tp4695836p4729806.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Temp Store is Full - False Positive?

khandelwalanuj
I am a bit confused here. Few questions,

1. Even if we are using levelDB for persistent store, does temp store is still kahadb ?
2. What is the location of temp store, is it same as levelDB store or something else ? (Similar question I asked http://activemq.2283324.n4.nabble.com/ActiveMQ-Temp-Store-location-td4729704.html)

Thanks,
Anuj
Reply | Threaded
Open this post in threaded view
|

Re: Temp Store is Full - False Positive?

Tim Bain
On Aug 18, 2017 7:34 AM, "khandelwalanuj" <[hidden email]>
wrote:

I am a bit confused here. Few questions,

1. Even if we are using levelDB for persistent store, does temp store is
still kahadb ?


FilePendingMessageCursor.store (
http://grepcode.com/file/repo1.maven.org/maven2/org.apache.activemq/activemq-broker/5.11.0/org/apache/activemq/broker/region/cursors/FilePendingMessageCursor.java#FilePendingMessageCursor.0store)
is of type PListStore, which is only implemented by KahaDB, so I believe
it's always KahaDB even if your persistent store is LevelDB. But you should
be able to confirm that: if you have one set of LevelDB files and one set
of KahaDB files then it's as I described here, and if there are two sets of
LevelDB files then I'm mistaken about how it works.

2. What is the location of temp store, is it same as levelDB store or
something else?


Per Torsten at
http://tmielke.blogspot.com/2011/02/observations-on-activemqs-temp-storage.html?m=1,
by default, the temp store's files are at data/localhost/tmp_storage within
the ActiveMQ installation. But it can be changed if necessary:
http://activemq.2283324.n4.nabble.com/Changing-disk-location-of-tempUsage-td4668124.html

(Similar question I asked
http://activemq.2283324.n4.nabble.com/ActiveMQ-Temp-
Store-location-td4729704.html)

Thanks,
Anuj



--
View this message in context: http://activemq.2283324.n4.
nabble.com/Temp-Store-is-Full-False-Positive-tp4695836p4729811.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: Temp Store is Full - False Positive?

khandelwalanuj
Thanks Tim for the explanation.


"tmpDataDirectory" property is not respected. Even after specifying this I can see that plist directory is created under levelDB directory only.

    <amq:broker tmpDataDirectory="/local/mqbroker/tmp_store"  persistent="true" brokerName="${activemq.broker.name}"
        useJmx="true" offlineDurableSubscriberTimeout="172800000"
        offlineDurableSubscriberTaskSchedule="3600000" advisorySupport="false">



[amqdev@app1a (jupiter.c) /local/mqbroker/leveldb]$ ls
0000000000000000.log  dirty.index  lock  nodeid.txt  plist.index  store-version.txt


[amqdev@app1a (jupiter.c) /local/mqbroker/leveldb/plist.index]$ ls
000005.sst  000019.sst  000033.sst  000047.sst  000061.sst  000075.sst  000089.sst  000103.sst  000117.sst  000131.sst  000145.sst  000159.sst
000007.sst  000021.sst  000035.sst  000049.sst  000063.sst  000077.sst  000091.sst  000105.sst  000119.sst  000133.sst  000147.sst  000160.log  CURRENT  LOCK  MANIFEST-000002

Thanks,
Anuj
Reply | Threaded
Open this post in threaded view
|

Re: Temp Store is Full - False Positive?

khandelwalanuj
The default location is also not respected. I can see that nothing is there under "data/localhost/tmp_storage "

Thanks,
Anuj
Reply | Threaded
Open this post in threaded view
|

Re: Temp Store is Full - False Positive?

khandelwalanuj
Can someone please check this ?
Reply | Threaded
Open this post in threaded view
|

Re: Temp Store is Full - False Positive?

khandelwalanuj
Can someone check this please ?



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Temp Store is Full - False Positive?

khandelwalanuj
I can see that "tmpDataDirectory" is not respected. Is there something I am
doing wrong ? Any suggestions will be helpful.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Temp Store is Full - False Positive?

Tim Bain
I finally got some time to look into this, and several parts of what I told
you were wrong.

I had said that the temp store was a KahaDB instance. That was true, but
imprecise and I didn't know it. Technically, the temp store is any store
that implements org.apache.activemq.store.PListStore, and if the
persistence store does, it will be used rather than creating a second,
separate database instance. LevelDB is a PListStore, so when LevelDB is in
use, it will be used as the temp store. When KahaDB is in use, the
persistence adapter it uses is not a PListStore, so a new
org.apache.activemq.store.kahadb.plist.PListStoreImpl will be created,
meaning that it's a KahaDB instance but its data files will look similar to
(the same as?) a LevelDB store's data files, so you'd have trouble telling
them apart using ls. Either way, you're using LevelDB, so you won't see
files for a new store created.

BTW, the tmpDataDirectory is indeed passed into the broker from the XML
config file (I confirmed that with a debugger), it's just that the code
never chooses to use that path to create a new database instance.

Tim

On Tue, Sep 5, 2017 at 11:43 PM, khandelwalanuj <
[hidden email]> wrote:

> I can see that "tmpDataDirectory" is not respected. Is there something I am
> doing wrong ? Any suggestions will be helpful.
>
>
>
> --
> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
> f2341805.html
>