Persistent store is Full, 100% of 107374182400. Stopping producer

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Persistent store is Full, 100% of 107374182400. Stopping producer

bharath
Hi All,

I'm new to Active MQ.

Today users repoted that Active MQ is stopped and not processing messages.
When i verified log file noticed below error.

2018-05-28 00:40:19,190 | INFO  | Usage(default:store:queue://xxxxxxx:store)
percentUsage=0%, usage=107430821727, limit=107374182400,
percentUsageMinDelta=1%;Parent:Usage(default:store) percentUsage=100%,
usage=107430821727, limit=107374182400, percentUsageMinDelta=1%: Persistent
store is Full, 100% of 107374182400. Stopping producer
(ID:xxxxxxxx-28816-1526103479544-17148:1:1:2) to prevent flooding
queue://xxxxxxx. See http://activemq.apache.org/producer-flow-control.html
for more info (blocking for: 182s) | org.apache.activemq.broker.region.Queue
| qtp947553027-132

activemq.xml file has value --> storeUsage limit="100 gb"
File system where KahaDB is residing has around 30 GB free space. Still
Active MQ failed with said errors.

I have requested Unix admins to add 50 GB extra space to File system and
updated value --> storeUsage limit="130 gb" and started Active MQ.

Then Active MQ is up and started processing messages.


Can some one help me to understand why Active MQ failed with message "
Persistent store is Full " even there is 30 GB free in File system.

Also i can see KahaDB consists 1 month older data files only. How is purging
happens in Active MQ. We are not using any script to purge old data.


Thanks in advance.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Persistent store is Full, 100% of 107374182400. Stopping producer

Tim Bain
That message doesn't mean the disk is full, it means you're using the
maximum amount of space that you've told the broker to allow, irrespective
of whether there's still space available on the disk. So that's why you hit
the message when using 100 GB of disk space even though there are still 30
GB available.

The better question is, why are your consumers leaving 100 GB of messages
on the broker? Does that indicate that you've misconfigured something about
your system?

Tim

On Mon, May 28, 2018, 5:30 PM bharath <[hidden email]> wrote:

> Hi All,
>
> I'm new to Active MQ.
>
> Today users repoted that Active MQ is stopped and not processing messages.
> When i verified log file noticed below error.
>
> 2018-05-28 00:40:19,190 | INFO  |
> Usage(default:store:queue://xxxxxxx:store)
> percentUsage=0%, usage=107430821727, limit=107374182400,
> percentUsageMinDelta=1%;Parent:Usage(default:store) percentUsage=100%,
> usage=107430821727, limit=107374182400, percentUsageMinDelta=1%: Persistent
> store is Full, 100% of 107374182400. Stopping producer
> (ID:xxxxxxxx-28816-1526103479544-17148:1:1:2) to prevent flooding
> queue://xxxxxxx. See http://activemq.apache.org/producer-flow-control.html
> for more info (blocking for: 182s) |
> org.apache.activemq.broker.region.Queue
> | qtp947553027-132
>
> activemq.xml file has value --> storeUsage limit="100 gb"
> File system where KahaDB is residing has around 30 GB free space. Still
> Active MQ failed with said errors.
>
> I have requested Unix admins to add 50 GB extra space to File system and
> updated value --> storeUsage limit="130 gb" and started Active MQ.
>
> Then Active MQ is up and started processing messages.
>
>
> Can some one help me to understand why Active MQ failed with message "
> Persistent store is Full " even there is 30 GB free in File system.
>
> Also i can see KahaDB consists 1 month older data files only. How is
> purging
> happens in Active MQ. We are not using any script to purge old data.
>
>
> Thanks in advance.
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Persistent store is Full, 100% of 107374182400. Stopping producer

Tim Bain
KahaDB purges a data file automatically when all messages in it have been
consumed and all acks in it refer to messages in data files that have been
deleted. If files are staying around, it means that you have old unconsumed
messages. Check your DLQ.

Tim

On Mon, May 28, 2018, 9:24 PM Tim Bain <[hidden email]> wrote:

> That message doesn't mean the disk is full, it means you're using the
> maximum amount of space that you've told the broker to allow, irrespective
> of whether there's still space available on the disk. So that's why you hit
> the message when using 100 GB of disk space even though there are still 30
> GB available.
>
> The better question is, why are your consumers leaving 100 GB of messages
> on the broker? Does that indicate that you've misconfigured something about
> your system?
>
> Tim
>
> On Mon, May 28, 2018, 5:30 PM bharath <[hidden email]> wrote:
>
>> Hi All,
>>
>> I'm new to Active MQ.
>>
>> Today users repoted that Active MQ is stopped and not processing messages.
>> When i verified log file noticed below error.
>>
>> 2018-05-28 00:40:19,190 | INFO  |
>> Usage(default:store:queue://xxxxxxx:store)
>> percentUsage=0%, usage=107430821727, limit=107374182400,
>> percentUsageMinDelta=1%;Parent:Usage(default:store) percentUsage=100%,
>> usage=107430821727, limit=107374182400, percentUsageMinDelta=1%:
>> Persistent
>> store is Full, 100% of 107374182400. Stopping producer
>> (ID:xxxxxxxx-28816-1526103479544-17148:1:1:2) to prevent flooding
>> queue://xxxxxxx. See
>> http://activemq.apache.org/producer-flow-control.html
>> for more info (blocking for: 182s) |
>> org.apache.activemq.broker.region.Queue
>> | qtp947553027-132
>>
>> activemq.xml file has value --> storeUsage limit="100 gb"
>> File system where KahaDB is residing has around 30 GB free space. Still
>> Active MQ failed with said errors.
>>
>> I have requested Unix admins to add 50 GB extra space to File system and
>> updated value --> storeUsage limit="130 gb" and started Active MQ.
>>
>> Then Active MQ is up and started processing messages.
>>
>>
>> Can some one help me to understand why Active MQ failed with message "
>> Persistent store is Full " even there is 30 GB free in File system.
>>
>> Also i can see KahaDB consists 1 month older data files only. How is
>> purging
>> happens in Active MQ. We are not using any script to purge old data.
>>
>>
>> Thanks in advance.
>>
>>
>>
>> --
>> Sent from:
>> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Persistent store is Full, 100% of 107374182400. Stopping producer

bharath
In reply to this post by Tim Bain
Hi Tim,

Thank you for looking into this.

In my previous update i've not mentioned about File System space allocation.
Let me explain you in detail.

Active MQ data directory resides in File System : /u01/ActiveMQ . Total
space allocated to File System : /u01/ActiveMQ  is 100GB

In activemq.xml we have systemUsage values as below

<systemUsage>
   <systemUsage>
        <memoryUsage>
            <memoryUsage percentOfJvmHeap="70" />
        </memoryUsage>
        <storeUsage>
            <storeUsage limit="100 gb"/>
        </storeUsage>
        <tempUsage>
            <tempUsage limit="50 gb"/>
        </tempUsage>
    </systemUsage>
</systemUsage>
           

From above context from activemq.xml, my understanding is broker will use
max of 100 GB in File system. But was not happen.

When issue happen, data directory size under File System : /u01/ActiveMQ is
32 GB and 30 GB is free in File System.

Why did we get 100% store when it could not have used 100 GB of spce from
filesystem as we know 30 GB was free

Also let me know how to check DLQ.



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
Reply | Threaded
Open this post in threaded view
|

Re: Persistent store is Full, 100% of 107374182400. Stopping producer

Tim Bain
For checking the DLQ, an easy option is to use the web console or a JMX
viewer such as JConsole to look at stats for the queue named ActiveMQ.DLQ.
Alternatively, you can follow the instructions on this page (
http://activemq.apache.org/why-do-kahadb-log-files-remain-after-cleanup.html)
to determine which destinations have unconsumed messages that are keeping
your KahaDB files alive, which has the advantage of telling you which
destination even if it's not the DLQ.

As for the question of why the broker believes that it's using more of the
persistence store than what is used on disk, it's very possible that
there's a bug, but I wonder if maybe you're using compressed messages and
so perhaps the amount of bytes on disk are fewer (due to compression) than
the size of their actual content. I've not been able to check the source
code yet to see whether the usage for the persistent store is based on the
uncompressed size, so this might all be moot, but are your messages being
sent compressed or uncompressed?

Also, does the broker show the same percent utilization after a restart? If
the number changes significantly following a broker restart, then this is
definitely a bug; it not, there's likely something else (such as
compression) going on.

Tim

On Tue, May 29, 2018, 5:12 AM bharath <[hidden email]> wrote:

> Hi Tim,
>
> Thank you for looking into this.
>
> In my previous update i've not mentioned about File System space
> allocation.
> Let me explain you in detail.
>
> Active MQ data directory resides in File System : /u01/ActiveMQ . Total
> space allocated to File System : /u01/ActiveMQ  is 100GB
>
> In activemq.xml we have systemUsage values as below
>
> <systemUsage>
>    <systemUsage>
>         <memoryUsage>
>             <memoryUsage percentOfJvmHeap="70" />
>         </memoryUsage>
>         <storeUsage>
>             <storeUsage limit="100 gb"/>
>         </storeUsage>
>         <tempUsage>
>             <tempUsage limit="50 gb"/>
>         </tempUsage>
>     </systemUsage>
> </systemUsage>
>
>
> From above context from activemq.xml, my understanding is broker will use
> max of 100 GB in File system. But was not happen.
>
> When issue happen, data directory size under File System : /u01/ActiveMQ is
> 32 GB and 30 GB is free in File System.
>
> Why did we get 100% store when it could not have used 100 GB of spce from
> filesystem as we know 30 GB was free
>
> Also let me know how to check DLQ.
>
>
>
> --
> Sent from:
> http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html
>