how can a producer detect that temp disk is full

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

how can a producer detect that temp disk is full

joesmithcomm
Hi,

I'm trying to impl a use case where a non-persistent msg producer can
send msgs until no more disk space.   We want to flow the msgs to disk with a limit - and to detect when the space is full (instead of being throttled).

The attempt is to set tempUsage limit with sendFailIfNoSpace on the systemUsage.  The target queue is myqueue.

        <destinationPolicy>
            <policyMap>
              <policyEntries>
                <policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb">
                  <pendingSubscriberPolicy>
                    <vmCursor />
                  </pendingSubscriberPolicy>
                </policyEntry>
                <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb">
                </policyEntry>
                <policyEntry queue="myqueue" producerFlowControl="true" memoryLimit="1mb">
                  <pendingQueuePolicy>
                    <fileQueueCursor/>
                  </pendingQueuePolicy>
                </policyEntry>
              </policyEntries>
            </policyMap>
        </destinationPolicy>

        <systemUsage>
            <systemUsage sendFailIfNoSpace="true" >
                <memoryUsage>
                    <memoryUsage limit="20 mb"/>
                </memoryUsage>
                <storeUsa
                    <storeUsage limit="1 gb"/>
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="33 mb"/>
                </tempUsage>
            </systemUsage>
        </systemUsage>


But the outcome was not exact.

1 - fileQueueCursor, flow control on, producer jms.useAsyncSend=false

The producer stopped after sending about 2600 msgs.  When a consumer was connected, no msgs were received, even though the queue has about 700 msgs.


2 - fileQueueCursor, flow control off, producer jms.useAsyncSend=false

Same as above - the flow control did not affect the outcome.
3 - vmQueueCursor, flow control on

Producer received onException call
back after 1mb memory was exhausted.  No msg
flowed to disk.  Producer can continue to send w/o error after a consumer had drained msgs from memory.

4 - vmQueueCursor, flow control off
Producer continued to send - beyond 32mb tempUsage limit.

Broker version 5.5.1
Java 1.6
64-bit linux system


Any suggestions on how to utilize all disk space but with notification when space is near depletion?


Thanks
Reply | Threaded
Open this post in threaded view
|

Re: how can a producer detect that temp disk is full

gtully
Simplest config is to use the default cursor, which is the store
cursor that has an embedded file cursor for non persistent messages.

To have the file cursor spool to disk, it needs to reach 70% of its
destination limit, so the system usage memory limits need to exceed
this value, which the appear to do in your config below: 20mb > 1mb

The spooling will continue till the temp usage limit is reached and
the resource exception is throws as a result of sendFailIfNoSpace.

I though there was a simple junit test case for this but it seems not,
I just added one to an existing test case, see:
http://svn.apache.org/viewvc?view=revision&revision=1233367

The new test is:
org.apache.activemq.bugs.TempStorageBlockedBrokerTest#testFillTempAndConsume

play around with that or compare to what you are doing and if you find
any problem use that test case as the basis for a bug report.

Note, the vm cursors do not spool to disk, so they will not use any
temp store (disk space)

On 19 January 2012 10:45, Joe Smith <[hidden email]> wrote:

> Hi,
>
> I'm trying to impl a use case where a non-persistent msg producer can
> send msgs until no more disk space.   We want to flow the msgs to disk with a limit - and to detect when the space is full (instead of being throttled).
>
> The attempt is to set tempUsage limit with sendFailIfNoSpace on the systemUsage.  The target queue is myqueue.
>
>         <destinationPolicy>
>             <policyMap>
>               <policyEntries>
>                 <policyEntry topic=">" producerFlowControl="true" memoryLimit="1mb">
>                   <pendingSubscriberPolicy>
>                     <vmCursor />
>                   </pendingSubscriberPolicy>
>                 </policyEntry>
>                 <policyEntry queue=">" producerFlowControl="true" memoryLimit="1mb">
>                 </policyEntry>
>                 <policyEntry queue="myqueue" producerFlowControl="true" memoryLimit="1mb">
>                   <pendingQueuePolicy>
>                     <fileQueueCursor/>
>                   </pendingQueuePolicy>
>                 </policyEntry>
>               </policyEntries>
>             </policyMap>
>         </destinationPolicy>
>
>         <systemUsage>
>             <systemUsage sendFailIfNoSpace="true" >
>                 <memoryUsage>
>                     <memoryUsage limit="20 mb"/>
>                 </memoryUsage>
>                 <storeUsa
>                     <storeUsage limit="1 gb"/>
>                 </storeUsage>
>                 <tempUsage>
>                     <tempUsage limit="33 mb"/>
>                 </tempUsage>
>             </systemUsage>
>         </systemUsage>
>
>
> But the outcome was not exact.
>
> 1 - fileQueueCursor, flow control on, producer jms.useAsyncSend=false
>
> The producer stopped after sending about 2600 msgs.  When a consumer was connected, no msgs were received, even though the queue has about 700 msgs.
>
>
> 2 - fileQueueCursor, flow control off, producer jms.useAsyncSend=false
>
> Same as above - the flow control did not affect the outcome.
> 3 - vmQueueCursor, flow control on
>
> Producer received onException call
> back after 1mb memory was exhausted.  No msg
> flowed to disk.  Producer can continue to send w/o error after a consumer had drained msgs from memory.
>
> 4 - vmQueueCursor, flow control off
> Producer continued to send - beyond 32mb tempUsage limit.
>
> Broker version 5.5.1
> Java 1.6
> 64-bit linux system
>
>
> Any suggestions on how to utilize all disk space but with notification when space is near depletion?
>
>
> Thanks



--
http://fusesource.com
http://blog.garytully.com
Reply | Threaded
Open this post in threaded view
|

Re: how can a producer detect that temp disk is full

joesmithcomm
Hi Gary,

Thanks for the link to test source file.  Finally got a chance to add a test method and ran some tests.

Scenario 1:
1. Producer sends msgs
2. Pause 30 sec for Producer to send msgs
3. Consumer receives msgs but at a slower rate than Producer
4. Compare # msgs sent, received, and # of ResourceAllocationException received.

Result:
1. 1600 msgs were sent by Producer
2. 1499 msgs were received by Consumer
3.   101 ResourceAlloctionExcpetions were received by Producer on the onException() callback.

However, the disk remained full - space was not released.  When sending a 2nd batch of 1600 msgs, Producer received 1600 ResourceAllocationExcetions.

I wasn't sure if there are other settings that need to be satisfied to release the space or the connection need to be closed...

Here is a snippet of the exception from producer2 sending:

2012-01-19 17:27:31,114 [127.0.0.1:50460] - WARN  Service                        - Async error occurred: javax.jms.ResourceAllocationException: Usage Manager Temp Store is Full (99% of 33554432). Stopping producer (ID:US137440.local-50458-1327011882287-1:2:2:1) to prevent flooding queue://testqueue. See http://activemq.apache.org/producer-flow-control.html for more info
javax.jms.ResourceAllocationException: Usage Manager Temp Store is Full (99% of 33554432). Stopping producer (ID:US137440.local-50458-1327011882287-1:2:2:1) to prevent flooding queue://testqueue. See http://activemq.apache.org/producer-flow-control.html for more info
        at org.apache.activemq.broker.region.BaseDestination.waitForSpace(BaseDestination.java:579)
        at org.apache.activemq.broker.region.BaseDestination.waitForSpace(BaseDestination.java:573)
        at org.apache.activemq.broker.region.Queue.checkUsage(Queue.java:757)
        at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:674)
        at org.apache.activemq.broker.region.Queue.send(Queue.java:653)
        at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:365)


Scenario 2:
- Same as 1, but I didn't start the consumer until Producer had sent all 1600 messages.  Consumer read all messages, but the temp store remained at 100% usage - and not more msgs could be sent.

I read somewhere the min allocation for temp store should be 32mb or greater.  I increased the temp store limit to 38mb and ran w/increased number of msgs to max out the space.  Resulting behavior is the same.

----------------------------

Also, when I used the same settings and ran against the downloaded binary broker, the behavior seems to be differ.  Producer received ResourceAllocationException as expected, and it maxed out temp store.  I then started a Consumer to read from the queue, the responses were:
1. Consumer received some messages but then stopped.
2. Consumer received no message at all.

Guess:

Once the temp store is maxed out, somehow the space was not freed up.  Subsequent connection/sends were blocked while waiting for spaces.  Let me know if I mis-configured some options.

I am attaching the src code (*.java), full log (activemq-test.log) and the summary log w/o the error/warn trace in the zip attachment  src-log.zip.  I'm not sure if you will get it.  Let me know if I should  post it somewhere.

Thanks for any help.
Jason



From: Gary Tully <[hidden email]>
To: [hidden email]; Joe Smith <[hidden email]>
Sent: Thursday, January 19, 2012 8:39 AM
Subject: Re: how can a producer detect that temp disk is full

Simplest config is to use the default cursor, which is the store
cursor that has an embedded file cursor for non persistent messages.

To have the file cursor spool to disk, it needs to reach 70% of its
destination limit, so the system usage memory limits need to exceed
this value, which the appear to do in your config below: 20mb > 1mb

The spooling will continue till the temp usage limit is reached and
the resource exception is throws as a result of sendFailIfNoSpace.

I though there was a simple junit test case for this but it seems not,
I just added one to an existing test case, see:
http://svn.apache.org/viewvc?view=revision&revision=1233367

The new test is:
org.apache.activemq.bugs.TempStorageBlockedBrokerTest#testFillTempAndConsume

play around with that or compare to what you are doing and if you find
any problem use that test case as the basis for a bug report.

Note, the vm cursors do not spool to disk, so they will not use any
temp store (disk space)



src-log.zip (51K) Download Attachment