why it used up all the heap space and hang even I specified memoryUsage limit?

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

why it used up all the heap space and hang even I specified memoryUsage limit?

Li Li
    I gave 3GB  total heap space of activemq, the XX:NewRatio=2. So
new generation 1GB and old generation 2GB. And I also config
memoryUsage limit="1 gb"


ACTIVEMQ_OPTS_MEMORY="-Xms1G -Xmx3G -XX:+UseConcMarkSweepGC
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=70 -XX:NewRatio=2"
There are only two queues.

<policyEntry queue=">" optimizedDispatch="true"
producerFlowControl="false" cursorMemoryHighWaterMark="70"
memoryLimit="100 mb" />

        <systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage limit="1 gb" />
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit="50 gb"/>
                </storeUsage>
                <tempUsage>
                    <tempUsage limit="50 gb"/>
                </tempUsage>
            </systemUsage>
        </systemUsage>


$ jstat -gcutil 10345 1000
  S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT
 99.72   0.00 100.00 100.00  59.94    293   14.149    69  115.657  129.806
 99.72   0.00 100.00 100.00  59.94    293   14.149    69  115.657  129.806
  0.00   0.00  58.20 100.00  59.94    293   14.149    70  129.375  143.524
  0.00   0.00  62.86 100.00  59.95    293   14.149    70  130.325  144.474
  0.00   0.00  72.10 100.00  59.98    293   14.149    70  130.325  144.474
  0.00   0.00  85.59 100.00  59.98    293   14.149    70  130.325  144.474
  0.00   0.00  99.82 100.00  59.98    293   14.149    70  130.325  144.474
 46.40   0.00 100.00 100.00  59.99    293   14.149    70  130.325  144.474
 99.96   0.00 100.00 100.00  60.00    293   14.149    71  130.325  144.474
 99.96   0.00 100.00 100.00  60.00    293   14.149    71  130.325  144.474
 99.96   0.00 100.00 100.00  60.00    293   14.149    71  130.325  144.474

$ jmap -heap 10345
Attaching to process ID 10345, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.45-b08

using parallel threads in the new generation.
using thread-local object allocation.
Concurrent Mark-Sweep GC

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 3221225472 (3072.0MB)
   NewSize          = 1310720 (1.25MB)
   MaxNewSize       = 17592186044415 MB
   OldSize          = 5439488 (5.1875MB)
   NewRatio         = 2
   SurvivorRatio    = 8
   PermSize         = 21757952 (20.75MB)
   MaxPermSize      = 85983232 (82.0MB)
   G1HeapRegionSize = 0 (0.0MB)

Heap Usage:
New Generation (Eden + 1 Survivor Space):
   capacity = 966393856 (921.625MB)
   used     = 966388976 (921.6203460693359MB)
   free     = 4880 (0.0046539306640625MB)
   99.99949502990218% used
Eden Space:
   capacity = 859045888 (819.25MB)
   used     = 859045888 (819.25MB)
   free     = 0 (0.0MB)
   100.0% used
From Space:
   capacity = 107347968 (102.375MB)
   used     = 107343088 (102.37034606933594MB)
   free     = 4880 (0.0046539306640625MB)
   99.99545403598138% used
To Space:
   capacity = 107347968 (102.375MB)
   used     = 0 (0.0MB)
   free     = 107347968 (102.375MB)
   0.0% used
concurrent mark-sweep generation:
   capacity = 2147483648 (2048.0MB)
   used     = 2147483608 (2047.9999618530273MB)
   free     = 40 (3.814697265625E-5MB)
   99.99999813735485% used
Perm Generation:
   capacity = 52916224 (50.46484375MB)
   used     = 31660712 (30.194007873535156MB)
   free     = 21255512 (20.270835876464844MB)
   59.83176728558712% used
Reply | Threaded
Open this post in threaded view
|

Re: why it used up all the heap space and hang even I specified memoryUsage limit?

ceposta
Just curious why you have producer flow control off if you expect it to
honor memory limits?

On Wednesday, June 25, 2014, Li Li <[hidden email]> wrote:

>     I gave 3GB  total heap space of activemq, the XX:NewRatio=2. So
> new generation 1GB and old generation 2GB. And I also config
> memoryUsage limit="1 gb"
>
>
> ACTIVEMQ_OPTS_MEMORY="-Xms1G -Xmx3G -XX:+UseConcMarkSweepGC
> -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=70 -XX:NewRatio=2"
> There are only two queues.
>
> <policyEntry queue=">" optimizedDispatch="true"
> producerFlowControl="false" cursorMemoryHighWaterMark="70"
> memoryLimit="100 mb" />
>
>         <systemUsage>
>             <systemUsage>
>                 <memoryUsage>
>                     <memoryUsage limit="1 gb" />
>                 </memoryUsage>
>                 <storeUsage>
>                     <storeUsage limit="50 gb"/>
>                 </storeUsage>
>                 <tempUsage>
>                     <tempUsage limit="50 gb"/>
>                 </tempUsage>
>             </systemUsage>
>         </systemUsage>
>
>
> $ jstat -gcutil 10345 1000
>   S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT
>  99.72   0.00 100.00 100.00  59.94    293   14.149    69  115.657  129.806
>  99.72   0.00 100.00 100.00  59.94    293   14.149    69  115.657  129.806
>   0.00   0.00  58.20 100.00  59.94    293   14.149    70  129.375  143.524
>   0.00   0.00  62.86 100.00  59.95    293   14.149    70  130.325  144.474
>   0.00   0.00  72.10 100.00  59.98    293   14.149    70  130.325  144.474
>   0.00   0.00  85.59 100.00  59.98    293   14.149    70  130.325  144.474
>   0.00   0.00  99.82 100.00  59.98    293   14.149    70  130.325  144.474
>  46.40   0.00 100.00 100.00  59.99    293   14.149    70  130.325  144.474
>  99.96   0.00 100.00 100.00  60.00    293   14.149    71  130.325  144.474
>  99.96   0.00 100.00 100.00  60.00    293   14.149    71  130.325  144.474
>  99.96   0.00 100.00 100.00  60.00    293   14.149    71  130.325  144.474
>
> $ jmap -heap 10345
> Attaching to process ID 10345, please wait...
> Debugger attached successfully.
> Server compiler detected.
> JVM version is 24.45-b08
>
> using parallel threads in the new generation.
> using thread-local object allocation.
> Concurrent Mark-Sweep GC
>
> Heap Configuration:
>    MinHeapFreeRatio = 40
>    MaxHeapFreeRatio = 70
>    MaxHeapSize      = 3221225472 (3072.0MB)
>    NewSize          = 1310720 (1.25MB)
>    MaxNewSize       = 17592186044415 MB
>    OldSize          = 5439488 (5.1875MB)
>    NewRatio         = 2
>    SurvivorRatio    = 8
>    PermSize         = 21757952 (20.75MB)
>    MaxPermSize      = 85983232 (82.0MB)
>    G1HeapRegionSize = 0 (0.0MB)
>
> Heap Usage:
> New Generation (Eden + 1 Survivor Space):
>    capacity = 966393856 (921.625MB)
>    used     = 966388976 (921.6203460693359MB)
>    free     = 4880 (0.0046539306640625MB)
>    99.99949502990218% used
> Eden Space:
>    capacity = 859045888 (819.25MB)
>    used     = 859045888 (819.25MB)
>    free     = 0 (0.0MB)
>    100.0% used
> From Space:
>    capacity = 107347968 (102.375MB)
>    used     = 107343088 (102.37034606933594MB)
>    free     = 4880 (0.0046539306640625MB)
>    99.99545403598138% used
> To Space:
>    capacity = 107347968 (102.375MB)
>    used     = 0 (0.0MB)
>    free     = 107347968 (102.375MB)
>    0.0% used
> concurrent mark-sweep generation:
>    capacity = 2147483648 (2048.0MB)
>    used     = 2147483608 (2047.9999618530273MB)
>    free     = 40 (3.814697265625E-5MB)
>    99.99999813735485% used
> Perm Generation:
>    capacity = 52916224 (50.46484375MB)
>    used     = 31660712 (30.194007873535156MB)
>    free     = 21255512 (20.270835876464844MB)
>    59.83176728558712% used
>


--
*Christian Posta*
http://www.christianposta.com/blog
http://fabric8.io
twitter: @christianposta
Reply | Threaded
Open this post in threaded view
|

Re: why it used up all the heap space and hang even I specified memoryUsage limit?

Li Li
The producer is much faster than consumer. I will do speed control --
when the producer find the queue size is above a threshold, it will
stop sending messages
I just want the producer sent as many messages as broker can store. I
want the broker save messages in broker's hard disk. So the producer
don't have to consider how to save the messages itself.

On Thu, Jun 26, 2014 at 8:35 PM, Christian Posta
<[hidden email]> wrote:

> Just curious why you have producer flow control off if you expect it to
> honor memory limits?
>
> On Wednesday, June 25, 2014, Li Li <[hidden email]> wrote:
>
>>     I gave 3GB  total heap space of activemq, the XX:NewRatio=2. So
>> new generation 1GB and old generation 2GB. And I also config
>> memoryUsage limit="1 gb"
>>
>>
>> ACTIVEMQ_OPTS_MEMORY="-Xms1G -Xmx3G -XX:+UseConcMarkSweepGC
>> -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=70 -XX:NewRatio=2"
>> There are only two queues.
>>
>> <policyEntry queue=">" optimizedDispatch="true"
>> producerFlowControl="false" cursorMemoryHighWaterMark="70"
>> memoryLimit="100 mb" />
>>
>>         <systemUsage>
>>             <systemUsage>
>>                 <memoryUsage>
>>                     <memoryUsage limit="1 gb" />
>>                 </memoryUsage>
>>                 <storeUsage>
>>                     <storeUsage limit="50 gb"/>
>>                 </storeUsage>
>>                 <tempUsage>
>>                     <tempUsage limit="50 gb"/>
>>                 </tempUsage>
>>             </systemUsage>
>>         </systemUsage>
>>
>>
>> $ jstat -gcutil 10345 1000
>>   S0     S1     E      O      P     YGC     YGCT    FGC    FGCT     GCT
>>  99.72   0.00 100.00 100.00  59.94    293   14.149    69  115.657  129.806
>>  99.72   0.00 100.00 100.00  59.94    293   14.149    69  115.657  129.806
>>   0.00   0.00  58.20 100.00  59.94    293   14.149    70  129.375  143.524
>>   0.00   0.00  62.86 100.00  59.95    293   14.149    70  130.325  144.474
>>   0.00   0.00  72.10 100.00  59.98    293   14.149    70  130.325  144.474
>>   0.00   0.00  85.59 100.00  59.98    293   14.149    70  130.325  144.474
>>   0.00   0.00  99.82 100.00  59.98    293   14.149    70  130.325  144.474
>>  46.40   0.00 100.00 100.00  59.99    293   14.149    70  130.325  144.474
>>  99.96   0.00 100.00 100.00  60.00    293   14.149    71  130.325  144.474
>>  99.96   0.00 100.00 100.00  60.00    293   14.149    71  130.325  144.474
>>  99.96   0.00 100.00 100.00  60.00    293   14.149    71  130.325  144.474
>>
>> $ jmap -heap 10345
>> Attaching to process ID 10345, please wait...
>> Debugger attached successfully.
>> Server compiler detected.
>> JVM version is 24.45-b08
>>
>> using parallel threads in the new generation.
>> using thread-local object allocation.
>> Concurrent Mark-Sweep GC
>>
>> Heap Configuration:
>>    MinHeapFreeRatio = 40
>>    MaxHeapFreeRatio = 70
>>    MaxHeapSize      = 3221225472 (3072.0MB)
>>    NewSize          = 1310720 (1.25MB)
>>    MaxNewSize       = 17592186044415 MB
>>    OldSize          = 5439488 (5.1875MB)
>>    NewRatio         = 2
>>    SurvivorRatio    = 8
>>    PermSize         = 21757952 (20.75MB)
>>    MaxPermSize      = 85983232 (82.0MB)
>>    G1HeapRegionSize = 0 (0.0MB)
>>
>> Heap Usage:
>> New Generation (Eden + 1 Survivor Space):
>>    capacity = 966393856 (921.625MB)
>>    used     = 966388976 (921.6203460693359MB)
>>    free     = 4880 (0.0046539306640625MB)
>>    99.99949502990218% used
>> Eden Space:
>>    capacity = 859045888 (819.25MB)
>>    used     = 859045888 (819.25MB)
>>    free     = 0 (0.0MB)
>>    100.0% used
>> From Space:
>>    capacity = 107347968 (102.375MB)
>>    used     = 107343088 (102.37034606933594MB)
>>    free     = 4880 (0.0046539306640625MB)
>>    99.99545403598138% used
>> To Space:
>>    capacity = 107347968 (102.375MB)
>>    used     = 0 (0.0MB)
>>    free     = 107347968 (102.375MB)
>>    0.0% used
>> concurrent mark-sweep generation:
>>    capacity = 2147483648 (2048.0MB)
>>    used     = 2147483608 (2047.9999618530273MB)
>>    free     = 40 (3.814697265625E-5MB)
>>    99.99999813735485% used
>> Perm Generation:
>>    capacity = 52916224 (50.46484375MB)
>>    used     = 31660712 (30.194007873535156MB)
>>    free     = 21255512 (20.270835876464844MB)
>>    59.83176728558712% used
>>
>
>
> --
> *Christian Posta*
> http://www.christianposta.com/blog
> http://fabric8.io
> twitter: @christianposta