ActiveMQ fails to gracefully shut down if ACTIVEMQ_OPTS_MEMORY uses > 50% of memory.

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

ActiveMQ fails to gracefully shut down if ACTIVEMQ_OPTS_MEMORY uses > 50% of memory.

Kevin Burton
Pretty sure this is a bug and I”ll create one if you guys agree.

I’m allocating about 6GB of RAM to ActiveMQ.  The problem is that this box
only has 4GB free after that.

If I try to gracefully shut down ActiveMQ tries to run:

          PID=`cat $ACTIVEMQ_PIDFILE`

          $EXEC_OPTION $DOIT_PREFIX "$JAVACMD $ACTIVEMQ_OPTS
$ACTIVEMQ_DEBUG_OPTS \

              -Dactivemq.classpath=\"${ACTIVEMQ_CLASSPATH}\" \

              -Dactivemq.home=\"${ACTIVEMQ_HOME}\" \

              -Dactivemq.base=\"${ACTIVEMQ_BASE}\" \

              -Dactivemq.conf=\"${ACTIVEMQ_CONF}\" \

              -Dactivemq.data=\"${ACTIVEMQ_DATA}\" \

              $ACTIVEMQ_CYGWIN \

              -jar \"${ACTIVEMQ_HOME}/bin/activemq.jar\" $COMMANDLINE_ARGS
--pid $PID &

              RET=\"\$?\"; APID=\"\$!\";

              echo \$APID > $ACTIVEMQ_DATA/stop.pid; exit \$RET"
$DOIT_POSTFIX

      RET="$?"


However, this will fail.  because ACTIVEMQ_OPTS includes
ACTIVEMQ_OPTS_MEMORY which is where I set the daemon to use large amounts
of memory.

I THINK the resolution to this is to ONLY use ACTIVEMQ_OPTS_MEMORY in
start…. wouldn’t you agree?

I might pull this script into our version control and fix it and then
submit back a patch.

This has a second issue where since I’ms eating -Xms and -Xmx and
allocating another 6GB during stop it could trigger an OOM killer and will
definitely invalidate a large chunk of the VFS page cache.

Kevin

--

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ fails to gracefully shut down if ACTIVEMQ_OPTS_MEMORY uses > 50% of memory.

Tim Bain
Similarly, we've noticed that when you enable the debugging options, they
get applied to the shutdown too, so you get a warning message saying the
port is already in use.  That doesn't prevent the JVM from starting like
the memory arguments would, but it still should be cleaned up at the same
time.  And I agree that the right implementation is to only use those
arguments on start (and the start portion of restart), but not on stop or
status.

Tim
On Feb 8, 2015 6:31 AM, "Kevin Burton" <[hidden email]> wrote:

> Pretty sure this is a bug and I”ll create one if you guys agree.
>
> I’m allocating about 6GB of RAM to ActiveMQ.  The problem is that this box
> only has 4GB free after that.
>
> If I try to gracefully shut down ActiveMQ tries to run:
>
>           PID=`cat $ACTIVEMQ_PIDFILE`
>
>           $EXEC_OPTION $DOIT_PREFIX "$JAVACMD $ACTIVEMQ_OPTS
> $ACTIVEMQ_DEBUG_OPTS \
>
>               -Dactivemq.classpath=\"${ACTIVEMQ_CLASSPATH}\" \
>
>               -Dactivemq.home=\"${ACTIVEMQ_HOME}\" \
>
>               -Dactivemq.base=\"${ACTIVEMQ_BASE}\" \
>
>               -Dactivemq.conf=\"${ACTIVEMQ_CONF}\" \
>
>               -Dactivemq.data=\"${ACTIVEMQ_DATA}\" \
>
>               $ACTIVEMQ_CYGWIN \
>
>               -jar \"${ACTIVEMQ_HOME}/bin/activemq.jar\" $COMMANDLINE_ARGS
> --pid $PID &
>
>               RET=\"\$?\"; APID=\"\$!\";
>
>               echo \$APID > $ACTIVEMQ_DATA/stop.pid; exit \$RET"
> $DOIT_POSTFIX
>
>       RET="$?"
>
>
> However, this will fail.  because ACTIVEMQ_OPTS includes
> ACTIVEMQ_OPTS_MEMORY which is where I set the daemon to use large amounts
> of memory.
>
> I THINK the resolution to this is to ONLY use ACTIVEMQ_OPTS_MEMORY in
> start…. wouldn’t you agree?
>
> I might pull this script into our version control and fix it and then
> submit back a patch.
>
> This has a second issue where since I’ms eating -Xms and -Xmx and
> allocating another 6GB during stop it could trigger an OOM killer and will
> definitely invalidate a large chunk of the VFS page cache.
>
> Kevin
>
> --
>
> Founder/CEO Spinn3r.com
> Location: *San Francisco, CA*
> blog: http://burtonator.wordpress.com
> … or check out my Google+ profile
> <https://plus.google.com/102718274791889610666/posts>
> <http://spinn3r.com>
>
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ fails to gracefully shut down if ACTIVEMQ_OPTS_MEMORY uses > 50% of memory.

Kevin Burton
I fixed this in my local version.

https://issues.apache.org/jira/browse/AMQ-5571

I created a JIRA on this.

Tim, My version will also fix the issue you see above with debugging.

I included a patch in the JIRA.

IF someone could apply this patch and close it out (and agrees that the
patch is a good solution) then I’ll be super happy!

Thanks!

On Sun, Feb 8, 2015 at 4:06 AM, Tim Bain <[hidden email]> wrote:

> Similarly, we've noticed that when you enable the debugging options, they
> get applied to the shutdown too, so you get a warning message saying the
> port is already in use.  That doesn't prevent the JVM from starting like
> the memory arguments would, but it still should be cleaned up at the same
> time.  And I agree that the right implementation is to only use those
> arguments on start (and the start portion of restart), but not on stop or
> status.
>
> Tim
> On Feb 8, 2015 6:31 AM, "Kevin Burton" <[hidden email]> wrote:
>
> > Pretty sure this is a bug and I”ll create one if you guys agree.
> >
> > I’m allocating about 6GB of RAM to ActiveMQ.  The problem is that this
> box
> > only has 4GB free after that.
> >
> > If I try to gracefully shut down ActiveMQ tries to run:
> >
> >           PID=`cat $ACTIVEMQ_PIDFILE`
> >
> >           $EXEC_OPTION $DOIT_PREFIX "$JAVACMD $ACTIVEMQ_OPTS
> > $ACTIVEMQ_DEBUG_OPTS \
> >
> >               -Dactivemq.classpath=\"${ACTIVEMQ_CLASSPATH}\" \
> >
> >               -Dactivemq.home=\"${ACTIVEMQ_HOME}\" \
> >
> >               -Dactivemq.base=\"${ACTIVEMQ_BASE}\" \
> >
> >               -Dactivemq.conf=\"${ACTIVEMQ_CONF}\" \
> >
> >               -Dactivemq.data=\"${ACTIVEMQ_DATA}\" \
> >
> >               $ACTIVEMQ_CYGWIN \
> >
> >               -jar \"${ACTIVEMQ_HOME}/bin/activemq.jar\"
> $COMMANDLINE_ARGS
> > --pid $PID &
> >
> >               RET=\"\$?\"; APID=\"\$!\";
> >
> >               echo \$APID > $ACTIVEMQ_DATA/stop.pid; exit \$RET"
> > $DOIT_POSTFIX
> >
> >       RET="$?"
> >
> >
> > However, this will fail.  because ACTIVEMQ_OPTS includes
> > ACTIVEMQ_OPTS_MEMORY which is where I set the daemon to use large amounts
> > of memory.
> >
> > I THINK the resolution to this is to ONLY use ACTIVEMQ_OPTS_MEMORY in
> > start…. wouldn’t you agree?
> >
> > I might pull this script into our version control and fix it and then
> > submit back a patch.
> >
> > This has a second issue where since I’ms eating -Xms and -Xmx and
> > allocating another 6GB during stop it could trigger an OOM killer and
> will
> > definitely invalidate a large chunk of the VFS page cache.
> >
> > Kevin
> >
> > --
> >
> > Founder/CEO Spinn3r.com
> > Location: *San Francisco, CA*
> > blog: http://burtonator.wordpress.com
> > … or check out my Google+ profile
> > <https://plus.google.com/102718274791889610666/posts>
> > <http://spinn3r.com>
> >
>



--

Founder/CEO Spinn3r.com
Location: *San Francisco, CA*
blog: http://burtonator.wordpress.com
… or check out my Google+ profile
<https://plus.google.com/102718274791889610666/posts>
<http://spinn3r.com>