kahadb index file larger than journal file

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

kahadb index file larger than journal file

Scorpio
Hi, All,

In my activemq instance, I find that the kahadb index file is larger than the journal file.

As far as I know, db.data is the meta store, index for the journal file, which should means it is much smaller than the journal file. However, it does not look like that. The index file seems to keep growing while the journal file is rotated. Do I miss something? Thanks.

Log attached:
root@host:~/run/apache-activemq-5.13.1/data/kahadb# du * -sh
16M     db-16.log
1.2G    db.data
3.2M    db.redo
4.0K    lock
root@host:~/run/apache-activemq-5.13.1/data/kahadb#

Kind regards,
Scorpio
Reply | Threaded
Open this post in threaded view
|

Re: kahadb index file larger than journal file

christopher.l.shannon
The index grows based off the amount of data it tracks in the journal,
but doesn't shrink.  This means if you had a lot of data in your
journal at some point (several gigabytes, etc) then the index would
have grown in size with enough page files to keep track of all of the
data.  This is usually not an issue because the index size generally
stabilizes at some point relative to the work load of the broker.   If
you really want to shrink it the only way right now is to stop the
broker, delete the index and restart so it does a replay and rebuilds
the index but the size would just grow again as messages flow.

On Mon, Jan 23, 2017 at 8:06 PM, Scorpio <[hidden email]> wrote:

> Hi, All,
>
> In my activemq instance, I find that the kahadb index file is larger than
> the journal file.
>
> As far as I know, /db.data/ is the meta store, index for the journal file,
> which should means it is much smaller than the journal file. However, it
> does not look like that. The index file seems to keep growing while the
> journal file is rotated. Do I miss something? Thanks.
>
> Log attached:
> root@host:~/run/apache-activemq-5.13.1/data/kahadb# du * -sh
> 16M     db-16.log
> 1.2G    db.data
> 3.2M    db.redo
> 4.0K    lock
> root@host:~/run/apache-activemq-5.13.1/data/kahadb#
>
> Kind regards,
> Scorpio
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/kahadb-index-file-larger-than-journal-file-tp4721284.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: kahadb index file larger than journal file

Scorpio

Christopher:


Thank you for your explanation!

Hope there would be other proper ways to shrink the index file in future ActiveMQ releases.

Once we monitored that our broker is killed by system due to "out of memory".

I know that meta cache in memory will be synced into index file at checking points. Does this mean there is as much memory as  the db.data file being used?

Even though our broker is idle,  we still find that it uses up much memory. Could they be relevant associated? Thanks


Kind regards,

Scorpio

On 2017/1/24 20:16, christopher.l.shannon [via ActiveMQ] wrote:
The index grows based off the amount of data it tracks in the journal,
but doesn't shrink.  This means if you had a lot of data in your
journal at some point (several gigabytes, etc) then the index would
have grown in size with enough page files to keep track of all of the
data.  This is usually not an issue because the index size generally
stabilizes at some point relative to the work load of the broker.   If
you really want to shrink it the only way right now is to stop the
broker, delete the index and restart so it does a replay and rebuilds
the index but the size would just grow again as messages flow.

On Mon, Jan 23, 2017 at 8:06 PM, Scorpio <[hidden email]> wrote:

> Hi, All,
>
> In my activemq instance, I find that the kahadb index file is larger than
> the journal file.
>
> As far as I know, /db.data/ is the meta store, index for the journal file,
> which should means it is much smaller than the journal file. However, it
> does not look like that. The index file seems to keep growing while the
> journal file is rotated. Do I miss something? Thanks.
>
> Log attached:
> [hidden email] du * -sh
> 16M     db-16.log
> 1.2G    db.data
> 3.2M    db.redo
> 4.0K    lock
> [hidden email]
>
> Kind regards,
> Scorpio
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/kahadb-index-file-larger-than-journal-file-tp4721284.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.



If you reply to this email, your message will be added to the discussion below:
http://activemq.2283324.n4.nabble.com/kahadb-index-file-larger-than-journal-file-tp4721284p4721300.html
To start a new topic under ActiveMQ - User, email [hidden email]
To unsubscribe from kahadb index file larger than journal file, click here.
NAML

Reply | Threaded
Open this post in threaded view
|

Re: kahadb index file larger than journal file

christopher.l.shannon
Sorry for taking a while to get back to you.

Based on your originally message and after analyzing some index files I
have actually discovered a bug with KahaDB that causes the index file to
grow too large when things like out of memory errors occur or any unclean
shutdown.  On recovery the free pages in the index file are not being
properly tracked which is leading to excessive index file usage.  So this
is probably the issue that you are seeing.  I will be fixing this in
version 5.14.4.  Upon upgrade you will need to delete the old index and
have it be rebuilt and then going forward it should work as intended.  You
can track the status of the bug here:
https://issues.apache.org/jira/browse/AMQ-6590

In terms of the in memory cache, only a fixed number of pages are cached.
The default is 10 thousand pages and each page is 4KB.  So roughly 40 MB of
memory will be used at most.  If the number of pages in use grows beyond
that then the cache will swap pages based on the defined eviction
strategy.  You can increase the number of pages in memory (which would help
performance at the cost of eating up memory) by tweaking the indexCacheSize
value of KahaDB.  See http://activemq.apache.org/kahadb.html for more info.

On Tue, Jan 24, 2017 at 11:00 PM, Scorpio <[hidden email]> wrote:

> Christopher:
>
>
> Thank you for your explanation!
>
> Hope there would be other proper ways to shrink the index file in future
> ActiveMQ releases.
>
> Once we monitored that our broker is killed by system due to "out of
> memory".
>
> I know that meta cache in memory will be synced into index file at
> checking points. _*Does this mean there is as much memory as  the
> db.data file being used?*_
>
> Even though our broker is idle,  we still find that it uses up much
> memory. Could they be relevant associated? Thanks
>
>
> Kind regards,
>
> Scorpio
>
> On 2017/1/24 20:16, christopher.l.shannon [via ActiveMQ] wrote:
> > The index grows based off the amount of data it tracks in the journal,
> > but doesn't shrink.  This means if you had a lot of data in your
> > journal at some point (several gigabytes, etc) then the index would
> > have grown in size with enough page files to keep track of all of the
> > data.  This is usually not an issue because the index size generally
> > stabilizes at some point relative to the work load of the broker.   If
> > you really want to shrink it the only way right now is to stop the
> > broker, delete the index and restart so it does a replay and rebuilds
> > the index but the size would just grow again as messages flow.
> >
> > On Mon, Jan 23, 2017 at 8:06 PM, Scorpio <[hidden email]
> > </user/SendEmail.jtp?type=node&node=4721300&i=0>> wrote:
> >
> > > Hi, All,
> > >
> > > In my activemq instance, I find that the kahadb index file is larger
> > than
> > > the journal file.
> > >
> > > As far as I know, /db.data/ is the meta store, index for the journal
> > file,
> > > which should means it is much smaller than the journal file.
> > However, it
> > > does not look like that. The index file seems to keep growing while the
> > > journal file is rotated. Do I miss something? Thanks.
> > >
> > > Log attached:
> > > root@host:~/run/apache-activemq-5.13.1/data/kahadb# du * -sh
> > > 16M     db-16.log
> > > 1.2G    db.data
> > > 3.2M    db.redo
> > > 4.0K    lock
> > > root@host:~/run/apache-activemq-5.13.1/data/kahadb#
> > >
> > > Kind regards,
> > > Scorpio
> > >
> > >
> > >
> > > --
> > > View this message in context:
> > http://activemq.2283324.n4.nabble.com/kahadb-index-file-
> larger-than-journal-file-tp4721284.html
> > > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
> >
> > ------------------------------------------------------------------------
> > If you reply to this email, your message will be added to the
> > discussion below:
> > http://activemq.2283324.n4.nabble.com/kahadb-index-file-
> larger-than-journal-file-tp4721284p4721300.html
> >
> > To start a new topic under ActiveMQ - User, email
> > [hidden email]
> > To unsubscribe from kahadb index file larger than journal file, click
> > here
> > <http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=
> unsubscribe_by_code&node=4721284&code=bnlAeGlhbmdqaWFiYW8uY29tfDQ3Mj
> EyODR8MTkzNzMzOTQ5NQ==>.
> > NAML
> > <http://activemq.2283324.n4.nabble.com/template/
> NamlServlet.jtp?macro=macro_viewer&id=instant_html%
> 21nabble%3Aemail.naml&base=nabble.naml.namespaces.
> BasicNamespace-nabble.view.web.template.NabbleNamespace-
> nabble.view.web.template.NodeNamespace&breadcrumbs=
> notify_subscribers%21nabble%3Aemail.naml-instant_emails%
> 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> >
>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/kahadb-index-file-larger-than-journal-file-
> tp4721284p4721315.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: kahadb index file larger than journal file

Tim Bain
Chris,

What will be the behavior for anyone who doesn't delete their index after
upgrading? Will the broker continue to function as-is in that scenario, or
is this a required procedure during an upgrade beyond 5.14.3?

Tim

On Feb 2, 2017 5:02 AM, "Christopher Shannon" <
[hidden email]> wrote:

> Sorry for taking a while to get back to you.
>
> Based on your originally message and after analyzing some index files I
> have actually discovered a bug with KahaDB that causes the index file to
> grow too large when things like out of memory errors occur or any unclean
> shutdown.  On recovery the free pages in the index file are not being
> properly tracked which is leading to excessive index file usage.  So this
> is probably the issue that you are seeing.  I will be fixing this in
> version 5.14.4.  Upon upgrade you will need to delete the old index and
> have it be rebuilt and then going forward it should work as intended.  You
> can track the status of the bug here:
> https://issues.apache.org/jira/browse/AMQ-6590
>
> In terms of the in memory cache, only a fixed number of pages are cached.
> The default is 10 thousand pages and each page is 4KB.  So roughly 40 MB of
> memory will be used at most.  If the number of pages in use grows beyond
> that then the cache will swap pages based on the defined eviction
> strategy.  You can increase the number of pages in memory (which would help
> performance at the cost of eating up memory) by tweaking the indexCacheSize
> value of KahaDB.  See http://activemq.apache.org/kahadb.html for more
> info.
>
> On Tue, Jan 24, 2017 at 11:00 PM, Scorpio <[hidden email]> wrote:
>
> > Christopher:
> >
> >
> > Thank you for your explanation!
> >
> > Hope there would be other proper ways to shrink the index file in future
> > ActiveMQ releases.
> >
> > Once we monitored that our broker is killed by system due to "out of
> > memory".
> >
> > I know that meta cache in memory will be synced into index file at
> > checking points. _*Does this mean there is as much memory as  the
> > db.data file being used?*_
> >
> > Even though our broker is idle,  we still find that it uses up much
> > memory. Could they be relevant associated? Thanks
> >
> >
> > Kind regards,
> >
> > Scorpio
> >
> > On 2017/1/24 20:16, christopher.l.shannon [via ActiveMQ] wrote:
> > > The index grows based off the amount of data it tracks in the journal,
> > > but doesn't shrink.  This means if you had a lot of data in your
> > > journal at some point (several gigabytes, etc) then the index would
> > > have grown in size with enough page files to keep track of all of the
> > > data.  This is usually not an issue because the index size generally
> > > stabilizes at some point relative to the work load of the broker.   If
> > > you really want to shrink it the only way right now is to stop the
> > > broker, delete the index and restart so it does a replay and rebuilds
> > > the index but the size would just grow again as messages flow.
> > >
> > > On Mon, Jan 23, 2017 at 8:06 PM, Scorpio <[hidden email]
> > > </user/SendEmail.jtp?type=node&node=4721300&i=0>> wrote:
> > >
> > > > Hi, All,
> > > >
> > > > In my activemq instance, I find that the kahadb index file is larger
> > > than
> > > > the journal file.
> > > >
> > > > As far as I know, /db.data/ is the meta store, index for the journal
> > > file,
> > > > which should means it is much smaller than the journal file.
> > > However, it
> > > > does not look like that. The index file seems to keep growing while
> the
> > > > journal file is rotated. Do I miss something? Thanks.
> > > >
> > > > Log attached:
> > > > root@host:~/run/apache-activemq-5.13.1/data/kahadb# du * -sh
> > > > 16M     db-16.log
> > > > 1.2G    db.data
> > > > 3.2M    db.redo
> > > > 4.0K    lock
> > > > root@host:~/run/apache-activemq-5.13.1/data/kahadb#
> > > >
> > > > Kind regards,
> > > > Scorpio
> > > >
> > > >
> > > >
> > > > --
> > > > View this message in context:
> > > http://activemq.2283324.n4.nabble.com/kahadb-index-file-
> > larger-than-journal-file-tp4721284.html
> > > > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> > >
> > >
> > > ------------------------------------------------------------
> ------------
> > > If you reply to this email, your message will be added to the
> > > discussion below:
> > > http://activemq.2283324.n4.nabble.com/kahadb-index-file-
> > larger-than-journal-file-tp4721284p4721300.html
> > >
> > > To start a new topic under ActiveMQ - User, email
> > > [hidden email]
> > > To unsubscribe from kahadb index file larger than journal file, click
> > > here
> > > <http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=
> > unsubscribe_by_code&node=4721284&code=bnlAeGlhbmdqaWFiYW8uY29tfDQ3Mj
> > EyODR8MTkzNzMzOTQ5NQ==>.
> > > NAML
> > > <http://activemq.2283324.n4.nabble.com/template/
> > NamlServlet.jtp?macro=macro_viewer&id=instant_html%
> > 21nabble%3Aemail.naml&base=nabble.naml.namespaces.
> > BasicNamespace-nabble.view.web.template.NabbleNamespace-
> > nabble.view.web.template.NodeNamespace&breadcrumbs=
> > notify_subscribers%21nabble%3Aemail.naml-instant_emails%
> > 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> > >
> >
> >
> >
> >
> >
> > --
> > View this message in context: http://activemq.2283324.n4.
> > nabble.com/kahadb-index-file-larger-than-journal-file-
> > tp4721284p4721315.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: kahadb index file larger than journal file

christopher.l.shannon
The broker will continue to work just fine and the index deletion is not
required.  You only need to delete the index file if you want to reclaim
the wasted space otherwise the fix will at least prevent the issue from
getting worse.

On Thu, Feb 2, 2017 at 8:27 AM, Tim Bain <[hidden email]> wrote:

> Chris,
>
> What will be the behavior for anyone who doesn't delete their index after
> upgrading? Will the broker continue to function as-is in that scenario, or
> is this a required procedure during an upgrade beyond 5.14.3?
>
> Tim
>
> On Feb 2, 2017 5:02 AM, "Christopher Shannon" <
> [hidden email]> wrote:
>
> > Sorry for taking a while to get back to you.
> >
> > Based on your originally message and after analyzing some index files I
> > have actually discovered a bug with KahaDB that causes the index file to
> > grow too large when things like out of memory errors occur or any unclean
> > shutdown.  On recovery the free pages in the index file are not being
> > properly tracked which is leading to excessive index file usage.  So this
> > is probably the issue that you are seeing.  I will be fixing this in
> > version 5.14.4.  Upon upgrade you will need to delete the old index and
> > have it be rebuilt and then going forward it should work as intended.
> You
> > can track the status of the bug here:
> > https://issues.apache.org/jira/browse/AMQ-6590
> >
> > In terms of the in memory cache, only a fixed number of pages are cached.
> > The default is 10 thousand pages and each page is 4KB.  So roughly 40 MB
> of
> > memory will be used at most.  If the number of pages in use grows beyond
> > that then the cache will swap pages based on the defined eviction
> > strategy.  You can increase the number of pages in memory (which would
> help
> > performance at the cost of eating up memory) by tweaking the
> indexCacheSize
> > value of KahaDB.  See http://activemq.apache.org/kahadb.html for more
> > info.
> >
> > On Tue, Jan 24, 2017 at 11:00 PM, Scorpio <[hidden email]> wrote:
> >
> > > Christopher:
> > >
> > >
> > > Thank you for your explanation!
> > >
> > > Hope there would be other proper ways to shrink the index file in
> future
> > > ActiveMQ releases.
> > >
> > > Once we monitored that our broker is killed by system due to "out of
> > > memory".
> > >
> > > I know that meta cache in memory will be synced into index file at
> > > checking points. _*Does this mean there is as much memory as  the
> > > db.data file being used?*_
> > >
> > > Even though our broker is idle,  we still find that it uses up much
> > > memory. Could they be relevant associated? Thanks
> > >
> > >
> > > Kind regards,
> > >
> > > Scorpio
> > >
> > > On 2017/1/24 20:16, christopher.l.shannon [via ActiveMQ] wrote:
> > > > The index grows based off the amount of data it tracks in the
> journal,
> > > > but doesn't shrink.  This means if you had a lot of data in your
> > > > journal at some point (several gigabytes, etc) then the index would
> > > > have grown in size with enough page files to keep track of all of the
> > > > data.  This is usually not an issue because the index size generally
> > > > stabilizes at some point relative to the work load of the broker.
>  If
> > > > you really want to shrink it the only way right now is to stop the
> > > > broker, delete the index and restart so it does a replay and rebuilds
> > > > the index but the size would just grow again as messages flow.
> > > >
> > > > On Mon, Jan 23, 2017 at 8:06 PM, Scorpio <[hidden email]
> > > > </user/SendEmail.jtp?type=node&node=4721300&i=0>> wrote:
> > > >
> > > > > Hi, All,
> > > > >
> > > > > In my activemq instance, I find that the kahadb index file is
> larger
> > > > than
> > > > > the journal file.
> > > > >
> > > > > As far as I know, /db.data/ is the meta store, index for the
> journal
> > > > file,
> > > > > which should means it is much smaller than the journal file.
> > > > However, it
> > > > > does not look like that. The index file seems to keep growing while
> > the
> > > > > journal file is rotated. Do I miss something? Thanks.
> > > > >
> > > > > Log attached:
> > > > > root@host:~/run/apache-activemq-5.13.1/data/kahadb# du * -sh
> > > > > 16M     db-16.log
> > > > > 1.2G    db.data
> > > > > 3.2M    db.redo
> > > > > 4.0K    lock
> > > > > root@host:~/run/apache-activemq-5.13.1/data/kahadb#
> > > > >
> > > > > Kind regards,
> > > > > Scorpio
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > View this message in context:
> > > > http://activemq.2283324.n4.nabble.com/kahadb-index-file-
> > > larger-than-journal-file-tp4721284.html
> > > > > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> > > >
> > > >
> > > > ------------------------------------------------------------
> > ------------
> > > > If you reply to this email, your message will be added to the
> > > > discussion below:
> > > > http://activemq.2283324.n4.nabble.com/kahadb-index-file-
> > > larger-than-journal-file-tp4721284p4721300.html
> > > >
> > > > To start a new topic under ActiveMQ - User, email
> > > > [hidden email]
> > > > To unsubscribe from kahadb index file larger than journal file, click
> > > > here
> > > > <http://activemq.2283324.n4.nabble.com/template/
> NamlServlet.jtp?macro=
> > > unsubscribe_by_code&node=4721284&code=bnlAeGlhbmdqaWFiYW8uY29tfDQ3Mj
> > > EyODR8MTkzNzMzOTQ5NQ==>.
> > > > NAML
> > > > <http://activemq.2283324.n4.nabble.com/template/
> > > NamlServlet.jtp?macro=macro_viewer&id=instant_html%
> > > 21nabble%3Aemail.naml&base=nabble.naml.namespaces.
> > > BasicNamespace-nabble.view.web.template.NabbleNamespace-
> > > nabble.view.web.template.NodeNamespace&breadcrumbs=
> > > notify_subscribers%21nabble%3Aemail.naml-instant_emails%
> > > 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> > > >
> > >
> > >
> > >
> > >
> > >
> > > --
> > > View this message in context: http://activemq.2283324.n4.
> > > nabble.com/kahadb-index-file-larger-than-journal-file-
> > > tp4721284p4721315.html
> > > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: kahadb index file larger than journal file

Tim Bain
Thanks for clarifying.

On Feb 2, 2017 6:33 AM, "Christopher Shannon" <
[hidden email]> wrote:

> The broker will continue to work just fine and the index deletion is not
> required.  You only need to delete the index file if you want to reclaim
> the wasted space otherwise the fix will at least prevent the issue from
> getting worse.
>
> On Thu, Feb 2, 2017 at 8:27 AM, Tim Bain <[hidden email]> wrote:
>
> > Chris,
> >
> > What will be the behavior for anyone who doesn't delete their index after
> > upgrading? Will the broker continue to function as-is in that scenario,
> or
> > is this a required procedure during an upgrade beyond 5.14.3?
> >
> > Tim
> >
> > On Feb 2, 2017 5:02 AM, "Christopher Shannon" <
> > [hidden email]> wrote:
> >
> > > Sorry for taking a while to get back to you.
> > >
> > > Based on your originally message and after analyzing some index files I
> > > have actually discovered a bug with KahaDB that causes the index file
> to
> > > grow too large when things like out of memory errors occur or any
> unclean
> > > shutdown.  On recovery the free pages in the index file are not being
> > > properly tracked which is leading to excessive index file usage.  So
> this
> > > is probably the issue that you are seeing.  I will be fixing this in
> > > version 5.14.4.  Upon upgrade you will need to delete the old index and
> > > have it be rebuilt and then going forward it should work as intended.
> > You
> > > can track the status of the bug here:
> > > https://issues.apache.org/jira/browse/AMQ-6590
> > >
> > > In terms of the in memory cache, only a fixed number of pages are
> cached.
> > > The default is 10 thousand pages and each page is 4KB.  So roughly 40
> MB
> > of
> > > memory will be used at most.  If the number of pages in use grows
> beyond
> > > that then the cache will swap pages based on the defined eviction
> > > strategy.  You can increase the number of pages in memory (which would
> > help
> > > performance at the cost of eating up memory) by tweaking the
> > indexCacheSize
> > > value of KahaDB.  See http://activemq.apache.org/kahadb.html for more
> > > info.
> > >
> > > On Tue, Jan 24, 2017 at 11:00 PM, Scorpio <[hidden email]> wrote:
> > >
> > > > Christopher:
> > > >
> > > >
> > > > Thank you for your explanation!
> > > >
> > > > Hope there would be other proper ways to shrink the index file in
> > future
> > > > ActiveMQ releases.
> > > >
> > > > Once we monitored that our broker is killed by system due to "out of
> > > > memory".
> > > >
> > > > I know that meta cache in memory will be synced into index file at
> > > > checking points. _*Does this mean there is as much memory as  the
> > > > db.data file being used?*_
> > > >
> > > > Even though our broker is idle,  we still find that it uses up much
> > > > memory. Could they be relevant associated? Thanks
> > > >
> > > >
> > > > Kind regards,
> > > >
> > > > Scorpio
> > > >
> > > > On 2017/1/24 20:16, christopher.l.shannon [via ActiveMQ] wrote:
> > > > > The index grows based off the amount of data it tracks in the
> > journal,
> > > > > but doesn't shrink.  This means if you had a lot of data in your
> > > > > journal at some point (several gigabytes, etc) then the index would
> > > > > have grown in size with enough page files to keep track of all of
> the
> > > > > data.  This is usually not an issue because the index size
> generally
> > > > > stabilizes at some point relative to the work load of the broker.
> >  If
> > > > > you really want to shrink it the only way right now is to stop the
> > > > > broker, delete the index and restart so it does a replay and
> rebuilds
> > > > > the index but the size would just grow again as messages flow.
> > > > >
> > > > > On Mon, Jan 23, 2017 at 8:06 PM, Scorpio <[hidden email]
> > > > > </user/SendEmail.jtp?type=node&node=4721300&i=0>> wrote:
> > > > >
> > > > > > Hi, All,
> > > > > >
> > > > > > In my activemq instance, I find that the kahadb index file is
> > larger
> > > > > than
> > > > > > the journal file.
> > > > > >
> > > > > > As far as I know, /db.data/ is the meta store, index for the
> > journal
> > > > > file,
> > > > > > which should means it is much smaller than the journal file.
> > > > > However, it
> > > > > > does not look like that. The index file seems to keep growing
> while
> > > the
> > > > > > journal file is rotated. Do I miss something? Thanks.
> > > > > >
> > > > > > Log attached:
> > > > > > root@host:~/run/apache-activemq-5.13.1/data/kahadb# du * -sh
> > > > > > 16M     db-16.log
> > > > > > 1.2G    db.data
> > > > > > 3.2M    db.redo
> > > > > > 4.0K    lock
> > > > > > root@host:~/run/apache-activemq-5.13.1/data/kahadb#
> > > > > >
> > > > > > Kind regards,
> > > > > > Scorpio
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > View this message in context:
> > > > > http://activemq.2283324.n4.nabble.com/kahadb-index-file-
> > > > larger-than-journal-file-tp4721284.html
> > > > > > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> > > > >
> > > > >
> > > > > ------------------------------------------------------------
> > > ------------
> > > > > If you reply to this email, your message will be added to the
> > > > > discussion below:
> > > > > http://activemq.2283324.n4.nabble.com/kahadb-index-file-
> > > > larger-than-journal-file-tp4721284p4721300.html
> > > > >
> > > > > To start a new topic under ActiveMQ - User, email
> > > > > [hidden email]
> > > > > To unsubscribe from kahadb index file larger than journal file,
> click
> > > > > here
> > > > > <http://activemq.2283324.n4.nabble.com/template/
> > NamlServlet.jtp?macro=
> > > > unsubscribe_by_code&node=4721284&code=bnlAeGlhbmdqaWFiYW8uY29tfDQ3Mj
> > > > EyODR8MTkzNzMzOTQ5NQ==>.
> > > > > NAML
> > > > > <http://activemq.2283324.n4.nabble.com/template/
> > > > NamlServlet.jtp?macro=macro_viewer&id=instant_html%
> > > > 21nabble%3Aemail.naml&base=nabble.naml.namespaces.
> > > > BasicNamespace-nabble.view.web.template.NabbleNamespace-
> > > > nabble.view.web.template.NodeNamespace&breadcrumbs=
> > > > notify_subscribers%21nabble%3Aemail.naml-instant_emails%
> > > > 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > --
> > > > View this message in context: http://activemq.2283324.n4.
> > > > nabble.com/kahadb-index-file-larger-than-journal-file-
> > > > tp4721284p4721315.html
> > > > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: kahadb index file larger than journal file

Scorpio
In reply to this post by christopher.l.shannon

Christopher,


Thank you very much!

:), I will track this bug and verify the fix after 5.14.4 released.


Regards,

Scorpio

On 2017/2/2 19:59, christopher.l.shannon [via ActiveMQ] wrote:
Sorry for taking a while to get back to you.

Based on your originally message and after analyzing some index files I
have actually discovered a bug with KahaDB that causes the index file to
grow too large when things like out of memory errors occur or any unclean
shutdown.  On recovery the free pages in the index file are not being
properly tracked which is leading to excessive index file usage.  So this
is probably the issue that you are seeing.  I will be fixing this in
version 5.14.4.  Upon upgrade you will need to delete the old index and
have it be rebuilt and then going forward it should work as intended.  You
can track the status of the bug here:
https://issues.apache.org/jira/browse/AMQ-6590

In terms of the in memory cache, only a fixed number of pages are cached.
The default is 10 thousand pages and each page is 4KB.  So roughly 40 MB of
memory will be used at most.  If the number of pages in use grows beyond
that then the cache will swap pages based on the defined eviction
strategy.  You can increase the number of pages in memory (which would help
performance at the cost of eating up memory) by tweaking the indexCacheSize
value of KahaDB.  See http://activemq.apache.org/kahadb.html for more info.

On Tue, Jan 24, 2017 at 11:00 PM, Scorpio <[hidden email]> wrote:

> Christopher:
>
>
> Thank you for your explanation!
>
> Hope there would be other proper ways to shrink the index file in future
> ActiveMQ releases.
>
> Once we monitored that our broker is killed by system due to "out of
> memory".
>
> I know that meta cache in memory will be synced into index file at
> checking points. _*Does this mean there is as much memory as  the
> db.data file being used?*_
>
> Even though our broker is idle,  we still find that it uses up much
> memory. Could they be relevant associated? Thanks
>
>
> Kind regards,
>
> Scorpio
>
> On 2017/1/24 20:16, christopher.l.shannon [via ActiveMQ] wrote:
> > The index grows based off the amount of data it tracks in the journal,
> > but doesn't shrink.  This means if you had a lot of data in your
> > journal at some point (several gigabytes, etc) then the index would
> > have grown in size with enough page files to keep track of all of the
> > data.  This is usually not an issue because the index size generally
> > stabilizes at some point relative to the work load of the broker.   If
> > you really want to shrink it the only way right now is to stop the
> > broker, delete the index and restart so it does a replay and rebuilds
> > the index but the size would just grow again as messages flow.
> >
> > On Mon, Jan 23, 2017 at 8:06 PM, Scorpio <[hidden email]
> > </user/SendEmail.jtp?type=node&node=4721300&i=0>> wrote:
> >
> > > Hi, All,
> > >
> > > In my activemq instance, I find that the kahadb index file is larger
> > than
> > > the journal file.
> > >
> > > As far as I know, /db.data/ is the meta store, index for the journal
> > file,
> > > which should means it is much smaller than the journal file.
> > However, it
> > > does not look like that. The index file seems to keep growing while the
> > > journal file is rotated. Do I miss something? Thanks.
> > >
> > > Log attached:
> > > [hidden email] du * -sh
> > > 16M     db-16.log
> > > 1.2G    db.data
> > > 3.2M    db.redo
> > > 4.0K    lock
> > > [hidden email]
> > >
> > > Kind regards,
> > > Scorpio
> > >
> > >
> > >
> > > --
> > > View this message in context:
> > http://activemq.2283324.n4.nabble.com/kahadb-index-file-
> larger-than-journal-file-tp4721284.html
> > > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
> >
> > ------------------------------------------------------------------------
> > If you reply to this email, your message will be added to the
> > discussion below:
> > http://activemq.2283324.n4.nabble.com/kahadb-index-file-
> larger-than-journal-file-tp4721284p4721300.html
> >
> > To start a new topic under ActiveMQ - User, email
> > [hidden email]
> > To unsubscribe from kahadb index file larger than journal file, click
> > here
> > <http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=
> unsubscribe_by_code&node=4721284&code=bnlAeGlhbmdqaWFiYW8uY29tfDQ3Mj
> EyODR8MTkzNzMzOTQ5NQ==>.
> > NAML
> > <http://activemq.2283324.n4.nabble.com/template/
> NamlServlet.jtp?macro=macro_viewer&id=instant_html%
> 21nabble%3Aemail.naml&base=nabble.naml.namespaces.
> BasicNamespace-nabble.view.web.template.NabbleNamespace-
> nabble.view.web.template.NodeNamespace&breadcrumbs=
> notify_subscribers%21nabble%3Aemail.naml-instant_emails%
> 21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
> >
>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/kahadb-index-file-larger-than-journal-file-
> tp4721284p4721315.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



If you reply to this email, your message will be added to the discussion below:
http://activemq.2283324.n4.nabble.com/kahadb-index-file-larger-than-journal-file-tp4721284p4721502.html
To start a new topic under ActiveMQ - User, email [hidden email]
To unsubscribe from kahadb index file larger than journal file, click here.
NAML