Active MQ Shared File System Master Slave with Elastic File System

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Active MQ Shared File System Master Slave with Elastic File System

ishmeister
I'm setting up Active MQ in production with a shared file system master/slave configuration (KahaDB). I've set everything up and mounted the EFS on both EC2 instances.

When I check the disk free stats I get 8 exabytes for the shared file system:

$ df -h
eu-west-1a.***.efs.eu-west-1.amazonaws.com:/  8.0E     0  8.0E   0% /mnt/efs
Unfortunately, ActiveMQ cannot interpret this number:

Store limit is 102400 mb (current store usage is 0 mb). The data directory: /mnt/efs/kahadb only has -8796093022208 mb of usable space - resetting to maximum available disk space: -8796093022207 mb
Store limit is -8796093022207 mb, whilst the max journal file size for the store is: 32 mb, the store will not accept any data when used.

However, this has not prevented ActiveMQ from working as far as I can tell. Is it a problem that "the store will not accept any data when used"?
Reply | Threaded
Open this post in threaded view
|

Re: Active MQ Shared File System Master Slave with Elastic File System

Tim Bain
I'd guess that the way ActiveMQ is "working" is that your persistent
messages, which should have been stored in the persistence store, are
instead being stored in the memory store.  This means they will not survive
a broker restart/failover even though they should, and you'll be subject to
the memory store's limit rather than the persistence store's.  So yes,
messages are getting delivered, but that doesn't mean ActiveMQ is working.

Could you submit a bug in JIRA to capture the fact that we're not handling
a disk size that large?  I suspect the problem is that we're using an int
where we should be using a long.  Till the bug is fixed, I'd suggest
allocating an EBS volume just for the persistence store that's sized to
match whatever limit you're going to put on the KahaDB store.

Tim

On Aug 25, 2016 2:18 AM, "ishmeister" <[hidden email]> wrote:

> I'm setting up Active MQ in production with a shared file system
> master/slave
> configuration (KahaDB). I've set everything up and mounted the EFS on both
> EC2 instances.
>
> When I check the disk free stats I get 8 exabytes for the shared file
> system:
>
> $ df -h
> eu-west-1a.***.efs.eu-west-1.amazonaws.com:/  8.0E     0  8.0E   0%
> /mnt/efs
> Unfortunately, ActiveMQ cannot interpret this number:
>
> Store limit is 102400 mb (current store usage is 0 mb). The data directory:
> /mnt/efs/kahadb only has -8796093022208 mb of usable space - resetting to
> maximum available disk space: -8796093022207 mb
> Store limit is -8796093022207 mb, whilst the max journal file size for the
> store is: 32 mb, the store will not accept any data when used.
>
> However, this has not prevented ActiveMQ from working as far as I can tell.
> Is it a problem that "the store will not accept any data when used"?
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Active-MQ-Shared-File-System-Master-Slave-with-
> Elastic-File-System-tp4715818.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Active MQ Shared File System Master Slave with Elastic File System

ishmeister
Thanks Tim. I'll get this up on JIRA soon.
Reply | Threaded
Open this post in threaded view
|

Re: Active MQ Shared File System Master Slave with Elastic File System

Korny Sietsma
In reply to this post by ishmeister
Interesting - we set up ActiveMQ 5.9 successfully on EFS, I'm guessing
these error messages relate to checks made in later versions of the product.

Note that we gave up on this approach - we found that shared filesystem
master/slave had several problems for us when we tested it with network lag
and splits between the master and the EFS file system.

We have since moved to testing LevelDB + Zookeeper, which has been rather
more reliable in failure scenarios.

- Korny

On 25 August 2016 at 09:16, ishmeister <[hidden email]> wrote:

> I'm setting up Active MQ in production with a shared file system
> master/slave
> configuration (KahaDB). I've set everything up and mounted the EFS on both
> EC2 instances.
>
> When I check the disk free stats I get 8 exabytes for the shared file
> system:
>
> $ df -h
> eu-west-1a.***.efs.eu-west-1.amazonaws.com:/  8.0E     0  8.0E   0%
> /mnt/efs
> Unfortunately, ActiveMQ cannot interpret this number:
>
> Store limit is 102400 mb (current store usage is 0 mb). The data directory:
> /mnt/efs/kahadb only has -8796093022208 mb of usable space - resetting to
> maximum available disk space: -8796093022207 mb
> Store limit is -8796093022207 mb, whilst the max journal file size for the
> store is: 32 mb, the store will not accept any data when used.
>
> However, this has not prevented ActiveMQ from working as far as I can tell.
> Is it a problem that "the store will not accept any data when used"?
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Active-MQ-Shared-File-System-Master-Slave-with-
> Elastic-File-System-tp4715818.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>



--
Kornelis Sietsma  korny at my surname dot com http://korny.info
.fnord { display: none !important; }
Reply | Threaded
Open this post in threaded view
|

Re: Active MQ Shared File System Master Slave with Elastic File System

Tim Bain
Was your EFS as large as the one the OP described?  The perceived problem
was that the code couldn't handle a volume that size, not that it couldn't
handle an EFS volume.

On Sep 2, 2016 5:17 AM, "Korny Sietsma" <[hidden email]> wrote:

> Interesting - we set up ActiveMQ 5.9 successfully on EFS, I'm guessing
> these error messages relate to checks made in later versions of the
> product.
>
> Note that we gave up on this approach - we found that shared filesystem
> master/slave had several problems for us when we tested it with network lag
> and splits between the master and the EFS file system.
>
> We have since moved to testing LevelDB + Zookeeper, which has been rather
> more reliable in failure scenarios.
>
> - Korny
>
> On 25 August 2016 at 09:16, ishmeister <[hidden email]> wrote:
>
> > I'm setting up Active MQ in production with a shared file system
> > master/slave
> > configuration (KahaDB). I've set everything up and mounted the EFS on
> both
> > EC2 instances.
> >
> > When I check the disk free stats I get 8 exabytes for the shared file
> > system:
> >
> > $ df -h
> > eu-west-1a.***.efs.eu-west-1.amazonaws.com:/  8.0E     0  8.0E   0%
> > /mnt/efs
> > Unfortunately, ActiveMQ cannot interpret this number:
> >
> > Store limit is 102400 mb (current store usage is 0 mb). The data
> directory:
> > /mnt/efs/kahadb only has -8796093022208 mb of usable space - resetting to
> > maximum available disk space: -8796093022207 mb
> > Store limit is -8796093022207 mb, whilst the max journal file size for
> the
> > store is: 32 mb, the store will not accept any data when used.
> >
> > However, this has not prevented ActiveMQ from working as far as I can
> tell.
> > Is it a problem that "the store will not accept any data when used"?
> >
> >
> >
> > --
> > View this message in context: http://activemq.2283324.n4.
> > nabble.com/Active-MQ-Shared-File-System-Master-Slave-with-
> > Elastic-File-System-tp4715818.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
>
>
> --
> Kornelis Sietsma  korny at my surname dot com http://korny.info
> .fnord { display: none !important; }
>
Reply | Threaded
Open this post in threaded view
|

Re: Active MQ Shared File System Master Slave with Elastic File System

ishmeister
Tim,

I contacted AWS tech support and their response was this:

"I've checked with our EFS team to see if there is any EFS-related workaround they can offer. Unfortunately, they have confirmed that this is EFS intended behavior and there is no way of mounting a filesystem with "predetermined size" or change the way it is reported by the OS that they or myself know of."

The OS in my case was Ubuntu 14.04.

Korny, thanks for the tips.
Reply | Threaded
Open this post in threaded view
|

Re: Active MQ Shared File System Master Slave with Elastic File System

Korny Sietsma
In reply to this post by Tim Bain
I think all EFS volumes are that size. It's a play-by-usage model - it did
strike me as risky, a runaway process could cost you a lot of money!

On 2 Sep 2016 4:19 p.m., "Tim Bain" <[hidden email]> wrote:

Was your EFS as large as the one the OP described?  The perceived problem
was that the code couldn't handle a volume that size, not that it couldn't
handle an EFS volume.

On Sep 2, 2016 5:17 AM, "Korny Sietsma" <[hidden email]> wrote:

> Interesting - we set up ActiveMQ 5.9 successfully on EFS, I'm guessing
> these error messages relate to checks made in later versions of the
> product.
>
> Note that we gave up on this approach - we found that shared filesystem
> master/slave had several problems for us when we tested it with network
lag

> and splits between the master and the EFS file system.
>
> We have since moved to testing LevelDB + Zookeeper, which has been rather
> more reliable in failure scenarios.
>
> - Korny
>
> On 25 August 2016 at 09:16, ishmeister <[hidden email]> wrote:
>
> > I'm setting up Active MQ in production with a shared file system
> > master/slave
> > configuration (KahaDB). I've set everything up and mounted the EFS on
> both
> > EC2 instances.
> >
> > When I check the disk free stats I get 8 exabytes for the shared file
> > system:
> >
> > $ df -h
> > eu-west-1a.***.efs.eu-west-1.amazonaws.com:/  8.0E     0  8.0E   0%
> > /mnt/efs
> > Unfortunately, ActiveMQ cannot interpret this number:
> >
> > Store limit is 102400 mb (current store usage is 0 mb). The data
> directory:
> > /mnt/efs/kahadb only has -8796093022208 mb of usable space - resetting
to

> > maximum available disk space: -8796093022207 mb
> > Store limit is -8796093022207 mb, whilst the max journal file size for
> the
> > store is: 32 mb, the store will not accept any data when used.
> >
> > However, this has not prevented ActiveMQ from working as far as I can
> tell.
> > Is it a problem that "the store will not accept any data when used"?
> >
> >
> >
> > --
> > View this message in context: http://activemq.2283324.n4.
> > nabble.com/Active-MQ-Shared-File-System-Master-Slave-with-
> > Elastic-File-System-tp4715818.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
>
>
> --
> Kornelis Sietsma  korny at my surname dot com http://korny.info
> .fnord { display: none !important; }
>
Reply | Threaded
Open this post in threaded view
|

Re: Active MQ Shared File System Master Slave with Elastic File System

Justin Reock-2
Be careful of this, in my own load testing the lock state can be lost over EFS leading to a “master-master” scenario under heavy load, which can lead to journal corruption.  Check out the lease locking pluggable storage locker, which is a viable workaround for this problem.

Thanks,
Justin  

On 9/2/16, 2:08 PM, "[hidden email] on behalf of Korny Sietsma" <[hidden email] on behalf of [hidden email]> wrote:

    I think all EFS volumes are that size. It's a play-by-usage model - it did
    strike me as risky, a runaway process could cost you a lot of money!
   
    On 2 Sep 2016 4:19 p.m., "Tim Bain" <[hidden email]> wrote:
   
    Was your EFS as large as the one the OP described?  The perceived problem
    was that the code couldn't handle a volume that size, not that it couldn't
    handle an EFS volume.
   
    On Sep 2, 2016 5:17 AM, "Korny Sietsma" <[hidden email]> wrote:
   
    > Interesting - we set up ActiveMQ 5.9 successfully on EFS, I'm guessing
    > these error messages relate to checks made in later versions of the
    > product.
    >
    > Note that we gave up on this approach - we found that shared filesystem
    > master/slave had several problems for us when we tested it with network
    lag
    > and splits between the master and the EFS file system.
    >
    > We have since moved to testing LevelDB + Zookeeper, which has been rather
    > more reliable in failure scenarios.
    >
    > - Korny
    >
    > On 25 August 2016 at 09:16, ishmeister <[hidden email]> wrote:
    >
    > > I'm setting up Active MQ in production with a shared file system
    > > master/slave
    > > configuration (KahaDB). I've set everything up and mounted the EFS on
    > both
    > > EC2 instances.
    > >
    > > When I check the disk free stats I get 8 exabytes for the shared file
    > > system:
    > >
    > > $ df -h
    > > eu-west-1a.***.efs.eu-west-1.amazonaws.com:/  8.0E     0  8.0E   0%
    > > /mnt/efs
    > > Unfortunately, ActiveMQ cannot interpret this number:
    > >
    > > Store limit is 102400 mb (current store usage is 0 mb). The data
    > directory:
    > > /mnt/efs/kahadb only has -8796093022208 mb of usable space - resetting
    to
    > > maximum available disk space: -8796093022207 mb
    > > Store limit is -8796093022207 mb, whilst the max journal file size for
    > the
    > > store is: 32 mb, the store will not accept any data when used.
    > >
    > > However, this has not prevented ActiveMQ from working as far as I can
    > tell.
    > > Is it a problem that "the store will not accept any data when used"?
    > >
    > >
    > >
    > > --
    > > View this message in context: http://activemq.2283324.n4.
    > > nabble.com/Active-MQ-Shared-File-System-Master-Slave-with-
    > > Elastic-File-System-tp4715818.html
    > > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
    > >
    >
    >
    >
    > --
    > Kornelis Sietsma  korny at my surname dot com http://korny.info
    > .fnord { display: none !important; }
    >
   

Reply | Threaded
Open this post in threaded view
|

Re: Active MQ Shared File System Master Slave with Elastic File System

ishmeister
The issue is perhaps that EFS supports NFS4 but with the following limitations:

http://docs.aws.amazon.com/efs/latest/ug/nfs4-unsupported-features.html

In particular: "All locks in Amazon EFS are advisory, which means that READ and WRITE operations do not check for conflicting locks before the operation is executed."

Whereas the ActiveMQ docs seem to be quite clear on this matter: "Note that the requirements of this failover system are a distributed file system like a SAN for which exclusive file locks work reliably."

http://activemq.apache.org/shared-file-system-master-slave.html

So I guess that effectively rules out using EFS.
Reply | Threaded
Open this post in threaded view
|

Re: Active MQ Shared File System Master Slave with Elastic File System

Tim Bain
It rules out using EFS for your locker implementation, but as Justin said,
pluggable lockers would allow you to put your data into an EFS but use
another mechanism for locking to determine ownership.  But obviously there
would need to be a fix to the code that figures out whether there's enough
space on the volume before that would work.

Tim

On Fri, Sep 2, 2016 at 4:17 PM, ishmeister <[hidden email]> wrote:

> The issue is perhaps that EFS supports NFS4 but with the following
> limitations:
>
> http://docs.aws.amazon.com/efs/latest/ug/nfs4-unsupported-features.html
>
> In particular: "All locks in Amazon EFS are advisory, which means that READ
> and WRITE operations do not check for conflicting locks before the
> operation
> is executed."
>
> Whereas the ActiveMQ docs seem to be quite clear on this matter: "Note that
> the requirements of this failover system are a distributed file system like
> a SAN for which exclusive file locks work reliably."
>
> http://activemq.apache.org/shared-file-system-master-slave.html
>
> So I guess that effectively rules out using EFS.
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Active-MQ-Shared-File-System-Master-Slave-with-
> Elastic-File-System-tp4715818p4716132.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Active MQ Shared File System Master Slave with Elastic File System

Justin Reock-2
Thanks Tim! I recently posted a blog article on this subject, if the user community would like some details about the pluggable storage locker implementation:  http://blog.klocwork.com/open-source/pluggable-storage-lockers-for-activemq/

I hope it’s helpful.

-Justin


On 9/2/16, 11:53 PM, "[hidden email] on behalf of Tim Bain" <[hidden email] on behalf of [hidden email]> wrote:

    It rules out using EFS for your locker implementation, but as Justin said,
    pluggable lockers would allow you to put your data into an EFS but use
    another mechanism for locking to determine ownership.  But obviously there
    would need to be a fix to the code that figures out whether there's enough
    space on the volume before that would work.
   
    Tim
   
    On Fri, Sep 2, 2016 at 4:17 PM, ishmeister <[hidden email]> wrote:
   
    > The issue is perhaps that EFS supports NFS4 but with the following
    > limitations:
    >
    > http://docs.aws.amazon.com/efs/latest/ug/nfs4-unsupported-features.html
    >
    > In particular: "All locks in Amazon EFS are advisory, which means that READ
    > and WRITE operations do not check for conflicting locks before the
    > operation
    > is executed."
    >
    > Whereas the ActiveMQ docs seem to be quite clear on this matter: "Note that
    > the requirements of this failover system are a distributed file system like
    > a SAN for which exclusive file locks work reliably."
    >
    > http://activemq.apache.org/shared-file-system-master-slave.html
    >
    > So I guess that effectively rules out using EFS.
    >
    >
    >
    > --
    > View this message in context: http://activemq.2283324.n4.
    > nabble.com/Active-MQ-Shared-File-System-Master-Slave-with-
    > Elastic-File-System-tp4715818p4716132.html
    > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
    >
   

Reply | Threaded
Open this post in threaded view
|

Re: Active MQ Shared File System Master Slave with Elastic File System

Tim Bain
Nice article.  One suggestion: you should link to the relevant wiki page (
http://activemq.apache.org/pluggable-storage-lockers.html) for anyone who
wants more information.

On Sep 8, 2016 11:34 PM, "Justin Reock" <[hidden email]> wrote:

Thanks Tim! I recently posted a blog article on this subject, if the user
community would like some details about the pluggable storage locker
implementation:  http://blog.klocwork.com/open-source/pluggable-storage-
lockers-for-activemq/

I hope it’s helpful.

-Justin


On 9/2/16, 11:53 PM, "[hidden email] on behalf of Tim Bain" <
[hidden email] on behalf of [hidden email]> wrote:

    It rules out using EFS for your locker implementation, but as Justin
said,
    pluggable lockers would allow you to put your data into an EFS but use
    another mechanism for locking to determine ownership.  But obviously
there
    would need to be a fix to the code that figures out whether there's
enough
    space on the volume before that would work.

    Tim

    On Fri, Sep 2, 2016 at 4:17 PM, ishmeister <[hidden email]>
wrote:

    > The issue is perhaps that EFS supports NFS4 but with the following
    > limitations:
    >
    > http://docs.aws.amazon.com/efs/latest/ug/nfs4-
unsupported-features.html
    >
    > In particular: "All locks in Amazon EFS are advisory, which means
that READ
    > and WRITE operations do not check for conflicting locks before the
    > operation
    > is executed."
    >
    > Whereas the ActiveMQ docs seem to be quite clear on this matter:
"Note that
    > the requirements of this failover system are a distributed file
system like
    > a SAN for which exclusive file locks work reliably."
    >
    > http://activemq.apache.org/shared-file-system-master-slave.html
    >
    > So I guess that effectively rules out using EFS.
    >
    >
    >
    > --
    > View this message in context: http://activemq.2283324.n4.
    > nabble.com/Active-MQ-Shared-File-System-Master-Slave-with-
    > Elastic-File-System-tp4715818p4716132.html
    > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
    >
Reply | Threaded
Open this post in threaded view
|

Re: Active MQ Shared File System Master Slave with Elastic File System

khokhani
This post was updated on .
In reply to this post by Justin Reock-2
Thanks Tim! Thanks Justin! for insight on this topic.
I have setup Amazon EFS as LevelDB persistence store and MySql based locker. All functional and failover testing are positive and message are available and delivered without any failure or loss while switching from master to slave as well as restarting all brokers.
The below concerns stays as activemq is not setting the store usage parameter correctly.

2016-09-17 16:06:00,994 | WARN | Store limit is 102400 mb (current store usage is 100 mb). The data directory: /mnt/efs/amq001/data/leveldb only has -8796093022110 mb of usable space. - resetting to maximum available disk space: -8796093022110 mb | org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain
2016-09-17 16:06:01,016 | WARN | Temporary Store limit is 10240 mb (current store usage is 100 mb). The data directory: /mnt/efs/amq001/data only has -8796093022110 mb of usable space. - resetting to maximum available disk space: -8796093022110 mb | org.apache.activemq.broker.BrokerService | WrapperSimpleAppMain
Is this something can be expected to be fixed?
Reply | Threaded
Open this post in threaded view
|

Re: Active MQ Shared File System Master Slave with Elastic File System

Tim Bain
I hadn't heard of anyone using EFS before you asked about it.  That's not
to say that no one has, just that I've got less idea than you do about what
if any pitfalls you might find along the way.

On Sep 21, 2016 10:33 AM, "khokhani" <[hidden email]> wrote:

> Thanks Tim!  Thanks Justin! for insight on this topic.
>
> I have setup Amazon EFS as LevelDB persistence store and MySql based
> locker.
> All functional and failover testing are positive and message are available
> and delivered without any failure or loss while switching from master to
> slave as well as restarting all brokers.
>
> Is there any other danger using EFS with RDBMS based locker?
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Active-MQ-Shared-File-System-Master-Slave-with-
> Elastic-File-System-tp4715818p4716712.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Active MQ Shared File System Master Slave with Elastic File System

wcrowell
In reply to this post by ishmeister
Did a JIRA issue get created on this?
Reply | Threaded
Open this post in threaded view
|

Re: Active MQ Shared File System Master Slave with Elastic File System

ishmeister
No not yet, but I will do so now.
Reply | Threaded
Open this post in threaded view
|

Re: Active MQ Shared File System Master Slave with Elastic File System

ishmeister
Done: https://issues.apache.org/jira/browse/AMQ-6441

If anyone wants to update the ticket, feel free.
Reply | Threaded
Open this post in threaded view
|

Re: Active MQ Shared File System Master Slave with Elastic File System

wcrowell
Thank you!!!  I was going to create one if it had not been done already.
Reply | Threaded
Open this post in threaded view
|

Re: Active MQ Shared File System Master Slave with Elastic File System

wcrowell
Please take a look at the comment I made on: https://issues.apache.org/jira/browse/AMQ-6441

I may know what the issue is.  The particular client I am working with is running JDK 7.
Reply | Threaded
Open this post in threaded view
|

Re: Active MQ Shared File System Master Slave with Elastic File System

Tim Bain
In reply to this post by wcrowell
Thanks for following up on this to make sure that one got created.

On Sep 23, 2016 9:34 AM, "wcrowell" <[hidden email]> wrote:

> Thank you!!!  I was going to create one if it had not been done already.
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Active-MQ-Shared-File-System-Master-Slave-with-
> Elastic-File-System-tp4715818p4716853.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
12