[DISCUSS] Memory Mapped Journal Type

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

[DISCUSS] Memory Mapped Journal Type

nigro_franz
Hi guys!

In the last few months there are a lot of talks around Persistent Storages  solutions and with the new Intel's 3D XPoints starting to have affordable prices I was thinking that having a memory mapped solution for the Artemis's Journal would be great: a mmap call against a file mounted on a persistent storage will mean direct access to the storage memory while writing, with super-fast flushes/commit on disk, when durability is needed...
Having such kind of journal could be useful as it is right now, making a lot of operations against the journal more straightforward to be implemented (eg the encoding of messages could be directly addressed on the mapped bytes) and it will be future-proof too considering the arrival of this new type of storage...what do you think about it?
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Memory Mapped Journal Type

clebertsuconic
that is really great!

I was recently able to fix libaio up to a point that is performing
really fast on SSDs.. but when we stop having disks and start to have
persistent memory, a new approach will be needed for that kind of
hardware.


I can't wait till we can test the PR you sent against some of that hardware:

https://github.com/apache/activemq-artemis/pull/981


I have been testing your PR. I may need to rebase it and do some stuff
with it, but we should merge your contribution really soon. that's a
really great contribution, and I'm happy to have you doing this kind
of thing.. keep up the good work you're doing.



On Thu, Feb 2, 2017 at 4:12 AM, nigro_franz <[hidden email]> wrote:

> Hi guys!
>
> In the last few months there are a lot of talks around  Persistent Storages
> <http://pmem.io/>    solutions and with the new Intel's 3D XPoints starting
> to have affordable prices I was thinking that having a memory mapped
> solution for the Artemis's Journal would be great: a mmap call against a
> file mounted on a persistent storage will mean direct access to the storage
> memory while writing, with super-fast flushes/commit on disk, when
> durability is needed...
> Having such kind of journal could be useful as it is right now, making a lot
> of operations against the journal more straightforward to be implemented (eg
> the encoding of messages could be directly addressed on the mapped bytes)
> and it will be future-proof too considering the arrival of this new type of
> storage...what do you think about it?
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Memory-Mapped-Journal-Type-tp4721494.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Memory Mapped Journal Type

nigro_franz
Thanks !
I've started here a thread around the Java implementations of that persistent memory, just to understand what will be the direction and I'm too curious about the benchmarks too!
The Journal is assuming the shape of a stand-alone project by itself!!

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Memory Mapped Journal Type

clebertsuconic
I always had requests to spin of the journal as a separate project.


We could create a Collection in top of the journal. it would make it
very useful.

On Thu, Feb 2, 2017 at 1:39 PM, nigro_franz <[hidden email]> wrote:

> Thanks !
> I've started  here
> <https://groups.google.com/forum/#!topic/pmem/2LJFFpoc8gA>   a thread around
> the Java implementations of that persistent memory, just to understand what
> will be the direction and I'm too curious about the benchmarks too!
> The Journal is assuming the shape of a stand-alone project by itself!!
>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Memory-Mapped-Journal-Type-tp4721494p4721536.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Memory Mapped Journal Type

clebertsuconic
On Thu, Feb 2, 2017 at 1:43 PM, Clebert Suconic
<[hidden email]> wrote:
> I always had requests to spin of the journal as a separate project.
>


I mean.. these were always personal requests...
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Memory Mapped Journal Type

clebertsuconic
WOW... if you disable persistence.. (in case you don't actually need
syncs every time).. this thing kicks some (put your favorite word
here)...


on a temporary branch I"m working before putting this on master:



in one screen:

./artemis consumer --message-count 500000



and on another screen:

./artemis producer --message-count 500000



#notice! the producer is still synchronizing, meaning we could disable
sync on producer as well:


journal-type = MAPPED, sync=false:
Producer ActiveMQQueue[TEST], thread=0 Produced: 500000 messages
Producer ActiveMQQueue[TEST], thread=0 Elapsed time in second : 36 s
Producer ActiveMQQueue[TEST], thread=0 Elapsed time in milli second :
36345 milli seconds


You Rock @Francesco!


On Thu, Feb 2, 2017 at 1:44 PM, Clebert Suconic
<[hidden email]> wrote:
> On Thu, Feb 2, 2017 at 1:43 PM, Clebert Suconic
> <[hidden email]> wrote:
>> I always had requests to spin of the journal as a separate project.
>>
>
>
> I mean.. these were always personal requests...



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Memory Mapped Journal Type

clebertsuconic
On Thu, Feb 2, 2017 at 6:51 PM, Clebert Suconic
<[hidden email]> wrote:
> WOW... if you disable persistence.. (in case you don't actually need
> syncs every time).. this thing kicks some (put your favorite word
> here)...


I mean.. if you disable syncs on the journal...


no-syncs on anything else won't run as fast as mapped.

>
>
> on a temporary branch I"m working before putting this on master:
>
>
>
> in one screen:
>
> ./artemis consumer --message-count 500000
>
>
>
> and on another screen:
>
> ./artemis producer --message-count 500000
>
>
>
> #notice! the producer is still synchronizing, meaning we could disable
> sync on producer as well:
>
>
> journal-type = MAPPED, sync=false:
> Producer ActiveMQQueue[TEST], thread=0 Produced: 500000 messages
> Producer ActiveMQQueue[TEST], thread=0 Elapsed time in second : 36 s
> Producer ActiveMQQueue[TEST], thread=0 Elapsed time in milli second :
> 36345 milli seconds
>
>
> You Rock @Francesco!
>
>
> On Thu, Feb 2, 2017 at 1:44 PM, Clebert Suconic
> <[hidden email]> wrote:
>> On Thu, Feb 2, 2017 at 1:43 PM, Clebert Suconic
>> <[hidden email]> wrote:
>>> I always had requests to spin of the journal as a separate project.
>>>
>>
>>
>> I mean.. these were always personal requests...
>
>
>
> --
> Clebert Suconic



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Memory Mapped Journal Type

clebertsuconic
Just as a reference, no-sync with AIO:



Producer ActiveMQQueue[TEST], thread=0 Produced: 500000 messages
Producer ActiveMQQueue[TEST], thread=0 Elapsed time in second : 124 s
Producer ActiveMQQueue[TEST], thread=0 Elapsed time in milli second :
124739 milli seconds


I'm not arguing how safe is to use either, but on my crash tests the
messages weren't lost if I waited some time before killing the server.


IAIO with no-sync probably has a higher level of guarantees of being
actually on the disk in case of crashes.. but if the user don't need
syncs, that's a rock & roll possibility.

On Thu, Feb 2, 2017 at 6:51 PM, Clebert Suconic
<[hidden email]> wrote:

> On Thu, Feb 2, 2017 at 6:51 PM, Clebert Suconic
> <[hidden email]> wrote:
>> WOW... if you disable persistence.. (in case you don't actually need
>> syncs every time).. this thing kicks some (put your favorite word
>> here)...
>
>
> I mean.. if you disable syncs on the journal...
>
>
> no-syncs on anything else won't run as fast as mapped.
>>
>>
>> on a temporary branch I"m working before putting this on master:
>>
>>
>>
>> in one screen:
>>
>> ./artemis consumer --message-count 500000
>>
>>
>>
>> and on another screen:
>>
>> ./artemis producer --message-count 500000
>>
>>
>>
>> #notice! the producer is still synchronizing, meaning we could disable
>> sync on producer as well:
>>
>>
>> journal-type = MAPPED, sync=false:
>> Producer ActiveMQQueue[TEST], thread=0 Produced: 500000 messages
>> Producer ActiveMQQueue[TEST], thread=0 Elapsed time in second : 36 s
>> Producer ActiveMQQueue[TEST], thread=0 Elapsed time in milli second :
>> 36345 milli seconds
>>
>>
>> You Rock @Francesco!
>>
>>
>> On Thu, Feb 2, 2017 at 1:44 PM, Clebert Suconic
>> <[hidden email]> wrote:
>>> On Thu, Feb 2, 2017 at 1:43 PM, Clebert Suconic
>>> <[hidden email]> wrote:
>>>> I always had requests to spin of the journal as a separate project.
>>>>
>>>
>>>
>>> I mean.. these were always personal requests...
>>
>>
>>
>> --
>> Clebert Suconic
>
>
>
> --
> Clebert Suconic



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Memory Mapped Journal Type

clebertsuconic
@Francesco, when  you do sync=false, is that actually writing to the
disk, or it's just putting it in memory until you call force or close
the disk?

if that's the case this is just doing memory and not actually writing
to the storage. Can you confirm what is doing?


Take a look on my branch, mapped at my fork.

On Thu, Feb 2, 2017 at 6:57 PM, Clebert Suconic
<[hidden email]> wrote:

> Just as a reference, no-sync with AIO:
>
>
>
> Producer ActiveMQQueue[TEST], thread=0 Produced: 500000 messages
> Producer ActiveMQQueue[TEST], thread=0 Elapsed time in second : 124 s
> Producer ActiveMQQueue[TEST], thread=0 Elapsed time in milli second :
> 124739 milli seconds
>
>
> I'm not arguing how safe is to use either, but on my crash tests the
> messages weren't lost if I waited some time before killing the server.
>
>
> IAIO with no-sync probably has a higher level of guarantees of being
> actually on the disk in case of crashes.. but if the user don't need
> syncs, that's a rock & roll possibility.
>
> On Thu, Feb 2, 2017 at 6:51 PM, Clebert Suconic
> <[hidden email]> wrote:
>> On Thu, Feb 2, 2017 at 6:51 PM, Clebert Suconic
>> <[hidden email]> wrote:
>>> WOW... if you disable persistence.. (in case you don't actually need
>>> syncs every time).. this thing kicks some (put your favorite word
>>> here)...
>>
>>
>> I mean.. if you disable syncs on the journal...
>>
>>
>> no-syncs on anything else won't run as fast as mapped.
>>>
>>>
>>> on a temporary branch I"m working before putting this on master:
>>>
>>>
>>>
>>> in one screen:
>>>
>>> ./artemis consumer --message-count 500000
>>>
>>>
>>>
>>> and on another screen:
>>>
>>> ./artemis producer --message-count 500000
>>>
>>>
>>>
>>> #notice! the producer is still synchronizing, meaning we could disable
>>> sync on producer as well:
>>>
>>>
>>> journal-type = MAPPED, sync=false:
>>> Producer ActiveMQQueue[TEST], thread=0 Produced: 500000 messages
>>> Producer ActiveMQQueue[TEST], thread=0 Elapsed time in second : 36 s
>>> Producer ActiveMQQueue[TEST], thread=0 Elapsed time in milli second :
>>> 36345 milli seconds
>>>
>>>
>>> You Rock @Francesco!
>>>
>>>
>>> On Thu, Feb 2, 2017 at 1:44 PM, Clebert Suconic
>>> <[hidden email]> wrote:
>>>> On Thu, Feb 2, 2017 at 1:43 PM, Clebert Suconic
>>>> <[hidden email]> wrote:
>>>>> I always had requests to spin of the journal as a separate project.
>>>>>
>>>>
>>>>
>>>> I mean.. these were always personal requests...
>>>
>>>
>>>
>>> --
>>> Clebert Suconic
>>
>>
>>
>> --
>> Clebert Suconic
>
>
>
> --
> Clebert Suconic



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Memory Mapped Journal Type

nigro_franz
This post was updated on .
@clebertsuconic But even if is a personal request, IMHO would be really great to have a separate (open source of course!) project that aims to provide a journal as a first class citizen, maybe with pluggable features like different binary encoders/decoders (SBE, ProtoBuf, FlatBuffers..), replication/compaction/paging and using different stores (redis, SQL, cassandra, elasticsearch, Kafka)...the possibilities could be infinite, wdyt?
when  you do sync=false, is that actually writing to the
disk, or it's just putting it in memory until you call force or close
the disk?
AFAIK it is really writing to the disk, but asynchronously and using the OS configuration for it (on linux: numa balancing, swappiness, dirty_expire_centisecs, dirty_ratio, dirty_writeback_centisecs and many more).
I'm not arguing how safe is to use either, but on my crash tests the
messages weren't lost if I waited some time before killing the server.
Cool! Effectively It's guaranteed to be safe with process failure, indeed AFAIK pretty almost any OS that provides mmap ensures that every writes performed on mapped memory will be persisted on physical storage even if the owner process will crash! The only way to force a mmap file to lose data is with power failures/OS hard-crashes (and sadly Murphy's law said that will happen for sure eheh).
IAIO with no-sync probably has a higher level of guarantees of being
actually on the disk in case of crashes.. but if the user don't need
syncs, that's a rock & roll possibility.
Yes, i think the same, it has a good balance between reliability (on process crashes, but on the cloud it could be enough :P) and speed but AIO is unbeatable from this POV: if the write is reported as performed it is no way on the disk, even on OS failures!
Another 2 points of difference will be memory and cpu usage: the writes you're seeing are flushed by the OS on the disk in background, using the virtual memory directly as a buffer (but on lAIO there is no need to use it!); indeed the CPU usage will be, for high writes rate, a little higher than the lAIO version, due to this async flushes, right?
Take a look on my branch, mapped at my fork
Thanks!For sure!!!
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Memory Mapped Journal Type

clebertsuconic
BTW: you're right.. the journal-sync tool was not actually syncing on NIO

Mapped and NIO are performing similarly. (Mapped being slight faster)


I am also changing AIO to not use O_DIRECT in case of sync=false...

(I don't know why someone would use AIO with sync=false, but in any
case.. i will make the change)

On Fri, Feb 3, 2017 at 3:58 AM, nigro_franz <[hidden email]> wrote:

> @clebertsuconic But even if is a personal request, IMHO would be really great
> to have a separate (open source of course!) project that aims to provide a
> journal as a first class citizen, maybe with pluggable features like
> different binary encoders/decoders (SBE, ProtoBuf, FlatBuffers..),
> replication/compaction/paging and using different stores (redis, SQL,
> cassandra, elasticsearch, Kafka)...the possibilities could be infinite,
> wdyt?
>
>> when  you do sync=false, is that actually writing to the
>> disk, or it's just putting it in memory until you call force or close
>> the disk?
>
> AFAIK it is really writing to the disk, but asynchronously and using the OS
> configuration for it (on linux: numa balancing, swappiness,
> dirty_expire_centisecs, dirty_ratio, dirty_writeback_centisecs and many
> more).
>
>> I'm not arguing how safe is to use either, but on my crash tests the
>> messages weren't lost if I waited some time before killing the server.
>
> Cool! Effectively It's guaranteed to be safe with process failure, indeed
> AFAIK pretty almost any OS that provides mmap ensures that every writes
> performed on mapped memory will be persisted on physical storage even if the
> owner process will crash! The only way to force a mmap file to lose data is
> with power failures/OS hard-crashes (that sadly Murphy's law said that will
> happen for sure eheh).
>
>> IAIO with no-sync probably has a higher level of guarantees of being
>>> actually on the disk in case of crashes.. but if the user don't need
>>> syncs, that's a rock & roll possibility.
>
> Yes, i think the same, it has a good balance between reliability (on process
> crashes, but on the cloud it could be enough :P) and speed but AIO is
> unbeatable from this POV: if the write is reported as performed it is no way
> on the disk, even on OS failures!
> Another 2 points of difference will be memory and cpu usage: the writes
> you're seeing are flushed by the OS on the disk in background, using the
> virtual memory directly as a buffer (but on lAIO there is no need to use
> it!); indeed the CPU usage will be, for high writes rate, a little higher
> than the lAIO version, due to this async flushes, right?
>
>> Take a look on my branch, mapped at my fork
>
> Thanks!For sure!!!
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Memory-Mapped-Journal-Type-tp4721494p4721560.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Memory Mapped Journal Type

nigro_franz
@clebertsuconic The guys of the Persistence library on pmem has answered on the pmem forum !
And about the current status of the MAPPED Journal, there is some need to perform additional tests or to change the README of Artemis to include it?
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Memory Mapped Journal Type

Matt Pavlovich-2
In reply to this post by clebertsuconic
Clebert-

Also, check out the Linux man page:
http://man7.org/linux/man-pages/man2/mmap.2.html

I could see mmap being handy for LAZY_PERSISTENCE and perhaps as a
back-end for mostly persistent indexes?

-Matt

On 2/3/17 11:20 AM, Clebert Suconic wrote:

> BTW: you're right.. the journal-sync tool was not actually syncing on NIO
>
> Mapped and NIO are performing similarly. (Mapped being slight faster)
>
>
> I am also changing AIO to not use O_DIRECT in case of sync=false...
>
> (I don't know why someone would use AIO with sync=false, but in any
> case.. i will make the change)
>
> On Fri, Feb 3, 2017 at 3:58 AM, nigro_franz <[hidden email]> wrote:
>> @clebertsuconic But even if is a personal request, IMHO would be really great
>> to have a separate (open source of course!) project that aims to provide a
>> journal as a first class citizen, maybe with pluggable features like
>> different binary encoders/decoders (SBE, ProtoBuf, FlatBuffers..),
>> replication/compaction/paging and using different stores (redis, SQL,
>> cassandra, elasticsearch, Kafka)...the possibilities could be infinite,
>> wdyt?
>>
>>> when  you do sync=false, is that actually writing to the
>>> disk, or it's just putting it in memory until you call force or close
>>> the disk?
>> AFAIK it is really writing to the disk, but asynchronously and using the OS
>> configuration for it (on linux: numa balancing, swappiness,
>> dirty_expire_centisecs, dirty_ratio, dirty_writeback_centisecs and many
>> more).
>>
>>> I'm not arguing how safe is to use either, but on my crash tests the
>>> messages weren't lost if I waited some time before killing the server.
>> Cool! Effectively It's guaranteed to be safe with process failure, indeed
>> AFAIK pretty almost any OS that provides mmap ensures that every writes
>> performed on mapped memory will be persisted on physical storage even if the
>> owner process will crash! The only way to force a mmap file to lose data is
>> with power failures/OS hard-crashes (that sadly Murphy's law said that will
>> happen for sure eheh).
>>
>>> IAIO with no-sync probably has a higher level of guarantees of being
>>>> actually on the disk in case of crashes.. but if the user don't need
>>>> syncs, that's a rock & roll possibility.
>> Yes, i think the same, it has a good balance between reliability (on process
>> crashes, but on the cloud it could be enough :P) and speed but AIO is
>> unbeatable from this POV: if the write is reported as performed it is no way
>> on the disk, even on OS failures!
>> Another 2 points of difference will be memory and cpu usage: the writes
>> you're seeing are flushed by the OS on the disk in background, using the
>> virtual memory directly as a buffer (but on lAIO there is no need to use
>> it!); indeed the CPU usage will be, for high writes rate, a little higher
>> than the lAIO version, due to this async flushes, right?
>>
>>> Take a look on my branch, mapped at my fork
>> Thanks!For sure!!!
>>
>>
>>
>> --
>> View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Memory-Mapped-Journal-Type-tp4721494p4721560.html
>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>
>

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Memory Mapped Journal Type

nigro_franz
Hi Matt!

yes, it could be a good candidate for that purpose too and AFAIK are used in MongoDb for the indexes!
Indeed the coolest thing is that if you disable data-sync on the broker (choosing to be reliable only to process failures) you could experience writes/reads with speed similar to those of plain memory access, with the bonus of reducing the context switches due to the elimination of any user-space->kernel-space interactions...