sun.misc.Unsafe VS io.netty.util.internal.PlatformDependent VS null

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

sun.misc.Unsafe VS io.netty.util.internal.PlatformDependent VS null

nigro_franz
Hi to everyone!

I've developed a memory mapped implementation of SequenceFile to be used inside Artemis's Journal and i've encountered a big question: can i use sun.misc.Unsafe?

I've noticed that between all the dependencies of Artemis there was Netty's libs with its PlatformDependent class and i've finally choose to use it instead of "naked" Unsafe,but i'm not completly satisfied by this solution because PlatformDependent does not provide all the methods of Unsafe (e.g. Unsafe::pageSize).
With such methods there are a lot of improvements and features that can be developed, considering that in the future of Java there will be a place for a safer Unsafe :)
There exists a good policy for all that cases in which something so "unsafe" and "platform dependend" is needed?What the other devs think about it?


With Regards,
Francesco

Reply | Threaded
Open this post in threaded view
|

Re: sun.misc.Unsafe VS io.netty.util.internal.PlatformDependent VS null

clebertsuconic
I would isolate it on a library. Netty does have it but it's well
controlled on a single place I think (easier to maintain it)

I had played with memory mapped years ago and I've decided to use
libaio instead. Did you run any micro benchmarks, have you made any
progress? it seems like a nice idea if you found a breakthrough on
making it performant.. I'm quite interested on it.

On Wed, Jun 8, 2016 at 10:52 AM, nigro_franz <[hidden email]> wrote:

> Hi to everyone!
>
> I've developed a memory mapped implementation of SequenceFile to be used
> inside Artemis's Journal and i've encountered a big question: can i use
> sun.misc.Unsafe?
>
> I've noticed that between all the dependencies of Artemis there was Netty's
> libs with its PlatformDependent class and i've finally choose to use it
> instead of "naked" Unsafe,but i'm not completly satisfied by this solution
> because PlatformDependent does not provide all the methods of Unsafe (e.g.
> Unsafe::pageSize).
> With such methods there are a lot of improvements and features that can be
> developed, considering that in the future of Java there will be a place for
> a safer Unsafe :)
> There exists a good policy for all that cases in which something so "unsafe"
> and "platform dependend" is needed?What the other devs think about it?
>
>
> With Regards,
> Francesco
>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/sun-misc-Unsafe-VS-io-netty-util-internal-PlatformDependent-VS-null-tp4712785.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: sun.misc.Unsafe VS io.netty.util.internal.PlatformDependent VS null

clebertsuconic
Like with Artemis, there's the Artemis-native, where I isolate the use
of the native module.

Maybe you could create a project to just control MapFiles.. it would
be easier to maintain it I think.

On Thu, Jun 9, 2016 at 2:20 PM, Clebert Suconic
<[hidden email]> wrote:

> I would isolate it on a library. Netty does have it but it's well
> controlled on a single place I think (easier to maintain it)
>
> I had played with memory mapped years ago and I've decided to use
> libaio instead. Did you run any micro benchmarks, have you made any
> progress? it seems like a nice idea if you found a breakthrough on
> making it performant.. I'm quite interested on it.
>
> On Wed, Jun 8, 2016 at 10:52 AM, nigro_franz <[hidden email]> wrote:
>> Hi to everyone!
>>
>> I've developed a memory mapped implementation of SequenceFile to be used
>> inside Artemis's Journal and i've encountered a big question: can i use
>> sun.misc.Unsafe?
>>
>> I've noticed that between all the dependencies of Artemis there was Netty's
>> libs with its PlatformDependent class and i've finally choose to use it
>> instead of "naked" Unsafe,but i'm not completly satisfied by this solution
>> because PlatformDependent does not provide all the methods of Unsafe (e.g.
>> Unsafe::pageSize).
>> With such methods there are a lot of improvements and features that can be
>> developed, considering that in the future of Java there will be a place for
>> a safer Unsafe :)
>> There exists a good policy for all that cases in which something so "unsafe"
>> and "platform dependend" is needed?What the other devs think about it?
>>
>>
>> With Regards,
>> Francesco
>>
>>
>>
>>
>>
>> --
>> View this message in context: http://activemq.2283324.n4.nabble.com/sun-misc-Unsafe-VS-io-netty-util-internal-PlatformDependent-VS-null-tp4712785.html
>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
>
>
>
> --
> Clebert Suconic



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: sun.misc.Unsafe VS io.netty.util.internal.PlatformDependent VS null

nigro_franz
Hi!

Thanks for the good advice, i'm happy to let Unsafe improve further the performances of this mapped implementation
Here https://issues.apache.org/jira/browse/ARTEMIS-508 there is the JIRA related to the memory mapped implementation of SequenceFile: i need to squash the (too many) commits into one before doing any PR and obviosly reimplement a "confined" version of PlatformDependent that let Android too to use the mmap implementation (crossing the fingers, ART VM is pure evil).
I've built for now only 2 benchmarks (latency focused), but without using JHM because of the coordinated omission issue, thus i've found JLBH (Java Latency Benchmark Harness) http://www.rationaljava.com/2016/04/a-series-of-posts-on-jlbh-java-latency.html that is best suited for this kind of measurements IMHO.
The results of the benchmarks are pretty good against NIO (if you push data in "burst" of less then 10% of your memory, for a default configured latest ubuntu linux box): about 2 order of magnitude less in the 99 %tile latency measurements.
With these benchs i could measure how much on different configured HW the mmap Sequence File could "keep the pace" with a sustained load of messages, thus producing something pretty similar to a real use case eg without any "math benefits" derived by the backpressure of the OS during the page faulting unavoidable pauses.
I've planned to solve the Page Faulting issue using a pure async implementation, using a single producer single consumer wait-free ringBuffer that feeds an event loop responsible to write the data on the MappedFile but the benchs that i've built (around Journal and SequenceFile) show that the higher layers (above the mmap impl of SequenceFile) slow even this pure-sync implementation, so i'm not sure that the Journal could receive a perceptible benefit from an async one considering that it will only reduce the hiccups when a page fault of few microseconds occour (anyway it could be tuned with Huge Pages and increasing the dirty ratio).
I'm very curious to test it against libaio and i will be very happy if you are interested to try it with me, in order to be objective: my expectations are that with burst of small messages the mmap impl will be faster (crossing JNI has higher cost than call an intrinsified tiny copy with Unsafe, in my experience) , the opposite while the size of the messages grows. If you need other benchs (throughput focused with JHM) or need more info about the implementation i'm ever available