[DISCUSS] Custom Object Serialisation Support

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
63 messages Options
1234
Reply | Threaded
Open this post in threaded view
|

[DISCUSS] Custom Object Serialisation Support

MichaelAndrePearce
Hi All,

Id like to start to discuss a way to customise object serialisation on ActiveMQ.

Currently Message Object serialisation doesn’t allow any pluggability, there are many serialisation formats organisations may want to use instead for different pros and cons.

A good example here would be AVRO with a Schema Registry, in Kafka.

What i was thinking would be to allow a way to inject a SerDes implementation into the connection factory.

If present then this would be used to serialise the Object instead of the default, and subsequently create/convert to a BytesMessage, with a header set to denote it was custom serialised.
The BytesMessage on consume, if custom serde present and header, would then be transformed back into an ObjectMessage, else pass through as a BytesMessage.

This way older consumers would still able to consume the message as a BytesMessage, and like wise can map to other protocols e.g. AMQP. We would also look to promote support in the QPID solution also. But first thought it best to get consensus here.

WDYT??

Cheers
Mike
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Custom Object Serialisation Support

clebertsuconic
that would be a nice addition to the JMS API for sure!!! +1


We would need to get qpid-jms also implementing it.. I want to listen
from @Tim Bish on this subject what he thinks.. but it's a +1 from me.


On Tue, May 30, 2017 at 5:03 PM, Michael André Pearce
<[hidden email]> wrote:

> Hi All,
>
> Id like to start to discuss a way to customise object serialisation on ActiveMQ.
>
> Currently Message Object serialisation doesn’t allow any pluggability, there are many serialisation formats organisations may want to use instead for different pros and cons.
>
> A good example here would be AVRO with a Schema Registry, in Kafka.
>
> What i was thinking would be to allow a way to inject a SerDes implementation into the connection factory.
>
> If present then this would be used to serialise the Object instead of the default, and subsequently create/convert to a BytesMessage, with a header set to denote it was custom serialised.
> The BytesMessage on consume, if custom serde present and header, would then be transformed back into an ObjectMessage, else pass through as a BytesMessage.
>
> This way older consumers would still able to consume the message as a BytesMessage, and like wise can map to other protocols e.g. AMQP. We would also look to promote support in the QPID solution also. But first thought it best to get consensus here.
>
> WDYT??
>
> Cheers
> Mike



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Custom Object Serialisation Support

Matt Pavlovich-2
In reply to this post by MichaelAndrePearce
Michael-

+1 dealing with bytes messages is preferred to object messages and custom object SerDes is super useful.

What do you think about considering a generic client-side plugin approach vs just a payload handler?

> On May 30, 2017, at 4:03 PM, Michael André Pearce <[hidden email]> wrote:
>
> If present then this would be used to serialise the Object instead of the default, and subsequently create/convert to a BytesMessage, with a header set to denote it was custom serialised.

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Custom Object Serialisation Support

MichaelAndrePearce
Hi Matt,

I think having just a SerDe interface for payload handling separate from a general interceptor / client side plugin, is beneficial as then it keeps the logic for serialsation well encapsulated. And also cleaner if we need to do anything (eg sending it as a bytesmessags with a header flag)

 I see this very much like difference in kafka where you have payload serialisation (serdes) and custom-plugin (interceptors)

Also having just a serde makes interface for people to implement and care about much simpler. Eg this is almost the interface I would expect:

byte[] serialize(Destination destination, Object o)

Object deserialize(Destination destination, byte[] bytes)

Nice and simple to implement without having to care about anything else.

Cheers
Mike


Sent from my iPhone

> On 30 May 2017, at 23:18, Matt Pavlovich <[hidden email]> wrote:
>
> Michael-
>
> +1 dealing with bytes messages is preferred to object messages and custom object SerDes is super useful.
>
> What do you think about considering a generic client-side plugin approach vs just a payload handler?
>
>> On May 30, 2017, at 4:03 PM, Michael André Pearce <[hidden email]> wrote:
>>
>> If present then this would be used to serialise the Object instead of the default, and subsequently create/convert to a BytesMessage, with a header set to denote it was custom serialised.
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Custom Object Serialisation Support

MichaelAndrePearce
Or even (as we use address now)

> byte[] serialize(String address, Object o)
>
> Object deserialize(String address, byte[] bytes)



Sent from my iPhone

> On 31 May 2017, at 03:00, Michael André Pearce <[hidden email]> wrote:
>
> Hi Matt,
>
> I think having just a SerDe interface for payload handling separate from a general interceptor / client side plugin, is beneficial as then it keeps the logic for serialsation well encapsulated. And also cleaner if we need to do anything (eg sending it as a bytesmessags with a header flag)
>
> I see this very much like difference in kafka where you have payload serialisation (serdes) and custom-plugin (interceptors)
>
> Also having just a serde makes interface for people to implement and care about much simpler. Eg this is almost the interface I would expect:
>
> byte[] serialize(Destination destination, Object o)
>
> Object deserialize(Destination destination, byte[] bytes)
>
> Nice and simple to implement without having to care about anything else.
>
> Cheers
> Mike
>
>
> Sent from my iPhone
>
>> On 30 May 2017, at 23:18, Matt Pavlovich <[hidden email]> wrote:
>>
>> Michael-
>>
>> +1 dealing with bytes messages is preferred to object messages and custom object SerDes is super useful.
>>
>> What do you think about considering a generic client-side plugin approach vs just a payload handler?
>>
>>> On May 30, 2017, at 4:03 PM, Michael André Pearce <[hidden email]> wrote:
>>>
>>> If present then this would be used to serialise the Object instead of the default, and subsequently create/convert to a BytesMessage, with a header set to denote it was custom serialised.
>>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Custom Object Serialisation Support

nigro_franz
+1
But about the API, I don't know how much freedom we could have here, but maybe could be nice to design something that allows zero garbage approaches from the beginning, like:

//SERIALIZATION SIDE

int expectedSerializedLength(CharSequence address, Object o);
int maxSerializedLength(CharSequence address, Object o);

//return the effective length of the serialized form, it enables reuse of byte[]
int serialize(CharSequence address, Object o, byte[] serializedForm);

default byte[] serialize(String address, Object o) {
 int expectedSerializedLength = expectedSerializedLength(address, o);
 final byte[] serializedForm = new byte[expectedSerializedLength];
 int realSize = serialize(address, o);
 assert realSize == expectedSerializedLength : "We have problems Huston!";
 return serializedForm;
}

//DESERIALIZATION SIDE

long typeId(String address, byte[] bytes, int length);

//enable reuse of byte[], delimiting the length of the serialized form 
//allows objFactory to pool Object if necessary
default Object deserialize(String address, byte[] bytes, int length, LongFunction<? extends Object> objFactory) {
    long typeId = typeId(address, bytes, length);
    final Object instance = objFactory.apply(typeId);
    deserialize(address, bytes, length, instance);
    return instance;
}

void deserialize(String address, byte[] bytes, int length, Object deserializedForm);

LongFunction<? extends Object> objFactory();

default Object deserialize(String address, byte[] bytes) {
    LongFunction<? extends Object> objFactory = objFactory();
    return deserialize(address, bytes, bytes.length, objFactory);
}

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Custom Object Serialisation Support

MichaelAndrePearce
Client side I think this is much less concern than broker side. Also serialisation to from Java objects will create objects there's not much to be done there the nature of the beast.

ByteMessage at best we can put a single byte[], we could try expose the internal netty buffer but then this gets a lot less user friendly, and out the box many serialisation tools output to a java buffer (not netty)or byte[]

I think the level they have in kafka which is essentially I'm proposing seems to be good balance here and is kind of proven as it has picked up very good traction there.

Sent from my iPhone

> On 31 May 2017, at 10:18, nigro_franz <[hidden email]> wrote:
>
> +1
> But about the API, I don't know how much freedom we could have here, but
> maybe could be nice to design something that allows zero garbage approaches
> from the beginning, like:
>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Custom-Object-Serialisation-Support-tp4726741p4726788.html
> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Custom Object Serialisation Support

MichaelAndrePearce
@franz was something meant to come after the "like:" in your response?

@all do we have an area to detail and design out before implementing/coding features that add or amend key apis or functionality? To aid the discussion and design processes.
Like an AMQ Improvement Proposals area?

Sent from my iPhone

> On 31 May 2017, at 13:24, Michael André Pearce <[hidden email]> wrote:
>
> Client side I think this is much less concern than broker side. Also serialisation to from Java objects will create objects there's not much to be done there the nature of the beast.
>
> ByteMessage at best we can put a single byte[], we could try expose the internal netty buffer but then this gets a lot less user friendly, and out the box many serialisation tools output to a java buffer (not netty)or byte[]
>
> I think the level they have in kafka which is essentially I'm proposing seems to be good balance here and is kind of proven as it has picked up very good traction there.
>
> Sent from my iPhone
>
>> On 31 May 2017, at 10:18, nigro_franz <[hidden email]> wrote:
>>
>> +1
>> But about the API, I don't know how much freedom we could have here, but
>> maybe could be nice to design something that allows zero garbage approaches
>> from the beginning, like:
>>
>>
>>
>>
>>
>> --
>> View this message in context: http://activemq.2283324.n4.nabble.com/DISCUSS-Custom-Object-Serialisation-Support-tp4726741p4726788.html
>> Sent from the ActiveMQ - Dev mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Custom Object Serialisation Support

nigro_franz
Yes but I don't know why wasn't shown in the email, only on nabble :O:

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Custom Object Serialisation Support

Matt Pavlovich-2
In reply to this post by MichaelAndrePearce
Hey Mike-

You bring up some good points about keeping the serialization simple and separate. I’m good w/ that approach.

The API you propose doesn’t seem to indicate a place to touch the message headers. Are you suggesting that the Artemis client-side SerDes handler would have a convention where it would set the classname or other in a standard header? BTW— I think that defining a standard convention would be preferred. users could still have access to headers / properties ahead of calling the .send(..).

-Thanks,
Matt

> On May 30, 2017, at 9:00 PM, Michael André Pearce <[hidden email]> wrote:
>
> Hi Matt,
>
> I think having just a SerDe interface for payload handling separate from a general interceptor / client side plugin, is beneficial as then it keeps the logic for serialsation well encapsulated. And also cleaner if we need to do anything (eg sending it as a bytesmessags with a header flag)
>
> I see this very much like difference in kafka where you have payload serialisation (serdes) and custom-plugin (interceptors)
>
> Also having just a serde makes interface for people to implement and care about much simpler. Eg this is almost the interface I would expect:
>
> byte[] serialize(Destination destination, Object o)
>
> Object deserialize(Destination destination, byte[] bytes)
>
> Nice and simple to implement without having to care about anything else.
>
> Cheers
> Mike
>
>
> Sent from my iPhone
>
>> On 30 May 2017, at 23:18, Matt Pavlovich <[hidden email]> wrote:
>>
>> Michael-
>>
>> +1 dealing with bytes messages is preferred to object messages and custom object SerDes is super useful.
>>
>> What do you think about considering a generic client-side plugin approach vs just a payload handler?
>>
>>> On May 30, 2017, at 4:03 PM, Michael André Pearce <[hidden email]> wrote:
>>>
>>> If present then this would be used to serialise the Object instead of the default, and subsequently create/convert to a BytesMessage, with a header set to denote it was custom serialised.
>>

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Custom Object Serialisation Support

MichaelAndrePearce
Hi Matt,

I would suggest that serdes have to provide a mime type, we will have to add method to expose this, and then we set content-type to the provided mime type string.

Maybe simple method:

String mimeType()

As such if on byte message on consume if custom serdes exist and content type header exists and matches that returned from the serdes we deserialise via custom.




Sent from my iPhone

> On 31 May 2017, at 17:58, Matt Pavlovich <[hidden email]> wrote:
>
> Hey Mike-
>
> You bring up some good points about keeping the serialization simple and separate. I’m good w/ that approach.
>
> The API you propose doesn’t seem to indicate a place to touch the message headers. Are you suggesting that the Artemis client-side SerDes handler would have a convention where it would set the classname or other in a standard header? BTW— I think that defining a standard convention would be preferred. users could still have access to headers / properties ahead of calling the .send(..).
>
> -Thanks,
> Matt
>
>> On May 30, 2017, at 9:00 PM, Michael André Pearce <[hidden email]> wrote:
>>
>> Hi Matt,
>>
>> I think having just a SerDe interface for payload handling separate from a general interceptor / client side plugin, is beneficial as then it keeps the logic for serialsation well encapsulated. And also cleaner if we need to do anything (eg sending it as a bytesmessags with a header flag)
>>
>> I see this very much like difference in kafka where you have payload serialisation (serdes) and custom-plugin (interceptors)
>>
>> Also having just a serde makes interface for people to implement and care about much simpler. Eg this is almost the interface I would expect:
>>
>> byte[] serialize(Destination destination, Object o)
>>
>> Object deserialize(Destination destination, byte[] bytes)
>>
>> Nice and simple to implement without having to care about anything else.
>>
>> Cheers
>> Mike
>>
>>
>> Sent from my iPhone
>>
>>> On 30 May 2017, at 23:18, Matt Pavlovich <[hidden email]> wrote:
>>>
>>> Michael-
>>>
>>> +1 dealing with bytes messages is preferred to object messages and custom object SerDes is super useful.
>>>
>>> What do you think about considering a generic client-side plugin approach vs just a payload handler?
>>>
>>>> On May 30, 2017, at 4:03 PM, Michael André Pearce <[hidden email]> wrote:
>>>>
>>>> If present then this would be used to serialise the Object instead of the default, and subsequently create/convert to a BytesMessage, with a header set to denote it was custom serialised.
>>>
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Custom Object Serialisation Support

Matt Pavlovich-2
Mike-

Is there a scenario where you envision the serdes being anything other than a Java POJO/model-style Object?  If so, perhaps we just use the className and optionally include a serialVersionUID/version number header?

-Matt

> On May 31, 2017, at 1:44 PM, Michael André Pearce <[hidden email]> wrote:
>
> Hi Matt,
>
> I would suggest that serdes have to provide a mime type, we will have to add method to expose this, and then we set content-type to the provided mime type string.
>
> Maybe simple method:
>
> String mimeType()
>
> As such if on byte message on consume if custom serdes exist and content type header exists and matches that returned from the serdes we deserialise via custom.
>
>
>
>
> Sent from my iPhone
>
>> On 31 May 2017, at 17:58, Matt Pavlovich <[hidden email]> wrote:
>>
>> Hey Mike-
>>
>> You bring up some good points about keeping the serialization simple and separate. I’m good w/ that approach.
>>
>> The API you propose doesn’t seem to indicate a place to touch the message headers. Are you suggesting that the Artemis client-side SerDes handler would have a convention where it would set the classname or other in a standard header? BTW— I think that defining a standard convention would be preferred. users could still have access to headers / properties ahead of calling the .send(..).
>>
>> -Thanks,
>> Matt
>>
>>> On May 30, 2017, at 9:00 PM, Michael André Pearce <[hidden email]> wrote:
>>>
>>> Hi Matt,
>>>
>>> I think having just a SerDe interface for payload handling separate from a general interceptor / client side plugin, is beneficial as then it keeps the logic for serialsation well encapsulated. And also cleaner if we need to do anything (eg sending it as a bytesmessags with a header flag)
>>>
>>> I see this very much like difference in kafka where you have payload serialisation (serdes) and custom-plugin (interceptors)
>>>
>>> Also having just a serde makes interface for people to implement and care about much simpler. Eg this is almost the interface I would expect:
>>>
>>> byte[] serialize(Destination destination, Object o)
>>>
>>> Object deserialize(Destination destination, byte[] bytes)
>>>
>>> Nice and simple to implement without having to care about anything else.
>>>
>>> Cheers
>>> Mike
>>>
>>>
>>> Sent from my iPhone
>>>
>>>> On 30 May 2017, at 23:18, Matt Pavlovich <[hidden email]> wrote:
>>>>
>>>> Michael-
>>>>
>>>> +1 dealing with bytes messages is preferred to object messages and custom object SerDes is super useful.
>>>>
>>>> What do you think about considering a generic client-side plugin approach vs just a payload handler?
>>>>
>>>>> On May 30, 2017, at 4:03 PM, Michael André Pearce <[hidden email]> wrote:
>>>>>
>>>>> If present then this would be used to serialise the Object instead of the default, and subsequently create/convert to a BytesMessage, with a header set to denote it was custom serialised.
>>>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Custom Object Serialisation Support

clebertsuconic
In reply to this post by MichaelAndrePearce
I'm a bit confused about what API are you talking about...

do you intend for having that as an extension of the JMS Client..
(including qpid).. or as a tooling in which you could convert bytes
and buffers to the intended format.

On Wed, May 31, 2017 at 2:44 PM, Michael André Pearce
<[hidden email]> wrote:

> Hi Matt,
>
> I would suggest that serdes have to provide a mime type, we will have to add method to expose this, and then we set content-type to the provided mime type string.
>
> Maybe simple method:
>
> String mimeType()
>
> As such if on byte message on consume if custom serdes exist and content type header exists and matches that returned from the serdes we deserialise via custom.
>
>
>
>
> Sent from my iPhone
>
>> On 31 May 2017, at 17:58, Matt Pavlovich <[hidden email]> wrote:
>>
>> Hey Mike-
>>
>> You bring up some good points about keeping the serialization simple and separate. I’m good w/ that approach.
>>
>> The API you propose doesn’t seem to indicate a place to touch the message headers. Are you suggesting that the Artemis client-side SerDes handler would have a convention where it would set the classname or other in a standard header? BTW— I think that defining a standard convention would be preferred. users could still have access to headers / properties ahead of calling the .send(..).
>>
>> -Thanks,
>> Matt
>>
>>> On May 30, 2017, at 9:00 PM, Michael André Pearce <[hidden email]> wrote:
>>>
>>> Hi Matt,
>>>
>>> I think having just a SerDe interface for payload handling separate from a general interceptor / client side plugin, is beneficial as then it keeps the logic for serialsation well encapsulated. And also cleaner if we need to do anything (eg sending it as a bytesmessags with a header flag)
>>>
>>> I see this very much like difference in kafka where you have payload serialisation (serdes) and custom-plugin (interceptors)
>>>
>>> Also having just a serde makes interface for people to implement and care about much simpler. Eg this is almost the interface I would expect:
>>>
>>> byte[] serialize(Destination destination, Object o)
>>>
>>> Object deserialize(Destination destination, byte[] bytes)
>>>
>>> Nice and simple to implement without having to care about anything else.
>>>
>>> Cheers
>>> Mike
>>>
>>>
>>> Sent from my iPhone
>>>
>>>> On 30 May 2017, at 23:18, Matt Pavlovich <[hidden email]> wrote:
>>>>
>>>> Michael-
>>>>
>>>> +1 dealing with bytes messages is preferred to object messages and custom object SerDes is super useful.
>>>>
>>>> What do you think about considering a generic client-side plugin approach vs just a payload handler?
>>>>
>>>>> On May 30, 2017, at 4:03 PM, Michael André Pearce <[hidden email]> wrote:
>>>>>
>>>>> If present then this would be used to serialise the Object instead of the default, and subsequently create/convert to a BytesMessage, with a header set to denote it was custom serialised.
>>>>
>>



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Custom Object Serialisation Support

MichaelAndrePearce
@Clebert

As a extension to the JMS Client.

Just to set the real use case

We want to send Avro records, we know that with a schema registry the serialised bytes are of orders of magnitude less and also faster at serialising than letting the java serialisation do it. As well it allows for version projection of the data.

Though obviously we know/aware that others may want to use other serialisation of their objects.

Intent is just to plug / configure the serialisation/deserialisation impl in the connection factory. So on ObjectMessage send if present instead of Java serialisation that occurs currently it uses the custom serdes.
Very similar to kafka producer/consumer.  









Sent from my iPhone

> On 31 May 2017, at 20:02, Clebert Suconic <[hidden email]> wrote:
>
> I'm a bit confused about what API are you talking about...
>
> do you intend for having that as an extension of the JMS Client..
> (including qpid).. or as a tooling in which you could convert bytes
> and buffers to the intended format.
>
> On Wed, May 31, 2017 at 2:44 PM, Michael André Pearce
> <[hidden email]> wrote:
>> Hi Matt,
>>
>> I would suggest that serdes have to provide a mime type, we will have to add method to expose this, and then we set content-type to the provided mime type string.
>>
>> Maybe simple method:
>>
>> String mimeType()
>>
>> As such if on byte message on consume if custom serdes exist and content type header exists and matches that returned from the serdes we deserialise via custom.
>>
>>
>>
>>
>> Sent from my iPhone
>>
>>> On 31 May 2017, at 17:58, Matt Pavlovich <[hidden email]> wrote:
>>>
>>> Hey Mike-
>>>
>>> You bring up some good points about keeping the serialization simple and separate. I’m good w/ that approach.
>>>
>>> The API you propose doesn’t seem to indicate a place to touch the message headers. Are you suggesting that the Artemis client-side SerDes handler would have a convention where it would set the classname or other in a standard header? BTW— I think that defining a standard convention would be preferred. users could still have access to headers / properties ahead of calling the .send(..).
>>>
>>> -Thanks,
>>> Matt
>>>
>>>> On May 30, 2017, at 9:00 PM, Michael André Pearce <[hidden email]> wrote:
>>>>
>>>> Hi Matt,
>>>>
>>>> I think having just a SerDe interface for payload handling separate from a general interceptor / client side plugin, is beneficial as then it keeps the logic for serialsation well encapsulated. And also cleaner if we need to do anything (eg sending it as a bytesmessags with a header flag)
>>>>
>>>> I see this very much like difference in kafka where you have payload serialisation (serdes) and custom-plugin (interceptors)
>>>>
>>>> Also having just a serde makes interface for people to implement and care about much simpler. Eg this is almost the interface I would expect:
>>>>
>>>> byte[] serialize(Destination destination, Object o)
>>>>
>>>> Object deserialize(Destination destination, byte[] bytes)
>>>>
>>>> Nice and simple to implement without having to care about anything else.
>>>>
>>>> Cheers
>>>> Mike
>>>>
>>>>
>>>> Sent from my iPhone
>>>>
>>>>> On 30 May 2017, at 23:18, Matt Pavlovich <[hidden email]> wrote:
>>>>>
>>>>> Michael-
>>>>>
>>>>> +1 dealing with bytes messages is preferred to object messages and custom object SerDes is super useful.
>>>>>
>>>>> What do you think about considering a generic client-side plugin approach vs just a payload handler?
>>>>>
>>>>>> On May 30, 2017, at 4:03 PM, Michael André Pearce <[hidden email]> wrote:
>>>>>>
>>>>>> If present then this would be used to serialise the Object instead of the default, and subsequently create/convert to a BytesMessage, with a header set to denote it was custom serialised.
>>>>>
>>>
>
>
>
> --
> Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Custom Object Serialisation Support

MichaelAndrePearce
In reply to this post by clebertsuconic
@matt

Very much so, case in point would be the Avro use case. It plays very nicely to interop object/data serialisation with c++ clients. Or like wise some of the other serialisation options out there.

This is why I suggest a not Java specific but more generic/widely adopted content-type media type header.

Sent from my iPhone

> On 31 May 2017, at 20:02, Clebert Suconic <[hidden email]> wrote:
>
> I'm a bit confused about what API are you talking about...
>
> do you intend for having that as an extension of the JMS Client..
> (including qpid).. or as a tooling in which you could convert bytes
> and buffers to the intended format.
>
> On Wed, May 31, 2017 at 2:44 PM, Michael André Pearce
> <[hidden email]> wrote:
>> Hi Matt,
>>
>> I would suggest that serdes have to provide a mime type, we will have to add method to expose this, and then we set content-type to the provided mime type string.
>>
>> Maybe simple method:
>>
>> String mimeType()
>>
>> As such if on byte message on consume if custom serdes exist and content type header exists and matches that returned from the serdes we deserialise via custom.
>>
>>
>>
>>
>> Sent from my iPhone
>>
>>> On 31 May 2017, at 17:58, Matt Pavlovich <[hidden email]> wrote:
>>>
>>> Hey Mike-
>>>
>>> You bring up some good points about keeping the serialization simple and separate. I’m good w/ that approach.
>>>
>>> The API you propose doesn’t seem to indicate a place to touch the message headers. Are you suggesting that the Artemis client-side SerDes handler would have a convention where it would set the classname or other in a standard header? BTW— I think that defining a standard convention would be preferred. users could still have access to headers / properties ahead of calling the .send(..).
>>>
>>> -Thanks,
>>> Matt
>>>
>>>> On May 30, 2017, at 9:00 PM, Michael André Pearce <[hidden email]> wrote:
>>>>
>>>> Hi Matt,
>>>>
>>>> I think having just a SerDe interface for payload handling separate from a general interceptor / client side plugin, is beneficial as then it keeps the logic for serialsation well encapsulated. And also cleaner if we need to do anything (eg sending it as a bytesmessags with a header flag)
>>>>
>>>> I see this very much like difference in kafka where you have payload serialisation (serdes) and custom-plugin (interceptors)
>>>>
>>>> Also having just a serde makes interface for people to implement and care about much simpler. Eg this is almost the interface I would expect:
>>>>
>>>> byte[] serialize(Destination destination, Object o)
>>>>
>>>> Object deserialize(Destination destination, byte[] bytes)
>>>>
>>>> Nice and simple to implement without having to care about anything else.
>>>>
>>>> Cheers
>>>> Mike
>>>>
>>>>
>>>> Sent from my iPhone
>>>>
>>>>> On 30 May 2017, at 23:18, Matt Pavlovich <[hidden email]> wrote:
>>>>>
>>>>> Michael-
>>>>>
>>>>> +1 dealing with bytes messages is preferred to object messages and custom object SerDes is super useful.
>>>>>
>>>>> What do you think about considering a generic client-side plugin approach vs just a payload handler?
>>>>>
>>>>>> On May 30, 2017, at 4:03 PM, Michael André Pearce <[hidden email]> wrote:
>>>>>>
>>>>>> If present then this would be used to serialise the Object instead of the default, and subsequently create/convert to a BytesMessage, with a header set to denote it was custom serialised.
>>>>>
>>>
>
>
>
> --
> Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Custom Object Serialisation Support

clebertsuconic
In reply to this post by MichaelAndrePearce
> Intent is just to plug / configure the serialisation/deserialisation impl in the connection factory. So on ObjectMessage send if present instead of Java serialisation that occurs currently it uses the custom serdes.
> Very similar to kafka producer/consumer.


Ok, thanks for clarifying...

That's how I originally understood but I thought the discussion had
diverged into making it differently.

Although an external class that would be reused inside the connection
factory, might be helpful.. So this would also be useful outside of
this context.


That makes a cool demo though. Send an Object Message, later develop a
Bridge.. send Artemis, consume Kafka... nice for a blog or demo.
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Custom Object Serialisation Support

MichaelAndrePearce
In reply to this post by MichaelAndrePearce
To help discussion,

A very very basic implementation just to simulate the idea.
 
https://github.com/michaelandrepearce/activemq-artemis/tree/CustomSerialisation <https://github.com/michaelandrepearce/activemq-artemis/tree/CustomSerialisation>

n.b. doesn’t fully compile is just pseudo impl, nor doesn’t include bits as discussed below like map/change type to a byte message for compatibility, nor media type idea.

Cheers
Mike

> On 31 May 2017, at 21:19, Michael André Pearce <[hidden email]> wrote:
>
> @matt
>
> Very much so, case in point would be the Avro use case. It plays very nicely to interop object/data serialisation with c++ clients. Or like wise some of the other serialisation options out there.
>
> This is why I suggest a not Java specific but more generic/widely adopted content-type media type header.
>
> Sent from my iPhone
>
>> On 31 May 2017, at 20:02, Clebert Suconic <[hidden email]> wrote:
>>
>> I'm a bit confused about what API are you talking about...
>>
>> do you intend for having that as an extension of the JMS Client..
>> (including qpid).. or as a tooling in which you could convert bytes
>> and buffers to the intended format.
>>
>> On Wed, May 31, 2017 at 2:44 PM, Michael André Pearce
>> <[hidden email]> wrote:
>>> Hi Matt,
>>>
>>> I would suggest that serdes have to provide a mime type, we will have to add method to expose this, and then we set content-type to the provided mime type string.
>>>
>>> Maybe simple method:
>>>
>>> String mimeType()
>>>
>>> As such if on byte message on consume if custom serdes exist and content type header exists and matches that returned from the serdes we deserialise via custom.
>>>
>>>
>>>
>>>
>>> Sent from my iPhone
>>>
>>>> On 31 May 2017, at 17:58, Matt Pavlovich <[hidden email]> wrote:
>>>>
>>>> Hey Mike-
>>>>
>>>> You bring up some good points about keeping the serialization simple and separate. I’m good w/ that approach.
>>>>
>>>> The API you propose doesn’t seem to indicate a place to touch the message headers. Are you suggesting that the Artemis client-side SerDes handler would have a convention where it would set the classname or other in a standard header? BTW— I think that defining a standard convention would be preferred. users could still have access to headers / properties ahead of calling the .send(..).
>>>>
>>>> -Thanks,
>>>> Matt
>>>>
>>>>> On May 30, 2017, at 9:00 PM, Michael André Pearce <[hidden email]> wrote:
>>>>>
>>>>> Hi Matt,
>>>>>
>>>>> I think having just a SerDe interface for payload handling separate from a general interceptor / client side plugin, is beneficial as then it keeps the logic for serialsation well encapsulated. And also cleaner if we need to do anything (eg sending it as a bytesmessags with a header flag)
>>>>>
>>>>> I see this very much like difference in kafka where you have payload serialisation (serdes) and custom-plugin (interceptors)
>>>>>
>>>>> Also having just a serde makes interface for people to implement and care about much simpler. Eg this is almost the interface I would expect:
>>>>>
>>>>> byte[] serialize(Destination destination, Object o)
>>>>>
>>>>> Object deserialize(Destination destination, byte[] bytes)
>>>>>
>>>>> Nice and simple to implement without having to care about anything else.
>>>>>
>>>>> Cheers
>>>>> Mike
>>>>>
>>>>>
>>>>> Sent from my iPhone
>>>>>
>>>>>> On 30 May 2017, at 23:18, Matt Pavlovich <[hidden email]> wrote:
>>>>>>
>>>>>> Michael-
>>>>>>
>>>>>> +1 dealing with bytes messages is preferred to object messages and custom object SerDes is super useful.
>>>>>>
>>>>>> What do you think about considering a generic client-side plugin approach vs just a payload handler?
>>>>>>
>>>>>>> On May 30, 2017, at 4:03 PM, Michael André Pearce <[hidden email]> wrote:
>>>>>>>
>>>>>>> If present then this would be used to serialise the Object instead of the default, and subsequently create/convert to a BytesMessage, with a header set to denote it was custom serialised.
>>>>>>
>>>>
>>
>>
>>
>> --
>> Clebert Suconic

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Custom Object Serialisation Support

Timothy Nodine
Should the interface require the underlying class to be Serializable?
One use case might be to provide serialization to classes that aren't
natively serializable.



Michael André Pearce wrote:

> To help discussion,
>
> A very very basic implementation just to simulate the idea.
>  
> https://github.com/michaelandrepearce/activemq-artemis/tree/CustomSerialisation <https://github.com/michaelandrepearce/activemq-artemis/tree/CustomSerialisation>
>
> n.b. doesn’t fully compile is just pseudo impl, nor doesn’t include bits as discussed below like map/change type to a byte message for compatibility, nor media type idea.
>
> Cheers
> Mike
>  

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Custom Object Serialisation Support

MichaelAndrePearce
Jms api dictates class set in object message to be serializable.


> On 31 May 2017, at 22:37, Timothy Nodine <[hidden email]> wrote:
>
> Should the interface require the underlying class to be Serializable? One use case might be to provide serialization to classes that aren't natively serializable.
>
>
>
> Michael André Pearce wrote:
>> To help discussion,
>> A very very basic implementation just to simulate the idea.  https://github.com/michaelandrepearce/activemq-artemis/tree/CustomSerialisation <https://github.com/michaelandrepearce/activemq-artemis/tree/CustomSerialisation>
>>
>> n.b. doesn’t fully compile is just pseudo impl, nor doesn’t include bits as discussed below like map/change type to a byte message for compatibility, nor media type idea.
>>
>> Cheers
>> Mike
>>  
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] Custom Object Serialisation Support

clebertsuconic
On Wed, May 31, 2017 at 10:47 PM Michael André Pearce <
[hidden email]> wrote:

> Jms api dictates class set in object message to be serializable.


We could make an extension. It could be an extra message this actually.

>
>
>
> > On 31 May 2017, at 22:37, Timothy Nodine <[hidden email]> wrote:
> >
> > Should the interface require the underlying class to be Serializable?
> One use case might be to provide serialization to classes that aren't
> natively serializable.
> >
> >
> >
> > Michael André Pearce wrote:
> >> To help discussion,
> >> A very very basic implementation just to simulate the idea.
> https://github.com/michaelandrepearce/activemq-artemis/tree/CustomSerialisation
> <
> https://github.com/michaelandrepearce/activemq-artemis/tree/CustomSerialisation
> >
> >>
> >> n.b. doesn’t fully compile is just pseudo impl, nor doesn’t include
> bits as discussed below like map/change type to a byte message for
> compatibility, nor media type idea.
> >>
> >> Cheers
> >> Mike
> >>
> >
>
--
Clebert Suconic
1234