MQTT retained messages with weird characters

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

MQTT retained messages with weird characters

francesco81
Hello,

I'm trying the Apache Artemis Broker (ver. 1.5.1) for MQTT protocol.

Exactly I'm testing the retained feature for messages of MQTT protocol.

While the broker is up it seems to work well:

a) a client publishes a retained message to a specific topic

b) any client which then subscribes that topic receives the retained message (any time it re-subscribes the topic it receives the last retained message ...perfect!)

But it happens a strange thing when I restart the broker. And I mean I restart the broker without any change on configuration: simply I make a "artemis-service stop" and "artemis-service start". After The broker is up and running, if a client subscribes the previous topic it still receives the retained message, but with weird characters appended to it. For new retained messages published it work well again... but the previous ones (before the broker restart) are sent all with these weird characters. E.g., the following is a retained message published before the broker restart:

"test 25 for a retained message 20170110"

And this is exactly how clients get it before the broker restart.

Instead, after broker restart, the same retained message is received by clients in the following way:

test 25 for a retained message 20170110?
                                        ?6$sys.mqtt.retain..cro.test5?&mqtt.message.retain?mqtt.qos.level


Is this my bad configuration (but I don't know ehere)? Or anything else? Or what?

Someone can help me?


Thanks in advance.


Francesco

________________________________

Ce message, ainsi que tous les fichiers joints à ce message, peuvent contenir des informations sensibles et/ ou confidentielles ne devant pas être divulguées. Si vous n'êtes pas le destinataire de ce message (ou que vous recevez ce message par erreur), nous vous remercions de le notifier immédiatement à son expéditeur, et de détruire ce message. Toute copie, divulgation, modification, utilisation ou diffusion, non autorisée, directe ou indirecte, de tout ou partie de ce message, est strictement interdite.


This e-mail, and any document attached hereby, may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized, direct or indirect, copying, disclosure, distribution or other use of the material or parts thereof is strictly forbidden.
Reply | Threaded
Open this post in threaded view
|

Re: MQTT retained messages with weird characters

clebertsuconic
Do you have a working test you can share?

On Tue, Jan 10, 2017 at 12:07 PM, Francesco PADOVANI
<[hidden email]> wrote:

> Hello,
>
> I'm trying the Apache Artemis Broker (ver. 1.5.1) for MQTT protocol.
>
> Exactly I'm testing the retained feature for messages of MQTT protocol.
>
> While the broker is up it seems to work well:
>
> a) a client publishes a retained message to a specific topic
>
> b) any client which then subscribes that topic receives the retained message (any time it re-subscribes the topic it receives the last retained message ...perfect!)
>
> But it happens a strange thing when I restart the broker. And I mean I restart the broker without any change on configuration: simply I make a "artemis-service stop" and "artemis-service start". After The broker is up and running, if a client subscribes the previous topic it still receives the retained message, but with weird characters appended to it. For new retained messages published it work well again... but the previous ones (before the broker restart) are sent all with these weird characters. E.g., the following is a retained message published before the broker restart:
>
> "test 25 for a retained message 20170110"
>
> And this is exactly how clients get it before the broker restart.
>
> Instead, after broker restart, the same retained message is received by clients in the following way:
>
> test 25 for a retained message 20170110?
>                                         ? 6$sys.mqtt.retain..cro.test5 ?  &mqtt.message.retain ? mqtt.qos.level
>
>
> Is this my bad configuration (but I don't know ehere)? Or anything else? Or what?
>
> Someone can help me?
>
>
> Thanks in advance.
>
>
> Francesco
>
> ________________________________
>
> Ce message, ainsi que tous les fichiers joints à ce message, peuvent contenir des informations sensibles et/ ou confidentielles ne devant pas être divulguées. Si vous n'êtes pas le destinataire de ce message (ou que vous recevez ce message par erreur), nous vous remercions de le notifier immédiatement à son expéditeur, et de détruire ce message. Toute copie, divulgation, modification, utilisation ou diffusion, non autorisée, directe ou indirecte, de tout ou partie de ce message, est strictement interdite.
>
>
> This e-mail, and any document attached hereby, may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized, direct or indirect, copying, disclosure, distribution or other use of the material or parts thereof is strictly forbidden.



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: MQTT retained messages with weird characters

francesco81

Unfortunately no (not yet).

Anyway, it's a clean Artemis installation of version 1.5.1, made by following the user manual instructions at https://activemq.apache.org/artemis/docs/1.5.1/using-server.html.

The installation was done by using a dedicated user and also the artemis java process runs with this dedicated user. Attached you can find my broker.xml configuration file: it's pretty much the same default created during the installation procedure, but for the acceptors (which I've customized for my MQTT purpose) and the addition of parameter "<last-value-queue>true</last-value-queue>" inside the address-setting section (but I tried with and without it and my issue persists).

The only other configuration I changed is the heap size dedicated to the process, by setting "-Xms4096M -Xmx4096M" among the JAVA_ARGS in the artemis.profile file.


The platform where Artemis broker is installed is:


- Machine:

It's an EC2 instance on AWS cloud of type m4.large: 2 vCPU, 8G RAM and SSD storage.


- OS:

CentOS Linux release 7.3.1611 (Core) - x86_64


- JVM:

java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)


- Distro package version of libaio:

Arch        : x86_64
Version     : 0.3.109
Release     : 13.el7


It seems like the problem relates the Persistence (File Journal?). Until the broker keeps the retained messages in ram it works. When it needs to recover the retained messages  from the file system, clients receive them with weird chars.
And I suspect the same issue appears when RAM is full and messages start to be paged... but this case is a bit more difficult to test in a systematic way.


Francesco




From: Clebert Suconic <[hidden email]>
Sent: Tuesday, January 10, 2017 7:43 PM
To: [hidden email]
Subject: Re: MQTT retained messages with weird characters
 
Do you have a working test you can share?

On Tue, Jan 10, 2017 at 12:07 PM, Francesco PADOVANI
<[hidden email]> wrote:
> Hello,
>
> I'm trying the Apache Artemis Broker (ver. 1.5.1) for MQTT protocol.
>
> Exactly I'm testing the retained feature for messages of MQTT protocol.
>
> While the broker is up it seems to work well:
>
> a) a client publishes a retained message to a specific topic
>
> b) any client which then subscribes that topic receives the retained message (any time it re-subscribes the topic it receives the last retained message ...perfect!)
>
> But it happens a strange thing when I restart the broker. And I mean I restart the broker without any change on configuration: simply I make a "artemis-service stop" and "artemis-service start". After The broker is up and running, if a client subscribes the previous topic it still receives the retained message, but with weird characters appended to it. For new retained messages published it work well again... but the previous ones (before the broker restart) are sent all with these weird characters. E.g., the following is a retained message published before the broker restart:
>
> "test 25 for a retained message 20170110"
>
> And this is exactly how clients get it before the broker restart.
>
> Instead, after broker restart, the same retained message is received by clients in the following way:
>
> test 25 for a retained message 20170110?
>                                         ? 6$sys.mqtt.retain..cro.test5 ?  &mqtt.message.retain ? mqtt.qos.level
>
>
> Is this my bad configuration (but I don't know ehere)? Or anything else? Or what?
>
> Someone can help me?
>
>
> Thanks in advance.
>
>
> Francesco
>
> ________________________________
>
> Ce message, ainsi que tous les fichiers joints à ce message, peuvent contenir des informations sensibles et/ ou confidentielles ne devant pas être divulguées. Si vous n'êtes pas le destinataire de ce message (ou que vous recevez ce message par erreur), nous vous remercions de le notifier immédiatement à son expéditeur, et de détruire ce message. Toute copie, divulgation, modification, utilisation ou diffusion, non autorisée, directe ou indirecte, de tout ou partie de ce message, est strictement interdite.
>
>
> This e-mail, and any document attached hereby, may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized, direct or indirect, copying, disclosure, distribution or other use of the material or parts thereof is strictly forbidden.



--
Clebert Suconic


Ce message, ainsi que tous les fichiers joints à ce message, peuvent contenir des informations sensibles et/ ou confidentielles ne devant pas être divulguées. Si vous n'êtes pas le destinataire de ce message (ou que vous recevez ce message par erreur), nous vous remercions de le notifier immédiatement à son expéditeur, et de détruire ce message. Toute copie, divulgation, modification, utilisation ou diffusion, non autorisée, directe ou indirecte, de tout ou partie de ce message, est strictement interdite.


This e-mail, and any document attached hereby, may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized, direct or indirect, copying, disclosure, distribution or other use of the material or parts thereof is strictly forbidden.


broker.xml (9K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: MQTT retained messages with weird characters

clebertsuconic
the MQTT ProtocolManager does some playing with retaining, perhaps
there's something wrong with reloading from journal on retaining. I
was wondering if you could send us a test showing the issue with MQTT
so we could see if there's anything specific to your test while using
MQTT.

On Wed, Jan 11, 2017 at 4:22 AM, Francesco PADOVANI
<[hidden email]> wrote:

> Unfortunately no (not yet).
>
> Anyway, it's a clean Artemis installation of version 1.5.1, made by
> following the user manual instructions at
> https://activemq.apache.org/artemis/docs/1.5.1/using-server.html.
>
> The installation was done by using a dedicated user and also the artemis
> java process runs with this dedicated user. Attached you can find my
> broker.xml configuration file: it's pretty much the same default created
> during the installation procedure, but for the acceptors (which I've
> customized for my MQTT purpose) and the addition of parameter
> "<last-value-queue>true</last-value-queue>" inside the address-setting
> section (but I tried with and without it and my issue persists).
>
> The only other configuration I changed is the heap size dedicated to the
> process, by setting "-Xms4096M -Xmx4096M" among the JAVA_ARGS in the
> artemis.profile file.
>
>
> The platform where Artemis broker is installed is:
>
>
> - Machine:
>
> It's an EC2 instance on AWS cloud of type m4.large: 2 vCPU, 8G RAM and SSD
> storage.
>
>
> - OS:
>
> CentOS Linux release 7.3.1611 (Core) - x86_64
>
>
> - JVM:
>
> java version "1.8.0_111"
> Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
> Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
>
>
> - Distro package version of libaio:
>
> Arch        : x86_64
> Version     : 0.3.109
> Release     : 13.el7
>
>
> It seems like the problem relates the Persistence (File Journal?). Until the
> broker keeps the retained messages in ram it works. When it needs to recover
> the retained messages  from the file system, clients receive them with weird
> chars.
> And I suspect the same issue appears when RAM is full and messages start to
> be paged... but this case is a bit more difficult to test in a systematic
> way.
>
>
> Francesco
>
>
>
> ________________________________
> From: Clebert Suconic <[hidden email]>
> Sent: Tuesday, January 10, 2017 7:43 PM
> To: [hidden email]
> Subject: Re: MQTT retained messages with weird characters
>
> Do you have a working test you can share?
>
> On Tue, Jan 10, 2017 at 12:07 PM, Francesco PADOVANI
> <[hidden email]> wrote:
>> Hello,
>>
>> I'm trying the Apache Artemis Broker (ver. 1.5.1) for MQTT protocol.
>>
>> Exactly I'm testing the retained feature for messages of MQTT protocol.
>>
>> While the broker is up it seems to work well:
>>
>> a) a client publishes a retained message to a specific topic
>>
>> b) any client which then subscribes that topic receives the retained
>> message (any time it re-subscribes the topic it receives the last retained
>> message ...perfect!)
>>
>> But it happens a strange thing when I restart the broker. And I mean I
>> restart the broker without any change on configuration: simply I make a
>> "artemis-service stop" and "artemis-service start". After The broker is up
>> and running, if a client subscribes the previous topic it still receives the
>> retained message, but with weird characters appended to it. For new retained
>> messages published it work well again... but the previous ones (before the
>> broker restart) are sent all with these weird characters. E.g., the
>> following is a retained message published before the broker restart:
>>
>> "test 25 for a retained message 20170110"
>>
>> And this is exactly how clients get it before the broker restart.
>>
>> Instead, after broker restart, the same retained message is received by
>> clients in the following way:
>>
>> test 25 for a retained message 20170110?
>>                                         ? 6$sys.mqtt.retain..cro.test5 ?
>> &mqtt.message.retain ? mqtt.qos.level
>>
>>
>> Is this my bad configuration (but I don't know ehere)? Or anything else?
>> Or what?
>>
>> Someone can help me?
>>
>>
>> Thanks in advance.
>>
>>
>> Francesco
>>
>> ________________________________
>>
>> Ce message, ainsi que tous les fichiers joints à ce message, peuvent
>> contenir des informations sensibles et/ ou confidentielles ne devant pas
>> être divulguées. Si vous n'êtes pas le destinataire de ce message (ou que
>> vous recevez ce message par erreur), nous vous remercions de le notifier
>> immédiatement à son expéditeur, et de détruire ce message. Toute copie,
>> divulgation, modification, utilisation ou diffusion, non autorisée, directe
>> ou indirecte, de tout ou partie de ce message, est strictement interdite.
>>
>>
>> This e-mail, and any document attached hereby, may contain confidential
>> and/or privileged information. If you are not the intended recipient (or
>> have received this e-mail in error) please notify the sender immediately and
>> destroy this e-mail. Any unauthorized, direct or indirect, copying,
>> disclosure, distribution or other use of the material or parts thereof is
>> strictly forbidden.
>
>
>
> --
> Clebert Suconic
>
> ________________________________
>
> Ce message, ainsi que tous les fichiers joints à ce message, peuvent
> contenir des informations sensibles et/ ou confidentielles ne devant pas
> être divulguées. Si vous n'êtes pas le destinataire de ce message (ou que
> vous recevez ce message par erreur), nous vous remercions de le notifier
> immédiatement à son expéditeur, et de détruire ce message. Toute copie,
> divulgation, modification, utilisation ou diffusion, non autorisée, directe
> ou indirecte, de tout ou partie de ce message, est strictement interdite.
>
>
> This e-mail, and any document attached hereby, may contain confidential
> and/or privileged information. If you are not the intended recipient (or
> have received this e-mail in error) please notify the sender immediately and
> destroy this e-mail. Any unauthorized, direct or indirect, copying,
> disclosure, distribution or other use of the material or parts thereof is
> strictly forbidden.



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: MQTT retained messages with weird characters

Martyn Taylor
It looks to me like the full message buffer (not just the body) is getting
returned to the client.  Clebert I think you're right in that we do some
hackery to handle retained messages.  I suspect this is where the error is.

Francesco could you create a JIRA with some reproducer steps and I'll aim
to take a look.

Thanks

On Wed, Jan 11, 2017 at 2:03 PM, Clebert Suconic <[hidden email]>
wrote:

> the MQTT ProtocolManager does some playing with retaining, perhaps
> there's something wrong with reloading from journal on retaining. I
> was wondering if you could send us a test showing the issue with MQTT
> so we could see if there's anything specific to your test while using
> MQTT.
>
> On Wed, Jan 11, 2017 at 4:22 AM, Francesco PADOVANI
> <[hidden email]> wrote:
> > Unfortunately no (not yet).
> >
> > Anyway, it's a clean Artemis installation of version 1.5.1, made by
> > following the user manual instructions at
> > https://activemq.apache.org/artemis/docs/1.5.1/using-server.html.
> >
> > The installation was done by using a dedicated user and also the artemis
> > java process runs with this dedicated user. Attached you can find my
> > broker.xml configuration file: it's pretty much the same default created
> > during the installation procedure, but for the acceptors (which I've
> > customized for my MQTT purpose) and the addition of parameter
> > "<last-value-queue>true</last-value-queue>" inside the address-setting
> > section (but I tried with and without it and my issue persists).
> >
> > The only other configuration I changed is the heap size dedicated to the
> > process, by setting "-Xms4096M -Xmx4096M" among the JAVA_ARGS in the
> > artemis.profile file.
> >
> >
> > The platform where Artemis broker is installed is:
> >
> >
> > - Machine:
> >
> > It's an EC2 instance on AWS cloud of type m4.large: 2 vCPU, 8G RAM and
> SSD
> > storage.
> >
> >
> > - OS:
> >
> > CentOS Linux release 7.3.1611 (Core) - x86_64
> >
> >
> > - JVM:
> >
> > java version "1.8.0_111"
> > Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
> > Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
> >
> >
> > - Distro package version of libaio:
> >
> > Arch        : x86_64
> > Version     : 0.3.109
> > Release     : 13.el7
> >
> >
> > It seems like the problem relates the Persistence (File Journal?). Until
> the
> > broker keeps the retained messages in ram it works. When it needs to
> recover
> > the retained messages  from the file system, clients receive them with
> weird
> > chars.
> > And I suspect the same issue appears when RAM is full and messages start
> to
> > be paged... but this case is a bit more difficult to test in a systematic
> > way.
> >
> >
> > Francesco
> >
> >
> >
> > ________________________________
> > From: Clebert Suconic <[hidden email]>
> > Sent: Tuesday, January 10, 2017 7:43 PM
> > To: [hidden email]
> > Subject: Re: MQTT retained messages with weird characters
> >
> > Do you have a working test you can share?
> >
> > On Tue, Jan 10, 2017 at 12:07 PM, Francesco PADOVANI
> > <[hidden email]> wrote:
> >> Hello,
> >>
> >> I'm trying the Apache Artemis Broker (ver. 1.5.1) for MQTT protocol.
> >>
> >> Exactly I'm testing the retained feature for messages of MQTT protocol.
> >>
> >> While the broker is up it seems to work well:
> >>
> >> a) a client publishes a retained message to a specific topic
> >>
> >> b) any client which then subscribes that topic receives the retained
> >> message (any time it re-subscribes the topic it receives the last
> retained
> >> message ...perfect!)
> >>
> >> But it happens a strange thing when I restart the broker. And I mean I
> >> restart the broker without any change on configuration: simply I make a
> >> "artemis-service stop" and "artemis-service start". After The broker is
> up
> >> and running, if a client subscribes the previous topic it still
> receives the
> >> retained message, but with weird characters appended to it. For new
> retained
> >> messages published it work well again... but the previous ones (before
> the
> >> broker restart) are sent all with these weird characters. E.g., the
> >> following is a retained message published before the broker restart:
> >>
> >> "test 25 for a retained message 20170110"
> >>
> >> And this is exactly how clients get it before the broker restart.
> >>
> >> Instead, after broker restart, the same retained message is received by
> >> clients in the following way:
> >>
> >> test 25 for a retained message 20170110?
> >>                                         ? 6$sys.mqtt.retain..cro.test5 ?
> >> &mqtt.message.retain ? mqtt.qos.level
> >>
> >>
> >> Is this my bad configuration (but I don't know ehere)? Or anything else?
> >> Or what?
> >>
> >> Someone can help me?
> >>
> >>
> >> Thanks in advance.
> >>
> >>
> >> Francesco
> >>
> >> ________________________________
> >>
> >> Ce message, ainsi que tous les fichiers joints à ce message, peuvent
> >> contenir des informations sensibles et/ ou confidentielles ne devant pas
> >> être divulguées. Si vous n'êtes pas le destinataire de ce message (ou
> que
> >> vous recevez ce message par erreur), nous vous remercions de le notifier
> >> immédiatement à son expéditeur, et de détruire ce message. Toute copie,
> >> divulgation, modification, utilisation ou diffusion, non autorisée,
> directe
> >> ou indirecte, de tout ou partie de ce message, est strictement
> interdite.
> >>
> >>
> >> This e-mail, and any document attached hereby, may contain confidential
> >> and/or privileged information. If you are not the intended recipient (or
> >> have received this e-mail in error) please notify the sender
> immediately and
> >> destroy this e-mail. Any unauthorized, direct or indirect, copying,
> >> disclosure, distribution or other use of the material or parts thereof
> is
> >> strictly forbidden.
> >
> >
> >
> > --
> > Clebert Suconic
> >
> > ________________________________
> >
> > Ce message, ainsi que tous les fichiers joints à ce message, peuvent
> > contenir des informations sensibles et/ ou confidentielles ne devant pas
> > être divulguées. Si vous n'êtes pas le destinataire de ce message (ou que
> > vous recevez ce message par erreur), nous vous remercions de le notifier
> > immédiatement à son expéditeur, et de détruire ce message. Toute copie,
> > divulgation, modification, utilisation ou diffusion, non autorisée,
> directe
> > ou indirecte, de tout ou partie de ce message, est strictement interdite.
> >
> >
> > This e-mail, and any document attached hereby, may contain confidential
> > and/or privileged information. If you are not the intended recipient (or
> > have received this e-mail in error) please notify the sender immediately
> and
> > destroy this e-mail. Any unauthorized, direct or indirect, copying,
> > disclosure, distribution or other use of the material or parts thereof is
> > strictly forbidden.
>
>
>
> --
> Clebert Suconic
>
Reply | Threaded
Open this post in threaded view
|

Re: MQTT retained messages with weird characters

francesco81
Hi Martyn,

I've just opened a JIRA related to this issue: it's the tkt ARTEMIS-917 (https://issues.apache.org/jira/browse/ARTEMIS-917). Of course it's currently unassigned: let me know if I have to assign it to someone in particular.


A Thing That may help you (even if I understand you've already figured out where the problem lies): a developer of ours did put the Artemis code under debug and he has noticed that the message is yes, written correctly on the file system, but then it is read in a wrong way. It seems a matter of pointer position, which is kept to the end of metadata instead of the end of message body while reading it. The involved classes should be the followings:


AbstractJournalStorageManager.java

--> this is the class that probably doesn't correctly values the pointer.


MessageImpl.java

--> this (if I understand well) is the class containing the "decode" method used to read the message. A question: do you think the last bold line, below, could be a solution?


public void decode(final ActiveMQBuffer buff) {

      int start = buff.readerIndex();

      endOfBodyPosition = buff.readInt();

      endOfMessagePosition = buff.getInt(endOfBodyPosition - BUFFER_HEADER_SPACE + start);

      int length = endOfMessagePosition - BUFFER_HEADER_SPACE;

      buffer.setIndex(0, BUFFER_HEADER_SPACE);

      buffer.writeBytes(buff, start, length);

      decode();

      buff.readerIndex(start + length);

      buff.writerIndex(start +  endOfBodyPosition);

   }



Thanks a lot  for your help and your patience.


Francesco



________________________________
From: Martyn Taylor <[hidden email]>
Sent: Wednesday, January 11, 2017 3:53:00 PM
To: [hidden email]
Subject: Re: MQTT retained messages with weird characters

It looks to me like the full message buffer (not just the body) is getting
returned to the client.  Clebert I think you're right in that we do some
hackery to handle retained messages.  I suspect this is where the error is.

Francesco could you create a JIRA with some reproducer steps and I'll aim
to take a look.

Thanks

On Wed, Jan 11, 2017 at 2:03 PM, Clebert Suconic <[hidden email]>
wrote:

> the MQTT ProtocolManager does some playing with retaining, perhaps
> there's something wrong with reloading from journal on retaining. I
> was wondering if you could send us a test showing the issue with MQTT
> so we could see if there's anything specific to your test while using
> MQTT.
>
> On Wed, Jan 11, 2017 at 4:22 AM, Francesco PADOVANI
> <[hidden email]> wrote:
> > Unfortunately no (not yet).
> >
> > Anyway, it's a clean Artemis installation of version 1.5.1, made by
> > following the user manual instructions at
> > https://activemq.apache.org/artemis/docs/1.5.1/using-server.html.
> >
> > The installation was done by using a dedicated user and also the artemis
> > java process runs with this dedicated user. Attached you can find my
> > broker.xml configuration file: it's pretty much the same default created
> > during the installation procedure, but for the acceptors (which I've
> > customized for my MQTT purpose) and the addition of parameter
> > "<last-value-queue>true</last-value-queue>" inside the address-setting
> > section (but I tried with and without it and my issue persists).
> >
> > The only other configuration I changed is the heap size dedicated to the
> > process, by setting "-Xms4096M -Xmx4096M" among the JAVA_ARGS in the
> > artemis.profile file.
> >
> >
> > The platform where Artemis broker is installed is:
> >
> >
> > - Machine:
> >
> > It's an EC2 instance on AWS cloud of type m4.large: 2 vCPU, 8G RAM and
> SSD
> > storage.
> >
> >
> > - OS:
> >
> > CentOS Linux release 7.3.1611 (Core) - x86_64
> >
> >
> > - JVM:
> >
> > java version "1.8.0_111"
> > Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
> > Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
> >
> >
> > - Distro package version of libaio:
> >
> > Arch        : x86_64
> > Version     : 0.3.109
> > Release     : 13.el7
> >
> >
> > It seems like the problem relates the Persistence (File Journal?). Until
> the
> > broker keeps the retained messages in ram it works. When it needs to
> recover
> > the retained messages  from the file system, clients receive them with
> weird
> > chars.
> > And I suspect the same issue appears when RAM is full and messages start
> to
> > be paged... but this case is a bit more difficult to test in a systematic
> > way.
> >
> >
> > Francesco
> >
> >
> >
> > ________________________________
> > From: Clebert Suconic <[hidden email]>
> > Sent: Tuesday, January 10, 2017 7:43 PM
> > To: [hidden email]
> > Subject: Re: MQTT retained messages with weird characters
> >
> > Do you have a working test you can share?
> >
> > On Tue, Jan 10, 2017 at 12:07 PM, Francesco PADOVANI
> > <[hidden email]> wrote:
> >> Hello,
> >>
> >> I'm trying the Apache Artemis Broker (ver. 1.5.1) for MQTT protocol.
> >>
> >> Exactly I'm testing the retained feature for messages of MQTT protocol.
> >>
> >> While the broker is up it seems to work well:
> >>
> >> a) a client publishes a retained message to a specific topic
> >>
> >> b) any client which then subscribes that topic receives the retained
> >> message (any time it re-subscribes the topic it receives the last
> retained
> >> message ...perfect!)
> >>
> >> But it happens a strange thing when I restart the broker. And I mean I
> >> restart the broker without any change on configuration: simply I make a
> >> "artemis-service stop" and "artemis-service start". After The broker is
> up
> >> and running, if a client subscribes the previous topic it still
> receives the
> >> retained message, but with weird characters appended to it. For new
> retained
> >> messages published it work well again... but the previous ones (before
> the
> >> broker restart) are sent all with these weird characters. E.g., the
> >> following is a retained message published before the broker restart:
> >>
> >> "test 25 for a retained message 20170110"
> >>
> >> And this is exactly how clients get it before the broker restart.
> >>
> >> Instead, after broker restart, the same retained message is received by
> >> clients in the following way:
> >>
> >> test 25 for a retained message 20170110?
> >>                                         ? 6$sys.mqtt.retain..cro.test5 ?
> >> &mqtt.message.retain ? mqtt.qos.level
> >>
> >>
> >> Is this my bad configuration (but I don't know ehere)? Or anything else?
> >> Or what?
> >>
> >> Someone can help me?
> >>
> >>
> >> Thanks in advance.
> >>
> >>
> >> Francesco
> >>
> >> ________________________________
> >>
> >> Ce message, ainsi que tous les fichiers joints à ce message, peuvent
> >> contenir des informations sensibles et/ ou confidentielles ne devant pas
> >> être divulguées. Si vous n'êtes pas le destinataire de ce message (ou
> que
> >> vous recevez ce message par erreur), nous vous remercions de le notifier
> >> immédiatement à son expéditeur, et de détruire ce message. Toute copie,
> >> divulgation, modification, utilisation ou diffusion, non autorisée,
> directe
> >> ou indirecte, de tout ou partie de ce message, est strictement
> interdite.
> >>
> >>
> >> This e-mail, and any document attached hereby, may contain confidential
> >> and/or privileged information. If you are not the intended recipient (or
> >> have received this e-mail in error) please notify the sender
> immediately and
> >> destroy this e-mail. Any unauthorized, direct or indirect, copying,
> >> disclosure, distribution or other use of the material or parts thereof
> is
> >> strictly forbidden.
> >
> >
> >
> > --
> > Clebert Suconic
> >
> > ________________________________
> >
> > Ce message, ainsi que tous les fichiers joints à ce message, peuvent
> > contenir des informations sensibles et/ ou confidentielles ne devant pas
> > être divulguées. Si vous n'êtes pas le destinataire de ce message (ou que
> > vous recevez ce message par erreur), nous vous remercions de le notifier
> > immédiatement à son expéditeur, et de détruire ce message. Toute copie,
> > divulgation, modification, utilisation ou diffusion, non autorisée,
> directe
> > ou indirecte, de tout ou partie de ce message, est strictement interdite.
> >
> >
> > This e-mail, and any document attached hereby, may contain confidential
> > and/or privileged information. If you are not the intended recipient (or
> > have received this e-mail in error) please notify the sender immediately
> and
> > destroy this e-mail. Any unauthorized, direct or indirect, copying,
> > disclosure, distribution or other use of the material or parts thereof is
> > strictly forbidden.
>
>
>
> --
> Clebert Suconic
>

________________________________

Ce message, ainsi que tous les fichiers joints à ce message, peuvent contenir des informations sensibles et/ ou confidentielles ne devant pas être divulguées. Si vous n'êtes pas le destinataire de ce message (ou que vous recevez ce message par erreur), nous vous remercions de le notifier immédiatement à son expéditeur, et de détruire ce message. Toute copie, divulgation, modification, utilisation ou diffusion, non autorisée, directe ou indirecte, de tout ou partie de ce message, est strictement interdite.


This e-mail, and any document attached hereby, may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized, direct or indirect, copying, disclosure, distribution or other use of the material or parts thereof is strictly forbidden.
Reply | Threaded
Open this post in threaded view
|

Re: MQTT retained messages with weird characters

Martyn Taylor
Hi Francesco,

It appears that there is a bug in the MessageImpl code that is not properly
setting the endOfBodyPosition when the server restarts (reads from journal)
and then there's a subsequent message copy (i.e. retain).  I've enabled
correct pointer setting and added the fix to MQTT.

I'm running the full suite atm to ensure this doesn't affect other areas of
the code.  Should have this fix done shortly.

Thanks for all the info,
Martyn

On Thu, Jan 12, 2017 at 10:13 AM, Francesco PADOVANI <
[hidden email]> wrote:

> Hi Martyn,
>
> I've just opened a JIRA related to this issue: it's the tkt ARTEMIS-917 (
> https://issues.apache.org/jira/browse/ARTEMIS-917). Of course it's
> currently unassigned: let me know if I have to assign it to someone in
> particular.
>
>
> A Thing That may help you (even if I understand you've already figured out
> where the problem lies): a developer of ours did put the Artemis code under
> debug and he has noticed that the message is yes, written correctly on the
> file system, but then it is read in a wrong way. It seems a matter of
> pointer position, which is kept to the end of metadata instead of the end
> of message body while reading it. The involved classes should be the
> followings:
>
>
> AbstractJournalStorageManager.java
>
> --> this is the class that probably doesn't correctly values the pointer.
>
>
> MessageImpl.java
>
> --> this (if I understand well) is the class containing the "decode"
> method used to read the message. A question: do you think the last bold
> line, below, could be a solution?
>
>
> public void decode(final ActiveMQBuffer buff) {
>
>       int start = buff.readerIndex();
>
>       endOfBodyPosition = buff.readInt();
>
>       endOfMessagePosition = buff.getInt(endOfBodyPosition -
> BUFFER_HEADER_SPACE + start);
>
>       int length = endOfMessagePosition - BUFFER_HEADER_SPACE;
>
>       buffer.setIndex(0, BUFFER_HEADER_SPACE);
>
>       buffer.writeBytes(buff, start, length);
>
>       decode();
>
>       buff.readerIndex(start + length);
>
>       buff.writerIndex(start +  endOfBodyPosition);
>
>    }
>
>
>
> Thanks a lot  for your help and your patience.
>
>
> Francesco
>
>
>
> ________________________________
> From: Martyn Taylor <[hidden email]>
> Sent: Wednesday, January 11, 2017 3:53:00 PM
> To: [hidden email]
> Subject: Re: MQTT retained messages with weird characters
>
> It looks to me like the full message buffer (not just the body) is getting
> returned to the client.  Clebert I think you're right in that we do some
> hackery to handle retained messages.  I suspect this is where the error is.
>
> Francesco could you create a JIRA with some reproducer steps and I'll aim
> to take a look.
>
> Thanks
>
> On Wed, Jan 11, 2017 at 2:03 PM, Clebert Suconic <
> [hidden email]>
> wrote:
>
> > the MQTT ProtocolManager does some playing with retaining, perhaps
> > there's something wrong with reloading from journal on retaining. I
> > was wondering if you could send us a test showing the issue with MQTT
> > so we could see if there's anything specific to your test while using
> > MQTT.
> >
> > On Wed, Jan 11, 2017 at 4:22 AM, Francesco PADOVANI
> > <[hidden email]> wrote:
> > > Unfortunately no (not yet).
> > >
> > > Anyway, it's a clean Artemis installation of version 1.5.1, made by
> > > following the user manual instructions at
> > > https://activemq.apache.org/artemis/docs/1.5.1/using-server.html.
> > >
> > > The installation was done by using a dedicated user and also the
> artemis
> > > java process runs with this dedicated user. Attached you can find my
> > > broker.xml configuration file: it's pretty much the same default
> created
> > > during the installation procedure, but for the acceptors (which I've
> > > customized for my MQTT purpose) and the addition of parameter
> > > "<last-value-queue>true</last-value-queue>" inside the address-setting
> > > section (but I tried with and without it and my issue persists).
> > >
> > > The only other configuration I changed is the heap size dedicated to
> the
> > > process, by setting "-Xms4096M -Xmx4096M" among the JAVA_ARGS in the
> > > artemis.profile file.
> > >
> > >
> > > The platform where Artemis broker is installed is:
> > >
> > >
> > > - Machine:
> > >
> > > It's an EC2 instance on AWS cloud of type m4.large: 2 vCPU, 8G RAM and
> > SSD
> > > storage.
> > >
> > >
> > > - OS:
> > >
> > > CentOS Linux release 7.3.1611 (Core) - x86_64
> > >
> > >
> > > - JVM:
> > >
> > > java version "1.8.0_111"
> > > Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
> > > Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
> > >
> > >
> > > - Distro package version of libaio:
> > >
> > > Arch        : x86_64
> > > Version     : 0.3.109
> > > Release     : 13.el7
> > >
> > >
> > > It seems like the problem relates the Persistence (File Journal?).
> Until
> > the
> > > broker keeps the retained messages in ram it works. When it needs to
> > recover
> > > the retained messages  from the file system, clients receive them with
> > weird
> > > chars.
> > > And I suspect the same issue appears when RAM is full and messages
> start
> > to
> > > be paged... but this case is a bit more difficult to test in a
> systematic
> > > way.
> > >
> > >
> > > Francesco
> > >
> > >
> > >
> > > ________________________________
> > > From: Clebert Suconic <[hidden email]>
> > > Sent: Tuesday, January 10, 2017 7:43 PM
> > > To: [hidden email]
> > > Subject: Re: MQTT retained messages with weird characters
> > >
> > > Do you have a working test you can share?
> > >
> > > On Tue, Jan 10, 2017 at 12:07 PM, Francesco PADOVANI
> > > <[hidden email]> wrote:
> > >> Hello,
> > >>
> > >> I'm trying the Apache Artemis Broker (ver. 1.5.1) for MQTT protocol.
> > >>
> > >> Exactly I'm testing the retained feature for messages of MQTT
> protocol.
> > >>
> > >> While the broker is up it seems to work well:
> > >>
> > >> a) a client publishes a retained message to a specific topic
> > >>
> > >> b) any client which then subscribes that topic receives the retained
> > >> message (any time it re-subscribes the topic it receives the last
> > retained
> > >> message ...perfect!)
> > >>
> > >> But it happens a strange thing when I restart the broker. And I mean I
> > >> restart the broker without any change on configuration: simply I make
> a
> > >> "artemis-service stop" and "artemis-service start". After The broker
> is
> > up
> > >> and running, if a client subscribes the previous topic it still
> > receives the
> > >> retained message, but with weird characters appended to it. For new
> > retained
> > >> messages published it work well again... but the previous ones (before
> > the
> > >> broker restart) are sent all with these weird characters. E.g., the
> > >> following is a retained message published before the broker restart:
> > >>
> > >> "test 25 for a retained message 20170110"
> > >>
> > >> And this is exactly how clients get it before the broker restart.
> > >>
> > >> Instead, after broker restart, the same retained message is received
> by
> > >> clients in the following way:
> > >>
> > >> test 25 for a retained message 20170110?
> > >>                                         ?
> 6$sys.mqtt.retain..cro.test5 ?
> > >> &mqtt.message.retain ? mqtt.qos.level
> > >>
> > >>
> > >> Is this my bad configuration (but I don't know ehere)? Or anything
> else?
> > >> Or what?
> > >>
> > >> Someone can help me?
> > >>
> > >>
> > >> Thanks in advance.
> > >>
> > >>
> > >> Francesco
> > >>
> > >> ________________________________
> > >>
> > >> Ce message, ainsi que tous les fichiers joints à ce message, peuvent
> > >> contenir des informations sensibles et/ ou confidentielles ne devant
> pas
> > >> être divulguées. Si vous n'êtes pas le destinataire de ce message (ou
> > que
> > >> vous recevez ce message par erreur), nous vous remercions de le
> notifier
> > >> immédiatement à son expéditeur, et de détruire ce message. Toute
> copie,
> > >> divulgation, modification, utilisation ou diffusion, non autorisée,
> > directe
> > >> ou indirecte, de tout ou partie de ce message, est strictement
> > interdite.
> > >>
> > >>
> > >> This e-mail, and any document attached hereby, may contain
> confidential
> > >> and/or privileged information. If you are not the intended recipient
> (or
> > >> have received this e-mail in error) please notify the sender
> > immediately and
> > >> destroy this e-mail. Any unauthorized, direct or indirect, copying,
> > >> disclosure, distribution or other use of the material or parts thereof
> > is
> > >> strictly forbidden.
> > >
> > >
> > >
> > > --
> > > Clebert Suconic
> > >
> > > ________________________________
> > >
> > > Ce message, ainsi que tous les fichiers joints à ce message, peuvent
> > > contenir des informations sensibles et/ ou confidentielles ne devant
> pas
> > > être divulguées. Si vous n'êtes pas le destinataire de ce message (ou
> que
> > > vous recevez ce message par erreur), nous vous remercions de le
> notifier
> > > immédiatement à son expéditeur, et de détruire ce message. Toute copie,
> > > divulgation, modification, utilisation ou diffusion, non autorisée,
> > directe
> > > ou indirecte, de tout ou partie de ce message, est strictement
> interdite.
> > >
> > >
> > > This e-mail, and any document attached hereby, may contain confidential
> > > and/or privileged information. If you are not the intended recipient
> (or
> > > have received this e-mail in error) please notify the sender
> immediately
> > and
> > > destroy this e-mail. Any unauthorized, direct or indirect, copying,
> > > disclosure, distribution or other use of the material or parts thereof
> is
> > > strictly forbidden.
> >
> >
> >
> > --
> > Clebert Suconic
> >
>
> ________________________________
>
> Ce message, ainsi que tous les fichiers joints à ce message, peuvent
> contenir des informations sensibles et/ ou confidentielles ne devant pas
> être divulguées. Si vous n'êtes pas le destinataire de ce message (ou que
> vous recevez ce message par erreur), nous vous remercions de le notifier
> immédiatement à son expéditeur, et de détruire ce message. Toute copie,
> divulgation, modification, utilisation ou diffusion, non autorisée, directe
> ou indirecte, de tout ou partie de ce message, est strictement interdite.
>
>
> This e-mail, and any document attached hereby, may contain confidential
> and/or privileged information. If you are not the intended recipient (or
> have received this e-mail in error) please notify the sender immediately
> and destroy this e-mail. Any unauthorized, direct or indirect, copying,
> disclosure, distribution or other use of the material or parts thereof is
> strictly forbidden.
>
Reply | Threaded
Open this post in threaded view
|

Re: MQTT retained messages with weird characters

francesco81
Hi Martyn,

great to know it!

Thanks a lot for your work.


Francesco


________________________________
From: Martyn Taylor <[hidden email]>
Sent: Monday, February 6, 2017 4:55:37 PM
To: [hidden email]
Subject: Re: MQTT retained messages with weird characters

Hi Francesco,

It appears that there is a bug in the MessageImpl code that is not properly
setting the endOfBodyPosition when the server restarts (reads from journal)
and then there's a subsequent message copy (i.e. retain).  I've enabled
correct pointer setting and added the fix to MQTT.

I'm running the full suite atm to ensure this doesn't affect other areas of
the code.  Should have this fix done shortly.

Thanks for all the info,
Martyn

On Thu, Jan 12, 2017 at 10:13 AM, Francesco PADOVANI <
[hidden email]> wrote:

> Hi Martyn,
>
> I've just opened a JIRA related to this issue: it's the tkt ARTEMIS-917 (
> https://issues.apache.org/jira/browse/ARTEMIS-917). Of course it's
> currently unassigned: let me know if I have to assign it to someone in
> particular.
>
>
> A Thing That may help you (even if I understand you've already figured out
> where the problem lies): a developer of ours did put the Artemis code under
> debug and he has noticed that the message is yes, written correctly on the
> file system, but then it is read in a wrong way. It seems a matter of
> pointer position, which is kept to the end of metadata instead of the end
> of message body while reading it. The involved classes should be the
> followings:
>
>
> AbstractJournalStorageManager.java
>
> --> this is the class that probably doesn't correctly values the pointer.
>
>
> MessageImpl.java
>
> --> this (if I understand well) is the class containing the "decode"
> method used to read the message. A question: do you think the last bold
> line, below, could be a solution?
>
>
> public void decode(final ActiveMQBuffer buff) {
>
>       int start = buff.readerIndex();
>
>       endOfBodyPosition = buff.readInt();
>
>       endOfMessagePosition = buff.getInt(endOfBodyPosition -
> BUFFER_HEADER_SPACE + start);
>
>       int length = endOfMessagePosition - BUFFER_HEADER_SPACE;
>
>       buffer.setIndex(0, BUFFER_HEADER_SPACE);
>
>       buffer.writeBytes(buff, start, length);
>
>       decode();
>
>       buff.readerIndex(start + length);
>
>       buff.writerIndex(start +  endOfBodyPosition);
>
>    }
>
>
>
> Thanks a lot  for your help and your patience.
>
>
> Francesco
>
>
>
> ________________________________
> From: Martyn Taylor <[hidden email]>
> Sent: Wednesday, January 11, 2017 3:53:00 PM
> To: [hidden email]
> Subject: Re: MQTT retained messages with weird characters
>
> It looks to me like the full message buffer (not just the body) is getting
> returned to the client.  Clebert I think you're right in that we do some
> hackery to handle retained messages.  I suspect this is where the error is.
>
> Francesco could you create a JIRA with some reproducer steps and I'll aim
> to take a look.
>
> Thanks
>
> On Wed, Jan 11, 2017 at 2:03 PM, Clebert Suconic <
> [hidden email]>
> wrote:
>
> > the MQTT ProtocolManager does some playing with retaining, perhaps
> > there's something wrong with reloading from journal on retaining. I
> > was wondering if you could send us a test showing the issue with MQTT
> > so we could see if there's anything specific to your test while using
> > MQTT.
> >
> > On Wed, Jan 11, 2017 at 4:22 AM, Francesco PADOVANI
> > <[hidden email]> wrote:
> > > Unfortunately no (not yet).
> > >
> > > Anyway, it's a clean Artemis installation of version 1.5.1, made by
> > > following the user manual instructions at
> > > https://activemq.apache.org/artemis/docs/1.5.1/using-server.html.
> > >
> > > The installation was done by using a dedicated user and also the
> artemis
> > > java process runs with this dedicated user. Attached you can find my
> > > broker.xml configuration file: it's pretty much the same default
> created
> > > during the installation procedure, but for the acceptors (which I've
> > > customized for my MQTT purpose) and the addition of parameter
> > > "<last-value-queue>true</last-value-queue>" inside the address-setting
> > > section (but I tried with and without it and my issue persists).
> > >
> > > The only other configuration I changed is the heap size dedicated to
> the
> > > process, by setting "-Xms4096M -Xmx4096M" among the JAVA_ARGS in the
> > > artemis.profile file.
> > >
> > >
> > > The platform where Artemis broker is installed is:
> > >
> > >
> > > - Machine:
> > >
> > > It's an EC2 instance on AWS cloud of type m4.large: 2 vCPU, 8G RAM and
> > SSD
> > > storage.
> > >
> > >
> > > - OS:
> > >
> > > CentOS Linux release 7.3.1611 (Core) - x86_64
> > >
> > >
> > > - JVM:
> > >
> > > java version "1.8.0_111"
> > > Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
> > > Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
> > >
> > >
> > > - Distro package version of libaio:
> > >
> > > Arch        : x86_64
> > > Version     : 0.3.109
> > > Release     : 13.el7
> > >
> > >
> > > It seems like the problem relates the Persistence (File Journal?).
> Until
> > the
> > > broker keeps the retained messages in ram it works. When it needs to
> > recover
> > > the retained messages  from the file system, clients receive them with
> > weird
> > > chars.
> > > And I suspect the same issue appears when RAM is full and messages
> start
> > to
> > > be paged... but this case is a bit more difficult to test in a
> systematic
> > > way.
> > >
> > >
> > > Francesco
> > >
> > >
> > >
> > > ________________________________
> > > From: Clebert Suconic <[hidden email]>
> > > Sent: Tuesday, January 10, 2017 7:43 PM
> > > To: [hidden email]
> > > Subject: Re: MQTT retained messages with weird characters
> > >
> > > Do you have a working test you can share?
> > >
> > > On Tue, Jan 10, 2017 at 12:07 PM, Francesco PADOVANI
> > > <[hidden email]> wrote:
> > >> Hello,
> > >>
> > >> I'm trying the Apache Artemis Broker (ver. 1.5.1) for MQTT protocol.
> > >>
> > >> Exactly I'm testing the retained feature for messages of MQTT
> protocol.
> > >>
> > >> While the broker is up it seems to work well:
> > >>
> > >> a) a client publishes a retained message to a specific topic
> > >>
> > >> b) any client which then subscribes that topic receives the retained
> > >> message (any time it re-subscribes the topic it receives the last
> > retained
> > >> message ...perfect!)
> > >>
> > >> But it happens a strange thing when I restart the broker. And I mean I
> > >> restart the broker without any change on configuration: simply I make
> a
> > >> "artemis-service stop" and "artemis-service start". After The broker
> is
> > up
> > >> and running, if a client subscribes the previous topic it still
> > receives the
> > >> retained message, but with weird characters appended to it. For new
> > retained
> > >> messages published it work well again... but the previous ones (before
> > the
> > >> broker restart) are sent all with these weird characters. E.g., the
> > >> following is a retained message published before the broker restart:
> > >>
> > >> "test 25 for a retained message 20170110"
> > >>
> > >> And this is exactly how clients get it before the broker restart.
> > >>
> > >> Instead, after broker restart, the same retained message is received
> by
> > >> clients in the following way:
> > >>
> > >> test 25 for a retained message 20170110?
> > >>                                         ?
> 6$sys.mqtt.retain..cro.test5 ?
> > >> &mqtt.message.retain ? mqtt.qos.level
> > >>
> > >>
> > >> Is this my bad configuration (but I don't know ehere)? Or anything
> else?
> > >> Or what?
> > >>
> > >> Someone can help me?
> > >>
> > >>
> > >> Thanks in advance.
> > >>
> > >>
> > >> Francesco
> > >>
> > >> ________________________________
> > >>
> > >> Ce message, ainsi que tous les fichiers joints à ce message, peuvent
> > >> contenir des informations sensibles et/ ou confidentielles ne devant
> pas
> > >> être divulguées. Si vous n'êtes pas le destinataire de ce message (ou
> > que
> > >> vous recevez ce message par erreur), nous vous remercions de le
> notifier
> > >> immédiatement à son expéditeur, et de détruire ce message. Toute
> copie,
> > >> divulgation, modification, utilisation ou diffusion, non autorisée,
> > directe
> > >> ou indirecte, de tout ou partie de ce message, est strictement
> > interdite.
> > >>
> > >>
> > >> This e-mail, and any document attached hereby, may contain
> confidential
> > >> and/or privileged information. If you are not the intended recipient
> (or
> > >> have received this e-mail in error) please notify the sender
> > immediately and
> > >> destroy this e-mail. Any unauthorized, direct or indirect, copying,
> > >> disclosure, distribution or other use of the material or parts thereof
> > is
> > >> strictly forbidden.
> > >
> > >
> > >
> > > --
> > > Clebert Suconic
> > >
> > > ________________________________
> > >
> > > Ce message, ainsi que tous les fichiers joints à ce message, peuvent
> > > contenir des informations sensibles et/ ou confidentielles ne devant
> pas
> > > être divulguées. Si vous n'êtes pas le destinataire de ce message (ou
> que
> > > vous recevez ce message par erreur), nous vous remercions de le
> notifier
> > > immédiatement à son expéditeur, et de détruire ce message. Toute copie,
> > > divulgation, modification, utilisation ou diffusion, non autorisée,
> > directe
> > > ou indirecte, de tout ou partie de ce message, est strictement
> interdite.
> > >
> > >
> > > This e-mail, and any document attached hereby, may contain confidential
> > > and/or privileged information. If you are not the intended recipient
> (or
> > > have received this e-mail in error) please notify the sender
> immediately
> > and
> > > destroy this e-mail. Any unauthorized, direct or indirect, copying,
> > > disclosure, distribution or other use of the material or parts thereof
> is
> > > strictly forbidden.
> >
> >
> >
> > --
> > Clebert Suconic
> >
>
> ________________________________
>
> Ce message, ainsi que tous les fichiers joints à ce message, peuvent
> contenir des informations sensibles et/ ou confidentielles ne devant pas
> être divulguées. Si vous n'êtes pas le destinataire de ce message (ou que
> vous recevez ce message par erreur), nous vous remercions de le notifier
> immédiatement à son expéditeur, et de détruire ce message. Toute copie,
> divulgation, modification, utilisation ou diffusion, non autorisée, directe
> ou indirecte, de tout ou partie de ce message, est strictement interdite.
>
>
> This e-mail, and any document attached hereby, may contain confidential
> and/or privileged information. If you are not the intended recipient (or
> have received this e-mail in error) please notify the sender immediately
> and destroy this e-mail. Any unauthorized, direct or indirect, copying,
> disclosure, distribution or other use of the material or parts thereof is
> strictly forbidden.
>

________________________________

Ce message, ainsi que tous les fichiers joints à ce message, peuvent contenir des informations sensibles et/ ou confidentielles ne devant pas être divulguées. Si vous n'êtes pas le destinataire de ce message (ou que vous recevez ce message par erreur), nous vous remercions de le notifier immédiatement à son expéditeur, et de détruire ce message. Toute copie, divulgation, modification, utilisation ou diffusion, non autorisée, directe ou indirecte, de tout ou partie de ce message, est strictement interdite.


This e-mail, and any document attached hereby, may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized, direct or indirect, copying, disclosure, distribution or other use of the material or parts thereof is strictly forbidden.
Reply | Threaded
Open this post in threaded view
|

Re: MQTT retained messages with weird characters

clebertsuconic
In reply to this post by Martyn Taylor
Perhaps we could add this to the next 1.5.3, if you cherry-pick the fix in 1.x

On Mon, Feb 6, 2017 at 10:55 AM, Martyn Taylor <[hidden email]> wrote:

> Hi Francesco,
>
> It appears that there is a bug in the MessageImpl code that is not properly
> setting the endOfBodyPosition when the server restarts (reads from journal)
> and then there's a subsequent message copy (i.e. retain).  I've enabled
> correct pointer setting and added the fix to MQTT.
>
> I'm running the full suite atm to ensure this doesn't affect other areas of
> the code.  Should have this fix done shortly.
>
> Thanks for all the info,
> Martyn
>
> On Thu, Jan 12, 2017 at 10:13 AM, Francesco PADOVANI <
> [hidden email]> wrote:
>
>> Hi Martyn,
>>
>> I've just opened a JIRA related to this issue: it's the tkt ARTEMIS-917 (
>> https://issues.apache.org/jira/browse/ARTEMIS-917). Of course it's
>> currently unassigned: let me know if I have to assign it to someone in
>> particular.
>>
>>
>> A Thing That may help you (even if I understand you've already figured out
>> where the problem lies): a developer of ours did put the Artemis code under
>> debug and he has noticed that the message is yes, written correctly on the
>> file system, but then it is read in a wrong way. It seems a matter of
>> pointer position, which is kept to the end of metadata instead of the end
>> of message body while reading it. The involved classes should be the
>> followings:
>>
>>
>> AbstractJournalStorageManager.java
>>
>> --> this is the class that probably doesn't correctly values the pointer.
>>
>>
>> MessageImpl.java
>>
>> --> this (if I understand well) is the class containing the "decode"
>> method used to read the message. A question: do you think the last bold
>> line, below, could be a solution?
>>
>>
>> public void decode(final ActiveMQBuffer buff) {
>>
>>       int start = buff.readerIndex();
>>
>>       endOfBodyPosition = buff.readInt();
>>
>>       endOfMessagePosition = buff.getInt(endOfBodyPosition -
>> BUFFER_HEADER_SPACE + start);
>>
>>       int length = endOfMessagePosition - BUFFER_HEADER_SPACE;
>>
>>       buffer.setIndex(0, BUFFER_HEADER_SPACE);
>>
>>       buffer.writeBytes(buff, start, length);
>>
>>       decode();
>>
>>       buff.readerIndex(start + length);
>>
>>       buff.writerIndex(start +  endOfBodyPosition);
>>
>>    }
>>
>>
>>
>> Thanks a lot  for your help and your patience.
>>
>>
>> Francesco
>>
>>
>>
>> ________________________________
>> From: Martyn Taylor <[hidden email]>
>> Sent: Wednesday, January 11, 2017 3:53:00 PM
>> To: [hidden email]
>> Subject: Re: MQTT retained messages with weird characters
>>
>> It looks to me like the full message buffer (not just the body) is getting
>> returned to the client.  Clebert I think you're right in that we do some
>> hackery to handle retained messages.  I suspect this is where the error is.
>>
>> Francesco could you create a JIRA with some reproducer steps and I'll aim
>> to take a look.
>>
>> Thanks
>>
>> On Wed, Jan 11, 2017 at 2:03 PM, Clebert Suconic <
>> [hidden email]>
>> wrote:
>>
>> > the MQTT ProtocolManager does some playing with retaining, perhaps
>> > there's something wrong with reloading from journal on retaining. I
>> > was wondering if you could send us a test showing the issue with MQTT
>> > so we could see if there's anything specific to your test while using
>> > MQTT.
>> >
>> > On Wed, Jan 11, 2017 at 4:22 AM, Francesco PADOVANI
>> > <[hidden email]> wrote:
>> > > Unfortunately no (not yet).
>> > >
>> > > Anyway, it's a clean Artemis installation of version 1.5.1, made by
>> > > following the user manual instructions at
>> > > https://activemq.apache.org/artemis/docs/1.5.1/using-server.html.
>> > >
>> > > The installation was done by using a dedicated user and also the
>> artemis
>> > > java process runs with this dedicated user. Attached you can find my
>> > > broker.xml configuration file: it's pretty much the same default
>> created
>> > > during the installation procedure, but for the acceptors (which I've
>> > > customized for my MQTT purpose) and the addition of parameter
>> > > "<last-value-queue>true</last-value-queue>" inside the address-setting
>> > > section (but I tried with and without it and my issue persists).
>> > >
>> > > The only other configuration I changed is the heap size dedicated to
>> the
>> > > process, by setting "-Xms4096M -Xmx4096M" among the JAVA_ARGS in the
>> > > artemis.profile file.
>> > >
>> > >
>> > > The platform where Artemis broker is installed is:
>> > >
>> > >
>> > > - Machine:
>> > >
>> > > It's an EC2 instance on AWS cloud of type m4.large: 2 vCPU, 8G RAM and
>> > SSD
>> > > storage.
>> > >
>> > >
>> > > - OS:
>> > >
>> > > CentOS Linux release 7.3.1611 (Core) - x86_64
>> > >
>> > >
>> > > - JVM:
>> > >
>> > > java version "1.8.0_111"
>> > > Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
>> > > Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)
>> > >
>> > >
>> > > - Distro package version of libaio:
>> > >
>> > > Arch        : x86_64
>> > > Version     : 0.3.109
>> > > Release     : 13.el7
>> > >
>> > >
>> > > It seems like the problem relates the Persistence (File Journal?).
>> Until
>> > the
>> > > broker keeps the retained messages in ram it works. When it needs to
>> > recover
>> > > the retained messages  from the file system, clients receive them with
>> > weird
>> > > chars.
>> > > And I suspect the same issue appears when RAM is full and messages
>> start
>> > to
>> > > be paged... but this case is a bit more difficult to test in a
>> systematic
>> > > way.
>> > >
>> > >
>> > > Francesco
>> > >
>> > >
>> > >
>> > > ________________________________
>> > > From: Clebert Suconic <[hidden email]>
>> > > Sent: Tuesday, January 10, 2017 7:43 PM
>> > > To: [hidden email]
>> > > Subject: Re: MQTT retained messages with weird characters
>> > >
>> > > Do you have a working test you can share?
>> > >
>> > > On Tue, Jan 10, 2017 at 12:07 PM, Francesco PADOVANI
>> > > <[hidden email]> wrote:
>> > >> Hello,
>> > >>
>> > >> I'm trying the Apache Artemis Broker (ver. 1.5.1) for MQTT protocol.
>> > >>
>> > >> Exactly I'm testing the retained feature for messages of MQTT
>> protocol.
>> > >>
>> > >> While the broker is up it seems to work well:
>> > >>
>> > >> a) a client publishes a retained message to a specific topic
>> > >>
>> > >> b) any client which then subscribes that topic receives the retained
>> > >> message (any time it re-subscribes the topic it receives the last
>> > retained
>> > >> message ...perfect!)
>> > >>
>> > >> But it happens a strange thing when I restart the broker. And I mean I
>> > >> restart the broker without any change on configuration: simply I make
>> a
>> > >> "artemis-service stop" and "artemis-service start". After The broker
>> is
>> > up
>> > >> and running, if a client subscribes the previous topic it still
>> > receives the
>> > >> retained message, but with weird characters appended to it. For new
>> > retained
>> > >> messages published it work well again... but the previous ones (before
>> > the
>> > >> broker restart) are sent all with these weird characters. E.g., the
>> > >> following is a retained message published before the broker restart:
>> > >>
>> > >> "test 25 for a retained message 20170110"
>> > >>
>> > >> And this is exactly how clients get it before the broker restart.
>> > >>
>> > >> Instead, after broker restart, the same retained message is received
>> by
>> > >> clients in the following way:
>> > >>
>> > >> test 25 for a retained message 20170110?
>> > >>                                         ?
>> 6$sys.mqtt.retain..cro.test5 ?
>> > >> &mqtt.message.retain ? mqtt.qos.level
>> > >>
>> > >>
>> > >> Is this my bad configuration (but I don't know ehere)? Or anything
>> else?
>> > >> Or what?
>> > >>
>> > >> Someone can help me?
>> > >>
>> > >>
>> > >> Thanks in advance.
>> > >>
>> > >>
>> > >> Francesco
>> > >>
>> > >> ________________________________
>> > >>
>> > >> Ce message, ainsi que tous les fichiers joints à ce message, peuvent
>> > >> contenir des informations sensibles et/ ou confidentielles ne devant
>> pas
>> > >> être divulguées. Si vous n'êtes pas le destinataire de ce message (ou
>> > que
>> > >> vous recevez ce message par erreur), nous vous remercions de le
>> notifier
>> > >> immédiatement à son expéditeur, et de détruire ce message. Toute
>> copie,
>> > >> divulgation, modification, utilisation ou diffusion, non autorisée,
>> > directe
>> > >> ou indirecte, de tout ou partie de ce message, est strictement
>> > interdite.
>> > >>
>> > >>
>> > >> This e-mail, and any document attached hereby, may contain
>> confidential
>> > >> and/or privileged information. If you are not the intended recipient
>> (or
>> > >> have received this e-mail in error) please notify the sender
>> > immediately and
>> > >> destroy this e-mail. Any unauthorized, direct or indirect, copying,
>> > >> disclosure, distribution or other use of the material or parts thereof
>> > is
>> > >> strictly forbidden.
>> > >
>> > >
>> > >
>> > > --
>> > > Clebert Suconic
>> > >
>> > > ________________________________
>> > >
>> > > Ce message, ainsi que tous les fichiers joints à ce message, peuvent
>> > > contenir des informations sensibles et/ ou confidentielles ne devant
>> pas
>> > > être divulguées. Si vous n'êtes pas le destinataire de ce message (ou
>> que
>> > > vous recevez ce message par erreur), nous vous remercions de le
>> notifier
>> > > immédiatement à son expéditeur, et de détruire ce message. Toute copie,
>> > > divulgation, modification, utilisation ou diffusion, non autorisée,
>> > directe
>> > > ou indirecte, de tout ou partie de ce message, est strictement
>> interdite.
>> > >
>> > >
>> > > This e-mail, and any document attached hereby, may contain confidential
>> > > and/or privileged information. If you are not the intended recipient
>> (or
>> > > have received this e-mail in error) please notify the sender
>> immediately
>> > and
>> > > destroy this e-mail. Any unauthorized, direct or indirect, copying,
>> > > disclosure, distribution or other use of the material or parts thereof
>> is
>> > > strictly forbidden.
>> >
>> >
>> >
>> > --
>> > Clebert Suconic
>> >
>>
>> ________________________________
>>
>> Ce message, ainsi que tous les fichiers joints à ce message, peuvent
>> contenir des informations sensibles et/ ou confidentielles ne devant pas
>> être divulguées. Si vous n'êtes pas le destinataire de ce message (ou que
>> vous recevez ce message par erreur), nous vous remercions de le notifier
>> immédiatement à son expéditeur, et de détruire ce message. Toute copie,
>> divulgation, modification, utilisation ou diffusion, non autorisée, directe
>> ou indirecte, de tout ou partie de ce message, est strictement interdite.
>>
>>
>> This e-mail, and any document attached hereby, may contain confidential
>> and/or privileged information. If you are not the intended recipient (or
>> have received this e-mail in error) please notify the sender immediately
>> and destroy this e-mail. Any unauthorized, direct or indirect, copying,
>> disclosure, distribution or other use of the material or parts thereof is
>> strictly forbidden.
>>



--
Clebert Suconic