ARTEMIS: bad-performance behaviour after 7-10 days of usage

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
23 messages Options
12
Reply | Threaded
Open this post in threaded view
|

ARTEMIS: bad-performance behaviour after 7-10 days of usage

francesco81

Hello,
I'm using Apache Artemis as MQTT broker for our IOT projects.
It's a clean Artemis installation of version 1.5.1., on a server (CentOS 7) which has 2 vCPU, 8 GB of RAM (4GB of Heap Space dedicated to Artemis) and 50 GB of SSD data disk.
After the installation of artemis Broker we started to test it with about 10-15 clients constantly connected, 150 subscribed topics and an average of 2 messages per minute per client. I think these are not huge numbers, right?
For the few days after the installation, all was good and the broker worked perfectly: it was fast and reliable. But day by day performances have decreased and after about 10 days of usage it is became almost unusable: due to resources consumption. The following is my "top" situation on the server:


top - 08:59:50 up 2 days, 22:09,  1 user,  load average: 2.56, 2.88, 3.00
Tasks:  92 total,   1 running,  91 sleeping,   0 stopped,   0 zombie
%Cpu(s): 99.5 us,  0.5 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  7747288 total,   290968 free,  4697976 used,  2758344 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  2734768 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND  
 4352 mqtt      20   0 7043380 4.348g  15192 S 200.0 58.9   6153:54 java                                                                                           
    1 root      20   0  128096   5176   2420 S   0.0  0.1   0:04.01 systemd

Now I can see the broker starting to page on disk almost always. For sure it's a wrong configuration of ours. Currently, it seems that very old address/queue (and the related retained messages) are always kept, making memory and cpu consuption growing more and more. And even after a restart, the broker takes so much to get up. Before to become reachable, it starts to make many management operations like, for example, retrieve data from paging, etc.. But I also see the broker that starts to register old topics and queues we don't need. How to clean them? How to make a topic/queue expire?

Inside my broker.xml I did set up the parameter last-value-queue=true, thinking that this was the problem ....but it doesn't solve. Or better: probably it's me that I've not understood the correct meaning of the parameter. I did check also the clients' parameters when they connect to the broker, to be sure they don't set, for example, clean-session = false (saying the broker to keep all messages also when a client disconnects). But they make it in the right way. The only thing is that they don't specify a client-id. They connect by using a username/password and certificate (over tls). So, every time a client connects, Artemis automatically provide a random client-id for it (if I understood well).

Attached you can find my broker.xml configuration file: it's pretty much the same default created during the installation procedure, but for the acceptors (which I've customized for my MQTT purpose) and the addition of parameter last-value-queue = true inside the address-setting section.


Please: some of you could help me? How I have to configure my broker instance to understand and solve these performance issues?


Thanks in advance


Francesco





Ce message, ainsi que tous les fichiers joints à ce message, peuvent contenir des informations sensibles et/ ou confidentielles ne devant pas être divulguées. Si vous n'êtes pas le destinataire de ce message (ou que vous recevez ce message par erreur), nous vous remercions de le notifier immédiatement à son expéditeur, et de détruire ce message. Toute copie, divulgation, modification, utilisation ou diffusion, non autorisée, directe ou indirecte, de tout ou partie de ce message, est strictement interdite.


This e-mail, and any document attached hereby, may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized, direct or indirect, copying, disclosure, distribution or other use of the material or parts thereof is strictly forbidden.


broker.xml (9K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage

clebertsuconic
An easy way to check what is going on is with


Artemis data print


(Preferably with the broker stopped)



You can then check what is not being asked and why it became paging.


You could maybe replicate a similar load pattern on a test? Like sending
many messages a second instead of 2 per minute?


I wonder what is being retained not acked leaving many messages in memory.
On Mon, Jan 23, 2017 at 5:55 AM Francesco PADOVANI <
[hidden email]> wrote:

>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Hello,
>
>
> I'm using Apache Artemis as MQTT broker for our IOT projects.
>
>
> It's a clean Artemis installation of version 1.5.1., on a server (CentOS
> 7) which has 2 vCPU, 8 GB of RAM (4GB of Heap Space dedicated to Artemis)
> and 50 GB of SSD data disk.
>
>
> After the installation of artemis Broker we started to test it with about
> 10-15 clients constantly connected, 150 subscribed topics and an average of
> 2 messages per minute per client. I think these are not huge numbers, right?
>
>
> For the few days after the installation, all was good and the broker
> worked perfectly: it was fast and reliable. But day by day performances
> have decreased and after about 10 days of usage it is became almost
> unusable: due to resources consumption. The following
>
> is my "top" situation on the server:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *top - 08:59:50 up 2 days, 22:09,  1 user,  load average: 2.56, 2.88,
> 3.00Tasks:  92 total,   1 running,  91 sleeping,   0 stopped,   0
> zombie%Cpu(s): 99.5 us,  0.5 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0
> si,  0.0 stKiB Mem :  7747288 total,   290968 free,  4697976 used,  2758344
> buff/cacheKiB Swap:        0 total,        0 free,        0 used.  2734768
> avail Mem  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM
> TIME+ COMMAND   4352 mqtt      20   0 7043380 4.348g  15192 S 200.0 58.9
> 6153:54
> java
>      1 root      20   0  128096   5176   2420 S   0.0  0.1   0:04.01
> systemd*
>
>
>
>
>
>
>
>
>
> Now I can see the broker starting to page on disk almost always. For sure
> it's a wrong configuration of ours. Currently, it seems that very old
> address/queue (and the related retained messages) are always kept, making
> memory and cpu consuption growing
>
> more and more. And even after a restart, the broker takes so much to get
> up.
>
> Before to become reachable, it starts to make many management operations
> like, for example, retrieve data from paging, etc.. But I also see the
> broker that starts to register old topics and queues we don't need. How to
> clean them? How to make a topic/queue
>
> expire?
>
>
>
>
>
>
>
> Inside my broker.xml I did set up the parameter last-value-queue=true,
> thinking that this was the problem ....but it doesn't solve. Or better:
> probably it's me that I've not understood the correct meaning of the
> parameter.
>
> I did check also the clients' parameters when they connect to the broker,
> to be sure they don't set, for example, clean-session = false (saying the
> broker to keep all messages also when a client disconnects). But they make
> it in the right way. The only
>
> thing is that they don't specify a client-id. They connect by using a
> username/password and certificate (over tls). So, every time a client
> connects, Artemis automatically provide a random client-id for it (if I
> understood well).
>
>
> Attached you can find my broker.xml configuration file: it's pretty much
> the same default created during the installation procedure, but for the
> acceptors (which I've customized for my MQTT purpose) and the addition of
> parameter
>
> last-value-queue = true inside the address-setting
>
> section.
>
>
>
>
>
>
>
>
>
> Please: some of you could help me? How I have to configure my broker
> instance to understand and solve these performance issues?
>
>
>
>
>
>
>
> Thanks in advance
>
>
>
>
>
>
>
> Francesco
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> ------------------------------
>
>
>
>
> Ce message, ainsi que tous les fichiers joints à ce message, peuvent
> contenir des informations sensibles et/ ou confidentielles ne devant pas
> être divulguées. Si vous n'êtes pas le destinataire de ce message (ou que
> vous recevez ce message par erreur), nous
>
> vous remercions de le notifier immédiatement à son expéditeur, et de
> détruire ce message. Toute copie, divulgation, modification, utilisation ou
> diffusion, non autorisée, directe ou indirecte, de tout ou partie de ce
> message, est strictement interdite.
>
>
>
>
>
>
> This e-mail, and any document attached hereby, may contain confidential
> and/or privileged information. If you are not the intended recipient (or
> have received this e-mail in error) please notify the sender immediately
> and destroy this e-mail. Any unauthorized,
>
> direct or indirect, copying, disclosure, distribution or other use of the
> material or parts thereof is strictly forbidden.
>
>
>
>
>
>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage

clebertsuconic
In reply to this post by francesco81
A question. Are u consuming from the subscriptions or u intend to leave
them hinging?


While paging. Do u need transactions? If u start to page not using
transactions would make paging to act like partitions on Kafka.



I will be waiting for more data from you before we can help some more.



On Mon, Jan 23, 2017 at 5:55 AM Francesco PADOVANI <
[hidden email]> wrote:

>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> Hello,
>
>
> I'm using Apache Artemis as MQTT broker for our IOT projects.
>
>
> It's a clean Artemis installation of version 1.5.1., on a server (CentOS
> 7) which has 2 vCPU, 8 GB of RAM (4GB of Heap Space dedicated to Artemis)
> and 50 GB of SSD data disk.
>
>
> After the installation of artemis Broker we started to test it with about
> 10-15 clients constantly connected, 150 subscribed topics and an average of
> 2 messages per minute per client. I think these are not huge numbers, right?
>
>
> For the few days after the installation, all was good and the broker
> worked perfectly: it was fast and reliable. But day by day performances
> have decreased and after about 10 days of usage it is became almost
> unusable: due to resources consumption. The following
>
> is my "top" situation on the server:
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *top - 08:59:50 up 2 days, 22:09,  1 user,  load average: 2.56, 2.88,
> 3.00Tasks:  92 total,   1 running,  91 sleeping,   0 stopped,   0
> zombie%Cpu(s): 99.5 us,  0.5 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0
> si,  0.0 stKiB Mem :  7747288 total,   290968 free,  4697976 used,  2758344
> buff/cacheKiB Swap:        0 total,        0 free,        0 used.  2734768
> avail Mem  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM
> TIME+ COMMAND   4352 mqtt      20   0 7043380 4.348g  15192 S 200.0 58.9
> 6153:54
> java
>      1 root      20   0  128096   5176   2420 S   0.0  0.1   0:04.01
> systemd*
>
>
>
>
>
>
>
>
>
> Now I can see the broker starting to page on disk almost always. For sure
> it's a wrong configuration of ours. Currently, it seems that very old
> address/queue (and the related retained messages) are always kept, making
> memory and cpu consuption growing
>
> more and more. And even after a restart, the broker takes so much to get
> up.
>
> Before to become reachable, it starts to make many management operations
> like, for example, retrieve data from paging, etc.. But I also see the
> broker that starts to register old topics and queues we don't need. How to
> clean them? How to make a topic/queue
>
> expire?
>
>
>
>
>
>
>
> Inside my broker.xml I did set up the parameter last-value-queue=true,
> thinking that this was the problem ....but it doesn't solve. Or better:
> probably it's me that I've not understood the correct meaning of the
> parameter.
>
> I did check also the clients' parameters when they connect to the broker,
> to be sure they don't set, for example, clean-session = false (saying the
> broker to keep all messages also when a client disconnects). But they make
> it in the right way. The only
>
> thing is that they don't specify a client-id. They connect by using a
> username/password and certificate (over tls). So, every time a client
> connects, Artemis automatically provide a random client-id for it (if I
> understood well).
>
>
> Attached you can find my broker.xml configuration file: it's pretty much
> the same default created during the installation procedure, but for the
> acceptors (which I've customized for my MQTT purpose) and the addition of
> parameter
>
> last-value-queue = true inside the address-setting
>
> section.
>
>
>
>
>
>
>
>
>
> Please: some of you could help me? How I have to configure my broker
> instance to understand and solve these performance issues?
>
>
>
>
>
>
>
> Thanks in advance
>
>
>
>
>
>
>
> Francesco
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> ------------------------------
>
>
>
>
> Ce message, ainsi que tous les fichiers joints à ce message, peuvent
> contenir des informations sensibles et/ ou confidentielles ne devant pas
> être divulguées. Si vous n'êtes pas le destinataire de ce message (ou que
> vous recevez ce message par erreur), nous
>
> vous remercions de le notifier immédiatement à son expéditeur, et de
> détruire ce message. Toute copie, divulgation, modification, utilisation ou
> diffusion, non autorisée, directe ou indirecte, de tout ou partie de ce
> message, est strictement interdite.
>
>
>
>
>
>
> This e-mail, and any document attached hereby, may contain confidential
> and/or privileged information. If you are not the intended recipient (or
> have received this e-mail in error) please notify the sender immediately
> and destroy this e-mail. Any unauthorized,
>
> direct or indirect, copying, disclosure, distribution or other use of the
> material or parts thereof is strictly forbidden.
>
>
>
>
>
>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage

francesco81
Hello,
just an update about this topic.

By disabling persistence, the broker starts again to work fast and reliable. But just for about 2 more weeks: then the memory come bake to be full and the broker starts to page constantly, and cpu usage goes to 100%.

I tried to run the command "artemis data print" as you suggested, but I have to admit my ignorance to read the results.
I did redirect the command result to a file and it produced about 758 MB of data.
I noticed no strange things among the initial lines but this:

17:39:47,214 FINE  [io.netty.util.internal.PlatformDependent] Javassist: unavailable
17:39:47,215 FINE  [io.netty.util.internal.PlatformDependent] You don't have Javassist in your class path or you don't have enough permission to load dynamically generated classes.  Please check the configuration for better performance.


...I don't know how much this will weigh on performances.

Then I have a "B I N D I N G S  J O U R N A L" section with thousands of lines like the followings:

operation@AddRecordTX;txID=109,recordID=110;userRecordType=21;isUpdate=false;compactCount=0;PersistentQueueBindingEncoding [id=110, name=b7478faf-f0fd-481a-93ef-b6109dc3a689..cro.plantid.1073403.gwid.25324.device_profile.04.set, address=.cro.plantid.1073403.gwid.25324.device_profile.04.set, filterString=NOT ((AMQAddress = 'jms.queue.activemq.management') OR (AMQAddress = 'activemq.notifications')), user=gateway, autoCreated=false]
operation@Commit;txID=109,numberOfRecords=1


...where "cro.plantid.1073403.gwid.25324.device_profile.04.set" is an example of a topic for a specific plant (there are about 48 topic for plant).
What does "isUpdate=false;compactCount=0;PersistentQueueBindingEncoding" mean?

Then I have a "### Surviving Records Summary ###" section, again with thousands of lines like this:

recordID=5205;userRecordType=21;isUpdate=false;compactCount=0;PersistentQueueBindingEncoding [id=5205, name=1447f6a659bc9bc0df2e55..cro.plantid.1071617.gwid.30889.
device_profile.02.err, address=.cro.plantid.1071617.gwid.30889.device_profile.02.err, filterString=NOT ((AMQAddress = 'jms.queue.activemq.management') OR (AMQAddre
ss = 'activemq.notifications')), user=gateway, autoCreated=false]


Then I have the section "### Prepared TX ###", which is empty, and the ### Message Counts ### as following:
message count=0
message reference count
prepared message count=0

Then I have the "M E S S A G E S   J O U R N A L" section with thousands of lines like the followings:

peration@AddRecord;recordID=10088;userRecordType=31;isUpdate=false;compactCount=0;Message(messageID=10086;msg=ServerMessage[messageID=10086,durable=true,userID=nu
ll,priority=0, bodySize=512, timestamp=0,expiration=0, durable=true, address=$sys.mqtt.retain..cro.plantid.1073620.gwid.32646.device_state.err,properties=TypedProp
erties[mqtt.message.retain=true,mqtt.qos.level=1]]@1144068272
operation@Update;recordID=10088;userRecordType=32;isUpdate=true;compactCount=0;AddRef;QueueEncoding [queueID=65]
operation@Update;recordID=10088;userRecordType=32;isUpdate=true;compactCount=0;AddRef;QueueEncoding [queueID=9747]
operation@Update;recordID=10088;userRecordType=32;isUpdate=true;compactCount=0;AddRef;QueueEncoding [queueID=7774]

... and no deleteRecord operation for that id (10088): I don't know if it is right.

Then I have again the "### Surviving Records Summary ###" with thousands of line like the followings;

recordID=196;userRecordType=31;isUpdate=false;compactCount=0;Message(messageID=196;msg=ServerMessage[messageID=196,durable=true,userID=null,priority=0, bodySize=2048, timestamp=0,expiration=0, durable=true, address=.cro.plantid.1073403.gwid.25324.device_profile.04,properties=TypedProperties[mqtt.message.retain=true,mqtt.qos.level=2]]@207430813
recordID=196;userRecordType=32;isUpdate=true;compactCount=0;AddRef;QueueEncoding [queueID=65]

...again my ignorance: why "durable=true"? because retained? because qos2?

for the MESSAGES JOURNAL I have the ### Message Counts ### with thousands of lines like these:
message count=55958
message reference count
queue id 2,count=11
queue id 196608,count=2
queue id 4,count=1
queue id 196619,count=4
queue id 65547,count=1
queue id 196625,count=2
queue id 196630,count=2
queue id 65559,count=1
queue id 1081351,count=1
queue id 163869,count=17
queue id 163875,count=2307
... what does this mean?

Finally I have the "P A G I N G" section with thousands of lines like the following:

Exploring store .cro.plantid.1074918.gwid.33300.device_profile.02 folder = /app/mqtt/activeMQ/brokers/eltbroker01/./data/paging/5ac5a57d-dcc1-11e6-a52b-069133a4f5c
f
*******   Page 1
17:40:26,550 DEBUG [org.apache.activemq.artemis.core.paging.impl.Page] reading page 1 on address = .cro.plantid.1074918.gwid.33300.device_profile.02
pg=1, msg=0,pgTX=-1,userMessageID=, msg=ServerMessage[messageID=1034283,durable=true,userID=null,priority=0, bodySize=2048, timestamp=0,expiration=0, durable=true,
address=.cro.plantid.1074918.gwid.33300.device_profile.02,properties=TypedProperties[mqtt.message.retain=true,mqtt.qos.level=2]]@1029639001,Queues = 1034178
pg=1, msg=1,pgTX=-1,userMessageID=, msg=ServerMessage[messageID=1036477,durable=true,userID=null,priority=0, bodySize=2048, timestamp=0,expiration=0, durable=true,
address=.cro.plantid.1074918.gwid.33300.device_profile.02,properties=TypedProperties[mqtt.message.retain=true,mqtt.qos.level=2]]@745530122,Queues = 1036418,103417
8,1035409
pg=1, msg=2,pgTX=-1,userMessageID=, msg=ServerMessage[messageID=1038280,durable=true,userID=null,priority=0, bodySize=2048, timestamp=0,expiration=0, durable=true,
address=.cro.plantid.1074918.gwid.33300.device_profile.02,properties=TypedProperties[mqtt.message.retain=true,mqtt.qos.level=2]]@752575846,Queues = 1036418,103797
9,1034178,1035409,1036769,1037082
...


...but, in this case for example, since on topic/address "cro.plantid.1074918.gwid.33300.device_profile.02" travel only retained messages, I thought that at anytime there could be at most only 1 message per time (because the next retained will substitute the previous one)... It's not so?
And also here: why durable true? Because QoS2? Does "durable=true" mean that these messages will persist in memory forever?

sorry for these patches of information ...but the result of "data print" command was too big for an upload.
Anyway, hope can help you to help me.

Francesco
Reply | Threaded
Open this post in threaded view
|

Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage

clebertsuconic
apparently there's a bug MQTT retained queues. when you disabled
persistence on the broker we removed the capacity of validating or
debugging it.


On Mon, Jan 30, 2017 at 9:58 AM, francesco81
<[hidden email]> wrote:

> Hello,
> just an update about this topic.
>
> By disabling persistence, the broker starts again to work fast and reliable.
> But just for about 2 more weeks: then the memory come bake to be full and
> the broker starts to page constantly, and cpu usage goes to 100%.
>
> I tried to run the command "artemis data print" as you suggested, but I have
> to admit my ignorance to read the results.
> I did redirect the command result to a file and it produced about 758 MB of
> data.
> I noticed no strange things among the initial lines but this:
>
> /*17:39:47,214 FINE  [io.netty.util.internal.PlatformDependent] Javassist:
> unavailable
> 17:39:47,215 FINE  [io.netty.util.internal.PlatformDependent] You don't have
> Javassist in your class path or you don't have enough permission to load
> dynamically generated classes.  Please check the configuration for better
> performance.*/
>
> ...I don't know how much this will weigh on performances.
>
> Then I have a "B I N D I N G S  J O U R N A L" section with thousands of
> lines like the followings:
>
> /*operation@AddRecordTX;txID=109,recordID=110;userRecordType=21;isUpdate=false;compactCount=0;PersistentQueueBindingEncoding
> [id=110,
> name=b7478faf-f0fd-481a-93ef-b6109dc3a689..cro.plantid.1073403.gwid.25324.device_profile.04.set,
> address=.cro.plantid.1073403.gwid.25324.device_profile.04.set,
> filterString=NOT ((AMQAddress = 'jms.queue.activemq.management') OR
> (AMQAddress = 'activemq.notifications')), user=gateway, autoCreated=false]
> operation@Commit;txID=109,numberOfRecords=1*/
>
> ...where "cro.plantid.1073403.gwid.25324.device_profile.04.set" is an
> example of a topic for a specific plant (there are about 48 topic for
> plant).
> What does "isUpdate=false;compactCount=0;PersistentQueueBindingEncoding"
> mean?
>
> Then I have a "### Surviving Records Summary ###" section, again with
> thousands of lines like this:
>
> /*recordID=5205;userRecordType=21;isUpdate=false;compactCount=0;PersistentQueueBindingEncoding
> [id=5205, name=1447f6a659bc9bc0df2e55..cro.plantid.1071617.gwid.30889.
> device_profile.02.err,
> address=.cro.plantid.1071617.gwid.30889.device_profile.02.err,
> filterString=NOT ((AMQAddress = 'jms.queue.activemq.management') OR
> (AMQAddre
> ss = 'activemq.notifications')), user=gateway, autoCreated=false]*/
>
> Then I have the section "### Prepared TX ###", which is empty, and the ###
> Message Counts ### as following:
> message count=0
> message reference count
> prepared message count=0*/
>
> Then I have the "M E S S A G E S   J O U R N A L" section with thousands of
> lines like the followings:
>
> peration@AddRecord;recordID=10088;userRecordType=31;isUpdate=false;compactCount=0;Message(messageID=10086;msg=ServerMessage[messageID=10086,durable=true,userID=nu
> ll,priority=0, bodySize=512, timestamp=0,expiration=0, durable=true,
> address=$sys.mqtt.retain..cro.plantid.1073620.gwid.32646.device_state.err,properties=TypedProp
> erties[mqtt.message.retain=true,mqtt.qos.level=1]]@1144068272
> operation@Update;recordID=10088;userRecordType=32;isUpdate=true;compactCount=0;AddRef;QueueEncoding
> [queueID=65]
> operation@Update;recordID=10088;userRecordType=32;isUpdate=true;compactCount=0;AddRef;QueueEncoding
> [queueID=9747]
> operation@Update;recordID=10088;userRecordType=32;isUpdate=true;compactCount=0;AddRef;QueueEncoding
> [queueID=7774]
>
> ... and no deleteRecord operation for that id (10088): I don't know if it is
> right.
>
> Then I have again the "### Surviving Records Summary ###" with thousands of
> line like the followings;
>
> recordID=196;userRecordType=31;isUpdate=false;compactCount=0;Message(messageID=196;msg=ServerMessage[messageID=196,durable=true,userID=null,priority=0,
> bodySize=2048, timestamp=0,expiration=0, durable=true,
> address=.cro.plantid.1073403.gwid.25324.device_profile.04,properties=TypedProperties[mqtt.message.retain=true,mqtt.qos.level=2]]@207430813
> recordID=196;userRecordType=32;isUpdate=true;compactCount=0;AddRef;QueueEncoding
> [queueID=65]
>
> ...again my ignorance: why "durable=true"? because retained? because qos2?
>
> for the MESSAGES JOURNAL I have the ### Message Counts ### with thousands of
> lines like these:
> message count=55958
> message reference count
> queue id 2,count=11
> queue id 196608,count=2
> queue id 4,count=1
> queue id 196619,count=4
> queue id 65547,count=1
> queue id 196625,count=2
> queue id 196630,count=2
> queue id 65559,count=1
> queue id 1081351,count=1
> queue id 163869,count=17
> queue id 163875,count=2307
> ... what does this mean?
>
> Finally I have the "P A G I N G" section with thousands of lines like the
> following:
>
> /*Exploring store .cro.plantid.1074918.gwid.33300.device_profile.02 folder =
> /app/mqtt/activeMQ/brokers/eltbroker01/./data/paging/5ac5a57d-dcc1-11e6-a52b-069133a4f5c
> f
> *******   Page 1
> 17:40:26,550 DEBUG [org.apache.activemq.artemis.core.paging.impl.Page]
> reading page 1 on address =
> .cro.plantid.1074918.gwid.33300.device_profile.02
> pg=1, msg=0,pgTX=-1,userMessageID=,
> msg=ServerMessage[messageID=1034283,durable=true,userID=null,priority=0,
> bodySize=2048, timestamp=0,expiration=0, durable=true,
> address=.cro.plantid.1074918.gwid.33300.device_profile.02,properties=TypedProperties[mqtt.message.retain=true,mqtt.qos.level=2]]@1029639001,Queues
> = 1034178
> pg=1, msg=1,pgTX=-1,userMessageID=,
> msg=ServerMessage[messageID=1036477,durable=true,userID=null,priority=0,
> bodySize=2048, timestamp=0,expiration=0, durable=true,
> address=.cro.plantid.1074918.gwid.33300.device_profile.02,properties=TypedProperties[mqtt.message.retain=true,mqtt.qos.level=2]]@745530122,Queues
> = 1036418,103417
> 8,1035409
> pg=1, msg=2,pgTX=-1,userMessageID=,
> msg=ServerMessage[messageID=1038280,durable=true,userID=null,priority=0,
> bodySize=2048, timestamp=0,expiration=0, durable=true,
> address=.cro.plantid.1074918.gwid.33300.device_profile.02,properties=TypedProperties[mqtt.message.retain=true,mqtt.qos.level=2]]@752575846,Queues
> = 1036418,103797
> 9,1034178,1035409,1036769,1037082
> ...
> */
>
> ...but, in this case for example, since on topic/address
> "cro.plantid.1074918.gwid.33300.device_profile.02" travel only retained
> messages, I thought that at anytime there could be at most only 1 message
> per time (because the next retained will substitute the previous one)...
> It's not so?
> And also here: why durable true? Because QoS2? Does "durable=true" mean that
> these messages will persist in memory forever?
>
> sorry for these patches of information ...but the result of "data print"
> command was too big for an upload.
> Anyway, hope can help you to help me.
>
> Francesco
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-tp4721272p4721395.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage

francesco81
Hi Clebert,
good to know it.
But, just to be clear, I have the same problem also with persistence enabled (indeed, it's even worse).
Regardless of the persistence, the behavior is the same: after a certain window of usage, ram fills and the broker starts to work constantly in paging mode (cpu --> 100%), becoming almost unusable.
- If persistence is disabled, I can restart the broker and it comes back to work fast (ram is free) ..but the problem, of course, is only postponed for another window of usage (currently about 10 days, for my use case).
- If persistence is enabled, I can restart the broker, but it starts to retrieve all stored messages from disk, ram fills quickly again, it returns to work in paging mode and cpu goes to 100%. Therefore I'm pratically obliged to disable persistence.

Could it be something related to a misconfiguration of my broker instance? I notice also that all messages witha QoS > 0 (regardless whether retained or not) have the property durable=true: does this mean that all those messages will persist in memory forever? There is no one else who is facing the same issue by using Artemis as mqtt broker?

Following is my "address-settings" configuration section within broker.xml file. It's the only one that I changed in the standard configuration (apart the acceptor for mqtt of course). Hope can help:

      <address-settings>
         
         <address-setting match="#">
            <dead-letter-address>jms.queue.DLQ</dead-letter-address>
            <expiry-address>jms.queue.ExpiryQueue</expiry-address>
            <redelivery-delay>0</redelivery-delay>
           
            <max-size-bytes>-1</max-size-bytes>
            <message-counter-history-day-limit>10</message-counter-history-day-limit>
            <address-full-policy>PAGE</address-full-policy>
            <last-value-queue>true</last-value-queue>
         </address-setting>
      </address-settings>


Francesco
Reply | Threaded
Open this post in threaded view
|

Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage

clebertsuconic
The bug wouldn't be on the persistence layer itself. The MQTT protocol
manager is managing retainers by removing (acking) older messages. And
something is broken. I'm not sure if it's a general case or something
specifically to your use case. I was wondering if you could replicate
the issue on a simple testsuite...


Martyn has done a lot of work around the MQTT protocol manager. I know
him personally and he's not around today, so I'm wondering if he will
be able to pick up this issue tomorrow.

but if you had a test replicating the issue it would help anyone to
jump on the issue quicker.


On Tue, Jan 31, 2017 at 6:27 AM, francesco81
<[hidden email]> wrote:

> Hi Clebert,
> good to know it.
> But, just to be clear, I have the same problem also with persistence enabled
> (indeed, it's even worse).
> Regardless of the persistence, the behavior is the same: after a certain
> window of usage, ram fills and the broker starts to work constantly in
> paging mode (cpu --> 100%), becoming almost unusable.
> - If persistence is disabled, I can restart the broker and it comes back to
> work fast (ram is free) ..but the problem, of course, is only postponed for
> another window of usage (currently about 10 days, for my use case).
> - If persistence is enabled, I can restart the broker, but it starts to
> retrieve all stored messages from disk, ram fills quickly again, it returns
> to work in paging mode and cpu goes to 100%. Therefore I'm pratically
> obliged to disable persistence.
>
> Could it be something related to a misconfiguration of my broker instance? I
> notice also that all messages witha QoS > 0 (regardless whether retained or
> not) have the property durable=true: does this mean that all those messages
> will persist in memory forever? There is no one else who is facing the same
> issue by using Artemis as mqtt broker?
>
> Following is my "address-settings" configuration section within broker.xml
> file. It's the only one that I changed in the standard configuration (apart
> the acceptor for mqtt of course). Hope can help:
>
> *      <address-settings>
>
>          <address-setting match="#">
>             <dead-letter-address>jms.queue.DLQ</dead-letter-address>
>             <expiry-address>jms.queue.ExpiryQueue</expiry-address>
>             <redelivery-delay>0</redelivery-delay>
>
>             <max-size-bytes>-1</max-size-bytes>
>
> <message-counter-history-day-limit>10</message-counter-history-day-limit>
>             <address-full-policy>PAGE</address-full-policy>
>             <last-value-queue>true</last-value-queue>
>          </address-setting>
>       </address-settings>*
>
> Francesco
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-tp4721272p4721417.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage

francesco81
Ok, perfect.
I'd be very happy if Martyn could take a look on this.
Meanwile, I'll try to put in place a test environment on AWS, accessible to everyone of the community ...this way I hope you'll be able to help me to debug and solve the problem.

Thanks

Francesco
Reply | Threaded
Open this post in threaded view
|

Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage

clebertsuconic
I actually thought about a self contained test, that we can use to
debug at our dev environment.

Putting something complex to be executed over the AWS wouldn't help
much. We would still need to figure out your whole app framework.

Why don't you use some of the examples infra-structure to create such
thing? if you want to do an UnitTest on the integration testsuite..
it's even better.

On Wed, Feb 1, 2017 at 8:02 AM, francesco81
<[hidden email]> wrote:

> Ok, perfect.
> I'd be very happy if Martyn could take a look on this.
> Meanwile, I'll try to put in place a test environment on AWS, accessible to
> everyone of the community ...this way I hope you'll be able to help me to
> debug and solve the problem.
>
> Thanks
>
> Francesco
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-tp4721272p4721470.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage

francesco81
Hi Clebert,
ok, we will build some junit tests to replicate the problem, follwing the exemple section.
Meanwhile, I can confirm to you the bug(s): it seems that the reference to the message is never removed from the queue. If the message is not retained it persists 1 reference on the queue and every time my client reconnect to the broker it will receive always the same message. If the message is retained is even worse, because it goes to increment the references on the queue and every time the client connects to the broker it will receive n + 1 times the same retained message where n is the re-connection number n.
Following a simple test I made by using mosquitto client:

CLIENT 1 (publisher):
mosquitto_pub -h <host> -q 1 -i <client_id_pub> -t /cro/plantid/0000000/gwid/00001/device_state -m "test non retained"

CLIENT 2 (subscriber):
1 connection:
mosquitto_sub -h <host> -q 2 -i <client_id_sub> -t /cro/plantid/0000000/gwid/00001/device_state
--> received:
test non retained
2 connection:
mosquitto_sub -h <host> -q 2 -i <client_id_sub> -t /cro/plantid/0000000/gwid/00001/device_state
--> received again:
test non retained
3 connection:
mosquitto_sub -h <host> -q 2 -i <client_id_sub> -t /cro/plantid/0000000/gwid/00001/device_state
--> received again:
test non retained
...
and so on

CLIENT 1 (publisher):
mosquitto_pub -h <host> -q 1 -i <client_id_pub> -r -t /cro/plantid/0000000/gwid/00001/device_state -m "test retained"

CLIENT 2 (subscriber):
4 connection:
mosquitto_sub -h <host> -q 2 -i <client_id_sub> -t /cro/plantid/0000000/gwid/00001/device_state
--> received:
test retained
test non retained
test retained

5 connection:
mosquitto_sub -h <host> -q 2 -i <client_id_sub> -t /cro/plantid/0000000/gwid/00001/device_state
--> received:
test retained
test non retained
test retained
test retained

6 connection:
mosquitto_sub -h <host> -q 2 -i <client_id_sub> -t /cro/plantid/0000000/gwid/00001/device_state
--> received:
test non retained
test retained
test retained
test retained
test retained

...
and so on.

If you agree, I would open a bug for this.

Francesco
Reply | Threaded
Open this post in threaded view
|

Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage

Martyn Taylor
Francesco,

Thanks for providing this information.  I think there's a couple of things
going on and I'd like to work with you to create some reproducers, we could
have a chat IRC #[hidden email] and I'll see if I can
recreate your env / case in our test suite.

With the case above, the mosquitto client is sending a subscribe frame on
every connection.  Artemis is then treating this as a new subscription
(even though the subscription queue already exists), and is copying over
the retain messages.  It's a reasonably trivial change, I should have a fix
for you shortly.

Thanks
Martyn

On Fri, Feb 3, 2017 at 11:39 AM, francesco81 <[hidden email]>
wrote:

> Hi Clebert,
> ok, we will build some junit tests to replicate the problem, follwing the
> exemple section.
> Meanwhile, I can confirm to you the bug(s): it seems that the reference to
> the message is never removed from the queue. If the message is not retained
> it persists 1 reference on the queue and every time my client reconnect to
> the broker it will receive always the same message. If the message is
> retained is even worse, because it goes to increment the references on the
> queue and every time the client connects to the broker it will receive n +
> 1
> times the same retained message where n is the re-connection number n.
> Following a simple test I made by using mosquitto client:
>
> CLIENT 1 (publisher):
> mosquitto_pub -h <host> -q 1 -i <client_id_pub> -t
> /cro/plantid/0000000/gwid/00001/device_state -m "test non retained"
>
> CLIENT 2 (subscriber):
> 1 connection:
> mosquitto_sub -h <host> -q 2 -i <client_id_sub> -t
> /cro/plantid/0000000/gwid/00001/device_state
> --> received:
> *test non retained*
> 2 connection:
> mosquitto_sub -h <host> -q 2 -i <client_id_sub> -t
> /cro/plantid/0000000/gwid/00001/device_state
> --> received again:
> *test non retained*
> 3 connection:
> mosquitto_sub -h <host> -q 2 -i <client_id_sub> -t
> /cro/plantid/0000000/gwid/00001/device_state
> --> received again:
> *test non retained*
> ...
> and so on
>
> CLIENT 1 (publisher):
> mosquitto_pub -h <host> -q 1 -i <client_id_pub> -r -t
> /cro/plantid/0000000/gwid/00001/device_state -m "test retained"
>
> CLIENT 2 (subscriber):
> 4 connection:
> mosquitto_sub -h <host> -q 2 -i <client_id_sub> -t
> /cro/plantid/0000000/gwid/00001/device_state
> --> received:
> *test retained
> test non retained
> test retained*
> 5 connection:
> mosquitto_sub -h <host> -q 2 -i <client_id_sub> -t
> /cro/plantid/0000000/gwid/00001/device_state
> --> received:
> *test retained
> test non retained
> test retained
> test retained*
> 6 connection:
> mosquitto_sub -h <host> -q 2 -i <client_id_sub> -t
> /cro/plantid/0000000/gwid/00001/device_state
> --> received:
> *test non retained
> test retained
> test retained
> test retained
> test retained*
> ...
> and so on.
>
> If you agree, I would open a bug for this.
>
> Francesco
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-
> tp4721272p4721562.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage

francesco81
Hi Martyn,
I'll be happy to enjoy the IRC chat as soon as I can.
Effectively, your words about the "treating as new subscription" would explain the issue with retained messages.
However there's still something that I don't understand: for example why also the non-retained messages are resent on resubscription. Regardless if retained or not, it seems that all messages persist on queue (or, in case of retained, they persist on address and one more reference is added on queue at each resubscription... I don't know, I'm trying to guess) and are never discarded.
How are they managed by artemis? In a scenario where clients never connect with "cleanSession=false" (that is our usecase), I thought:
- non-retained messages are removed from address (and references from queues) once they are consumed by clients (or immediatly if no client is connected).
- retained messages are removed from address when a new one arrives (and the reference to the new one substitutes the previous on the queue).
Is not so? I'm wrong somewhere?

Again, thanks for your patience.

Francesco
Reply | Threaded
Open this post in threaded view
|

Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage

Martyn Taylor
Francesco,

I think I've identified the cause of this problem.  There were two issues
which are now fixed as part of:
https://github.com/apache/activemq-artemis/pull/1002

I'll get these fixes cherry-picked onto Artemis 1.x stream.

I plan on doing a 1.5.3 (with these changes included) within the next
couple of days.

Cheers
Martyn

On Tue, Feb 7, 2017 at 10:33 AM, francesco81 <[hidden email]>
wrote:

> Hi Martyn,
> I'll be happy to enjoy the IRC chat as soon as I can.
> Effectively, your words about the "treating as new subscription" would
> explain the issue with retained messages.
> However there's still something that I don't understand: for example why
> also the non-retained messages are resent on resubscription. Regardless if
> retained or not, it seems that all messages persist on queue (or, in case
> of
> retained, they persist on address and one more reference is added on queue
> at each resubscription... I don't know, I'm trying to guess) and are never
> discarded.
> How are they managed by artemis? In a scenario where clients never connect
> with "cleanSession=false" (that is our usecase), I thought:
> - non-retained messages are removed from address (and references from
> queues) once they are consumed by clients (or immediatly if no client is
> connected).
> - retained messages are removed from address when a new one arrives (and
> the
> reference to the new one substitutes the previous on the queue).
> Is not so? I'm wrong somewhere?
>
> Again, thanks for your patience.
>
> Francesco
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-
> tp4721272p4721678.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage

francesco81
Hi Martyn,

Thanks a lot for this!

It could save us just in time...


As soon as I can, I'll try to build artemis from your last commit and make some tests.

I'll give you a feedback.


Francesco

________________________________
From: Martyn Taylor <[hidden email]>
Sent: Thursday, February 9, 2017 1:22:52 PM
To: [hidden email]
Subject: Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage

Francesco,

I think I've identified the cause of this problem.  There were two issues
which are now fixed as part of:
https://github.com/apache/activemq-artemis/pull/1002

I'll get these fixes cherry-picked onto Artemis 1.x stream.

I plan on doing a 1.5.3 (with these changes included) within the next
couple of days.

Cheers
Martyn

On Tue, Feb 7, 2017 at 10:33 AM, francesco81 <[hidden email]>
wrote:

> Hi Martyn,
> I'll be happy to enjoy the IRC chat as soon as I can.
> Effectively, your words about the "treating as new subscription" would
> explain the issue with retained messages.
> However there's still something that I don't understand: for example why
> also the non-retained messages are resent on resubscription. Regardless if
> retained or not, it seems that all messages persist on queue (or, in case
> of
> retained, they persist on address and one more reference is added on queue
> at each resubscription... I don't know, I'm trying to guess) and are never
> discarded.
> How are they managed by artemis? In a scenario where clients never connect
> with "cleanSession=false" (that is our usecase), I thought:
> - non-retained messages are removed from address (and references from
> queues) once they are consumed by clients (or immediatly if no client is
> connected).
> - retained messages are removed from address when a new one arrives (and
> the
> reference to the new one substitutes the previous on the queue).
> Is not so? I'm wrong somewhere?
>
> Again, thanks for your patience.
>
> Francesco
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-
> tp4721272p4721678.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

________________________________

Ce message, ainsi que tous les fichiers joints ? ce message, peuvent contenir des informations sensibles et/ ou confidentielles ne devant pas ?tre divulgu?es. Si vous n'?tes pas le destinataire de ce message (ou que vous recevez ce message par erreur), nous vous remercions de le notifier imm?diatement ? son exp?diteur, et de d?truire ce message. Toute copie, divulgation, modification, utilisation ou diffusion, non autoris?e, directe ou indirecte, de tout ou partie de ce message, est strictement interdite.


This e-mail, and any document attached hereby, may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized, direct or indirect, copying, disclosure, distribution or other use of the material or parts thereof is strictly forbidden.
Reply | Threaded
Open this post in threaded view
|

Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage

Martyn Taylor
On Thu, Feb 9, 2017 at 3:54 PM, Francesco PADOVANI <
[hidden email]> wrote:

> Hi Martyn,
>
> Thanks a lot for this!
>
> It could save us just in time...

I hope so.

>
> As soon as I can, I'll try to build artemis from your last commit and make
> some tests.
>
I've created a snapshot build you can test with:

https://repository.apache.org/content/groups/snapshots/org/apache/activemq/apache-artemis/1.6.0-SNAPSHOT/apache-artemis-1.6.0-20170209.161917-15-bin.zip

Please let me know how you get on.

Thanks

> Francesco
>
> ________________________________
> From: Martyn Taylor <[hidden email]>
> Sent: Thursday, February 9, 2017 1:22:52 PM
> To: [hidden email]
> Subject: Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage
>
> Francesco,
>
> I think I've identified the cause of this problem.  There were two issues
> which are now fixed as part of:
> https://github.com/apache/activemq-artemis/pull/1002
>
> I'll get these fixes cherry-picked onto Artemis 1.x stream.
>
> I plan on doing a 1.5.3 (with these changes included) within the next
> couple of days.
>
> Cheers
> Martyn
>
> On Tue, Feb 7, 2017 at 10:33 AM, francesco81 <
> [hidden email]>
> wrote:
>
> > Hi Martyn,
> > I'll be happy to enjoy the IRC chat as soon as I can.
> > Effectively, your words about the "treating as new subscription" would
> > explain the issue with retained messages.
> > However there's still something that I don't understand: for example why
> > also the non-retained messages are resent on resubscription. Regardless
> if
> > retained or not, it seems that all messages persist on queue (or, in case
> > of
> > retained, they persist on address and one more reference is added on
> queue
> > at each resubscription... I don't know, I'm trying to guess) and are
> never
> > discarded.
> > How are they managed by artemis? In a scenario where clients never
> connect
> > with "cleanSession=false" (that is our usecase), I thought:
> > - non-retained messages are removed from address (and references from
> > queues) once they are consumed by clients (or immediatly if no client is
> > connected).
> > - retained messages are removed from address when a new one arrives (and
> > the
> > reference to the new one substitutes the previous on the queue).
> > Is not so? I'm wrong somewhere?
> >
> > Again, thanks for your patience.
> >
> > Francesco
> >
> >
> >
> > --
> > View this message in context: http://activemq.2283324.n4.
> > nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-
> > tp4721272p4721678.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
> ________________________________
>
> Ce message, ainsi que tous les fichiers joints ? ce message, peuvent
> contenir des informations sensibles et/ ou confidentielles ne devant pas
> ?tre divulgu?es. Si vous n'?tes pas le destinataire de ce message (ou que
> vous recevez ce message par erreur), nous vous remercions de le notifier
> imm?diatement ? son exp?diteur, et de d?truire ce message. Toute copie,
> divulgation, modification, utilisation ou diffusion, non autoris?e, directe
> ou indirecte, de tout ou partie de ce message, est strictement interdite.
>
>
> This e-mail, and any document attached hereby, may contain confidential
> and/or privileged information. If you are not the intended recipient (or
> have received this e-mail in error) please notify the sender immediately
> and destroy this e-mail. Any unauthorized, direct or indirect, copying,
> disclosure, distribution or other use of the material or parts thereof is
> strictly forbidden.
>
Reply | Threaded
Open this post in threaded view
|

Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage

francesco81
Hi Martyn,

we're testing your 1.6.0 snapshot.

The issues related to retained messages ACK and durable queues seem ok now. Great!

I noticed that this snapshot doesn't include the fix for weird chars when messages are recovered from journal: do you think it will be included in 1.5.3 release?


And last question: with artemis 1.5.1 we were able to connect also by using clientID longer than 23 chars. Now, with 1.6.0 (and maybe also with 1.5.2 ...I admit I've not yet tested it) we can't.

It is changed something about clientid length support?


Thanks in advance.


Francesco

________________________________
From: Martyn Taylor <[hidden email]>
Sent: Thursday, February 9, 2017 5:36:19 PM
To: [hidden email]
Subject: Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage

On Thu, Feb 9, 2017 at 3:54 PM, Francesco PADOVANI <
[hidden email]> wrote:

> Hi Martyn,
>
> Thanks a lot for this!
>
> It could save us just in time...

I hope so.

>
> As soon as I can, I'll try to build artemis from your last commit and make
> some tests.
>
I've created a snapshot build you can test with:

https://repository.apache.org/content/groups/snapshots/org/apache/activemq/apache-artemis/1.6.0-SNAPSHOT/apache-artemis-1.6.0-20170209.161917-15-bin.zip

Please let me know how you get on.

Thanks

> Francesco
>
> ________________________________
> From: Martyn Taylor <[hidden email]>
> Sent: Thursday, February 9, 2017 1:22:52 PM
> To: [hidden email]
> Subject: Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage
>
> Francesco,
>
> I think I've identified the cause of this problem.  There were two issues
> which are now fixed as part of:
> https://github.com/apache/activemq-artemis/pull/1002
>
> I'll get these fixes cherry-picked onto Artemis 1.x stream.
>
> I plan on doing a 1.5.3 (with these changes included) within the next
> couple of days.
>
> Cheers
> Martyn
>
> On Tue, Feb 7, 2017 at 10:33 AM, francesco81 <
> [hidden email]>
> wrote:
>
> > Hi Martyn,
> > I'll be happy to enjoy the IRC chat as soon as I can.
> > Effectively, your words about the "treating as new subscription" would
> > explain the issue with retained messages.
> > However there's still something that I don't understand: for example why
> > also the non-retained messages are resent on resubscription. Regardless
> if
> > retained or not, it seems that all messages persist on queue (or, in case
> > of
> > retained, they persist on address and one more reference is added on
> queue
> > at each resubscription... I don't know, I'm trying to guess) and are
> never
> > discarded.
> > How are they managed by artemis? In a scenario where clients never
> connect
> > with "cleanSession=false" (that is our usecase), I thought:
> > - non-retained messages are removed from address (and references from
> > queues) once they are consumed by clients (or immediatly if no client is
> > connected).
> > - retained messages are removed from address when a new one arrives (and
> > the
> > reference to the new one substitutes the previous on the queue).
> > Is not so? I'm wrong somewhere?
> >
> > Again, thanks for your patience.
> >
> > Francesco
> >
> >
> >
> > --
> > View this message in context: http://activemq.2283324.n4.
> > nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-
> > tp4721272p4721678.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
> ________________________________
>
> Ce message, ainsi que tous les fichiers joints ? ce message, peuvent
> contenir des informations sensibles et/ ou confidentielles ne devant pas
> ?tre divulgu?es. Si vous n'?tes pas le destinataire de ce message (ou que
> vous recevez ce message par erreur), nous vous remercions de le notifier
> imm?diatement ? son exp?diteur, et de d?truire ce message. Toute copie,
> divulgation, modification, utilisation ou diffusion, non autoris?e, directe
> ou indirecte, de tout ou partie de ce message, est strictement interdite.
>
>
> This e-mail, and any document attached hereby, may contain confidential
> and/or privileged information. If you are not the intended recipient (or
> have received this e-mail in error) please notify the sender immediately
> and destroy this e-mail. Any unauthorized, direct or indirect, copying,
> disclosure, distribution or other use of the material or parts thereof is
> strictly forbidden.
>

________________________________

Ce message, ainsi que tous les fichiers joints ? ce message, peuvent contenir des informations sensibles et/ ou confidentielles ne devant pas ?tre divulgu?es. Si vous n'?tes pas le destinataire de ce message (ou que vous recevez ce message par erreur), nous vous remercions de le notifier imm?diatement ? son exp?diteur, et de d?truire ce message. Toute copie, divulgation, modification, utilisation ou diffusion, non autoris?e, directe ou indirecte, de tout ou partie de ce message, est strictement interdite.


This e-mail, and any document attached hereby, may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized, direct or indirect, copying, disclosure, distribution or other use of the material or parts thereof is strictly forbidden.
Reply | Threaded
Open this post in threaded view
|

Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage

francesco81
Sorry Martyn,

about the clientID lenght: it was a problem of my mqtt Client which had the compatibility with mqtt 3.1.1 disabled (stupid me).


Instead, I confirm to you the lack of fixing for weird chars on messages recovered from journal.


Francesco

________________________________
From: Francesco PADOVANI
Sent: Friday, February 10, 2017 2:58:19 PM
To: [hidden email]
Subject: Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage


Hi Martyn,

we're testing your 1.6.0 snapshot.

The issues related to retained messages ACK and durable queues seem ok now. Great!

I noticed that this snapshot doesn't include the fix for weird chars when messages are recovered from journal: do you think it will be included in 1.5.3 release?


And last question: with artemis 1.5.1 we were able to connect also by using clientID longer than 23 chars. Now, with 1.6.0 (and maybe also with 1.5.2 ...I admit I've not yet tested it) we can't.

It is changed something about clientid length support?


Thanks in advance.


Francesco

________________________________
From: Martyn Taylor <[hidden email]>
Sent: Thursday, February 9, 2017 5:36:19 PM
To: [hidden email]
Subject: Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage

On Thu, Feb 9, 2017 at 3:54 PM, Francesco PADOVANI <
[hidden email]> wrote:

> Hi Martyn,
>
> Thanks a lot for this!
>
> It could save us just in time...

I hope so.

>
> As soon as I can, I'll try to build artemis from your last commit and make
> some tests.
>
I've created a snapshot build you can test with:

https://repository.apache.org/content/groups/snapshots/org/apache/activemq/apache-artemis/1.6.0-SNAPSHOT/apache-artemis-1.6.0-20170209.161917-15-bin.zip

Please let me know how you get on.

Thanks

> Francesco
>
> ________________________________
> From: Martyn Taylor <[hidden email]>
> Sent: Thursday, February 9, 2017 1:22:52 PM
> To: [hidden email]
> Subject: Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage
>
> Francesco,
>
> I think I've identified the cause of this problem.  There were two issues
> which are now fixed as part of:
> https://github.com/apache/activemq-artemis/pull/1002
>
> I'll get these fixes cherry-picked onto Artemis 1.x stream.
>
> I plan on doing a 1.5.3 (with these changes included) within the next
> couple of days.
>
> Cheers
> Martyn
>
> On Tue, Feb 7, 2017 at 10:33 AM, francesco81 <
> [hidden email]>
> wrote:
>
> > Hi Martyn,
> > I'll be happy to enjoy the IRC chat as soon as I can.
> > Effectively, your words about the "treating as new subscription" would
> > explain the issue with retained messages.
> > However there's still something that I don't understand: for example why
> > also the non-retained messages are resent on resubscription. Regardless
> if
> > retained or not, it seems that all messages persist on queue (or, in case
> > of
> > retained, they persist on address and one more reference is added on
> queue
> > at each resubscription... I don't know, I'm trying to guess) and are
> never
> > discarded.
> > How are they managed by artemis? In a scenario where clients never
> connect
> > with "cleanSession=false" (that is our usecase), I thought:
> > - non-retained messages are removed from address (and references from
> > queues) once they are consumed by clients (or immediatly if no client is
> > connected).
> > - retained messages are removed from address when a new one arrives (and
> > the
> > reference to the new one substitutes the previous on the queue).
> > Is not so? I'm wrong somewhere?
> >
> > Again, thanks for your patience.
> >
> > Francesco
> >
> >
> >
> > --
> > View this message in context: http://activemq.2283324.n4.
> > nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-
> > tp4721272p4721678.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> >
>
> ________________________________
>
> Ce message, ainsi que tous les fichiers joints ? ce message, peuvent
> contenir des informations sensibles et/ ou confidentielles ne devant pas
> ?tre divulgu?es. Si vous n'?tes pas le destinataire de ce message (ou que
> vous recevez ce message par erreur), nous vous remercions de le notifier
> imm?diatement ? son exp?diteur, et de d?truire ce message. Toute copie,
> divulgation, modification, utilisation ou diffusion, non autoris?e, directe
> ou indirecte, de tout ou partie de ce message, est strictement interdite.
>
>
> This e-mail, and any document attached hereby, may contain confidential
> and/or privileged information. If you are not the intended recipient (or
> have received this e-mail in error) please notify the sender immediately
> and destroy this e-mail. Any unauthorized, direct or indirect, copying,
> disclosure, distribution or other use of the material or parts thereof is
> strictly forbidden.
>

________________________________

Ce message, ainsi que tous les fichiers joints ? ce message, peuvent contenir des informations sensibles et/ ou confidentielles ne devant pas ?tre divulgu?es. Si vous n'?tes pas le destinataire de ce message (ou que vous recevez ce message par erreur), nous vous remercions de le notifier imm?diatement ? son exp?diteur, et de d?truire ce message. Toute copie, divulgation, modification, utilisation ou diffusion, non autoris?e, directe ou indirecte, de tout ou partie de ce message, est strictement interdite.


This e-mail, and any document attached hereby, may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized, direct or indirect, copying, disclosure, distribution or other use of the material or parts thereof is strictly forbidden.
Reply | Threaded
Open this post in threaded view
|

Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage

Martyn Taylor
Looks like I made a bit of a git foobar when cherry-picking the commits
from master on the 1.x branch.  I will rectify this.  Thanks for pointing
this out.

On Fri, Feb 10, 2017 at 2:06 PM, Francesco PADOVANI <
[hidden email]> wrote:

> Sorry Martyn,
>
> about the clientID lenght: it was a problem of my mqtt Client which had
> the compatibility with mqtt 3.1.1 disabled (stupid me).
>
>
> Instead, I confirm to you the lack of fixing for weird chars on messages
> recovered from journal.
>
>
> Francesco
>
> ________________________________
> From: Francesco PADOVANI
> Sent: Friday, February 10, 2017 2:58:19 PM
> To: [hidden email]
> Subject: Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage
>
>
> Hi Martyn,
>
> we're testing your 1.6.0 snapshot.
>
> The issues related to retained messages ACK and durable queues seem ok
> now. Great!
>
> I noticed that this snapshot doesn't include the fix for weird chars when
> messages are recovered from journal: do you think it will be included in
> 1.5.3 release?
>
>
> And last question: with artemis 1.5.1 we were able to connect also by
> using clientID longer than 23 chars. Now, with 1.6.0 (and maybe also with
> 1.5.2 ...I admit I've not yet tested it) we can't.
>
> It is changed something about clientid length support?
>
>
> Thanks in advance.
>
>
> Francesco
>
> ________________________________
> From: Martyn Taylor <[hidden email]>
> Sent: Thursday, February 9, 2017 5:36:19 PM
> To: [hidden email]
> Subject: Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage
>
> On Thu, Feb 9, 2017 at 3:54 PM, Francesco PADOVANI <
> [hidden email]> wrote:
>
> > Hi Martyn,
> >
> > Thanks a lot for this!
> >
> > It could save us just in time...
>
> I hope so.
>
> >
> > As soon as I can, I'll try to build artemis from your last commit and
> make
> > some tests.
> >
> I've created a snapshot build you can test with:
>
> https://repository.apache.org/content/groups/snapshots/org/
> apache/activemq/apache-artemis/1.6.0-SNAPSHOT/apache-
> artemis-1.6.0-20170209.161917-15-bin.zip
>
> Please let me know how you get on.
>
> Thanks
>
> > Francesco
> >
> > ________________________________
> > From: Martyn Taylor <[hidden email]>
> > Sent: Thursday, February 9, 2017 1:22:52 PM
> > To: [hidden email]
> > Subject: Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage
> >
> > Francesco,
> >
> > I think I've identified the cause of this problem.  There were two issues
> > which are now fixed as part of:
> > https://github.com/apache/activemq-artemis/pull/1002
> >
> > I'll get these fixes cherry-picked onto Artemis 1.x stream.
> >
> > I plan on doing a 1.5.3 (with these changes included) within the next
> > couple of days.
> >
> > Cheers
> > Martyn
> >
> > On Tue, Feb 7, 2017 at 10:33 AM, francesco81 <
> > [hidden email]>
> > wrote:
> >
> > > Hi Martyn,
> > > I'll be happy to enjoy the IRC chat as soon as I can.
> > > Effectively, your words about the "treating as new subscription" would
> > > explain the issue with retained messages.
> > > However there's still something that I don't understand: for example
> why
> > > also the non-retained messages are resent on resubscription. Regardless
> > if
> > > retained or not, it seems that all messages persist on queue (or, in
> case
> > > of
> > > retained, they persist on address and one more reference is added on
> > queue
> > > at each resubscription... I don't know, I'm trying to guess) and are
> > never
> > > discarded.
> > > How are they managed by artemis? In a scenario where clients never
> > connect
> > > with "cleanSession=false" (that is our usecase), I thought:
> > > - non-retained messages are removed from address (and references from
> > > queues) once they are consumed by clients (or immediatly if no client
> is
> > > connected).
> > > - retained messages are removed from address when a new one arrives
> (and
> > > the
> > > reference to the new one substitutes the previous on the queue).
> > > Is not so? I'm wrong somewhere?
> > >
> > > Again, thanks for your patience.
> > >
> > > Francesco
> > >
> > >
> > >
> > > --
> > > View this message in context: http://activemq.2283324.n4.
> > > nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-
> > > tp4721272p4721678.html
> > > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> > >
> >
> > ________________________________
> >
> > Ce message, ainsi que tous les fichiers joints ? ce message, peuvent
> > contenir des informations sensibles et/ ou confidentielles ne devant pas
> > ?tre divulgu?es. Si vous n'?tes pas le destinataire de ce message (ou que
> > vous recevez ce message par erreur), nous vous remercions de le notifier
> > imm?diatement ? son exp?diteur, et de d?truire ce message. Toute copie,
> > divulgation, modification, utilisation ou diffusion, non autoris?e,
> directe
> > ou indirecte, de tout ou partie de ce message, est strictement interdite.
> >
> >
> > This e-mail, and any document attached hereby, may contain confidential
> > and/or privileged information. If you are not the intended recipient (or
> > have received this e-mail in error) please notify the sender immediately
> > and destroy this e-mail. Any unauthorized, direct or indirect, copying,
> > disclosure, distribution or other use of the material or parts thereof is
> > strictly forbidden.
> >
>
> ________________________________
>
> Ce message, ainsi que tous les fichiers joints ? ce message, peuvent
> contenir des informations sensibles et/ ou confidentielles ne devant pas
> ?tre divulgu?es. Si vous n'?tes pas le destinataire de ce message (ou que
> vous recevez ce message par erreur), nous vous remercions de le notifier
> imm?diatement ? son exp?diteur, et de d?truire ce message. Toute copie,
> divulgation, modification, utilisation ou diffusion, non autoris?e, directe
> ou indirecte, de tout ou partie de ce message, est strictement interdite.
>
>
> This e-mail, and any document attached hereby, may contain confidential
> and/or privileged information. If you are not the intended recipient (or
> have received this e-mail in error) please notify the sender immediately
> and destroy this e-mail. Any unauthorized, direct or indirect, copying,
> disclosure, distribution or other use of the material or parts thereof is
> strictly forbidden.
>
Reply | Threaded
Open this post in threaded view
|

Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage

francesco81
Hi Martyn,
good Monday.

Unfortunately, I've an update about our tests on artemis snapshot 1.6 you g=
ave me last week.
It seems that after few time of use (sometimes an hour ...other times less)=
 it stops to accept connections. Resources on server are totally ok: we hav=
e about 40/50 clients, each one who sends 2/3 messages per minute., and cpu=
 and mem consumption is really low.
But I don't know why... at a certain point, clients can't connect any more =
to the Artemis broker. And I don't see any error on log (set to DEBUG level=
). Simply, whoever try to connect, stays in "CONNECT" status forever (perfo=
rming retry attempts).
I've done a thread dump of JVM in that situation: you can find it attached. You can see there are few threads that are BLOCKED, waiting to =
lock at "org.apache.activemq.artemis.core.protocol.mqtt.MQTTConnectionManag=
er.connect". I don't know if this can help to find the problem. Unfortunate=
ly I'm not yet able to identify the exact steps to reproduce it. I'm curren=
tly committed exactly on this. I'll let you know as sthreaddumps.tdumpoon as I can.

Meanwhile: have you any idea?

Francesco
Reply | Threaded
Open this post in threaded view
|

Re: ARTEMIS: bad-performance behaviour after 7-10 days of usage

Martyn Taylor
Hi Francesco,

Thanks for the Thread dump.  I think I know what the issue is, I'll get it
fixed today.  This fix won't make the 1.5.3 release but I can provide you
with another snapshot with all the required fixes.

We can try to do another revision release soon after including this fix.

Cheers
Martyn

On Mon, Feb 13, 2017 at 11:03 AM, francesco81 <[hidden email]
> wrote:

> Hi Martyn,
> good Monday.
>
> Unfortunately, I've an update about our tests on artemis snapshot 1.6 you
> g=
> ave me last week.
> It seems that after few time of use (sometimes an hour ...other times
> less)=
>  it stops to accept connections. Resources on server are totally ok: we
> hav=
> e about 40/50 clients, each one who sends 2/3 messages per minute., and
> cpu=
>  and mem consumption is really low.
> But I don't know why... at a certain point, clients can't connect any more
> =
> to the Artemis broker. And I don't see any error on log (set to DEBUG
> level=
> ). Simply, whoever try to connect, stays in "CONNECT" status forever
> (perfo=
> rming retry attempts).
> I've done a thread dump of JVM in that situation: you can find it attached.
> You can see there are few threads that are BLOCKED, waiting to =
> lock at "org.apache.activemq.artemis.core.protocol.mqtt.
> MQTTConnectionManag=
> er.connect". I don't know if this can help to find the problem.
> Unfortunate=
> ly I'm not yet able to identify the exact steps to reproduce it. I'm
> curren=
> tly committed exactly on this. I'll let you know as s threaddumps.tdump
> <http://activemq.2283324.n4.nabble.com/file/n4721908/threaddumps.tdump>
> oon
> as I can.
>
> Meanwhile: have you any idea?
>
> Francesco
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/ARTEMIS-bad-performance-behaviour-after-7-10-days-of-usage-
> tp4721272p4721908.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
12