Apache Artemis - Stress-test time

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
30 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Apache Artemis - Stress-test time

alessandro.zannini@bticino.it
Here attached the output of "jstack -F" command.
jstack.zip
Reply | Threaded
Open this post in threaded view
|

Re: Apache Artemis - Stress-test time

clebertsuconic
only thing I see on the thread dump is reained message is doing a
query on the queue every time:

 - org.apache.activemq.artemis.api.core.SimpleString.split(char)
@bci=100, line=314 (Compiled frame)

 - org.apache.activemq.artemis.core.postoffice.impl.AddressImpl.<init>(org.apache.activemq.artemis.api.core.SimpleString,
org.apache.activemq.artemis.core.config.WildcardConfiguration)
@bci=31, line=48 (Compiled frame)

 - org.apache.activemq.artemis.core.postoffice.impl.SimpleAddressManager.getMatchingBindings(org.apache.activemq.artemis.api.core.SimpleString)
@bci=77, line=121 (Compiled frame)

 - org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.getMatchingBindings(org.apache.activemq.artemis.api.core.SimpleString)
@bci=5, line=656 (Compiled frame)

 - org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.bindingQuery(org.apache.activemq.artemis.api.core.SimpleString)
@bci=110, line=722 (Compiled frame)

 - org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.executeBindingQuery(org.apache.activemq.artemis.api.core.SimpleString)
@bci=9, line=730 (Compiled frame)

 - org.apache.activemq.artemis.core.protocol.mqtt.MQTTRetainMessageManager.addRetainedMessagesToQueue(org.apache.activemq.artemis.core.server.Queue,
java.lang.String) @bci=27, line=74 (Compiled frame)

 -


@Mtaylor: any way to avoid it?

On Mon, Apr 10, 2017 at 11:42 AM, [hidden email]
<[hidden email]> wrote:
> Here attached the output of "jstack -F" command.
> jstack.zip <http://activemq.2283324.n4.nabble.com/file/n4724787/jstack.zip>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Apache-Artemis-Stress-test-time-tp4723999p4724787.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: Apache Artemis - Stress-test time

Martyn Taylor
Clebert: Yes we could optimize on this area of the code to avoid the
bindings query.  To determine whether or not this is the root cause of the
problem experience by Francesco and Alessandro, we just need to follow the
same test without using retained messages.

Francesco, Alessandro is this something you could test for us?

Could you also let us know if staggering the subscribe helps.  Like try
testing connecting 100 clients at a time, wait a while then attach another
100.  Once you have all your clients connected does the system perform as
normal?  (this will help us determine whether or not the bottleneck is the
subscribe or just normal function with that many consumers).

Another quick q, are your subscriptions matching most messages.  e.g. are
you using "#" or similar.  If using "#" or subscriptions that match most
messages I'd expect the performance to drop, since the broker needs to
iterate through every subscription queue.  If you have 20,000 subscription
queues matching every message you might see perf decrease.

Any more information on your use case would be helpful in diagnosing the
bottleneck

Thanks




On Mon, Apr 10, 2017 at 5:33 PM, Clebert Suconic <[hidden email]>
wrote:

> only thing I see on the thread dump is reained message is doing a
> query on the queue every time:
>
>  - org.apache.activemq.artemis.api.core.SimpleString.split(char)
> @bci=100, line=314 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.postoffice.impl.
> AddressImpl.<init>(org.apache.activemq.artemis.api.core.SimpleString,
> org.apache.activemq.artemis.core.config.WildcardConfiguration)
> @bci=31, line=48 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.postoffice.impl.SimpleAddressManager.
> getMatchingBindings(org.apache.activemq.artemis.api.core.SimpleString)
> @bci=77, line=121 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.
> getMatchingBindings(org.apache.activemq.artemis.api.core.SimpleString)
> @bci=5, line=656 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.
> bindingQuery(org.apache.activemq.artemis.api.core.SimpleString)
> @bci=110, line=722 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.
> executeBindingQuery(org.apache.activemq.artemis.api.core.SimpleString)
> @bci=9, line=730 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.protocol.mqtt.
> MQTTRetainMessageManager.addRetainedMessagesToQueue(
> org.apache.activemq.artemis.core.server.Queue,
> java.lang.String) @bci=27, line=74 (Compiled frame)
>
>  -
>
>
> @Mtaylor: any way to avoid it?
>
> On Mon, Apr 10, 2017 at 11:42 AM, [hidden email]
> <[hidden email]> wrote:
> > Here attached the output of "jstack -F" command.
> > jstack.zip <http://activemq.2283324.n4.nabble.com/file/n4724787/
> jstack.zip>
> >
> >
> >
> > --
> > View this message in context: http://activemq.2283324.n4.
> nabble.com/Apache-Artemis-Stress-test-time-tp4723999p4724787.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
>
> --
> Clebert Suconic
>
Reply | Threaded
Open this post in threaded view
|

Re: Apache Artemis - Stress-test time

alessandro.zannini@bticino.it
This post was updated on .
Hello,

we've made another test , just making some subscriptions (so, no retained messages involved) .

Nothing changed: the cpu has reached very quickly 100% of usage!

Attached the new jstack output.

Looking at jstack output it seems that artemis tries to check on retained messages also if there aren't retained messages ...

Can you confirm this "strange" behaviour of Artemis ?

jstack_20170411_2.log
Reply | Threaded
Open this post in threaded view
|

Re: Apache Artemis - Stress-test time

alessandro.zannini@bticino.it
@Mtaylor
@clebert

Any news for us ?
Thanks.
Reply | Threaded
Open this post in threaded view
|

Re: Apache Artemis - Stress-test time

clebertsuconic
In reply to this post by alessandro.zannini@bticino.it
I am very confused here...

We have a small issue on quering the queues... but from what I saw the
CPU issue was on your usage of whatever logger framework you were
using. I'm not sure what we can do there.




On Tue, Apr 11, 2017 at 11:11 AM, [hidden email]
<[hidden email]> wrote:

> Hello,
>
> we've made another test , just making some subscriptions (so, no retained messages involved) .
>
> Nothing changed: the cpu has reached very quickly 100% of usage!
>
> Attached the new jstack output.
>
> Looking at jstack output it seems that artemis tries to check on retained messages also if there aren't retained messages ...
>
> Can you confirm this "strange" behaviour of Artemis ?
>
>
>
>
>
> Alessandro ZANNINI
> Project Leader Portale My Home e Sviluppo Applicazioni WEB
> Tel: +(39)0332272966 Ext 9966
> Indirizzo: V.le Borri 231 Varese 21100 IT
> email: [hidden email]
> WebSite: www.bticino.it<http://www.bticino.it>
> <http://www.bticino.it>[1491549876074_OutlookEmoji-1485765781489_logo_bticino.resized.png]
> [X]
> ________________________________
> Please consider your environmental responsibility before printing this Email
> ________________________________
>
>
> ________________________________
> Da: Martyn Taylor [via ActiveMQ] <[hidden email]>
> Inviato: martedì 11 aprile 2017 12.40
> A: Alessandro ZANNINI
> Oggetto: Re: Apache Artemis - Stress-test time
>
> Clebert: Yes we could optimize on this area of the code to avoid the
> bindings query.  To determine whether or not this is the root cause of the
> problem experience by Francesco and Alessandro, we just need to follow the
> same test without using retained messages.
>
> Francesco, Alessandro is this something you could test for us?
>
> Could you also let us know if staggering the subscribe helps.  Like try
> testing connecting 100 clients at a time, wait a while then attach another
> 100.  Once you have all your clients connected does the system perform as
> normal?  (this will help us determine whether or not the bottleneck is the
> subscribe or just normal function with that many consumers).
>
> Another quick q, are your subscriptions matching most messages.  e.g. are
> you using "#" or similar.  If using "#" or subscriptions that match most
> messages I'd expect the performance to drop, since the broker needs to
> iterate through every subscription queue.  If you have 20,000 subscription
> queues matching every message you might see perf decrease.
>
> Any more information on your use case would be helpful in diagnosing the
> bottleneck
>
> Thanks
>
>
>
>
> On Mon, Apr 10, 2017 at 5:33 PM, Clebert Suconic <[hidden email]</user/SendEmail.jtp?type=node&node=4724831&i=0>>
> wrote:
>
>> only thing I see on the thread dump is reained message is doing a
>> query on the queue every time:
>>
>>  - org.apache.activemq.artemis.api.core.SimpleString.split(char)
>> @bci=100, line=314 (Compiled frame)
>>
>>  - org.apache.activemq.artemis.core.postoffice.impl.
>> AddressImpl.<init>(org.apache.activemq.artemis.api.core.SimpleString,
>> org.apache.activemq.artemis.core.config.WildcardConfiguration)
>> @bci=31, line=48 (Compiled frame)
>>
>>  - org.apache.activemq.artemis.core.postoffice.impl.SimpleAddressManager.
>> getMatchingBindings(org.apache.activemq.artemis.api.core.SimpleString)
>> @bci=77, line=121 (Compiled frame)
>>
>>  - org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.
>> getMatchingBindings(org.apache.activemq.artemis.api.core.SimpleString)
>> @bci=5, line=656 (Compiled frame)
>>
>>  - org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.
>> bindingQuery(org.apache.activemq.artemis.api.core.SimpleString)
>> @bci=110, line=722 (Compiled frame)
>>
>>  - org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.
>> executeBindingQuery(org.apache.activemq.artemis.api.core.SimpleString)
>> @bci=9, line=730 (Compiled frame)
>>
>>  - org.apache.activemq.artemis.core.protocol.mqtt.
>> MQTTRetainMessageManager.addRetainedMessagesToQueue(
>> org.apache.activemq.artemis.core.server.Queue,
>> java.lang.String) @bci=27, line=74 (Compiled frame)
>>
>>  -
>>
>>
>> @Mtaylor: any way to avoid it?
>>
>> On Mon, Apr 10, 2017 at 11:42 AM, [hidden email]</user/SendEmail.jtp?type=node&node=4724831&i=1>
>> <[hidden email]</user/SendEmail.jtp?type=node&node=4724831&i=2>> wrote:
>> > Here attached the output of "jstack -F" command.
>> > jstack.zip <http://activemq.2283324.n4.nabble.com/file/n4724787/
>> jstack.zip>
>> >
>> >
>> >
>> > --
>> > View this message in context: http://activemq.2283324.n4.
>> nabble.com/Apache-Artemis-Stress-test-time-tp4723999p4724787.html
>> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>>
>> --
>> Clebert Suconic
>>
>
>
> ________________________________
> If you reply to this email, your message will be added to the discussion below:
> http://activemq.2283324.n4.nabble.com/Apache-Artemis-Stress-test-time-tp4723999p4724831.html
> To unsubscribe from Apache Artemis - Stress-test time, click here<
> NAML<
http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
>
> ________________________________
>
> Ce message, ainsi que tous les fichiers joints à ce message, peuvent contenir des informations sensibles et/ ou confidentielles ne devant pas être divulguées. Si vous n'êtes pas le destinataire de ce message (ou que vous recevez ce message par erreur), nous vous remercions de le notifier immédiatement à son expéditeur, et de détruire ce message. Toute copie, divulgation, modification, utilisation ou diffusion, non autorisée, directe ou indirecte, de tout ou partie de ce message, est strictement interdite.
>
>
> This e-mail, and any document attached hereby, may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized, direct or indirect, copying, disclosure, distribution or other use of the material or parts thereof is strictly forbidden.
>
>
> OutlookEmoji-1491549876074_OutlookEmoji-1485765781489_logo_bticino.resized.png.png (2K) <http://activemq.2283324.n4.nabble.com/attachment/4724845/0/OutlookEmoji-1491549876074_OutlookEmoji-1485765781489_logo_bticino.resized.png.png>
> jstack_20170411_2.log (547K) <http://activemq.2283324.n4.nabble.com/attachment/4724845/1/jstack_20170411_2.log>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.nabble.com/Apache-Artemis-Stress-test-time-tp4723999p4724845.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.



--
Clebert Suconic
Reply | Threaded
Open this post in threaded view
|

Re: Apache Artemis - Stress-test time

alessandro.zannini@bticino.it
clebertsuconic wrote
I am very confused here...
We have a small issue on quering the queues... but from what I saw the
CPU issue was on your usage of whatever logger framework you were
using. I'm not sure what we can do there.
Hi,
we use a custom security plugin that loads security roles (for each topic) on startup; during subscription phase (when cpu usage grows) our custom plugin is not used (in jstack we can see only RedisPlugin$UpdateThread.run() wich is a listener on change in our redis server);
We use standard logging (like that in LegacyLDAPSecuritySettingPlugin): what make you think our classes cause the cpu usage growing?

If you need other information, just let me know!

Thanks for your support!
Reply | Threaded
Open this post in threaded view
|

Re: Apache Artemis - Stress-test time

francesco81
In reply to this post by Martyn Taylor
Hi Martyn,
I can confirm you that once we have all our clients connected, the system performs as
normal.

We've done some more tests.
First of all we've disabled our custom plugin for managing redis authentication/authorization. So we started from a 2.1.0 clean installation (build from hash commit 2bcc255f4ca85c179671bb4bd4f0e77bd09e3281).
By testing our usual use case (subscriber clients who subscribe 50 topics each one) we reach a serious performance problem around 8000 connections.
Instead, by trying with subscriber clients who make a single subscription each one (using the wildcard char '#') it seems there are no problems: we've reached 16000 connections (8000 pub and 8000 sub) in about 5 minutes (the time to start clients) with system resources always under control.
It seems that subscriptions are the key: or I'm wrong?
If so, is there a way to optimize the subscription phase? Maybe also by adjusting our configuration settings?

Thanks as usual.

Francesco

________________________________
From: Martyn Taylor <[hidden email]>
Sent: Tuesday, April 11, 2017 12:49:57 PM
To: [hidden email]
Subject: Re: Apache Artemis - Stress-test time

Clebert: Yes we could optimize on this area of the code to avoid the
bindings query.  To determine whether or not this is the root cause of the
problem experience by Francesco and Alessandro, we just need to follow the
same test without using retained messages.

Francesco, Alessandro is this something you could test for us?

Could you also let us know if staggering the subscribe helps.  Like try
testing connecting 100 clients at a time, wait a while then attach another
100.  Once you have all your clients connected does the system perform as
normal?  (this will help us determine whether or not the bottleneck is the
subscribe or just normal function with that many consumers).

Another quick q, are your subscriptions matching most messages.  e.g. are
you using "#" or similar.  If using "#" or subscriptions that match most
messages I'd expect the performance to drop, since the broker needs to
iterate through every subscription queue.  If you have 20,000 subscription
queues matching every message you might see perf decrease.

Any more information on your use case would be helpful in diagnosing the
bottleneck

Thanks




On Mon, Apr 10, 2017 at 5:33 PM, Clebert Suconic <[hidden email]>
wrote:

> only thing I see on the thread dump is reained message is doing a
> query on the queue every time:
>
>  - org.apache.activemq.artemis.api.core.SimpleString.split(char)
> @bci=100, line=314 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.postoffice.impl.
> AddressImpl.<init>(org.apache.activemq.artemis.api.core.SimpleString,
> org.apache.activemq.artemis.core.config.WildcardConfiguration)
> @bci=31, line=48 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.postoffice.impl.SimpleAddressManager.
> getMatchingBindings(org.apache.activemq.artemis.api.core.SimpleString)
> @bci=77, line=121 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.
> getMatchingBindings(org.apache.activemq.artemis.api.core.SimpleString)
> @bci=5, line=656 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.
> bindingQuery(org.apache.activemq.artemis.api.core.SimpleString)
> @bci=110, line=722 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.
> executeBindingQuery(org.apache.activemq.artemis.api.core.SimpleString)
> @bci=9, line=730 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.protocol.mqtt.
> MQTTRetainMessageManager.addRetainedMessagesToQueue(
> org.apache.activemq.artemis.core.server.Queue,
> java.lang.String) @bci=27, line=74 (Compiled frame)
>
>  -
>
>
> @Mtaylor: any way to avoid it?
>
> On Mon, Apr 10, 2017 at 11:42 AM, [hidden email]
> <[hidden email]> wrote:
> > Here attached the output of "jstack -F" command.
> > jstack.zip <http://activemq.2283324.n4.nabble.com/file/n4724787/
> jstack.zip>
> >
> >
> >
> > --
> > View this message in context: http://activemq.2283324.n4.
> nabble.com/Apache-Artemis-Stress-test-time-tp4723999p4724787.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
>
> --
> Clebert Suconic
>

________________________________

Ce message, ainsi que tous les fichiers joints ? ce message, peuvent contenir des informations sensibles et/ ou confidentielles ne devant pas ?tre divulgu?es. Si vous n'?tes pas le destinataire de ce message (ou que vous recevez ce message par erreur), nous vous remercions de le notifier imm?diatement ? son exp?diteur, et de d?truire ce message. Toute copie, divulgation, modification, utilisation ou diffusion, non autoris?e, directe ou indirecte, de tout ou partie de ce message, est strictement interdite.


This e-mail, and any document attached hereby, may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized, direct or indirect, copying, disclosure, distribution or other use of the material or parts thereof is strictly forbidden.
Reply | Threaded
Open this post in threaded view
|

Re: Apache Artemis - Stress-test time

francesco81
Martyn, sorry, one more thing,
for the previous test we didn't use security: we assigned the same role to all user and the role had all permissions on all addresses/queues (#).
Now we'd like to try by using security, but we're facing an issue when a client publish on a topic which another client has subscribed: the latter is disconnected due to permission violation error:

"
09:55:58,213 DEBUG [org.apache.activemq.artemis.core.protocol.mqtt] Error processing Control Packet, Disconnecting Client: ActiveMQSecurityException[errorType=SECU
RITY_EXCEPTION message=AMQ119032: User: 38316 does not have permission='CONSUME' on address $sys.mqtt.queue.qos2.franzsub.$sys.mqtt.queue.qos2.franzsub]
        at org.apache.activemq.artemis.core.security.impl.SecurityStoreImpl.check(SecurityStoreImpl.java:201) [artemis-server-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.securityCheck(ServerSessionImpl.java:403) [artemis-server-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSH
OT]
        at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.createConsumer(ServerSessionImpl.java:441) [artemis-server-2.1.0-SNAPSHOT.jar:2.1.0-SNAPS
HOT]
        at org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.createManagementConsumer(MQTTPublishManager.java:86) [artemis-mqtt-protocol-2.1.0-SNAP
SHOT.jar:]
        at org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.getManagementAddress(MQTTPublishManager.java:192) [artemis-mqtt-protocol-2.1.0-SNAPSHO
T.jar:]
        at org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.handlePubRec(MQTTPublishManager.java:201) [artemis-mqtt-protocol-2.1.0-SNAPSHOT.jar:]
        at org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.handlePubrec(MQTTProtocolHandler.java:235) [artemis-mqtt-protocol-2.1.0-SNAPSHOT.jar:
]
...
"

I see, you already resolved a similar bug (ARTEMIS-990, "User: Customer does not have permission='CREATE_DURABLE_QUEUE' on address $sys.mqtt.queue.qos2"). Nevertheles, we have the above problem on version we are using (2.1.o build starting from master, commit 310f953c790380b65d2f02081a1888309dc93ab5). And the below workaround

<security-setting match="$sys.mqtt.#">
<permission type="createDurableQueue" roles="<MQTT Client Roles Here>"/>
<permission type="deleteDurableQueue" roles="<MQTT Client Roles Here>"/>
<permission type="consume" roles="<MQTT Client Roles Here>"/>
<permission type="send" roles="<MQTT Client Roles Here>"/>
</security-setting>

seems not working.
Any suggestion? Where I'm wrong?

Thanks

Francesco





________________________________
From: Francesco PADOVANI <[hidden email]>
Sent: Wednesday, April 19, 2017 2:20:58 PM
To: [hidden email]
Subject: Re: Apache Artemis - Stress-test time

Hi Martyn,
I can confirm you that once we have all our clients connected, the system performs as
normal.

We've done some more tests.
First of all we've disabled our custom plugin for managing redis authentication/authorization. So we started from a 2.1.0 clean installation (build from hash commit 2bcc255f4ca85c179671bb4bd4f0e77bd09e3281).
By testing our usual use case (subscriber clients who subscribe 50 topics each one) we reach a serious performance problem around 8000 connections.
Instead, by trying with subscriber clients who make a single subscription each one (using the wildcard char '#') it seems there are no problems: we've reached 16000 connections (8000 pub and 8000 sub) in about 5 minutes (the time to start clients) with system resources always under control.
It seems that subscriptions are the key: or I'm wrong?
If so, is there a way to optimize the subscription phase? Maybe also by adjusting our configuration settings?

Thanks as usual.

Francesco

________________________________
From: Martyn Taylor <[hidden email]>
Sent: Tuesday, April 11, 2017 12:49:57 PM
To: [hidden email]
Subject: Re: Apache Artemis - Stress-test time

Clebert: Yes we could optimize on this area of the code to avoid the
bindings query.  To determine whether or not this is the root cause of the
problem experience by Francesco and Alessandro, we just need to follow the
same test without using retained messages.

Francesco, Alessandro is this something you could test for us?

Could you also let us know if staggering the subscribe helps.  Like try
testing connecting 100 clients at a time, wait a while then attach another
100.  Once you have all your clients connected does the system perform as
normal?  (this will help us determine whether or not the bottleneck is the
subscribe or just normal function with that many consumers).

Another quick q, are your subscriptions matching most messages.  e.g. are
you using "#" or similar.  If using "#" or subscriptions that match most
messages I'd expect the performance to drop, since the broker needs to
iterate through every subscription queue.  If you have 20,000 subscription
queues matching every message you might see perf decrease.

Any more information on your use case would be helpful in diagnosing the
bottleneck

Thanks




On Mon, Apr 10, 2017 at 5:33 PM, Clebert Suconic <[hidden email]>
wrote:

> only thing I see on the thread dump is reained message is doing a
> query on the queue every time:
>
>  - org.apache.activemq.artemis.api.core.SimpleString.split(char)
> @bci=100, line=314 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.postoffice.impl.
> AddressImpl.<init>(org.apache.activemq.artemis.api.core.SimpleString,
> org.apache.activemq.artemis.core.config.WildcardConfiguration)
> @bci=31, line=48 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.postoffice.impl.SimpleAddressManager.
> getMatchingBindings(org.apache.activemq.artemis.api.core.SimpleString)
> @bci=77, line=121 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.
> getMatchingBindings(org.apache.activemq.artemis.api.core.SimpleString)
> @bci=5, line=656 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.
> bindingQuery(org.apache.activemq.artemis.api.core.SimpleString)
> @bci=110, line=722 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.
> executeBindingQuery(org.apache.activemq.artemis.api.core.SimpleString)
> @bci=9, line=730 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.protocol.mqtt.
> MQTTRetainMessageManager.addRetainedMessagesToQueue(
> org.apache.activemq.artemis.core.server.Queue,
> java.lang.String) @bci=27, line=74 (Compiled frame)
>
>  -
>
>
> @Mtaylor: any way to avoid it?
>
> On Mon, Apr 10, 2017 at 11:42 AM, [hidden email]
> <[hidden email]> wrote:
> > Here attached the output of "jstack -F" command.
> > jstack.zip <http://activemq.2283324.n4.nabble.com/file/n4724787/
> jstack.zip>
> >
> >
> >
> > --
> > View this message in context: http://activemq.2283324.n4.
> nabble.com/Apache-Artemis-Stress-test-time-tp4723999p4724787.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
>
> --
> Clebert Suconic
>

________________________________

Ce message, ainsi que tous les fichiers joints ? ce message, peuvent contenir des informations sensibles et/ ou confidentielles ne devant pas ?tre divulgu?es. Si vous n'?tes pas le destinataire de ce message (ou que vous recevez ce message par erreur), nous vous remercions de le notifier imm?diatement ? son exp?diteur, et de d?truire ce message. Toute copie, divulgation, modification, utilisation ou diffusion, non autoris?e, directe ou indirecte, de tout ou partie de ce message, est strictement interdite.


This e-mail, and any document attached hereby, may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized, direct or indirect, copying, disclosure, distribution or other use of the material or parts thereof is strictly forbidden.

________________________________

Ce message, ainsi que tous les fichiers joints ? ce message, peuvent contenir des informations sensibles et/ ou confidentielles ne devant pas ?tre divulgu?es. Si vous n'?tes pas le destinataire de ce message (ou que vous recevez ce message par erreur), nous vous remercions de le notifier imm?diatement ? son exp?diteur, et de d?truire ce message. Toute copie, divulgation, modification, utilisation ou diffusion, non autoris?e, directe ou indirecte, de tout ou partie de ce message, est strictement interdite.


This e-mail, and any document attached hereby, may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized, direct or indirect, copying, disclosure, distribution or other use of the material or parts thereof is strictly forbidden.
Reply | Threaded
Open this post in threaded view
|

Re: Apache Artemis - Stress-test time

francesco81
A question: why the logout method of the jaas interface, is never invoked? I expected it was called in case of disconnection of a client. It has been never implemented?


Francesco


________________________________
From: Francesco PADOVANI <[hidden email]>
Sent: Friday, April 21, 2017 12:04:45 PM
To: [hidden email]
Subject: Re: Apache Artemis - Stress-test time

Martyn, sorry, one more thing,
for the previous test we didn't use security: we assigned the same role to all user and the role had all permissions on all addresses/queues (#).
Now we'd like to try by using security, but we're facing an issue when a client publish on a topic which another client has subscribed: the latter is disconnected due to permission violation error:

"
09:55:58,213 DEBUG [org.apache.activemq.artemis.core.protocol.mqtt] Error processing Control Packet, Disconnecting Client: ActiveMQSecurityException[errorType=SECU
RITY_EXCEPTION message=AMQ119032: User: 38316 does not have permission='CONSUME' on address $sys.mqtt.queue.qos2.franzsub.$sys.mqtt.queue.qos2.franzsub]
        at org.apache.activemq.artemis.core.security.impl.SecurityStoreImpl.check(SecurityStoreImpl.java:201) [artemis-server-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.securityCheck(ServerSessionImpl.java:403) [artemis-server-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSH
OT]
        at org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.createConsumer(ServerSessionImpl.java:441) [artemis-server-2.1.0-SNAPSHOT.jar:2.1.0-SNAPS
HOT]
        at org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.createManagementConsumer(MQTTPublishManager.java:86) [artemis-mqtt-protocol-2.1.0-SNAP
SHOT.jar:]
        at org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.getManagementAddress(MQTTPublishManager.java:192) [artemis-mqtt-protocol-2.1.0-SNAPSHO
T.jar:]
        at org.apache.activemq.artemis.core.protocol.mqtt.MQTTPublishManager.handlePubRec(MQTTPublishManager.java:201) [artemis-mqtt-protocol-2.1.0-SNAPSHOT.jar:]
        at org.apache.activemq.artemis.core.protocol.mqtt.MQTTProtocolHandler.handlePubrec(MQTTProtocolHandler.java:235) [artemis-mqtt-protocol-2.1.0-SNAPSHOT.jar:
]
...
"

I see, you already resolved a similar bug (ARTEMIS-990, "User: Customer does not have permission='CREATE_DURABLE_QUEUE' on address $sys.mqtt.queue.qos2"). Nevertheles, we have the above problem on version we are using (2.1.o build starting from master, commit 310f953c790380b65d2f02081a1888309dc93ab5). And the below workaround

<security-setting match="$sys.mqtt.#">
<permission type="createDurableQueue" roles="<MQTT Client Roles Here>"/>
<permission type="deleteDurableQueue" roles="<MQTT Client Roles Here>"/>
<permission type="consume" roles="<MQTT Client Roles Here>"/>
<permission type="send" roles="<MQTT Client Roles Here>"/>
</security-setting>

seems not working.
Any suggestion? Where I'm wrong?

Thanks

Francesco





________________________________
From: Francesco PADOVANI <[hidden email]>
Sent: Wednesday, April 19, 2017 2:20:58 PM
To: [hidden email]
Subject: Re: Apache Artemis - Stress-test time

Hi Martyn,
I can confirm you that once we have all our clients connected, the system performs as
normal.

We've done some more tests.
First of all we've disabled our custom plugin for managing redis authentication/authorization. So we started from a 2.1.0 clean installation (build from hash commit 2bcc255f4ca85c179671bb4bd4f0e77bd09e3281).
By testing our usual use case (subscriber clients who subscribe 50 topics each one) we reach a serious performance problem around 8000 connections.
Instead, by trying with subscriber clients who make a single subscription each one (using the wildcard char '#') it seems there are no problems: we've reached 16000 connections (8000 pub and 8000 sub) in about 5 minutes (the time to start clients) with system resources always under control.
It seems that subscriptions are the key: or I'm wrong?
If so, is there a way to optimize the subscription phase? Maybe also by adjusting our configuration settings?

Thanks as usual.

Francesco

________________________________
From: Martyn Taylor <[hidden email]>
Sent: Tuesday, April 11, 2017 12:49:57 PM
To: [hidden email]
Subject: Re: Apache Artemis - Stress-test time

Clebert: Yes we could optimize on this area of the code to avoid the
bindings query.  To determine whether or not this is the root cause of the
problem experience by Francesco and Alessandro, we just need to follow the
same test without using retained messages.

Francesco, Alessandro is this something you could test for us?

Could you also let us know if staggering the subscribe helps.  Like try
testing connecting 100 clients at a time, wait a while then attach another
100.  Once you have all your clients connected does the system perform as
normal?  (this will help us determine whether or not the bottleneck is the
subscribe or just normal function with that many consumers).

Another quick q, are your subscriptions matching most messages.  e.g. are
you using "#" or similar.  If using "#" or subscriptions that match most
messages I'd expect the performance to drop, since the broker needs to
iterate through every subscription queue.  If you have 20,000 subscription
queues matching every message you might see perf decrease.

Any more information on your use case would be helpful in diagnosing the
bottleneck

Thanks




On Mon, Apr 10, 2017 at 5:33 PM, Clebert Suconic <[hidden email]>
wrote:

> only thing I see on the thread dump is reained message is doing a
> query on the queue every time:
>
>  - org.apache.activemq.artemis.api.core.SimpleString.split(char)
> @bci=100, line=314 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.postoffice.impl.
> AddressImpl.<init>(org.apache.activemq.artemis.api.core.SimpleString,
> org.apache.activemq.artemis.core.config.WildcardConfiguration)
> @bci=31, line=48 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.postoffice.impl.SimpleAddressManager.
> getMatchingBindings(org.apache.activemq.artemis.api.core.SimpleString)
> @bci=77, line=121 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.postoffice.impl.PostOfficeImpl.
> getMatchingBindings(org.apache.activemq.artemis.api.core.SimpleString)
> @bci=5, line=656 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl.
> bindingQuery(org.apache.activemq.artemis.api.core.SimpleString)
> @bci=110, line=722 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.server.impl.ServerSessionImpl.
> executeBindingQuery(org.apache.activemq.artemis.api.core.SimpleString)
> @bci=9, line=730 (Compiled frame)
>
>  - org.apache.activemq.artemis.core.protocol.mqtt.
> MQTTRetainMessageManager.addRetainedMessagesToQueue(
> org.apache.activemq.artemis.core.server.Queue,
> java.lang.String) @bci=27, line=74 (Compiled frame)
>
>  -
>
>
> @Mtaylor: any way to avoid it?
>
> On Mon, Apr 10, 2017 at 11:42 AM, [hidden email]
> <[hidden email]> wrote:
> > Here attached the output of "jstack -F" command.
> > jstack.zip <http://activemq.2283324.n4.nabble.com/file/n4724787/
> jstack.zip>
> >
> >
> >
> > --
> > View this message in context: http://activemq.2283324.n4.
> nabble.com/Apache-Artemis-Stress-test-time-tp4723999p4724787.html
> > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>
>
> --
> Clebert Suconic
>

________________________________

Ce message, ainsi que tous les fichiers joints ? ce message, peuvent contenir des informations sensibles et/ ou confidentielles ne devant pas ?tre divulgu?es. Si vous n'?tes pas le destinataire de ce message (ou que vous recevez ce message par erreur), nous vous remercions de le notifier imm?diatement ? son exp?diteur, et de d?truire ce message. Toute copie, divulgation, modification, utilisation ou diffusion, non autoris?e, directe ou indirecte, de tout ou partie de ce message, est strictement interdite.


This e-mail, and any document attached hereby, may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized, direct or indirect, copying, disclosure, distribution or other use of the material or parts thereof is strictly forbidden.

________________________________

Ce message, ainsi que tous les fichiers joints ? ce message, peuvent contenir des informations sensibles et/ ou confidentielles ne devant pas ?tre divulgu?es. Si vous n'?tes pas le destinataire de ce message (ou que vous recevez ce message par erreur), nous vous remercions de le notifier imm?diatement ? son exp?diteur, et de d?truire ce message. Toute copie, divulgation, modification, utilisation ou diffusion, non autoris?e, directe ou indirecte, de tout ou partie de ce message, est strictement interdite.


This e-mail, and any document attached hereby, may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized, direct or indirect, copying, disclosure, distribution or other use of the material or parts thereof is strictly forbidden.

________________________________

Ce message, ainsi que tous les fichiers joints ? ce message, peuvent contenir des informations sensibles et/ ou confidentielles ne devant pas ?tre divulgu?es. Si vous n'?tes pas le destinataire de ce message (ou que vous recevez ce message par erreur), nous vous remercions de le notifier imm?diatement ? son exp?diteur, et de d?truire ce message. Toute copie, divulgation, modification, utilisation ou diffusion, non autoris?e, directe ou indirecte, de tout ou partie de ce message, est strictement interdite.


This e-mail, and any document attached hereby, may contain confidential and/or privileged information. If you are not the intended recipient (or have received this e-mail in error) please notify the sender immediately and destroy this e-mail. Any unauthorized, direct or indirect, copying, disclosure, distribution or other use of the material or parts thereof is strictly forbidden.
12