[Artemis] message flow with core bridges

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[Artemis] message flow with core bridges

Nicolae Mihalache
Hello,

I have setup with a core bridge configuration between two artemis servers.
The latency between the servers is 12-13 ms (round trip).
I have a producer that produces about 150 messages per second and my
problem is that  the bridge does not manage to keep up with the producer.
The sender address is configured for paging and the paging directory keeps
increasing in size.

When I run a tcpdump I see that after each tcp send there is a response
received so it looks to me that artemis sender waits for a confirmation
from the receiver before sending the next message.

I also see in the netstat that there is no tcp buffer on the sender, load
average is also very small so I'm very confident that it's not a network
bandwith or cpu load problem.

I tried all kind of configs:
 in the connector configuration  preAcknowledge=true
in the bridge configuration setting a large value for
confirmation-window-size

none of them seems to have any effect.

I'm using artemis core 1.5.4

Any idea?


thanks,
nicolae
Reply | Threaded
Open this post in threaded view
|

Re: [Artemis] message flow with core bridges

Nicolae Mihalache
Some more info:

I enabled logging and I see these messages in the logs indicating that the
sender is blockign after each message sent:
Nov 09 16:07:46.730 [30]
org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl
sendBlocking FINEST:Sending blocking PACKET(SessionSendMessage)[type=71,
channelID=10, packetObject=SessionSendMessage,
message=ServerMessage[messageID=30065833866,durable=false,userID=null,priority=4,
bodySize=1024, timestamp=Thu Nov 09 15:57:01 UTC 2017,expiration=0,
durable=false,
address=aces-sim.tm_dump,properties=TypedProperties[_AMQ_ORIG_MESSAGE_ID=30065833865,_y_uniqueid=1270719294,_AMQ_ORIG_ADDRESS=aces-sim.tm_dump]]@2011757486,
requiresResponse=true]

I tried to follow the sourcecode to see why it's blocking, without success
unfortunately. In
the  org.apache.activemq.artemis.core.server.cluster.impl.BridgeImpl:

https://github.com/apache/activemq-artemis/blob/1.5.4/artemis-server/src/main/java/org/apache/activemq/artemis/core/server/cluster/impl/BridgeImpl.java#L864


the producer session (from the bridge to the remote server) seems to be
created with pre-acknowledgment so I don't understand anything :(


thanks for any hint
nicolae


On Thu, Nov 9, 2017 at 4:28 PM, Nicolae Mihalache <[hidden email]>
wrote:

> Hello,
>
> I have setup with a core bridge configuration between two artemis servers.
> The latency between the servers is 12-13 ms (round trip).
> I have a producer that produces about 150 messages per second and my
> problem is that  the bridge does not manage to keep up with the producer.
> The sender address is configured for paging and the paging directory keeps
> increasing in size.
>
> When I run a tcpdump I see that after each tcp send there is a response
> received so it looks to me that artemis sender waits for a confirmation
> from the receiver before sending the next message.
>
> I also see in the netstat that there is no tcp buffer on the sender, load
> average is also very small so I'm very confident that it's not a network
> bandwith or cpu load problem.
>
> I tried all kind of configs:
>  in the connector configuration  preAcknowledge=true
> in the bridge configuration setting a large value for
> confirmation-window-size
>
> none of them seems to have any effect.
>
> I'm using artemis core 1.5.4
>
> Any idea?
>
>
> thanks,
> nicolae
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [Artemis] message flow with core bridges

Martyn Taylor
Hi Nicolae,

Is broker 2 blocking production to the target address?  This would result
in back pressure on the bridge, meaning that messages will build up in
broker 1.

Logs of the brokers and bridges (with trace enabled) would help diagnose.

Cheers
Martyn




On Thu, Nov 9, 2017 at 4:49 PM, Nicolae Mihalache <[hidden email]>
wrote:

> Some more info:
>
> I enabled logging and I see these messages in the logs indicating that the
> sender is blockign after each message sent:
> Nov 09 16:07:46.730 [30]
> org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl
> sendBlocking FINEST:Sending blocking PACKET(SessionSendMessage)[type=71,
> channelID=10, packetObject=SessionSendMessage,
> message=ServerMessage[messageID=30065833866,durable=
> false,userID=null,priority=4,
> bodySize=1024, timestamp=Thu Nov 09 15:57:01 UTC 2017,expiration=0,
> durable=false,
> address=aces-sim.tm_dump,properties=TypedProperties[_AMQ_ORIG_MESSAGE_ID=
> 30065833865,_y_uniqueid=1270719294,_AMQ_ORIG_ADDRESS=
> aces-sim.tm_dump]]@2011757486,
> requiresResponse=true]
>
> I tried to follow the sourcecode to see why it's blocking, without success
> unfortunately. In
> the  org.apache.activemq.artemis.core.server.cluster.impl.BridgeImpl:
>
> https://github.com/apache/activemq-artemis/blob/1.5.4/
> artemis-server/src/main/java/org/apache/activemq/artemis/
> core/server/cluster/impl/BridgeImpl.java#L864
>
>
> the producer session (from the bridge to the remote server) seems to be
> created with pre-acknowledgment so I don't understand anything :(
>
>
> thanks for any hint
> nicolae
>
>
> On Thu, Nov 9, 2017 at 4:28 PM, Nicolae Mihalache <[hidden email]>
> wrote:
>
> > Hello,
> >
> > I have setup with a core bridge configuration between two artemis
> servers.
> > The latency between the servers is 12-13 ms (round trip).
> > I have a producer that produces about 150 messages per second and my
> > problem is that  the bridge does not manage to keep up with the producer.
> > The sender address is configured for paging and the paging directory
> keeps
> > increasing in size.
> >
> > When I run a tcpdump I see that after each tcp send there is a response
> > received so it looks to me that artemis sender waits for a confirmation
> > from the receiver before sending the next message.
> >
> > I also see in the netstat that there is no tcp buffer on the sender, load
> > average is also very small so I'm very confident that it's not a network
> > bandwith or cpu load problem.
> >
> > I tried all kind of configs:
> >  in the connector configuration  preAcknowledge=true
> > in the bridge configuration setting a large value for
> > confirmation-window-size
> >
> > none of them seems to have any effect.
> >
> > I'm using artemis core 1.5.4
> >
> > Any idea?
> >
> >
> > thanks,
> > nicolae
> >
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: [Artemis] message flow with core bridges

Nicolae Mihalache
No, it was just because I disabled duplicate detection and that made it
switch to an synchronous delivery.
I didn't see that documented anywhere but I found it by debugging the
source code (long life open-source):


https://github.com/apache/activemq-artemis/blob/8886ec29247fdaf1500d5d216015c0cc9ea05f75/artemis-server/src/main/java/org/apache/activemq/artemis/core/server/cluster/ClusterManager.java#L457

On Fri, Nov 10, 2017 at 1:15 PM, Martyn Taylor <[hidden email]> wrote:

> Hi Nicolae,
>
> Is broker 2 blocking production to the target address?  This would result
> in back pressure on the bridge, meaning that messages will build up in
> broker 1.
>
> Logs of the brokers and bridges (with trace enabled) would help diagnose.
>
> Cheers
> Martyn
>
>
>
>
> On Thu, Nov 9, 2017 at 4:49 PM, Nicolae Mihalache <[hidden email]>
> wrote:
>
> > Some more info:
> >
> > I enabled logging and I see these messages in the logs indicating that
> the
> > sender is blockign after each message sent:
> > Nov 09 16:07:46.730 [30]
> > org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl
> > sendBlocking FINEST:Sending blocking PACKET(SessionSendMessage)[type=71,
> > channelID=10, packetObject=SessionSendMessage,
> > message=ServerMessage[messageID=30065833866,durable=
> > false,userID=null,priority=4,
> > bodySize=1024, timestamp=Thu Nov 09 15:57:01 UTC 2017,expiration=0,
> > durable=false,
> > address=aces-sim.tm_dump,properties=TypedProperties[_
> AMQ_ORIG_MESSAGE_ID=
> > 30065833865,_y_uniqueid=1270719294,_AMQ_ORIG_ADDRESS=
> > aces-sim.tm_dump]]@2011757486,
> > requiresResponse=true]
> >
> > I tried to follow the sourcecode to see why it's blocking, without
> success
> > unfortunately. In
> > the  org.apache.activemq.artemis.core.server.cluster.impl.BridgeImpl:
> >
> > https://github.com/apache/activemq-artemis/blob/1.5.4/
> > artemis-server/src/main/java/org/apache/activemq/artemis/
> > core/server/cluster/impl/BridgeImpl.java#L864
> >
> >
> > the producer session (from the bridge to the remote server) seems to be
> > created with pre-acknowledgment so I don't understand anything :(
> >
> >
> > thanks for any hint
> > nicolae
> >
> >
> > On Thu, Nov 9, 2017 at 4:28 PM, Nicolae Mihalache <[hidden email]>
> > wrote:
> >
> > > Hello,
> > >
> > > I have setup with a core bridge configuration between two artemis
> > servers.
> > > The latency between the servers is 12-13 ms (round trip).
> > > I have a producer that produces about 150 messages per second and my
> > > problem is that  the bridge does not manage to keep up with the
> producer.
> > > The sender address is configured for paging and the paging directory
> > keeps
> > > increasing in size.
> > >
> > > When I run a tcpdump I see that after each tcp send there is a response
> > > received so it looks to me that artemis sender waits for a confirmation
> > > from the receiver before sending the next message.
> > >
> > > I also see in the netstat that there is no tcp buffer on the sender,
> load
> > > average is also very small so I'm very confident that it's not a
> network
> > > bandwith or cpu load problem.
> > >
> > > I tried all kind of configs:
> > >  in the connector configuration  preAcknowledge=true
> > > in the bridge configuration setting a large value for
> > > confirmation-window-size
> > >
> > > none of them seems to have any effect.
> > >
> > > I'm using artemis core 1.5.4
> > >
> > > Any idea?
> > >
> > >
> > > thanks,
> > > nicolae
> > >
> > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: [Artemis] message flow with core bridges

Nicolae Mihalache
In reply to this post by Martyn Taylor
No, it was just because I disabled duplicate detection and that made it
switch to an synchronous delivery.
I didn't see that documented anywhere but I found it by debugging the
source code (long life open-source):


https://github.com/apache/activemq-artemis/blob/
8886ec29247fdaf1500d5d216015c0cc9ea05f75/artemis-server/src/
main/java/org/apache/activemq/artemis/core/server/cluster/
ClusterManager.java#L457

On Fri, Nov 10, 2017 at 1:15 PM, Martyn Taylor <[hidden email]> wrote:

> Hi Nicolae,
>
> Is broker 2 blocking production to the target address?  This would result
> in back pressure on the bridge, meaning that messages will build up in
> broker 1.
>
> Logs of the brokers and bridges (with trace enabled) would help diagnose.
>
> Cheers
> Martyn
>
>
>
>
> On Thu, Nov 9, 2017 at 4:49 PM, Nicolae Mihalache <[hidden email]>
> wrote:
>
> > Some more info:
> >
> > I enabled logging and I see these messages in the logs indicating that
> the
> > sender is blockign after each message sent:
> > Nov 09 16:07:46.730 [30]
> > org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl
> > sendBlocking FINEST:Sending blocking PACKET(SessionSendMessage)[type=71,
> > channelID=10, packetObject=SessionSendMessage,
> > message=ServerMessage[messageID=30065833866,durable=
> > false,userID=null,priority=4,
> > bodySize=1024, timestamp=Thu Nov 09 15:57:01 UTC 2017,expiration=0,
> > durable=false,
> > address=aces-sim.tm_dump,properties=TypedProperties[_AMQ_
> ORIG_MESSAGE_ID=
> > 30065833865,_y_uniqueid=1270719294,_AMQ_ORIG_ADDRESS=
> > aces-sim.tm_dump]]@2011757486,
> > requiresResponse=true]
> >
> > I tried to follow the sourcecode to see why it's blocking, without
> success
> > unfortunately. In
> > the  org.apache.activemq.artemis.core.server.cluster.impl.BridgeImpl:
> >
> > https://github.com/apache/activemq-artemis/blob/1.5.4/
> > artemis-server/src/main/java/org/apache/activemq/artemis/
> > core/server/cluster/impl/BridgeImpl.java#L864
> >
> >
> > the producer session (from the bridge to the remote server) seems to be
> > created with pre-acknowledgment so I don't understand anything :(
> >
> >
> > thanks for any hint
> > nicolae
> >
> >
> > On Thu, Nov 9, 2017 at 4:28 PM, Nicolae Mihalache <[hidden email]>
> > wrote:
> >
> > > Hello,
> > >
> > > I have setup with a core bridge configuration between two artemis
> > servers.
> > > The latency between the servers is 12-13 ms (round trip).
> > > I have a producer that produces about 150 messages per second and my
> > > problem is that  the bridge does not manage to keep up with the
> producer.
> > > The sender address is configured for paging and the paging directory
> > keeps
> > > increasing in size.
> > >
> > > When I run a tcpdump I see that after each tcp send there is a response
> > > received so it looks to me that artemis sender waits for a confirmation
> > > from the receiver before sending the next message.
> > >
> > > I also see in the netstat that there is no tcp buffer on the sender,
> load
> > > average is also very small so I'm very confident that it's not a
> network
> > > bandwith or cpu load problem.
> > >
> > > I tried all kind of configs:
> > >  in the connector configuration  preAcknowledge=true
> > > in the bridge configuration setting a large value for
> > > confirmation-window-size
> > >
> > > none of them seems to have any effect.
> > >
> > > I'm using artemis core 1.5.4
> > >
> > > Any idea?
> > >
> > >
> > > thanks,
> > > nicolae
> > >
> > >
> > >
> >
>