Help needed. About message loss in distributed queue in network of brokers environment

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Help needed. About message loss in distributed queue in network of brokers environment

Bryan.Shaw
We have a distributed architecture with two brokers connnected to each other through WAN.

We configured it using network of brokers.
One producer sends messages to a distributed queue named "aqueue" on broker A.
One consumer consume messages from the queue "aqueue" on broker B.

I thought activemq is responsible for the reliability of the message transfer.
Which means the when message are sent to aqueue at broker A, the messages are first store on broker A's persistent store and then forwarded to broker B's persistent store and afterwards consumed by the consumer on broker B. If network between these two brokers are not stable, it is activemq's job to ensure the message is delivered and delivered only once.

But in real production environment, we found message loss when network between the two brokers is down.

I need proffesional help to address this issue. Is this a bug of activemq or a configuration issue?
Can anybody give me some advice to avoid this kind of message loss?
Reply | Threaded
Open this post in threaded view
|

Re: Help needed. About message loss in distributed queue in network of brokers environment

Bryan.Shaw
I discovered that the message transfer between network of brokers  is handled by DemandForwardingBridgeSupport class and there is some comment on async send mode will be assumed between brokers if the producer send the message to the queue using async send mode which will cause small probability of message loss.

I am now testing the sync send mode on the producer side, hope this can resolve my problem.

Bryan.Shaw wrote
We have a distributed architecture with two brokers connnected to each other through WAN.

We configured it using network of brokers.
One producer sends messages to a distributed queue named "aqueue" on broker A.
One consumer consume messages from the queue "aqueue" on broker B.

I thought activemq is responsible for the reliability of the message transfer.
Which means the when message are sent to aqueue at broker A, the messages are first store on broker A's persistent store and then forwarded to broker B's persistent store and afterwards consumed by the consumer on broker B. If network between these two brokers are not stable, it is activemq's job to ensure the message is delivered and delivered only once.

But in real production environment, we found message loss when network between the two brokers is down.

I need proffesional help to address this issue. Is this a bug of activemq or a configuration issue?
Can anybody give me some advice to avoid this kind of message loss?
Reply | Threaded
Open this post in threaded view
|

Re: Help needed. About message loss in distributed queue in network of brokers environment

Bryan.Shaw
It is sad that the activemq message sending is default to sync send.
So this issue is still open.

Please provide help!

Bryan.Shaw wrote
I discovered that the message transfer between network of brokers  is handled by DemandForwardingBridgeSupport class and there is some comment on async send mode will be assumed between brokers if the producer send the message to the queue using async send mode which will cause small probability of message loss.

I am now testing the sync send mode on the producer side, hope this can resolve my problem.

Bryan.Shaw wrote
We have a distributed architecture with two brokers connnected to each other through WAN.

We configured it using network of brokers.
One producer sends messages to a distributed queue named "aqueue" on broker A.
One consumer consume messages from the queue "aqueue" on broker B.

I thought activemq is responsible for the reliability of the message transfer.
Which means the when message are sent to aqueue at broker A, the messages are first store on broker A's persistent store and then forwarded to broker B's persistent store and afterwards consumed by the consumer on broker B. If network between these two brokers are not stable, it is activemq's job to ensure the message is delivered and delivered only once.

But in real production environment, we found message loss when network between the two brokers is down.

I need proffesional help to address this issue. Is this a bug of activemq or a configuration issue?
Can anybody give me some advice to avoid this kind of message loss?
Reply | Threaded
Open this post in threaded view
|

Re: Help needed. About message loss in distributed queue in network of brokers environment

rajdavies
This is currently only the case if you are using duplex connections -  
or the message was originally sent using async send from the client  
(e.g. non-persistent) - as the network tries to maintain the  same QOS  
as used by the original producer

cheers,

Rob

On 10 Jul 2008, at 10:34, Bryan.Shaw wrote:

>
> It is sad that the activemq message sending is default to sync send.
> So this issue is still open.
>
> Please provide help!
>
>
> Bryan.Shaw wrote:
>>
>> I discovered that the message transfer between network of brokers  is
>> handled by DemandForwardingBridgeSupport class and there is some  
>> comment
>> on async send mode will be assumed between brokers if the producer  
>> send
>> the message to the queue using async send mode which will cause small
>> probability of message loss.
>>
>> I am now testing the sync send mode on the producer side, hope this  
>> can
>> resolve my problem.
>>
>>
>> Bryan.Shaw wrote:
>>>
>>> We have a distributed architecture with two brokers connnected to  
>>> each
>>> other through WAN.
>>>
>>> We configured it using network of brokers.
>>> One producer sends messages to a distributed queue named "aqueue" on
>>> broker A.
>>> One consumer consume messages from the queue "aqueue" on broker B.
>>>
>>> I thought activemq is responsible for the reliability of the message
>>> transfer.
>>> Which means the when message are sent to aqueue at broker A, the  
>>> messages
>>> are first store on broker A's persistent store and then forwarded to
>>> broker B's persistent store and afterwards consumed by the  
>>> consumer on
>>> broker B. If network between these two brokers are not stable, it is
>>> activemq's job to ensure the message is delivered and delivered only
>>> once.
>>>
>>> But in real production environment, we found message loss when  
>>> network
>>> between the two brokers is down.
>>>
>>> I need proffesional help to address this issue. Is this a bug of  
>>> activemq
>>> or a configuration issue?
>>> Can anybody give me some advice to avoid this kind of message loss?
>>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Help-needed.-About-message-loss-in-distributed-queue-in-network-of-brokers-environment-tp18377630p18379174.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Reply | Threaded
Open this post in threaded view
|

Re: Help needed. About message loss in distributed queue in network of brokers environment

Bryan.Shaw
I am not using async send from the client.
But, I am using duplex connection because one of the two brokers do not have fixed IP Address.
How can I avoid using duplex connnection if only one IP is fixed?


rajdavies wrote
This is currently only the case if you are using duplex connections -  
or the message was originally sent using async send from the client  
(e.g. non-persistent) - as the network tries to maintain the  same QOS  
as used by the original producer

cheers,

Rob

On 10 Jul 2008, at 10:34, Bryan.Shaw wrote:

>
> It is sad that the activemq message sending is default to sync send.
> So this issue is still open.
>
> Please provide help!
>
>
> Bryan.Shaw wrote:
>>
>> I discovered that the message transfer between network of brokers  is
>> handled by DemandForwardingBridgeSupport class and there is some  
>> comment
>> on async send mode will be assumed between brokers if the producer  
>> send
>> the message to the queue using async send mode which will cause small
>> probability of message loss.
>>
>> I am now testing the sync send mode on the producer side, hope this  
>> can
>> resolve my problem.
>>
>>
>> Bryan.Shaw wrote:
>>>
>>> We have a distributed architecture with two brokers connnected to  
>>> each
>>> other through WAN.
>>>
>>> We configured it using network of brokers.
>>> One producer sends messages to a distributed queue named "aqueue" on
>>> broker A.
>>> One consumer consume messages from the queue "aqueue" on broker B.
>>>
>>> I thought activemq is responsible for the reliability of the message
>>> transfer.
>>> Which means the when message are sent to aqueue at broker A, the  
>>> messages
>>> are first store on broker A's persistent store and then forwarded to
>>> broker B's persistent store and afterwards consumed by the  
>>> consumer on
>>> broker B. If network between these two brokers are not stable, it is
>>> activemq's job to ensure the message is delivered and delivered only
>>> once.
>>>
>>> But in real production environment, we found message loss when  
>>> network
>>> between the two brokers is down.
>>>
>>> I need proffesional help to address this issue. Is this a bug of  
>>> activemq
>>> or a configuration issue?
>>> Can anybody give me some advice to avoid this kind of message loss?
>>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Help-needed.-About-message-loss-in-distributed-queue-in-network-of-brokers-environment-tp18377630p18379174.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Help needed. About message loss in distributed queue in network of brokers environment

rajdavies
You can't avoid duplex in this case unfortunately - will see if can  
get this fixed for 5.2.
Would you like to raise an issue for it - so I don't forget ?

On 10 Jul 2008, at 17:01, Bryan.Shaw wrote:

>
> I am not using async send from the client.
> But, I am using duplex connection because one of the two brokers do  
> not have
> fixed IP Address.
> How can I avoid using duplex connnection if only one IP is fixed?
>
>
>
> rajdavies wrote:
>>
>> This is currently only the case if you are using duplex connections -
>> or the message was originally sent using async send from the client
>> (e.g. non-persistent) - as the network tries to maintain the  same  
>> QOS
>> as used by the original producer
>>
>> cheers,
>>
>> Rob
>>
>> On 10 Jul 2008, at 10:34, Bryan.Shaw wrote:
>>
>>>
>>> It is sad that the activemq message sending is default to sync send.
>>> So this issue is still open.
>>>
>>> Please provide help!
>>>
>>>
>>> Bryan.Shaw wrote:
>>>>
>>>> I discovered that the message transfer between network of  
>>>> brokers  is
>>>> handled by DemandForwardingBridgeSupport class and there is some
>>>> comment
>>>> on async send mode will be assumed between brokers if the producer
>>>> send
>>>> the message to the queue using async send mode which will cause  
>>>> small
>>>> probability of message loss.
>>>>
>>>> I am now testing the sync send mode on the producer side, hope this
>>>> can
>>>> resolve my problem.
>>>>
>>>>
>>>> Bryan.Shaw wrote:
>>>>>
>>>>> We have a distributed architecture with two brokers connnected to
>>>>> each
>>>>> other through WAN.
>>>>>
>>>>> We configured it using network of brokers.
>>>>> One producer sends messages to a distributed queue named  
>>>>> "aqueue" on
>>>>> broker A.
>>>>> One consumer consume messages from the queue "aqueue" on broker B.
>>>>>
>>>>> I thought activemq is responsible for the reliability of the  
>>>>> message
>>>>> transfer.
>>>>> Which means the when message are sent to aqueue at broker A, the
>>>>> messages
>>>>> are first store on broker A's persistent store and then  
>>>>> forwarded to
>>>>> broker B's persistent store and afterwards consumed by the
>>>>> consumer on
>>>>> broker B. If network between these two brokers are not stable,  
>>>>> it is
>>>>> activemq's job to ensure the message is delivered and delivered  
>>>>> only
>>>>> once.
>>>>>
>>>>> But in real production environment, we found message loss when
>>>>> network
>>>>> between the two brokers is down.
>>>>>
>>>>> I need proffesional help to address this issue. Is this a bug of
>>>>> activemq
>>>>> or a configuration issue?
>>>>> Can anybody give me some advice to avoid this kind of message  
>>>>> loss?
>>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Help-needed.-About-message-loss-in-distributed-queue-in-network-of-brokers-environment-tp18377630p18379174.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Help-needed.-About-message-loss-in-distributed-queue-in-network-of-brokers-environment-tp18377630p18385580.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Reply | Threaded
Open this post in threaded view
|

Re: Help needed. About message loss in distributed queue in network of brokers environment

Bryan.Shaw
I have already reported this issue in JIRA.
The issue key is AMQ-1845.


rajdavies wrote
You can't avoid duplex in this case unfortunately - will see if can  
get this fixed for 5.2.
Would you like to raise an issue for it - so I don't forget ?

On 10 Jul 2008, at 17:01, Bryan.Shaw wrote:

>
> I am not using async send from the client.
> But, I am using duplex connection because one of the two brokers do  
> not have
> fixed IP Address.
> How can I avoid using duplex connnection if only one IP is fixed?
>
>
>
> rajdavies wrote:
>>
>> This is currently only the case if you are using duplex connections -
>> or the message was originally sent using async send from the client
>> (e.g. non-persistent) - as the network tries to maintain the  same  
>> QOS
>> as used by the original producer
>>
>> cheers,
>>
>> Rob
>>
>> On 10 Jul 2008, at 10:34, Bryan.Shaw wrote:
>>
>>>
>>> It is sad that the activemq message sending is default to sync send.
>>> So this issue is still open.
>>>
>>> Please provide help!
>>>
>>>
>>> Bryan.Shaw wrote:
>>>>
>>>> I discovered that the message transfer between network of  
>>>> brokers  is
>>>> handled by DemandForwardingBridgeSupport class and there is some
>>>> comment
>>>> on async send mode will be assumed between brokers if the producer
>>>> send
>>>> the message to the queue using async send mode which will cause  
>>>> small
>>>> probability of message loss.
>>>>
>>>> I am now testing the sync send mode on the producer side, hope this
>>>> can
>>>> resolve my problem.
>>>>
>>>>
>>>> Bryan.Shaw wrote:
>>>>>
>>>>> We have a distributed architecture with two brokers connnected to
>>>>> each
>>>>> other through WAN.
>>>>>
>>>>> We configured it using network of brokers.
>>>>> One producer sends messages to a distributed queue named  
>>>>> "aqueue" on
>>>>> broker A.
>>>>> One consumer consume messages from the queue "aqueue" on broker B.
>>>>>
>>>>> I thought activemq is responsible for the reliability of the  
>>>>> message
>>>>> transfer.
>>>>> Which means the when message are sent to aqueue at broker A, the
>>>>> messages
>>>>> are first store on broker A's persistent store and then  
>>>>> forwarded to
>>>>> broker B's persistent store and afterwards consumed by the
>>>>> consumer on
>>>>> broker B. If network between these two brokers are not stable,  
>>>>> it is
>>>>> activemq's job to ensure the message is delivered and delivered  
>>>>> only
>>>>> once.
>>>>>
>>>>> But in real production environment, we found message loss when
>>>>> network
>>>>> between the two brokers is down.
>>>>>
>>>>> I need proffesional help to address this issue. Is this a bug of
>>>>> activemq
>>>>> or a configuration issue?
>>>>> Can anybody give me some advice to avoid this kind of message  
>>>>> loss?
>>>>>
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Help-needed.-About-message-loss-in-distributed-queue-in-network-of-brokers-environment-tp18377630p18379174.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Help-needed.-About-message-loss-in-distributed-queue-in-network-of-brokers-environment-tp18377630p18385580.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Help needed. About message loss in distributed queue in network of brokers environment

Bryan.Shaw
In reply to this post by rajdavies
I have finished a test with two brokers connected to each other using non-duplex connection.
Broker A have a network connector to broker B and vice versa.
But I still loose message during the transfer when I unplug the network cable, this is exactly the same as I using only one duplex network connector from broker B to broker A.

This is really unacceptable, and I wish you can solve it ASAP.



rajdavies wrote
This is currently only the case if you are using duplex connections -  
or the message was originally sent using async send from the client  
(e.g. non-persistent) - as the network tries to maintain the  same QOS  
as used by the original producer

cheers,

Rob

On 10 Jul 2008, at 10:34, Bryan.Shaw wrote:

>
> It is sad that the activemq message sending is default to sync send.
> So this issue is still open.
>
> Please provide help!
>
>
> Bryan.Shaw wrote:
>>
>> I discovered that the message transfer between network of brokers  is
>> handled by DemandForwardingBridgeSupport class and there is some  
>> comment
>> on async send mode will be assumed between brokers if the producer  
>> send
>> the message to the queue using async send mode which will cause small
>> probability of message loss.
>>
>> I am now testing the sync send mode on the producer side, hope this  
>> can
>> resolve my problem.
>>
>>
>> Bryan.Shaw wrote:
>>>
>>> We have a distributed architecture with two brokers connnected to  
>>> each
>>> other through WAN.
>>>
>>> We configured it using network of brokers.
>>> One producer sends messages to a distributed queue named "aqueue" on
>>> broker A.
>>> One consumer consume messages from the queue "aqueue" on broker B.
>>>
>>> I thought activemq is responsible for the reliability of the message
>>> transfer.
>>> Which means the when message are sent to aqueue at broker A, the  
>>> messages
>>> are first store on broker A's persistent store and then forwarded to
>>> broker B's persistent store and afterwards consumed by the  
>>> consumer on
>>> broker B. If network between these two brokers are not stable, it is
>>> activemq's job to ensure the message is delivered and delivered only
>>> once.
>>>
>>> But in real production environment, we found message loss when  
>>> network
>>> between the two brokers is down.
>>>
>>> I need proffesional help to address this issue. Is this a bug of  
>>> activemq
>>> or a configuration issue?
>>> Can anybody give me some advice to avoid this kind of message loss?
>>>
>>
>>
>
> --
> View this message in context: http://www.nabble.com/Help-needed.-About-message-loss-in-distributed-queue-in-network-of-brokers-environment-tp18377630p18379174.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Help needed. About message loss in distributed queue in network of brokers environment

Bryan.Shaw
I am deadly waiting for solution or a work around to this problem.
But till now that issue created on JIRA is still open and not assigned.
Reply | Threaded
Open this post in threaded view
|

Re: Help needed. About message loss in distributed queue in network of brokers environment

gtully
I will get a chance (hour or so)  to look at this later today and see
if I can reproduce and/or diagnose.

2008/7/25 Bryan.Shaw <[hidden email]>:
>
> I am deadly waiting for solution or a work around to this problem.
> But till now that issue created on JIRA is still open and not assigned.
> --
> View this message in context: http://www.nabble.com/Help-needed.-About-message-loss-in-distributed-queue-in-network-of-brokers-environment-tp18377630p18646207.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
>