Quantcast

Duplicate Messages in the Message Store

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Duplicate Messages in the Message Store

srivmeka
Hi,
Am facing issue with broker trying to insert duplicate message into activemq_msgs.

This was observed during
1) Broker fail over
2) Data center/Application fail over where the primary broker changes per data center, note there is no broker shutdown here.

I am using JDBC persistent Adapter and
<broker brokerName="partner-broker" xmlns="http://activemq.apache.org/schema/core">
                <persistenceAdapter>
                        <jdbcPersistenceAdapter dataSource="#amq-dataSource" useDatabaseLock="false"/>
                </persistenceAdapter>
                <transportConnectors>
                       
                        <transportConnector name="openwire1" uri="tcp://${activemq.broker1.hostname}:${activemq.broker1.portId}?maximumConnections=1000"/>
                </transportConnectors>               
        </broker>

I have difficulty in understanding how broker inserts messages into message queues.
The code in activemq jar does explain that it gets the max(ID) for next sequence generator but when is this really happening? Is there any caching mechanism here? What is the message is already consumed and deleted, how can it get the MAX(ID)?

Thanks,
Sri

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Duplicate Messages in the Message Store

Tim Bain
Based on the doGetLastMessageStoreSequenceId() method in
https://fisheye.apache.org/browse/activemq-6/activemq-jdbc-store/src/main/java/org/apache/activemq/store/jdbc/adapter/DefaultJDBCAdapter.java?r=d54d046b8a8f2e9e5c0a28e1f8c7634b3c8b18e4&r=d54d046b8a8f2e9e5c0a28e1f8c7634b3c8b18e4&r=d54d046b8a8f2e9e5c0a28e1f8c7634b3c8b18e4&r=c5cf03806d646045e01f6cd382504fa73dedfeee&r=d54d046b8a8f2e9e5c0a28e1f8c7634b3c8b18e4&r=d54d046b8a8f2e9e5c0a28e1f8c7634b3c8b18e4&r=d54d046b8a8f2e9e5c0a28e1f8c7634b3c8b18e4&r=d54d046b8a8f2e9e5c0a28e1f8c7634b3c8b18e4,
if all messages have been consumed, acked, and deleted, the ID 0 will be
computed as the max value in the store, so I believe that the next ID used
will be 1.  But I'm not sure that that information helps you investigate
the behavior you're seeing during a failover event.

Are you able to reproduce this behavior reliably, in a minimal test
configuration?  The smoothest path forward would be for you to find a way
to reliably reproduce it and then submit a JIRA entry with the full details
of how to reproduce the problem, so that someone else could figure out
what's going on and address any bug that turns out to be the root cause of
the behavior.

Tim

On Fri, Apr 14, 2017 at 1:51 AM, srivmeka <[hidden email]> wrote:

> Hi,
> Am facing issue with broker trying to insert duplicate message into
> activemq_msgs.
>
> This was observed during
> 1) Broker fail over
> 2) Data center/Application fail over where the primary broker changes per
> data center, note there is no broker shutdown here.
>
> I am using JDBC persistent Adapter and
> <broker brokerName="partner-broker"
> xmlns="http://activemq.apache.org/schema/core">
>                 <persistenceAdapter>
>                         <jdbcPersistenceAdapter
> dataSource="#amq-dataSource"
> useDatabaseLock="false"/>
>                 </persistenceAdapter>
>                 <transportConnectors>
>
>                         <transportConnector name="openwire1"
> uri="tcp://${activemq.broker1.hostname}:${activemq.broker1.
> portId}?maximumConnections=1000"/>
>                 </transportConnectors>
>         </broker>
>
> I have difficulty in understanding how broker inserts messages into message
> queues.
> The code in activemq jar does explain that it gets the max(ID) for next
> sequence generator but when is this really happening? Is there any caching
> mechanism here? What is the message is already consumed and deleted, how
> can
> it get the MAX(ID)?
>
> Thanks,
> Sri
>
>
>
>
>
> --
> View this message in context: http://activemq.2283324.n4.
> nabble.com/Duplicate-Messages-in-the-Message-Store-tp4724959.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Duplicate Messages in the Message Store

srivmeka
This is what i reproduced on lower environments, I have two instances of servicemix 5.3.0 each have one broker bundle. Broker 1 on instancce 1 is primary.

1. Broker1 when up inserted a message in to the queue with ID #2466
2. I brought Broker 1 down and inserted another message via Broker2 expecting the ID to be 2466 +1 (as the message is not consumed yet), but the message inserted was #2209
3. Brought Broker 1 up and inserted another message, it got inserted with #2247
4. My Consumer pulls the messages 100 at a time form queue of same type.  Thus was expecting consumer to pull all 3 messages, but it did consume only the once that inserted via Broker1.

This certainly explains me there is message store cached per Broker, which is not good and causing problems.  
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Duplicate Messages in the Message Store

Tim Bain
Please submit a bug in JIRA for this issue.

Tim

On Apr 18, 2017 1:16 AM, "srivmeka" <[hidden email]> wrote:

This is what i reproduced on lower environments, I have two instances of
servicemix 5.3.0 each have one broker bundle. Broker 1 on instancce 1 is
primary.

1. Broker1 when up inserted a message in to the queue with ID #2466
2. I brought Broker 1 down and inserted another message via Broker2
expecting the ID to be 2466 +1 (as the message is not consumed yet), but the
message inserted was #2209
3. Brought Broker 1 up and inserted another message, it got inserted with
#2247
4. My Consumer pulls the messages 100 at a time form queue of same type.
Thus was expecting consumer to pull all 3 messages, but it did consume only
the once that inserted via Broker1.

This certainly explains me there is message store cached per Broker, which
is not good and causing problems.



--
View this message in context: http://activemq.2283324.n4.
nabble.com/Duplicate-Messages-in-the-Message-Store-tp4724959p4725019.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Loading...