[jira] Created: (AMQ-674) Composite Destination persisted messages never get cleaned up, halt message producers

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[jira] Created: (AMQ-674) Composite Destination persisted messages never get cleaned up, halt message producers

JIRA jira@apache.org
Composite Destination persisted messages never get cleaned up, halt message producers
-------------------------------------------------------------------------------------

         Key: AMQ-674
         URL: https://issues.apache.org/activemq/browse/AMQ-674
     Project: ActiveMQ
        Type: Bug

  Components: Broker, Message Store  
    Versions: 3.2.1    
    Reporter: Paul Smith
    Priority: Blocker


well, we've just got bit by something huge.  

Previously we have been shipping messages from producer to consumer via a named Queue 'Index2.EntityIncrementalIndexer', where the machine index2 has a consumer called "EntityIncrementalIndexer" using persisted messages.

The design used Composite destinations so that the destination could be:

"Index1.EntityIncrementalIndexer,Index2.EntityIncrementalIndexer"

and have the 2 hosts get sent the same message simply, and if one goes down, it'll catch up later.  This gave us the ability to have a mirrored configuration, and go tertiary later if we want simply by adding a new name in the composite destation.

With a single name, this works great, been working fine for months.  Yesterday we activated the _true_ composite destination, and both machines started getting their messages fine.  

no problems so far, BUT, 12 hours later we have noticed that the broker has now stopped accepting messages, and a look at the activemq_msgs table shows 33228 messages

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (AMQ-674) Composite Destination persisted messages never get cleaned up, halt message producers

JIRA jira@apache.org
    [ https://issues.apache.org/activemq/browse/AMQ-674?page=comments#action_35963 ]

Paul Smith commented on AMQ-674:
--------------------------------

hmm, it snipped my SQL

select container, count(*) from activemq_msgs group by container

Index2.EntityIncrementalIndexer                                                                                                                                                                                                                            6
Index2.EntityIncrementalIndexer,Index1.EntityIncrementalIndexer                                                                                                                                                                                            33228


activemq.log:

14:45:03 WARN  Queue is full, waiting for it to be dequeued.
14:45:03 WARN  Queue is full, waiting for it to be dequeued.
14:45:03 WARN  Queue is full, waiting for it to be dequeued.
14:45:03 WARN  Queue is full, waiting for it to be dequeued.
14:45:03 WARN  Queue is full, waiting for it to be dequeued.
14:45:03 WARN  memory limit low - forced to remove expired messages: PROCESSED_REQUESTS
14:45:03 WARN  memory limit low - forced to remove expired messages: INCOMING_REQUESTS
14:45:03 WARN  memory limit low - forced to remove expired messages: Index2.EntityFullIndexer
14:45:03 WARN  memory limit low - forced to remove expired messages: Index1.EntityFullIndexer
14:45:03 WARN  memory limit low - forced to remove expired messages: Index2.EntityIncrementalIndexer,Index1.EntityIncrementalIndexer
14:45:03 WARN  memory limit low - forced to remove expired messages: Index1.EntityIncrementalIndexer
14:45:03 WARN  memory limit low - forced to remove expired messages: Index2.EntityIncrementalIndexer
14:45:03 WARN  Queue is full, waiting for it to be dequeued.
14:45:03 WARN  Queue is full, waiting for it to be dequeued.
14:45:03 WARN  Queue is full, waiting for it to be dequeued.
14:45:03 WARN  Queue is full, waiting for it to be dequeued.
14:45:03 WARN  Queue is full, waiting for it to be dequeued.
14:45:03 WARN  Queue is full, waiting for it to be dequeued.
14:45:03 WARN  Queue is full, waiting for it to be dequeued.
14:45:03 WARN  Queue is full, waiting for it to be dequeued.
14:45:03 WARN  Queue is full, waiting for it to be dequeued.
14:45:03 WARN  Queue is full, waiting for it to be dequeued.
14:45:03 WARN  Queue is full, waiting for it to be dequeued.

> Composite Destination persisted messages never get cleaned up, halt message producers
> -------------------------------------------------------------------------------------
>
>          Key: AMQ-674
>          URL: https://issues.apache.org/activemq/browse/AMQ-674
>      Project: ActiveMQ
>         Type: Bug

>   Components: Broker, Message Store
>     Versions: 3.2.1
>     Reporter: Paul Smith
>     Priority: Blocker

>
>
> well, we've just got bit by something huge.  
> Previously we have been shipping messages from producer to consumer via a named Queue 'Index2.EntityIncrementalIndexer', where the machine index2 has a consumer called "EntityIncrementalIndexer" using persisted messages.
> The design used Composite destinations so that the destination could be:
> "Index1.EntityIncrementalIndexer,Index2.EntityIncrementalIndexer"
> and have the 2 hosts get sent the same message simply, and if one goes down, it'll catch up later.  This gave us the ability to have a mirrored configuration, and go tertiary later if we want simply by adding a new name in the composite destation.
> With a single name, this works great, been working fine for months.  Yesterday we activated the _true_ composite destination, and both machines started getting their messages fine.  
> no problems so far, BUT, 12 hours later we have noticed that the broker has now stopped accepting messages, and a look at the activemq_msgs table shows 33228 messages

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (AMQ-674) Composite Destination persisted messages never get cleaned up, halt message producers

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
    [ https://issues.apache.org/activemq/browse/AMQ-674?page=comments#action_35964 ]

Paul Smith commented on AMQ-674:
--------------------------------

We're using SQL 2000 as the JDBC store btw.

> Composite Destination persisted messages never get cleaned up, halt message producers
> -------------------------------------------------------------------------------------
>
>          Key: AMQ-674
>          URL: https://issues.apache.org/activemq/browse/AMQ-674
>      Project: ActiveMQ
>         Type: Bug

>   Components: Broker, Message Store
>     Versions: 3.2.1
>     Reporter: Paul Smith
>     Priority: Blocker

>
>
> well, we've just got bit by something huge.  
> Previously we have been shipping messages from producer to consumer via a named Queue 'Index2.EntityIncrementalIndexer', where the machine index2 has a consumer called "EntityIncrementalIndexer" using persisted messages.
> The design used Composite destinations so that the destination could be:
> "Index1.EntityIncrementalIndexer,Index2.EntityIncrementalIndexer"
> and have the 2 hosts get sent the same message simply, and if one goes down, it'll catch up later.  This gave us the ability to have a mirrored configuration, and go tertiary later if we want simply by adding a new name in the composite destation.
> With a single name, this works great, been working fine for months.  Yesterday we activated the _true_ composite destination, and both machines started getting their messages fine.  
> no problems so far, BUT, 12 hours later we have noticed that the broker has now stopped accepting messages, and a look at the activemq_msgs table shows 33228 messages

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Resolved: (AMQ-674) Composite Destination persisted messages never get cleaned up, halt message producers

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
     [ https://issues.apache.org/activemq/browse/AMQ-674?page=all ]
     
james strachan resolved AMQ-674:
--------------------------------

    Fix Version: 4.0 RC1
     Resolution: Fixed

This is a known issue with the architecture of 3.x; once you hit queue full; the system generally pauses until you consume more messages. You're only option is to increase the RAM limit - but it is generally delaying the inevitable.

We recommend upgrading to 4.x which has this issue resolved

> Composite Destination persisted messages never get cleaned up, halt message producers
> -------------------------------------------------------------------------------------
>
>          Key: AMQ-674
>          URL: https://issues.apache.org/activemq/browse/AMQ-674
>      Project: ActiveMQ
>         Type: Bug

>   Components: Broker, Message Store
>     Versions: 3.2.1
>     Reporter: Paul Smith
>     Priority: Blocker
>      Fix For: 4.0 RC1

>
>
> well, we've just got bit by something huge.  
> Previously we have been shipping messages from producer to consumer via a named Queue 'Index2.EntityIncrementalIndexer', where the machine index2 has a consumer called "EntityIncrementalIndexer" using persisted messages.
> The design used Composite destinations so that the destination could be:
> "Index1.EntityIncrementalIndexer,Index2.EntityIncrementalIndexer"
> and have the 2 hosts get sent the same message simply, and if one goes down, it'll catch up later.  This gave us the ability to have a mirrored configuration, and go tertiary later if we want simply by adding a new name in the composite destation.
> With a single name, this works great, been working fine for months.  Yesterday we activated the _true_ composite destination, and both machines started getting their messages fine.  
> no problems so far, BUT, 12 hours later we have noticed that the broker has now stopped accepting messages, and a look at the activemq_msgs table shows 33228 messages

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (AMQ-674) Composite Destination persisted messages never get cleaned up, halt message producers

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
    [ https://issues.apache.org/activemq/browse/AMQ-674?page=comments#action_35966 ]

Paul Smith commented on AMQ-674:
--------------------------------

Just to be clear, so persistent Composite Destination support is basically useless in the 3.x series?

You see, the consumers are definitely keeping up with the producers, we definitely do not have slow consumers here.  THere's no way that there is 32,000 unconsumed messages.  (In fact the consumer nodes are sitting there doing nothing, not being delivered anything)

I don't mind upgrading to 4.x (they don't call me the upgrade-a-tron for nothing), just want to clarify that there is no point pursuing a 3.2x fix?

cheers,

Paul

> Composite Destination persisted messages never get cleaned up, halt message producers
> -------------------------------------------------------------------------------------
>
>          Key: AMQ-674
>          URL: https://issues.apache.org/activemq/browse/AMQ-674
>      Project: ActiveMQ
>         Type: Bug

>   Components: Broker, Message Store
>     Versions: 3.2.1
>     Reporter: Paul Smith
>     Priority: Blocker
>      Fix For: 4.0 RC1

>
>
> well, we've just got bit by something huge.  
> Previously we have been shipping messages from producer to consumer via a named Queue 'Index2.EntityIncrementalIndexer', where the machine index2 has a consumer called "EntityIncrementalIndexer" using persisted messages.
> The design used Composite destinations so that the destination could be:
> "Index1.EntityIncrementalIndexer,Index2.EntityIncrementalIndexer"
> and have the 2 hosts get sent the same message simply, and if one goes down, it'll catch up later.  This gave us the ability to have a mirrored configuration, and go tertiary later if we want simply by adding a new name in the composite destation.
> With a single name, this works great, been working fine for months.  Yesterday we activated the _true_ composite destination, and both machines started getting their messages fine.  
> no problems so far, BUT, 12 hours later we have noticed that the broker has now stopped accepting messages, and a look at the activemq_msgs table shows 33228 messages

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (AMQ-674) Composite Destination persisted messages never get cleaned up, halt message producers

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
    [ https://issues.apache.org/activemq/browse/AMQ-674?page=comments#action_35967 ]

james strachan commented on AMQ-674:
------------------------------------

:)

The issue isn't with Composite Destinations per se - they are pretty trivial things, just a 1-many relationship of destinations. The issue is when you run out of RAM due to too many messages stuck on some queue, things kinda lock up until messages get consumed.

So I definitely recommend you upgrade to 4.0-RC2...
http://people.apache.org/~chirino/incubator-activemq-4.0-RC2/

which has all these issues fixed.

Note I'm a little confused when you talk about Index1.Foo and Index2.Foo and machines index1/index2. Index1.Foo is a queue which will exist on the broker your producer talks to right? Are you networking 2 brokers together in a store/forward network?


BTW one other benefit of 4.x is you can use JMX to view all the queues & their queue depths to easily grok whats going on...
http://activemq.org/JMX


> Composite Destination persisted messages never get cleaned up, halt message producers
> -------------------------------------------------------------------------------------
>
>          Key: AMQ-674
>          URL: https://issues.apache.org/activemq/browse/AMQ-674
>      Project: ActiveMQ
>         Type: Bug

>   Components: Broker, Message Store
>     Versions: 3.2.1
>     Reporter: Paul Smith
>     Priority: Blocker
>      Fix For: 4.0 RC1

>
>
> well, we've just got bit by something huge.  
> Previously we have been shipping messages from producer to consumer via a named Queue 'Index2.EntityIncrementalIndexer', where the machine index2 has a consumer called "EntityIncrementalIndexer" using persisted messages.
> The design used Composite destinations so that the destination could be:
> "Index1.EntityIncrementalIndexer,Index2.EntityIncrementalIndexer"
> and have the 2 hosts get sent the same message simply, and if one goes down, it'll catch up later.  This gave us the ability to have a mirrored configuration, and go tertiary later if we want simply by adding a new name in the composite destation.
> With a single name, this works great, been working fine for months.  Yesterday we activated the _true_ composite destination, and both machines started getting their messages fine.  
> no problems so far, BUT, 12 hours later we have noticed that the broker has now stopped accepting messages, and a look at the activemq_msgs table shows 33228 messages

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (AMQ-674) Composite Destination persisted messages never get cleaned up, halt message producers

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
    [ https://issues.apache.org/activemq/browse/AMQ-674?page=comments#action_35968 ]

Paul Smith commented on AMQ-674:
--------------------------------

There is a logical queue within our system called "EntityIncrementalIndexer' (don't think JMS here, just thing in-out pipeline), which updates to mail & docs in our app are sent as incremental update messages to index nodes.

Each index node, (Index1 & Index2) need to have their own queue, and we use the Composite Destination feature to clone a message to both destinations:


producer -> JMS -> Index1
                               -> Index2

The producer is configured to know how many nodes in the cluster, and sends queue messages via composite destinations, at startup it sets aa variable about the index node list, and creates a Desitaniotn of the form "Index1.EntityIncrementalIndexer,Index2.EntityIncrementalIndex".  I thought each Index node would then read from it's own queue, with ActiveMQ pushing the 1 sent message into BOTH queues.

If your immediate thought is "Shouldn't this be done via Durable Subscription topics?", you are probably right, but I've been bitten by vendors Durable Subscription implementation before, and thought the Composite Destination approach might be simpler, and more elegant.  (wrong! :))

Now, I'm confused by your comment about RAM.  You see I don't believe any of teh messages sent since we started using 2 nodes in the composite destination ever got 'missed'.   That is from our logs I believe that all messages arrived and were consumed by the destination.  I believe the messages are being persisted, but not marked as consumed.



> Composite Destination persisted messages never get cleaned up, halt message producers
> -------------------------------------------------------------------------------------
>
>          Key: AMQ-674
>          URL: https://issues.apache.org/activemq/browse/AMQ-674
>      Project: ActiveMQ
>         Type: Bug

>   Components: Broker, Message Store
>     Versions: 3.2.1
>     Reporter: Paul Smith
>     Priority: Blocker
>      Fix For: 4.0 RC1

>
>
> well, we've just got bit by something huge.  
> Previously we have been shipping messages from producer to consumer via a named Queue 'Index2.EntityIncrementalIndexer', where the machine index2 has a consumer called "EntityIncrementalIndexer" using persisted messages.
> The design used Composite destinations so that the destination could be:
> "Index1.EntityIncrementalIndexer,Index2.EntityIncrementalIndexer"
> and have the 2 hosts get sent the same message simply, and if one goes down, it'll catch up later.  This gave us the ability to have a mirrored configuration, and go tertiary later if we want simply by adding a new name in the composite destation.
> With a single name, this works great, been working fine for months.  Yesterday we activated the _true_ composite destination, and both machines started getting their messages fine.  
> no problems so far, BUT, 12 hours later we have noticed that the broker has now stopped accepting messages, and a look at the activemq_msgs table shows 33228 messages

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (AMQ-674) Composite Destination persisted messages never get cleaned up, halt message producers

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
    [ https://issues.apache.org/activemq/browse/AMQ-674?page=comments#action_35970 ]

james strachan commented on AMQ-674:
------------------------------------

BTW in 4.x the queue dispatch mechanism has been completely changed so that we can deal with massive queues with millions of messages on them; we don't have to keep around the entire queue contents in RAM in 4.x so you should not see the "WARN Queue is full, waiting for it to be dequeued. " messages in 4.x

> Composite Destination persisted messages never get cleaned up, halt message producers
> -------------------------------------------------------------------------------------
>
>          Key: AMQ-674
>          URL: https://issues.apache.org/activemq/browse/AMQ-674
>      Project: ActiveMQ
>         Type: Bug

>   Components: Broker, Message Store
>     Versions: 3.2.1
>     Reporter: Paul Smith
>     Priority: Blocker
>      Fix For: 4.0 RC1

>
>
> well, we've just got bit by something huge.  
> Previously we have been shipping messages from producer to consumer via a named Queue 'Index2.EntityIncrementalIndexer', where the machine index2 has a consumer called "EntityIncrementalIndexer" using persisted messages.
> The design used Composite destinations so that the destination could be:
> "Index1.EntityIncrementalIndexer,Index2.EntityIncrementalIndexer"
> and have the 2 hosts get sent the same message simply, and if one goes down, it'll catch up later.  This gave us the ability to have a mirrored configuration, and go tertiary later if we want simply by adding a new name in the composite destation.
> With a single name, this works great, been working fine for months.  Yesterday we activated the _true_ composite destination, and both machines started getting their messages fine.  
> no problems so far, BUT, 12 hours later we have noticed that the broker has now stopped accepting messages, and a look at the activemq_msgs table shows 33228 messages

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira

Reply | Threaded
Open this post in threaded view
|

[jira] Commented: (AMQ-674) Composite Destination persisted messages never get cleaned up, halt message producers

JIRA jira@apache.org
In reply to this post by JIRA jira@apache.org
    [ https://issues.apache.org/activemq/browse/AMQ-674?page=comments#action_35969 ]

james strachan commented on AMQ-674:
------------------------------------

I'm with you now. Composite destinations should work fine for you - its a way of using multiple queues to simulate durable topics - which is a good thing in general. (I'd like to provide queue-topic unification at some point so you can see durable topics as being like multiple virtual queues which multiple consumers can consume on the same queue to get load balancing - while avoiding the multiple-persistence of the message to each queue which happens now with composite destinations - but thats another topic).

My mention of RAM is that this error messsage...

14:45:03 WARN Queue is full, waiting for it to be dequeued.

is created when the queue dispatcher uses up its available RAM because it keeps all unconsumed queue messages in RAM until they are consumed. So its sounding like some messages on some queue are not getting consumed. Unfortunately in 3.x there's no easy way to look and see for sure - if you used 4.x you could just use JConsole to see all the queues and what messages are on each etc.



> Composite Destination persisted messages never get cleaned up, halt message producers
> -------------------------------------------------------------------------------------
>
>          Key: AMQ-674
>          URL: https://issues.apache.org/activemq/browse/AMQ-674
>      Project: ActiveMQ
>         Type: Bug

>   Components: Broker, Message Store
>     Versions: 3.2.1
>     Reporter: Paul Smith
>     Priority: Blocker
>      Fix For: 4.0 RC1

>
>
> well, we've just got bit by something huge.  
> Previously we have been shipping messages from producer to consumer via a named Queue 'Index2.EntityIncrementalIndexer', where the machine index2 has a consumer called "EntityIncrementalIndexer" using persisted messages.
> The design used Composite destinations so that the destination could be:
> "Index1.EntityIncrementalIndexer,Index2.EntityIncrementalIndexer"
> and have the 2 hosts get sent the same message simply, and if one goes down, it'll catch up later.  This gave us the ability to have a mirrored configuration, and go tertiary later if we want simply by adding a new name in the composite destation.
> With a single name, this works great, been working fine for months.  Yesterday we activated the _true_ composite destination, and both machines started getting their messages fine.  
> no problems so far, BUT, 12 hours later we have noticed that the broker has now stopped accepting messages, and a look at the activemq_msgs table shows 33228 messages

--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   https://issues.apache.org/activemq/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira