[activemq-dev] Feature: cleanup of expired messages and dead letter policy

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[activemq-dev] Feature: cleanup of expired messages and dead letter policy

Ramzi Saba
I would like to recommend that we provide a hook for client apps to be
notified of expired messages and be given a chance to act accordingly.  
For instance, I think that many applications would want to reset a
database flag or perhaps send an email when a message expires.

In order to support this feature, there seems to be 2 things to address:

1- Add a "deadLetterPolicy" child element to the "broker" element of
activemq.dtd.  This would allow a client app to extend the default
DeadLetterPolicy class and override the "sendToDeadLetter" method.  Some
client apps may want to send dead letters to the same destination/queue,
say "org.activemq.deadletters", as opposed to using different
destinations depending on the physical destination of the original
message.  For instance, some apps may need to handle all dead letters at
once as part of a batch process.  But that's up to client apps to define.

2. JDBCPersistenceAdapter creates a thread that cleans up expired
messages from the db periodically via JDBCPersistenceAdapter.cleanup().  
However, this cleanup does not generate dead letters.  Also, it always
cleans up all dead letters since they are always persisted with an
expired timestamp.  I suggest that we modify the cleanup method to a)
not cleanup dead letters and b) cleanup all other expired messages from
the db and generate dead letters accordingly.

Am I missing anything else?

Does this sound like a good feature to have?  If so, please let me know
as I would like to help.

-ramzi
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [activemq-dev] Feature: cleanup of expired messages and dead letter policy

jstrachan

On 15 Jun 2005, at 04:49, Ramzi Saba wrote:

> I would like to recommend that we provide a hook for client apps to  
> be notified of expired messages and be given a chance to act  
> accordingly.  For instance, I think that many applications would  
> want to reset a database flag or perhaps send an email when a  
> message expires.
>
> In order to support this feature, there seems to be 2 things to  
> address:
>
> 1- Add a "deadLetterPolicy" child element to the "broker" element  
> of activemq.dtd.  This would allow a client app to extend the  
> default DeadLetterPolicy class and override the "sendToDeadLetter"  
> method.  Some client apps may want to send dead letters to the same  
> destination/queue, say "org.activemq.deadletters", as opposed to  
> using different destinations depending on the physical destination  
> of the original message.  For instance, some apps may need to  
> handle all dead letters at once as part of a batch process.  But  
> that's up to client apps to define.

I don't quite follow this requirement. e.g. you can subscribe to  
org.activemq.deadletters.> to get notified of every message sent to a  
dead letter. You could then apply a selector if you wish to filter  
them in some way. So right now today you can consume all timeout  
messages / dead letter messages easily with topic wildcards. Does  
that do what you need?



> 2. JDBCPersistenceAdapter creates a thread that cleans up expired  
> messages from the db periodically via JDBCPersistenceAdapter.cleanup
> ().  However, this cleanup does not generate dead letters.  Also,  
> it always cleans up all dead letters since they are always  
> persisted with an expired timestamp.  I suggest that we modify the  
> cleanup method to a) not cleanup dead letters and b) cleanup all  
> other expired messages from the db and generate dead letters  
> accordingly.

Sounds like a good idea to me. We should make this configurable, so  
we can enable or disable the deletion of expired messages.

> Am I missing anything else?

I can't think of anything else.

> Does this sound like a good feature to have?  If so, please let me  
> know as I would like to help.

Sure! :) Welcome aboard! :) Also if you want to try developing  
something on ActiveMQ and want some inspiration of features folks are  
interested in, you could check out the issue tracker for outstanding  
new features & issues:

http://jira.logicblaze.com/jira/browse/AMQ

James
-------
http://radio.weblogs.com/0112098/

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [activemq-dev] Feature: cleanup of expired messages and dead letter policy

Ramzi Saba
[hidden email] wrote:

>
> On 15 Jun 2005, at 04:49, Ramzi Saba wrote:
>
>> I would like to recommend that we provide a hook for client apps to  
>> be notified of expired messages and be given a chance to act  
>> accordingly.  For instance, I think that many applications would  
>> want to reset a database flag or perhaps send an email when a  
>> message expires.
>>
>> In order to support this feature, there seems to be 2 things to  
>> address:
>>
>> 1- Add a "deadLetterPolicy" child element to the "broker" element  of
>> activemq.dtd.  This would allow a client app to extend the  default
>> DeadLetterPolicy class and override the "sendToDeadLetter"  method.  
>> Some client apps may want to send dead letters to the same  
>> destination/queue, say "org.activemq.deadletters", as opposed to  
>> using different destinations depending on the physical destination  
>> of the original message.  For instance, some apps may need to  handle
>> all dead letters at once as part of a batch process.  But  that's up
>> to client apps to define.
>
>
> I don't quite follow this requirement. e.g. you can subscribe to  
> org.activemq.deadletters.> to get notified of every message sent to a  
> dead letter. You could then apply a selector if you wish to filter  
> them in some way. So right now today you can consume all timeout  
> messages / dead letter messages easily with topic wildcards. Does  
> that do what you need?
>
>
I tried to subscribe to a wildcard topic as you suggested, however it
appears that in my case the dead letter is never sent to a topic.  This
makes sense since my original message is based on queue destination (not
a topic).

Anything I am missing here?  If not, I still think we could send dead
letters to a configurable queue (and by default use the existing
stragegy), which can be monitored by a separate process that consumes
and processes dead letters.

Regardless, I still think we should expose a "deadLetterPolicy" element
in activemq.dtd to allow implementations to customize the behavior of
dead letters, e.g. disable dead letter generation by setting a
"deadEnabled" attribute to false.

>
>> 2. JDBCPersistenceAdapter creates a thread that cleans up expired  
>> messages from the db periodically via JDBCPersistenceAdapter.cleanup
>> ().  However, this cleanup does not generate dead letters.  Also,  it
>> always cleans up all dead letters since they are always  persisted
>> with an expired timestamp.  I suggest that we modify the  cleanup
>> method to a) not cleanup dead letters and b) cleanup all  other
>> expired messages from the db and generate dead letters  accordingly.
>
>
> Sounds like a good idea to me. We should make this configurable, so  
> we can enable or disable the deletion of expired messages.

I have made these code changes to the cleanup in my local environment:
- optionally delete expired messages in the db
- optionally send dead letters for expired messages; btw, I make sure
that dead letters never expire otherwise they too would get picked up by
the cleanup process (endless loop).  Also, to prevent sending dup dead
letters on the same expired message, I set a boolean property on the
original message indicating that a dead letter has been sent.  
Alternatively, I could add a db colum to track if a dead letter has been
sent - this actually makes the db retrieval of expired messages much
more efficient.

I plan to submit a patch to JIRA.  Please let me know if you have any
further thoughts.

--
Ramzi Saba
Optaros, Inc - www.optaros.com

Loading...