Getting a "Cannot publish to a deleted Destination" but eventually works after a couple of retries

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Getting a "Cannot publish to a deleted Destination" but eventually works after a couple of retries

tikboy
Hi

We have an application that runs on Glassfish 3.1. It uses Camel to publish a request/reply JMS message to an AMQ broker. The message is then consumed by a Camel JMS listener running on Fuse 7.0.

This setup works most of the time but sometimes, when the camel route on Fuse is sending the reply back, the temporary queue cannot be found and would throw an "org.springframework.jms.InvalidDestinationException: Cannot publish to a deleted Destination" exception. A redelivery would be attempted by the camel route on Fuse up to 5 times (base on our policy). Eventually the response would be delivered after a couple of retries.

I initially thought that the connection between the Glassfish Camel route to AMQ had already broken causing the temporary queue to be deleted but base on what had happened, Glassfish was able to process the response sent by Fuse even after it caught an InvalidDestinationException and doing a couple of retries.

Can somebody please enlighten me on what might have happened causing this behavior?

Camel route on Glassfish that sends messages to AMQ
<route autoStartup="true" id="SendEmailMapRoute">
        <from uri="direct:SendEMailMapStart"/>
        <to uri="activemq:queue:com.mycompany.router.RouterService?disableTimeToLive=true&amp;requestTimeout=180000"/>
        <camel:process ref="ClientResponseProcessor"/>
</route>


Camel route on Fuse that processes messages received from AMQ
@Override
public void configure() throws Exception {
       
        from("activemq:queue:com.mycompany.router.RouterService")
                .setExchangePattern(ExchangePattern.InOut)
                .doTry()
                        .log(LoggingLevel.INFO,  "Dedicated Channel JMS request received ${id}")
                        .log(LoggingLevel.INFO,  "Dedicated Channel JMS request headers: ${headers}")
                        .log(LoggingLevel.DEBUG,  "Dedicated Channel JMS request body: ${body}")
                       
                        .to("nmr:preProcessor")
                        .to("nmr:validateMessage")
                        .to("nmr:serviceEndpoint")
                        .to("nmr:postProcessor")
                        .to("nmr:processSuccess")
                       
                        .log(LoggingLevel.DEBUG,  "Dedicated Channel JMS response body: ${body}")
                .doCatch(HTTPException.class, SocketException.class, SocketTimeoutException.class)
                        .to("nmr:processFailException")
                        .throwException(new EaiPlatformException("Unable to communicate with the server."))
                .doCatch(SoapFault.class)
                        .to("nmr:processSuccessException")
                .doCatch(Exception.class)
                        .to("nmr:processFailException")
                ;
}

Stack Trace:
2014-06-25 19:19:10,158 | WARN  | TION_LIST_EVENT] | EndpointMessageListener          | 139 - org.apache.camel.camel-core - 2.9.0.fuse-70-097 | Execution of JMS message listener failed. Caused
 by: [org.apache.camel.RuntimeCamelException - org.springframework.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:pksprodcl1app01-64463-1403693883558
-0:3:1; nested exception is javax.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:pksprodcl1app01-64463-1403693883558-0:3:1]
org.apache.camel.RuntimeCamelException: org.springframework.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:pksprodcl1app01-64463-1403693883558-0:3:1;
 nested exception is javax.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:pksprodcl1app01-64463-1403693883558-0:3:1
        at org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1277)[139:org.apache.camel.camel-core:2.9.0.fuse-70-097]
        at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:125)[146:org.apache.camel.camel-jms:2.9.0.fuse-70-097]
        at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:560)[145:org.springframework.jms:3.0.7.RELEASE]
        at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:498)[145:org.springframework.jms:3.0.7.RELEASE]
        at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467)[145:org.springframework.jms:3.0.7.RELEASE]
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325)[145:org.springframework.jms:3.0.7.RELE
ASE]
        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:243)[145:org.springframework.jms:3.0.7.RELEAS
E]
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058)[145:org.springframework.jms:3.
0.7.RELEASE]
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050)[145:org.springframework.jm
s:3.0.7.RELEASE]
        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947)[145:org.springframework.jms:3.0.7.RELEASE]
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)[:1.6.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)[:1.6.0_45]
        at java.lang.Thread.run(Thread.java:662)[:1.6.0_45]
Caused by: org.springframework.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:pksprodcl1app01-64463-1403693883558-0:3:1; nested exception is javax.jm
s.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:pksprodcl1app01-64463-1403693883558-0:3:1
        at org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:285)[145:org.springframework.jms:3.0.7.RELEASE]
        at org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:168)[145:org.springframework.jms:3.0.7.RELEASE]
        at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:469)[145:org.springframework.jms:3.0.7.RELEASE]
        at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.send(JmsConfiguration.java:197)[146:org.apache.camel.camel-jms:2.9.0.fuse-70-097]
        at org.apache.camel.component.jms.EndpointMessageListener.sendReply(EndpointMessageListener.java:353)[146:org.apache.camel.camel-jms:2.9.0.fuse-70-097]
        at org.apache.camel.component.jms.EndpointMessageListener$EndpointMessageListenerAsyncCallback.done(EndpointMessageListener.java:206)[146:org.apache.camel.camel-jms:2.9.0.fuse-70-097]
        at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:108)[146:org.apache.camel.camel-jms:2.9.0.fuse-70-097]
        ... 11 more
Caused by: javax.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:pksprodcl1app01-64463-1403693883558-0:3:1
        at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1705)[120:org.apache.activemq.activemq-core:5.5.1.fuse-70-097]
        at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:231)[120:org.apache.activemq.activemq-core:5.5.1.fuse-70-097]
        at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:74)[114:org.apache.activemq.activemq-pool:5.5.1.fuse-70-097]
        at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:59)[114:org.apache.activemq.activemq-pool:5.5.1.fuse-70-097]
        at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:592)[145:org.springframework.jms:3.0.7.RELEASE]
        at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.doSend(JmsConfiguration.java:275)[146:org.apache.camel.camel-jms:2.9.0.fuse-70-097]
        at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.doSendToDestination(JmsConfiguration.java:214)[146:org.apache.camel.camel-jms:2.9.0.fuse-70-097]
        at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.access$100(JmsConfiguration.java:156)[146:org.apache.camel.camel-jms:2.9.0.fuse-70-097]
        at org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate$4.doInJms(JmsConfiguration.java:199)[146:org.apache.camel.camel-jms:2.9.0.fuse-70-097]
        at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:466)[145:org.springframework.jms:3.0.7.RELEASE]
        ... 15 more
Reply | Threaded
Open this post in threaded view
|

Re: Getting a "Cannot publish to a deleted Destination" but eventually works after a couple of retries

ceposta
How are you setting up the connection factory for these things? Is this a
single broker or some network or brokers?

On Wednesday, June 25, 2014, tikboy <[hidden email]> wrote:

> Hi
>
> We have an application that runs on Glassfish 3.1. It uses Camel to publish
> a request/reply JMS message to an AMQ broker. The message is then consumed
> by a Camel JMS listener running on Fuse 7.0.
>
> This setup works most of the time but sometimes, when the camel route on
> Fuse is sending the reply back, the temporary queue cannot be found and
> would throw an "org.springframework.jms.InvalidDestinationException: Cannot
> publish to a deleted Destination" exception. A redelivery would be
> attempted
> by the camel route on Fuse up to 5 times (base on our policy). Eventually
> the response would be delivered after a couple of retries.
>
> I initially thought that the connection between the Glassfish Camel route
> to
> AMQ had already broken causing the temporary queue to be deleted but base
> on
> what had happened, Glassfish was able to process the response sent by Fuse
> even after it caught an InvalidDestinationException and doing a couple of
> retries.
>
> Can somebody please enlighten me on what might have happened causing this
> behavior?
>
> Camel route on Glassfish that sends messages to AMQ
> <route autoStartup="true" id="SendEmailMapRoute">
>         <from uri="direct:SendEMailMapStart"/>
>         <to
>
> uri="activemq:queue:com.mycompany.router.RouterService?disableTimeToLive=true&amp;requestTimeout=180000"/>
>         <camel:process ref="ClientResponseProcessor"/>
> </route>
>
>
> Camel route on Fuse that processes messages received from AMQ
> @Override
> public void configure() throws Exception {
>
>         from("activemq:queue:com.mycompany.router.RouterService")
>                 .setExchangePattern(ExchangePattern.InOut)
>                 .doTry()
>                         .log(LoggingLevel.INFO,  "Dedicated Channel JMS
> request received ${id}")
>                         .log(LoggingLevel.INFO,  "Dedicated Channel JMS
> request headers:
> ${headers}")
>                         .log(LoggingLevel.DEBUG,  "Dedicated Channel JMS
> request body: ${body}")
>
>                         .to("nmr:preProcessor")
>                         .to("nmr:validateMessage")
>                         .to("nmr:serviceEndpoint")
>                         .to("nmr:postProcessor")
>                         .to("nmr:processSuccess")
>
>                         .log(LoggingLevel.DEBUG,  "Dedicated Channel JMS
> response body: ${body}")
>                 .doCatch(HTTPException.class, SocketException.class,
> SocketTimeoutException.class)
>                         .to("nmr:processFailException")
>                         .throwException(new EaiPlatformException("Unable
> to communicate with the
> server."))
>                 .doCatch(SoapFault.class)
>                         .to("nmr:processSuccessException")
>                 .doCatch(Exception.class)
>                         .to("nmr:processFailException")
>                 ;
> }
>
> Stack Trace:
> 2014-06-25 19:19:10,158 | WARN  | TION_LIST_EVENT] |
> EndpointMessageListener
> | 139 - org.apache.camel.camel-core - 2.9.0.fuse-70-097 | Execution of JMS
> message listener failed. Caused
>  by: [org.apache.camel.RuntimeCamelException -
> org.springframework.jms.InvalidDestinationException: Cannot publish to a
> deleted Destination: temp-queue://ID:pksprodcl1app01-64463-1403693883558
> -0:3:1; nested exception is javax.jms.InvalidDestinationException: Cannot
> publish to a deleted Destination:
> temp-queue://ID:pksprodcl1app01-64463-1403693883558-0:3:1]
> org.apache.camel.RuntimeCamelException:
> org.springframework.jms.InvalidDestinationException: Cannot publish to a
> deleted Destination:
> temp-queue://ID:pksprodcl1app01-64463-1403693883558-0:3:1;
>  nested exception is javax.jms.InvalidDestinationException: Cannot publish
> to a deleted Destination:
> temp-queue://ID:pksprodcl1app01-64463-1403693883558-0:3:1
>         at
>
> org.apache.camel.util.ObjectHelper.wrapRuntimeCamelException(ObjectHelper.java:1277)[139:org.apache.camel.camel-core:2.9.0.fuse-70-097]
>         at
>
> org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:125)[146:org.apache.camel.camel-jms:2.9.0.fuse-70-097]
>         at
>
> org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:560)[145:org.springframework.jms:3.0.7.RELEASE]
>         at
>
> org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:498)[145:org.springframework.jms:3.0.7.RELEASE]
>         at
>
> org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467)[145:org.springframework.jms:3.0.7.RELEASE]
>         at
>
> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325)[145:org.springframework.jms:3.0.7.RELE
> ASE]
>         at
>
> org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:243)[145:org.springframework.jms:3.0.7.RELEAS
> E]
>         at
>
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058)[145:org.springframework.jms:3.
> 0.7.RELEASE]
>         at
>
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050)[145:
> org.springframework.jm
> s:3.0.7.RELEASE]
>         at
>
> org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947)[145:org.springframework.jms:3.0.7.RELEASE]
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)[:1.6.0_45]
>         at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)[:1.6.0_45]
>         at java.lang.Thread.run(Thread.java:662)[:1.6.0_45]
> Caused by: org.springframework.jms.InvalidDestinationException: Cannot
> publish to a deleted Destination:
> temp-queue://ID:pksprodcl1app01-64463-1403693883558-0:3:1; nested exception
> is javax.jm
> s.InvalidDestinationException: Cannot publish to a deleted Destination:
> temp-queue://ID:pksprodcl1app01-64463-1403693883558-0:3:1
>         at
>
> org.springframework.jms.support.JmsUtils.convertJmsAccessException(JmsUtils.java:285)[145:org.springframework.jms:3.0.7.RELEASE]
>         at
>
> org.springframework.jms.support.JmsAccessor.convertJmsAccessException(JmsAccessor.java:168)[145:org.springframework.jms:3.0.7.RELEASE]
>         at
>
> org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:469)[145:org.springframework.jms:3.0.7.RELEASE]
>         at
>
> org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.send(JmsConfiguration.java:197)[146:org.apache.camel.camel-jms:2.9.0.fuse-70-097]
>         at
>
> org.apache.camel.component.jms.EndpointMessageListener.sendReply(EndpointMessageListener.java:353)[146:org.apache.camel.camel-jms:2.9.0.fuse-70-097]
>         at
>
> org.apache.camel.component.jms.EndpointMessageListener$EndpointMessageListenerAsyncCallback.done(EndpointMessageListener.java:206)[146:org.apache.camel.camel-jms:2.9.0.fuse-70-097]
>         at
>
> org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:108)[146:org.apache.camel.camel-jms:2.9.0.fuse-70-097]
>         ... 11 more
> Caused by: javax.jms.InvalidDestinationException: Cannot publish to a
> deleted Destination:
> temp-queue://ID:pksprodcl1app01-64463-1403693883558-0:3:1
>         at
>
> org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1705)[120:org.apache.activemq.activemq-core:5.5.1.fuse-70-097]
>         at
>
> org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:231)[120:org.apache.activemq.activemq-core:5.5.1.fuse-70-097]
>         at
>
> org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:74)[114:org.apache.activemq.activemq-pool:5.5.1.fuse-70-097]
>         at
>
> org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:59)[114:org.apache.activemq.activemq-pool:5.5.1.fuse-70-097]
>         at
>
> org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:592)[145:org.springframework.jms:3.0.7.RELEASE]
>         at
>
> org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.doSend(JmsConfiguration.java:275)[146:org.apache.camel.camel-jms:2.9.0.fuse-70-097]
>         at
>
> org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.doSendToDestination(JmsConfiguration.java:214)[146:org.apache.camel.camel-jms:2.9.0.fuse-70-097]
>         at
>
> org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate.access$100(JmsConfiguration.java:156)[146:org.apache.camel.camel-jms:2.9.0.fuse-70-097]
>         at
>
> org.apache.camel.component.jms.JmsConfiguration$CamelJmsTemplate$4.doInJms(JmsConfiguration.java:199)[146:org.apache.camel.camel-jms:2.9.0.fuse-70-097]
>         at
>
> org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:466)[145:org.springframework.jms:3.0.7.RELEASE]
>         ... 15 more
>
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Getting-a-Cannot-publish-to-a-deleted-Destination-but-eventually-works-after-a-couple-of-retries-tp4682497.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>


--
*Christian Posta*
http://www.christianposta.com/blog
http://fabric8.io
twitter: @christianposta
Reply | Threaded
Open this post in threaded view
|

Re: Getting a "Cannot publish to a deleted Destination" but eventually works after a couple of retries

tikboy
Apologies for not giving an immediate response. It seems that our problem was solved by increasing the open files limit setting for both Centos and Ubuntu hosting Glassfish, Fuse and ActiveMQ.

But to answer your question, yes we are using a broker network comprised by 4 Ubuntu nodes. 1 has a duplex connection to 2, 2 has a duplex connection to 3, 3 duplex connection to 4 and 4 duplex connection to 1.

Thanks!
Reply | Threaded
Open this post in threaded view
|

Re: Getting a "Cannot publish to a deleted Destination" but eventually works after a couple of retries

atActiveMQ
Not sure how increasing the number of files limit could have resolved this issue? Please help me understand.