[activemq-dev] Re: [activemq-user] Re: ActiveMQ hangs when initial connection to broker fails using reliable transport

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[activemq-dev] Re: [activemq-user] Re: ActiveMQ hangs when initial connection to broker fails using reliable transport

Ramzi Saba
Folks, does this sound like a decent proposal, any concerns before I log it in
JIRA?

Ramzi Saba ([hidden email]) wrote:

>
> Agreed, I suggest we make these changes:
> 1- Start the reliable tcp channel in its own thread (seems there was an
> attempt to do so anyway)
> 2- Synchronous and asynchronous client calls (via session, consumer, etc.) to
> the reliable tcp channel should simply verify if a reliable channel has been
> already established, else throw a JMSException, alternatively allow the client
> to configure a timeout (currently it's hardcoded for synchronous calls only I
> believe).
> 3- Other than starting the reliable channel, the client should not be
> responsible of reestablishing a lost or unavailable channel.  I would delegate
> reliability to the reliable channel itself.
>
> -ramzi
>
> Dennis Cook ([hidden email]) wrote:
> >
> > Ah, yes I see your point about hanging at startup.  I have never run
> > across that particular problem because I have multiple brokers in my
> > reliable list.  At any given time it is able to reach one of them.
> >
> > I doubt a fix that performs a background initial connection will be
> > forthcoming though.  From the applications stand point there are many
> > synchonous operation that must follow the connection creation such as
> > session, consumer, and producer creation.  These in turn would have to
> > queued up for processing.  This is getting ugly.
> >
> > I think I would rather just have an initial failure throw an exception
> > out just as it would when using the tcp protocol without reliable.  I
> > already handle that condition within the application. :D
> >
> > Matthew Sinclair-Day wrote:
> > > But in this case the app is not trying to send a message.  It is
> > > attempting to open a connection to consume a message, and now it blocks
> > > forever until the broker comes up.
> > >
> > > My main point is now there seems to be two behaviors in the system,
> > > depending on whether the connection had been initially established.
> > >
> > > Consider the semantics of MessageConsumer.receive(long).  It allows for
> > > a null message to be returned to the caller on timeout.  What the
> > > "reliable" feature made so nice was that the caller could be
> > > transparently resilient to connection failures.  It mitigated the
> > > effects of an inherently unreliable resource.  The caller could be
> > > written not to care  whether a null Message were caused by a connection
> > > failure.  Underneath ActiveMQ would work to reestablish the connection,
> > > but up top the caller could continue calling
> > > MessageConsumer.receive(long) and react to null or actual Message objects.
> > >
> > > It also meant that clients and broker could be started in any order.  A
> > > client could be brought up before the broker and independently of it.
> > > That's a VERY nice thing when managing a large, distributed system.  As
> > > it is now in the system I am building, the broker will now have to be
> > > running in order for the client to be started completely, and that will
> > > cause operational problems; of course, this is my problem to address.
> > >
> > > I would almost prefer a JMSException be thrown if the initial connection
> > > failed with "reliable."
> > >
> > > Matt
> > >
> > >
> > > On 2005-07-28 13:56:42 -0400, Dennis Cook
> > > <[hidden email]> said:
> > >
> > > But even in a reconnect case, where initial connection is lost.  It is
> > > usually the keep-alive thread that is trying to reconnect.  But if the
> > > application thread tried to send a message while this process was going
> > > on, it too would still block.  And I think it should.  Since the client
> > > cannot synchonously send the message it only has two options.  It can
> > > throw an exception back to the application or it can block the thread.
> > > By definition of reliable, I would say blocking for the reconnect is the
> > > correct solution.  Then again the useAsynSend can be used and the
> > > application can go on its merry way.  Of course there is always the
> > > potentional for loss of the message that way, if the connection is never
> > > reestablished.
> > >
> > >>
> > >>
> > >> Ramzi Saba wrote:
> > >> I also agree on the reliable aspect.  Though it would also be nice if the
> > >>
> > >>> client could transparently keep trying to connect to the broker
> > >>> behind the
> > >>> scenes on initial connect, just like it works when the broker
> > >>> restarts and the
> > >>> client automatically keeps trying to reconnect without blocking the
> > >>> client.
> > >>>
> > >>> Dennis Cook ([hidden email]) wrote:
> > >>>
> > >>>
> > >>> I would have to agree.  I expect reliable connections to wait for a
> > >>>
> > >>>> reconnection then send my message.  If the application thread cannot be
> > >>>> blocked on a send, then it should spawned another to do the send or I
> > >>>> think you there is an asyncSend option on the ActiveMQConnectcion.
> > >>>> Maybe that would help in this case.
> > >>>>
> > >>>>
> > >>>> Hiram Chirino wrote:
> > >>>>
> > >>>> Hi Matt,
> > >>>>
> > >>>>>
> > >>>>> I would think that this is the correct behavior of a reliable
> > >>>>> connection.  It's purpose is to hide connection failures for the JMS
> > >>>>> client and to recover correctly.  Anybody else have an opinion?
> > >>>>>
> > >>>>> Regards,
> > >>>>> Hiram
> > >>>>>
> > >>>>>
> > >>>>> On Jul 28, 2005, at 9:11 AM, Matthew Sinclair-Day wrote:
> > >>>>>
> > >>>>>
> > >>>>> Hiram,
> > >>>>>
> > >>>>>>
> > >>>>>> Yes, the client connection becomes un-stuck when the broker comes
> > >>>>>> back up.  The problem is that until it becomes un-stuck the rest of
> > >>>>>> my application hangs.  I guess I need to know whether ActiveMQ's
> > >>>>>> behavior is considered correct in this case, so that changes in my
> > >>>>>> code to address this situation can be made, or whether I can wait
> > >>>>>> for
> > >>>>>> another build that would fix it.
> > >>>>>>
> > >>>>>> Matt
> > >>>>>>
> > >>>>>> On 2005-07-27 17:00:45 -0400, Hiram Chirino
> > >>>>>> <[hidden email]> said:
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> Hi Matt,
> > >>>>>>
> > >>>>>>> Does the client connection get un-stuck if the broker comes back up?
> > >>>>>>> Regards,
> > >>>>>>> Hiram
> > >>>>>>> On Jul 26, 2005, at 5:27 PM, Matthew Sinclair-Day wrote:
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Hello.
> > >>>>>>>
> > >>>>>>>> I am trying to track down a problem with ActiveMQ 3.1-M6 using
> > >>>>>>>> "reliable:tcp" transport to a broker.  In this case, the broker  is
> > >>>>>>>> not running when the client comes up.  When the client is  setting
> > >>>>>>>> up its connection, it calls Connection.setClientID()  prior to
> > >>>>>>>> creating a Session.  In this case, because the broker  is not
> > >>>>>>>> running, ActiveMQ seems to be stuck in
> > >>>>>>>> org.activemq.transport.composite.CompositeTransportChannel.
> > >>>>>>>> Relevant portion from thread's call list:
> > >>>>>>>> WSDMSGRECEIVER_SERVICE@3e1 daemon prio=5, in group "main",  status:
> > >>>>>>>> RUNNING
> > >>>>>>>>      sleep():-1, Thread.java
> > >>>>>>>>      establishConnection():284, CompositeTransportChannel.java
> > >>>>>>>>      start():86, CompositeTransportChannel.java
> > >>>>>>>>      checkClosed():805, ActiveMQConnection.java
> > >>>>>>>>      setClientID():554, ActiveMQConnection.java
> > >>>>>>>> When "reliable" transport is removed then the connection fails
> > >>>>>>>> fast, as expected.
> > >>>>>>>> I believe this is new behavior as of the M6 build or maybe M5 (I
> > >>>>>>>> have not tested M5).
> > >>>>>>>> Matt
> > >>>>>>>>
> > >
> > >
> > >
> >
> >
>
>

Reply | Threaded
Open this post in threaded view
|

RE: [activemq-dev] Re: [activemq-user] Re: ActiveMQ hangs when initial connection to broker fails using reliable transport

Dennis Cook
While this seems like it should be the most logical solution, but I would like to share a couple of thoughts.

If I provide a list of brokers in a reliable mode and at startup it cannot find any of the brokers specified, I consider this to be a severe error that I want to know about immediately.  It usually means that I have a configuration problem or a network problem. If the client quietly enters into the "wait for a broker to become available" mode, then my application start up in a crippled mode and I don't get the immediate notification.

So I am sort of leaning toward a solution that just throws an exception back to the client at connection creation time if no broker is available.  Or at least some notification via the TransportStatusEventListener that the initial connection could not be immediately established.

-----Original Message-----
From: Ramzi Saba [mailto:[hidden email]]
Sent: Tuesday, August 02, 2005 12:21 AM
To: [hidden email]
Subject: [activemq-dev] Re: [activemq-user] Re: ActiveMQ hangs when
initial connection to broker fails using reliable transport


Folks, does this sound like a decent proposal, any concerns before I log it in
JIRA?

Ramzi Saba ([hidden email]) wrote:

>
> Agreed, I suggest we make these changes:
> 1- Start the reliable tcp channel in its own thread (seems there was an
> attempt to do so anyway)
> 2- Synchronous and asynchronous client calls (via session, consumer, etc.) to
> the reliable tcp channel should simply verify if a reliable channel has been
> already established, else throw a JMSException, alternatively allow the client
> to configure a timeout (currently it's hardcoded for synchronous calls only I
> believe).
> 3- Other than starting the reliable channel, the client should not be
> responsible of reestablishing a lost or unavailable channel.  I would delegate
> reliability to the reliable channel itself.
>
> -ramzi
>
> Dennis Cook ([hidden email]) wrote:
> >
> > Ah, yes I see your point about hanging at startup.  I have never run
> > across that particular problem because I have multiple brokers in my
> > reliable list.  At any given time it is able to reach one of them.
> >
> > I doubt a fix that performs a background initial connection will be
> > forthcoming though.  From the applications stand point there are many
> > synchonous operation that must follow the connection creation such as
> > session, consumer, and producer creation.  These in turn would have to
> > queued up for processing.  This is getting ugly.
> >
> > I think I would rather just have an initial failure throw an exception
> > out just as it would when using the tcp protocol without reliable.  I
> > already handle that condition within the application. :D
> >
> > Matthew Sinclair-Day wrote:
> > > But in this case the app is not trying to send a message.  It is
> > > attempting to open a connection to consume a message, and now it blocks
> > > forever until the broker comes up.
> > >
> > > My main point is now there seems to be two behaviors in the system,
> > > depending on whether the connection had been initially established.
> > >
> > > Consider the semantics of MessageConsumer.receive(long).  It allows for
> > > a null message to be returned to the caller on timeout.  What the
> > > "reliable" feature made so nice was that the caller could be
> > > transparently resilient to connection failures.  It mitigated the
> > > effects of an inherently unreliable resource.  The caller could be
> > > written not to care  whether a null Message were caused by a connection
> > > failure.  Underneath ActiveMQ would work to reestablish the connection,
> > > but up top the caller could continue calling
> > > MessageConsumer.receive(long) and react to null or actual Message objects.
> > >
> > > It also meant that clients and broker could be started in any order.  A
> > > client could be brought up before the broker and independently of it.
> > > That's a VERY nice thing when managing a large, distributed system.  As
> > > it is now in the system I am building, the broker will now have to be
> > > running in order for the client to be started completely, and that will
> > > cause operational problems; of course, this is my problem to address.
> > >
> > > I would almost prefer a JMSException be thrown if the initial connection
> > > failed with "reliable."
> > >
> > > Matt
> > >
> > >
> > > On 2005-07-28 13:56:42 -0400, Dennis Cook
> > > <[hidden email]> said:
> > >
> > > But even in a reconnect case, where initial connection is lost.  It is
> > > usually the keep-alive thread that is trying to reconnect.  But if the
> > > application thread tried to send a message while this process was going
> > > on, it too would still block.  And I think it should.  Since the client
> > > cannot synchonously send the message it only has two options.  It can
> > > throw an exception back to the application or it can block the thread.
> > > By definition of reliable, I would say blocking for the reconnect is the
> > > correct solution.  Then again the useAsynSend can be used and the
> > > application can go on its merry way.  Of course there is always the
> > > potentional for loss of the message that way, if the connection is never
> > > reestablished.
> > >
> > >>
> > >>
> > >> Ramzi Saba wrote:
> > >> I also agree on the reliable aspect.  Though it would also be nice if the
> > >>
> > >>> client could transparently keep trying to connect to the broker
> > >>> behind the
> > >>> scenes on initial connect, just like it works when the broker
> > >>> restarts and the
> > >>> client automatically keeps trying to reconnect without blocking the
> > >>> client.
> > >>>
> > >>> Dennis Cook ([hidden email]) wrote:
> > >>>
> > >>>
> > >>> I would have to agree.  I expect reliable connections to wait for a
> > >>>
> > >>>> reconnection then send my message.  If the application thread cannot be
> > >>>> blocked on a send, then it should spawned another to do the send or I
> > >>>> think you there is an asyncSend option on the ActiveMQConnectcion.
> > >>>> Maybe that would help in this case.
> > >>>>
> > >>>>
> > >>>> Hiram Chirino wrote:
> > >>>>
> > >>>> Hi Matt,
> > >>>>
> > >>>>>
> > >>>>> I would think that this is the correct behavior of a reliable
> > >>>>> connection.  It's purpose is to hide connection failures for the JMS
> > >>>>> client and to recover correctly.  Anybody else have an opinion?
> > >>>>>
> > >>>>> Regards,
> > >>>>> Hiram
> > >>>>>
> > >>>>>
> > >>>>> On Jul 28, 2005, at 9:11 AM, Matthew Sinclair-Day wrote:
> > >>>>>
> > >>>>>
> > >>>>> Hiram,
> > >>>>>
> > >>>>>>
> > >>>>>> Yes, the client connection becomes un-stuck when the broker comes
> > >>>>>> back up.  The problem is that until it becomes un-stuck the rest of
> > >>>>>> my application hangs.  I guess I need to know whether ActiveMQ's
> > >>>>>> behavior is considered correct in this case, so that changes in my
> > >>>>>> code to address this situation can be made, or whether I can wait
> > >>>>>> for
> > >>>>>> another build that would fix it.
> > >>>>>>
> > >>>>>> Matt
> > >>>>>>
> > >>>>>> On 2005-07-27 17:00:45 -0400, Hiram Chirino
> > >>>>>> <[hidden email]> said:
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> Hi Matt,
> > >>>>>>
> > >>>>>>> Does the client connection get un-stuck if the broker comes back up?
> > >>>>>>> Regards,
> > >>>>>>> Hiram
> > >>>>>>> On Jul 26, 2005, at 5:27 PM, Matthew Sinclair-Day wrote:
> > >>>>>>>
> > >>>>>>>
> > >>>>>>> Hello.
> > >>>>>>>
> > >>>>>>>> I am trying to track down a problem with ActiveMQ 3.1-M6 using
> > >>>>>>>> "reliable:tcp" transport to a broker.  In this case, the broker  is
> > >>>>>>>> not running when the client comes up.  When the client is  setting
> > >>>>>>>> up its connection, it calls Connection.setClientID()  prior to
> > >>>>>>>> creating a Session.  In this case, because the broker  is not
> > >>>>>>>> running, ActiveMQ seems to be stuck in
> > >>>>>>>> org.activemq.transport.composite.CompositeTransportChannel.
> > >>>>>>>> Relevant portion from thread's call list:
> > >>>>>>>> WSDMSGRECEIVER_SERVICE@3e1 daemon prio=5, in group "main",  status:
> > >>>>>>>> RUNNING
> > >>>>>>>>      sleep():-1, Thread.java
> > >>>>>>>>      establishConnection():284, CompositeTransportChannel.java
> > >>>>>>>>      start():86, CompositeTransportChannel.java
> > >>>>>>>>      checkClosed():805, ActiveMQConnection.java
> > >>>>>>>>      setClientID():554, ActiveMQConnection.java
> > >>>>>>>> When "reliable" transport is removed then the connection fails
> > >>>>>>>> fast, as expected.
> > >>>>>>>> I believe this is new behavior as of the M6 build or maybe M5 (I
> > >>>>>>>> have not tested M5).
> > >>>>>>>> Matt
> > >>>>>>>>
> > >
> > >
> > >
> >
> >
>
>

Reply | Threaded
Open this post in threaded view
|

RE: [activemq-dev] Re: [activemq-user] Re: ActiveMQ hangs when initial connection to broker fails using reliable transport

Frik Strecker
Hi,

I can definitely see the case for what you propose.

For our purposes, we have a well defined setup and this may simply mean that
the servers are started in a different order.  So, we prefer the silent
retry.  Sounds like there should be some option or maybe a listener for your
case.

Kind regards,
Frik Strecker

Voice: +1-603-672-1472
http://www.GatherPlace.NET
Effective On-line Presentations

 

> -----Original Message-----
> From: Dennis Cook [mailto:[hidden email]]
> Sent: Tuesday, August 02, 2005 3:10 PM
> To: [hidden email]
> Subject: RE: [activemq-dev] Re: [activemq-user] Re: ActiveMQ
> hangs when initial connection to broker fails using reliable transport
>
> While this seems like it should be the most logical solution,
> but I would like to share a couple of thoughts.
>
> If I provide a list of brokers in a reliable mode and at
> startup it cannot find any of the brokers specified, I
> consider this to be a severe error that I want to know about
> immediately.  It usually means that I have a configuration
> problem or a network problem. If the client quietly enters
> into the "wait for a broker to become available" mode, then
> my application start up in a crippled mode and I don't get
> the immediate notification.
>
> So I am sort of leaning toward a solution that just throws an
> exception back to the client at connection creation time if
> no broker is available.  Or at least some notification via
> the TransportStatusEventListener that the initial connection
> could not be immediately established.
>
> -----Original Message-----
> From: Ramzi Saba [mailto:[hidden email]]
> Sent: Tuesday, August 02, 2005 12:21 AM
> To: [hidden email]
> Subject: [activemq-dev] Re: [activemq-user] Re: ActiveMQ hangs when
> initial connection to broker fails using reliable transport
>
>
> Folks, does this sound like a decent proposal, any concerns
> before I log it in
> JIRA?
>
> Ramzi Saba ([hidden email]) wrote:
> >
> > Agreed, I suggest we make these changes:
> > 1- Start the reliable tcp channel in its own thread (seems
> there was an
> > attempt to do so anyway)
> > 2- Synchronous and asynchronous client calls (via session,
> consumer, etc.) to
> > the reliable tcp channel should simply verify if a reliable
> channel has been
> > already established, else throw a JMSException,
> alternatively allow the client
> > to configure a timeout (currently it's hardcoded for
> synchronous calls only I
> > believe).
> > 3- Other than starting the reliable channel, the client
> should not be
> > responsible of reestablishing a lost or unavailable
> channel.  I would delegate
> > reliability to the reliable channel itself.
> >
> > -ramzi
> >
> > Dennis Cook ([hidden email]) wrote:
> > >
> > > Ah, yes I see your point about hanging at startup.  I
> have never run
> > > across that particular problem because I have multiple
> brokers in my
> > > reliable list.  At any given time it is able to reach one of them.
> > >
> > > I doubt a fix that performs a background initial
> connection will be
> > > forthcoming though.  From the applications stand point
> there are many
> > > synchonous operation that must follow the connection
> creation such as
> > > session, consumer, and producer creation.  These in turn
> would have to
> > > queued up for processing.  This is getting ugly.
> > >
> > > I think I would rather just have an initial failure throw
> an exception
> > > out just as it would when using the tcp protocol without
> reliable.  I
> > > already handle that condition within the application. :D
> > >
> > > Matthew Sinclair-Day wrote:
> > > > But in this case the app is not trying to send a message.  It is
> > > > attempting to open a connection to consume a message,
> and now it blocks
> > > > forever until the broker comes up.
> > > >
> > > > My main point is now there seems to be two behaviors in
> the system,
> > > > depending on whether the connection had been initially
> established.
> > > >
> > > > Consider the semantics of
> MessageConsumer.receive(long).  It allows for
> > > > a null message to be returned to the caller on timeout.
>  What the
> > > > "reliable" feature made so nice was that the caller could be
> > > > transparently resilient to connection failures.  It
> mitigated the
> > > > effects of an inherently unreliable resource.  The
> caller could be
> > > > written not to care  whether a null Message were caused
> by a connection
> > > > failure.  Underneath ActiveMQ would work to reestablish
> the connection,
> > > > but up top the caller could continue calling
> > > > MessageConsumer.receive(long) and react to null or
> actual Message objects.
> > > >
> > > > It also meant that clients and broker could be started
> in any order.  A
> > > > client could be brought up before the broker and
> independently of it.
> > > > That's a VERY nice thing when managing a large,
> distributed system.  As
> > > > it is now in the system I am building, the broker will
> now have to be
> > > > running in order for the client to be started
> completely, and that will
> > > > cause operational problems; of course, this is my
> problem to address.
> > > >
> > > > I would almost prefer a JMSException be thrown if the
> initial connection
> > > > failed with "reliable."
> > > >
> > > > Matt
> > > >
> > > >
> > > > On 2005-07-28 13:56:42 -0400, Dennis Cook
> > > > <[hidden email]> said:
> > > >
> > > > But even in a reconnect case, where initial connection
> is lost.  It is
> > > > usually the keep-alive thread that is trying to
> reconnect.  But if the
> > > > application thread tried to send a message while this
> process was going
> > > > on, it too would still block.  And I think it should.  
> Since the client
> > > > cannot synchonously send the message it only has two
> options.  It can
> > > > throw an exception back to the application or it can
> block the thread.
> > > > By definition of reliable, I would say blocking for the
> reconnect is the
> > > > correct solution.  Then again the useAsynSend can be
> used and the
> > > > application can go on its merry way.  Of course there
> is always the
> > > > potentional for loss of the message that way, if the
> connection is never
> > > > reestablished.
> > > >
> > > >>
> > > >>
> > > >> Ramzi Saba wrote:
> > > >> I also agree on the reliable aspect.  Though it would
> also be nice if the
> > > >>
> > > >>> client could transparently keep trying to connect to
> the broker
> > > >>> behind the
> > > >>> scenes on initial connect, just like it works when the broker
> > > >>> restarts and the
> > > >>> client automatically keeps trying to reconnect
> without blocking the
> > > >>> client.
> > > >>>
> > > >>> Dennis Cook ([hidden email]) wrote:
> > > >>>
> > > >>>
> > > >>> I would have to agree.  I expect reliable connections
> to wait for a
> > > >>>
> > > >>>> reconnection then send my message.  If the
> application thread cannot be
> > > >>>> blocked on a send, then it should spawned another to
> do the send or I
> > > >>>> think you there is an asyncSend option on the
> ActiveMQConnectcion.
> > > >>>> Maybe that would help in this case.
> > > >>>>
> > > >>>>
> > > >>>> Hiram Chirino wrote:
> > > >>>>
> > > >>>> Hi Matt,
> > > >>>>
> > > >>>>>
> > > >>>>> I would think that this is the correct behavior of
> a reliable
> > > >>>>> connection.  It's purpose is to hide connection
> failures for the JMS
> > > >>>>> client and to recover correctly.  Anybody else have
> an opinion?
> > > >>>>>
> > > >>>>> Regards,
> > > >>>>> Hiram
> > > >>>>>
> > > >>>>>
> > > >>>>> On Jul 28, 2005, at 9:11 AM, Matthew Sinclair-Day wrote:
> > > >>>>>
> > > >>>>>
> > > >>>>> Hiram,
> > > >>>>>
> > > >>>>>>
> > > >>>>>> Yes, the client connection becomes un-stuck when
> the broker comes
> > > >>>>>> back up.  The problem is that until it becomes
> un-stuck the rest of
> > > >>>>>> my application hangs.  I guess I need to know
> whether ActiveMQ's
> > > >>>>>> behavior is considered correct in this case, so
> that changes in my
> > > >>>>>> code to address this situation can be made, or
> whether I can wait
> > > >>>>>> for
> > > >>>>>> another build that would fix it.
> > > >>>>>>
> > > >>>>>> Matt
> > > >>>>>>
> > > >>>>>> On 2005-07-27 17:00:45 -0400, Hiram Chirino
> > > >>>>>> <[hidden email]> said:
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> Hi Matt,
> > > >>>>>>
> > > >>>>>>> Does the client connection get un-stuck if the
> broker comes back up?
> > > >>>>>>> Regards,
> > > >>>>>>> Hiram
> > > >>>>>>> On Jul 26, 2005, at 5:27 PM, Matthew Sinclair-Day wrote:
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> Hello.
> > > >>>>>>>
> > > >>>>>>>> I am trying to track down a problem with
> ActiveMQ 3.1-M6 using
> > > >>>>>>>> "reliable:tcp" transport to a broker.  In this
> case, the broker  is
> > > >>>>>>>> not running when the client comes up.  When the
> client is  setting
> > > >>>>>>>> up its connection, it calls
> Connection.setClientID()  prior to
> > > >>>>>>>> creating a Session.  In this case, because the
> broker  is not
> > > >>>>>>>> running, ActiveMQ seems to be stuck in
> > > >>>>>>>>
> org.activemq.transport.composite.CompositeTransportChannel.
> > > >>>>>>>> Relevant portion from thread's call list:
> > > >>>>>>>> WSDMSGRECEIVER_SERVICE@3e1 daemon prio=5, in
> group "main",  status:
> > > >>>>>>>> RUNNING
> > > >>>>>>>>      sleep():-1, Thread.java
> > > >>>>>>>>      establishConnection():284,
> CompositeTransportChannel.java
> > > >>>>>>>>      start():86, CompositeTransportChannel.java
> > > >>>>>>>>      checkClosed():805, ActiveMQConnection.java
> > > >>>>>>>>      setClientID():554, ActiveMQConnection.java
> > > >>>>>>>> When "reliable" transport is removed then the
> connection fails
> > > >>>>>>>> fast, as expected.
> > > >>>>>>>> I believe this is new behavior as of the M6
> build or maybe M5 (I
> > > >>>>>>>> have not tested M5).
> > > >>>>>>>> Matt
> > > >>>>>>>>
> > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>

Reply | Threaded
Open this post in threaded view
|

RE: [activemq-dev] Re: [activemq-user] Re: ActiveMQ hangs when initial connection to broker fails using reliable transport

Dennis Cook
In reply to this post by Ramzi Saba
Yeah, I was actually thinking of this problem being from the perspective of a JMS client.  But if reliable is used for inter broker network connections, then a quite startup would be desireable.  

I do think the listener event is the most desirable all around.

-----Original Message-----
From: Frik Strecker [mailto:[hidden email]]
Sent: Tuesday, August 02, 2005 3:30 PM
To: [hidden email]
Subject: RE: [activemq-dev] Re: [activemq-user] Re: ActiveMQ hangs when
initial connection to broker fails using reliable transport


Hi,

I can definitely see the case for what you propose.

For our purposes, we have a well defined setup and this may simply mean that
the servers are started in a different order.  So, we prefer the silent
retry.  Sounds like there should be some option or maybe a listener for your
case.

Kind regards,
Frik Strecker

Voice: +1-603-672-1472
http://www.GatherPlace.NET
Effective On-line Presentations

 

> -----Original Message-----
> From: Dennis Cook [mailto:[hidden email]]
> Sent: Tuesday, August 02, 2005 3:10 PM
> To: [hidden email]
> Subject: RE: [activemq-dev] Re: [activemq-user] Re: ActiveMQ
> hangs when initial connection to broker fails using reliable transport
>
> While this seems like it should be the most logical solution,
> but I would like to share a couple of thoughts.
>
> If I provide a list of brokers in a reliable mode and at
> startup it cannot find any of the brokers specified, I
> consider this to be a severe error that I want to know about
> immediately.  It usually means that I have a configuration
> problem or a network problem. If the client quietly enters
> into the "wait for a broker to become available" mode, then
> my application start up in a crippled mode and I don't get
> the immediate notification.
>
> So I am sort of leaning toward a solution that just throws an
> exception back to the client at connection creation time if
> no broker is available.  Or at least some notification via
> the TransportStatusEventListener that the initial connection
> could not be immediately established.
>
> -----Original Message-----
> From: Ramzi Saba [mailto:[hidden email]]
> Sent: Tuesday, August 02, 2005 12:21 AM
> To: [hidden email]
> Subject: [activemq-dev] Re: [activemq-user] Re: ActiveMQ hangs when
> initial connection to broker fails using reliable transport
>
>
> Folks, does this sound like a decent proposal, any concerns
> before I log it in
> JIRA?
>
> Ramzi Saba ([hidden email]) wrote:
> >
> > Agreed, I suggest we make these changes:
> > 1- Start the reliable tcp channel in its own thread (seems
> there was an
> > attempt to do so anyway)
> > 2- Synchronous and asynchronous client calls (via session,
> consumer, etc.) to
> > the reliable tcp channel should simply verify if a reliable
> channel has been
> > already established, else throw a JMSException,
> alternatively allow the client
> > to configure a timeout (currently it's hardcoded for
> synchronous calls only I
> > believe).
> > 3- Other than starting the reliable channel, the client
> should not be
> > responsible of reestablishing a lost or unavailable
> channel.  I would delegate
> > reliability to the reliable channel itself.
> >
> > -ramzi
> >
> > Dennis Cook ([hidden email]) wrote:
> > >
> > > Ah, yes I see your point about hanging at startup.  I
> have never run
> > > across that particular problem because I have multiple
> brokers in my
> > > reliable list.  At any given time it is able to reach one of them.
> > >
> > > I doubt a fix that performs a background initial
> connection will be
> > > forthcoming though.  From the applications stand point
> there are many
> > > synchonous operation that must follow the connection
> creation such as
> > > session, consumer, and producer creation.  These in turn
> would have to
> > > queued up for processing.  This is getting ugly.
> > >
> > > I think I would rather just have an initial failure throw
> an exception
> > > out just as it would when using the tcp protocol without
> reliable.  I
> > > already handle that condition within the application. :D
> > >
> > > Matthew Sinclair-Day wrote:
> > > > But in this case the app is not trying to send a message.  It is
> > > > attempting to open a connection to consume a message,
> and now it blocks
> > > > forever until the broker comes up.
> > > >
> > > > My main point is now there seems to be two behaviors in
> the system,
> > > > depending on whether the connection had been initially
> established.
> > > >
> > > > Consider the semantics of
> MessageConsumer.receive(long).  It allows for
> > > > a null message to be returned to the caller on timeout.
>  What the
> > > > "reliable" feature made so nice was that the caller could be
> > > > transparently resilient to connection failures.  It
> mitigated the
> > > > effects of an inherently unreliable resource.  The
> caller could be
> > > > written not to care  whether a null Message were caused
> by a connection
> > > > failure.  Underneath ActiveMQ would work to reestablish
> the connection,
> > > > but up top the caller could continue calling
> > > > MessageConsumer.receive(long) and react to null or
> actual Message objects.
> > > >
> > > > It also meant that clients and broker could be started
> in any order.  A
> > > > client could be brought up before the broker and
> independently of it.
> > > > That's a VERY nice thing when managing a large,
> distributed system.  As
> > > > it is now in the system I am building, the broker will
> now have to be
> > > > running in order for the client to be started
> completely, and that will
> > > > cause operational problems; of course, this is my
> problem to address.
> > > >
> > > > I would almost prefer a JMSException be thrown if the
> initial connection
> > > > failed with "reliable."
> > > >
> > > > Matt
> > > >
> > > >
> > > > On 2005-07-28 13:56:42 -0400, Dennis Cook
> > > > <[hidden email]> said:
> > > >
> > > > But even in a reconnect case, where initial connection
> is lost.  It is
> > > > usually the keep-alive thread that is trying to
> reconnect.  But if the
> > > > application thread tried to send a message while this
> process was going
> > > > on, it too would still block.  And I think it should.  
> Since the client
> > > > cannot synchonously send the message it only has two
> options.  It can
> > > > throw an exception back to the application or it can
> block the thread.
> > > > By definition of reliable, I would say blocking for the
> reconnect is the
> > > > correct solution.  Then again the useAsynSend can be
> used and the
> > > > application can go on its merry way.  Of course there
> is always the
> > > > potentional for loss of the message that way, if the
> connection is never
> > > > reestablished.
> > > >
> > > >>
> > > >>
> > > >> Ramzi Saba wrote:
> > > >> I also agree on the reliable aspect.  Though it would
> also be nice if the
> > > >>
> > > >>> client could transparently keep trying to connect to
> the broker
> > > >>> behind the
> > > >>> scenes on initial connect, just like it works when the broker
> > > >>> restarts and the
> > > >>> client automatically keeps trying to reconnect
> without blocking the
> > > >>> client.
> > > >>>
> > > >>> Dennis Cook ([hidden email]) wrote:
> > > >>>
> > > >>>
> > > >>> I would have to agree.  I expect reliable connections
> to wait for a
> > > >>>
> > > >>>> reconnection then send my message.  If the
> application thread cannot be
> > > >>>> blocked on a send, then it should spawned another to
> do the send or I
> > > >>>> think you there is an asyncSend option on the
> ActiveMQConnectcion.
> > > >>>> Maybe that would help in this case.
> > > >>>>
> > > >>>>
> > > >>>> Hiram Chirino wrote:
> > > >>>>
> > > >>>> Hi Matt,
> > > >>>>
> > > >>>>>
> > > >>>>> I would think that this is the correct behavior of
> a reliable
> > > >>>>> connection.  It's purpose is to hide connection
> failures for the JMS
> > > >>>>> client and to recover correctly.  Anybody else have
> an opinion?
> > > >>>>>
> > > >>>>> Regards,
> > > >>>>> Hiram
> > > >>>>>
> > > >>>>>
> > > >>>>> On Jul 28, 2005, at 9:11 AM, Matthew Sinclair-Day wrote:
> > > >>>>>
> > > >>>>>
> > > >>>>> Hiram,
> > > >>>>>
> > > >>>>>>
> > > >>>>>> Yes, the client connection becomes un-stuck when
> the broker comes
> > > >>>>>> back up.  The problem is that until it becomes
> un-stuck the rest of
> > > >>>>>> my application hangs.  I guess I need to know
> whether ActiveMQ's
> > > >>>>>> behavior is considered correct in this case, so
> that changes in my
> > > >>>>>> code to address this situation can be made, or
> whether I can wait
> > > >>>>>> for
> > > >>>>>> another build that would fix it.
> > > >>>>>>
> > > >>>>>> Matt
> > > >>>>>>
> > > >>>>>> On 2005-07-27 17:00:45 -0400, Hiram Chirino
> > > >>>>>> <[hidden email]> said:
> > > >>>>>>
> > > >>>>>>
> > > >>>>>>
> > > >>>>>> Hi Matt,
> > > >>>>>>
> > > >>>>>>> Does the client connection get un-stuck if the
> broker comes back up?
> > > >>>>>>> Regards,
> > > >>>>>>> Hiram
> > > >>>>>>> On Jul 26, 2005, at 5:27 PM, Matthew Sinclair-Day wrote:
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> Hello.
> > > >>>>>>>
> > > >>>>>>>> I am trying to track down a problem with
> ActiveMQ 3.1-M6 using
> > > >>>>>>>> "reliable:tcp" transport to a broker.  In this
> case, the broker  is
> > > >>>>>>>> not running when the client comes up.  When the
> client is  setting
> > > >>>>>>>> up its connection, it calls
> Connection.setClientID()  prior to
> > > >>>>>>>> creating a Session.  In this case, because the
> broker  is not
> > > >>>>>>>> running, ActiveMQ seems to be stuck in
> > > >>>>>>>>
> org.activemq.transport.composite.CompositeTransportChannel.
> > > >>>>>>>> Relevant portion from thread's call list:
> > > >>>>>>>> WSDMSGRECEIVER_SERVICE@3e1 daemon prio=5, in
> group "main",  status:
> > > >>>>>>>> RUNNING
> > > >>>>>>>>      sleep():-1, Thread.java
> > > >>>>>>>>      establishConnection():284,
> CompositeTransportChannel.java
> > > >>>>>>>>      start():86, CompositeTransportChannel.java
> > > >>>>>>>>      checkClosed():805, ActiveMQConnection.java
> > > >>>>>>>>      setClientID():554, ActiveMQConnection.java
> > > >>>>>>>> When "reliable" transport is removed then the
> connection fails
> > > >>>>>>>> fast, as expected.
> > > >>>>>>>> I believe this is new behavior as of the M6
> build or maybe M5 (I
> > > >>>>>>>> have not tested M5).
> > > >>>>>>>> Matt
> > > >>>>>>>>
> > > >
> > > >
> > > >
> > >
> > >
> >
> >
>
>