"Wire format negotiation timeout: peer did not send his wire format"

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

"Wire format negotiation timeout: peer did not send his wire format"

Larry Meadors
I'm running AMQ 5.7.0 with a camel application (using camel 2.10.7, in
case that's relevant) and recently, the app has started logging these
messages:

WARN  o.a.c.c.j.DefaultJmsMessageListenerContainer - Setup of JMS
message listener invoker failed for destination 'play.save' - trying
to recover. Cause: Wire format negotiation timeout: peer did not send
his wire format.

Once that starts happening, it seems that AMQ drops all connections,
and won't allow any new ones to be created. The result is that nothing
can send or consume messages and I end up restarting AMQ to get things
rolling again.

That's not epic.

The odd thing is that this app ran for close to a year with the same
AMQ configuration, then started throwing these down about the same
time I increased the JVM memory limits for it and AMQ.

I'm kind of stumped and looking for options. I considered dropping the
memory limits back to where they were before, but the reason I
increased them is because we were getting a lot of volume, and that
appeared to be causing OOMs.

Any suggestions?

Larry
Reply | Threaded
Open this post in threaded view
|

Re: "Wire format negotiation timeout: peer did not send his wire format"

Larry Meadors
No hints?

This is killing me - I'm restarting AMQ several times a week now. :-/

On Wed, May 7, 2014 at 11:18 AM, Larry Meadors <[hidden email]> wrote:

> I'm running AMQ 5.7.0 with a camel application (using camel 2.10.7, in
> case that's relevant) and recently, the app has started logging these
> messages:
>
> WARN  o.a.c.c.j.DefaultJmsMessageListenerContainer - Setup of JMS
> message listener invoker failed for destination 'play.save' - trying
> to recover. Cause: Wire format negotiation timeout: peer did not send
> his wire format.
>
> Once that starts happening, it seems that AMQ drops all connections,
> and won't allow any new ones to be created. The result is that nothing
> can send or consume messages and I end up restarting AMQ to get things
> rolling again.
>
> That's not epic.
>
> The odd thing is that this app ran for close to a year with the same
> AMQ configuration, then started throwing these down about the same
> time I increased the JVM memory limits for it and AMQ.
>
> I'm kind of stumped and looking for options. I considered dropping the
> memory limits back to where they were before, but the reason I
> increased them is because we were getting a lot of volume, and that
> appeared to be causing OOMs.
>
> Any suggestions?
>
> Larry
Reply | Threaded
Open this post in threaded view
|

Re: "Wire format negotiation timeout: peer did not send his wire format"

ceposta
What networking/firewalls do you have between your clients and broker?

On Sun, May 11, 2014 at 7:30 AM, Larry Meadors <[hidden email]> wrote:

> No hints?
>
> This is killing me - I'm restarting AMQ several times a week now. :-/
>
> On Wed, May 7, 2014 at 11:18 AM, Larry Meadors <[hidden email]> wrote:
>> I'm running AMQ 5.7.0 with a camel application (using camel 2.10.7, in
>> case that's relevant) and recently, the app has started logging these
>> messages:
>>
>> WARN  o.a.c.c.j.DefaultJmsMessageListenerContainer - Setup of JMS
>> message listener invoker failed for destination 'play.save' - trying
>> to recover. Cause: Wire format negotiation timeout: peer did not send
>> his wire format.
>>
>> Once that starts happening, it seems that AMQ drops all connections,
>> and won't allow any new ones to be created. The result is that nothing
>> can send or consume messages and I end up restarting AMQ to get things
>> rolling again.
>>
>> That's not epic.
>>
>> The odd thing is that this app ran for close to a year with the same
>> AMQ configuration, then started throwing these down about the same
>> time I increased the JVM memory limits for it and AMQ.
>>
>> I'm kind of stumped and looking for options. I considered dropping the
>> memory limits back to where they were before, but the reason I
>> increased them is because we were getting a lot of volume, and that
>> appeared to be causing OOMs.
>>
>> Any suggestions?
>>
>> Larry



--
Christian Posta
http://www.christianposta.com/blog
twitter: @christianposta
Reply | Threaded
Open this post in threaded view
|

Re: "Wire format negotiation timeout: peer did not send his wire format"

Larry Meadors
Thanks for your time, Christian!

Firewall/networking between them: None. :-\

They are running on the same (virtual) machine - it's an EC2 host
running ubuntu.

I am running 3 instances of AMQ on it (it's a dev/test/qa type of
environment) - the AMQ endpoints are configured as localhost:61617,
localhost:61618, and localhost:61619.

It's so strange - it'll work perfectly for several days, then the
error rate starts climbing. I monitor it with AWS Cloudwatch, and I
see solid performance for some time (I'm not seeing a pattern yet),
then it starts rejecting messages (not all, just some), and then it
kind of snowballs until it's rejecting 50-60% of them and things go
bad from there. :(

If there's any other info I can provide to help track this down, just
ask - I'm looking to figure this out and understand why it's failing.

I'm guessing it's something I buggered up in my configuration. :-/

If it would help, I can gist one of them on github and share a link to that.

Larry
Reply | Threaded
Open this post in threaded view
|

Re: "Wire format negotiation timeout: peer did not send his wire format"

Larry Meadors
Just for posterity, this was me being a bonehead - I didn't have a JMS
connection pool in my camel config - once I added one things got MUCH
happier.

Larry


On Mon, May 12, 2014 at 11:07 AM, Larry Meadors <[hidden email]> wrote:

> Thanks for your time, Christian!
>
> Firewall/networking between them: None. :-\
>
> They are running on the same (virtual) machine - it's an EC2 host
> running ubuntu.
>
> I am running 3 instances of AMQ on it (it's a dev/test/qa type of
> environment) - the AMQ endpoints are configured as localhost:61617,
> localhost:61618, and localhost:61619.
>
> It's so strange - it'll work perfectly for several days, then the
> error rate starts climbing. I monitor it with AWS Cloudwatch, and I
> see solid performance for some time (I'm not seeing a pattern yet),
> then it starts rejecting messages (not all, just some), and then it
> kind of snowballs until it's rejecting 50-60% of them and things go
> bad from there. :(
>
> If there's any other info I can provide to help track this down, just
> ask - I'm looking to figure this out and understand why it's failing.
>
> I'm guessing it's something I buggered up in my configuration. :-/
>
> If it would help, I can gist one of them on github and share a link to that.
>
> Larry