Quantcast

AMQ 5.4.1 File Descriptor (FD) leak of ticots loopback networks

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

AMQ 5.4.1 File Descriptor (FD) leak of ticots loopback networks

mickhayes
Hi folks,

This (after plenty searching) is not a request for help so much as an early start/nucleation point for this issue: somewhere to find and hang more info.
[Checked: We're not using Stomp + NIO; We do not have the symptoms of AMQ-2999 (don't use peer transport.); There is no thread leak]

No reproducible test case yet, but when isolated, we will update. First seen on migration to 5.4.1 from 4.1.1, but various changes in amq xml configuration were made at the same time. We have some 4.1.1 clients remaining, and understand that the combination of 4.1.1 client & 5.4.1 broker is "tested by the community" i.e. not tested by the Apache project.

Environment: Solaris 10 u9,  Java HotSpot(TM) 64-Bit Server VM version 17.0-b16.
Single, standalone broker 5.4.1 running as SMF service. Multiple clients on same machine and others.
We have leaked up to 2000 ticots FDs so far on the busiest of our test deployments - and it's only running approx. 5K messages per minute at the moment. [During load test we would put of the order of 1M messages per minute through the broker.]

Symptom: new FDs of the form (as seen from output of "lsof -p AMQpid | grep ticots" ):
COMMAND  PID   USER   FD   TYPE        DEVICE    SIZE/OFF     NODE NAME
java    12345 some_user   someFDnum  unix       105,xyz         0t0 55050244 /devices/pseudo/tl@0:ticots->(socketpair: 0xsomenum) (0xsomenum)

I've bolded the non-varying parts.

This leak is suspected to be triggered by particular JMS/Openwire traffic. We now have full DEBUG logging on and are trying to corellate the creation of a new ticots FD with particular events in the log - and perhaps linking the leak to a particular client.
Also using Dtrace to look at FD creation and disposal.

Changes from previous broker deployment, where the issue was not seen:
ActiveMQ version: 4.1.1 -> 5.4.1
TCP transport -> NIO transport (broker only change)
discardingDLQ plugin introduced.

All the best,
Mick Hayes.



Michael Hayes B.Sc. (NUI), M.Sc. (DCU), SCSA SCNA
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: AMQ 5.4.1 File Descriptor (FD) leak of ticots loopback networks

mickhayes
Update on this File Descriptor Leak with 5.4.1 broker and NIO transport - how to workaround/avoid.
The leak stops when transport changed to TCP in broker xml.
The leak resumes when transport changed back to NIO.
I.e. this fault is only seen with NIO transport.
I will update with a test case when I have one.
Michael Hayes B.Sc. (NUI), M.Sc. (DCU), SCSA SCNA
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: AMQ 5.4.1 File Descriptor (FD) leak of ticots loopback networks

dejanb
Hi,

is there a chance you can test nio with 5.5.0 or the latest 5.6-SNAPSHOT and
see if problem is still there? A test case would be great if you manage to
provide it.

Regards
--
Dejan Bosanac - http://twitter.com/dejanb
-----------------
The experts in open source integration and messaging - http://fusesource.com
ActiveMQ in Action - http://www.manning.com/snyder/
Blog - http://www.nighttale.net


On Fri, Jul 29, 2011 at 9:18 AM, mickhayes <[hidden email]> wrote:

> Update on this File Descriptor Leak with 5.4.1 broker and NIO transport -
> how
> to workaround/avoid.
> The leak *stops* when transport changed to TCP in broker xml.
> The leak *resumes *when transport changed back to NIO.
> I.e. this fault is only seen with NIO transport.
> I will update with a test case when I have one.
>
>
> -----
> Michael Hayes B.Sc. (NUI), M.Sc. (DCU), SCSA SCNA
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/AMQ-5-4-1-File-Descriptor-FD-leak-of-ticots-loopback-networks-tp3653804p3703254.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>
Loading...