AMQ 5.4.1 File Descriptor (FD) leak of ticots loopback networks
This (after plenty searching) is not a request for help so much as an early start/nucleation point for this issue: somewhere to find and hang more info.
[Checked: We're not using Stomp + NIO; We do not have the symptoms of AMQ-2999 (don't use peer transport.); There is no thread leak]
No reproducible test case yet, but when isolated, we will update. First seen on migration to 5.4.1 from 4.1.1, but various changes in amq xml configuration were made at the same time. We have some 4.1.1 clients remaining, and understand that the combination of 4.1.1 client & 5.4.1 broker is "tested by the community" i.e. not tested by the Apache project.
Environment: Solaris 10 u9, Java HotSpot(TM) 64-Bit Server VM version 17.0-b16.
Single, standalone broker 5.4.1 running as SMF service. Multiple clients on same machine and others.
We have leaked up to 2000 ticots FDs so far on the busiest of our test deployments - and it's only running approx. 5K messages per minute at the moment. [During load test we would put of the order of 1M messages per minute through the broker.]
Symptom: new FDs of the form (as seen from output of "lsof -p AMQpid | grep ticots" ):
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 12345 some_user someFDnum unix 105,xyz 0t055050244/devices/pseudo/tl@0:ticots->(socketpair: 0xsomenum) (0xsomenum)
I've bolded the non-varying parts.
This leak is suspected to be triggered by particular JMS/Openwire traffic. We now have full DEBUG logging on and are trying to corellate the creation of a new ticots FD with particular events in the log - and perhaps linking the leak to a particular client.
Also using Dtrace to look at FD creation and disposal.
Changes from previous broker deployment, where the issue was not seen:
ActiveMQ version: 4.1.1 -> 5.4.1
TCP transport -> NIO transport (broker only change)
discardingDLQ plugin introduced.
Update on this File Descriptor Leak with 5.4.1 broker and NIO transport - how to workaround/avoid.
The leak stops when transport changed to TCP in broker xml.
The leak resumes when transport changed back to NIO.
I.e. this fault is only seen with NIO transport.
I will update with a test case when I have one.
> Update on this File Descriptor Leak with 5.4.1 broker and NIO transport -
> to workaround/avoid.
> The leak *stops* when transport changed to TCP in broker xml.
> The leak *resumes *when transport changed back to NIO.
> I.e. this fault is only seen with NIO transport.
> I will update with a test case when I have one.
> Michael Hayes B.Sc. (NUI), M.Sc. (DCU), SCSA SCNA
> View this message in context:
> http://activemq.2283324.n4.nabble.com/AMQ-5-4-1-File-Descriptor-FD-leak-of-ticots-loopback-networks-tp3653804p3703254.html > Sent from the ActiveMQ - User mailing list archive at Nabble.com.