ActiveMQ 5.1.0 and 5.2.0 runs out of file descriptors due to CLOSE_WAIT bug

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

ActiveMQ 5.1.0 and 5.2.0 runs out of file descriptors due to CLOSE_WAIT bug

vjavarap
We have been experiencing severe issues with ActiveMQ 5.1.0 running out of file descriptors and netstat reveals thousands of tcp connections stuck in CLOSE_WAIT within a few minutes or hours or days of starting up based on the event volume.

Clearly, other activemq users are seeing the same issue: https://issues.apache.org/activemq/browse/AMQ-1739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel

The ActiveMQ runs on a Linux (CentOS) server. We are using stomp as the transport connector and production and consumption are via PHP clients.

Problem Definition
- When we add thousands of events onto the queue with no consumption, the events are registered and things work well.
- When events are consumed later when no events are being added at the same time, things work smoothly.
- However, when events are being added and consumed at the same time, the CLOSE_WAITs shoot up into the thousands (or tens of thousands) based on event activity.

We have tried upgrading to ActiveMQ 5.2.0 and having this in the config:

<transportConnector name="stomp" uri="stomp://localhost:61613?transport.closeAsync=false"/>

Once the CLOSE_WAITs shoot up into the thousands, the event queue stops accepting events.

I've set the ulimit for open files to 32K and we are nowhere close to the upper limit.

Any thoughts?
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ 5.1.0 and 5.2.0 runs out of file descriptors due to CLOSE_WAIT bug

Arjen van der Meijden
Did you also remove the limits for queue sizes (most notably the one
that limits all queues to only 5MB)?

With the ulimit up to 10240, the async closing turned off and the limits
for the seperate queues removed (we just disabled the entire
'destinationPolicy'-element from the default config) and the total
storage size increased we haven't seen any of those problems.

Best regards,

Arjen

On 27-2-2009 23:54 vjavarap wrote:

> We have been experiencing severe issues with ActiveMQ 5.1.0 running out of
> file descriptors and netstat reveals thousands of tcp connections stuck in
> CLOSE_WAIT within a few minutes or hours or days of starting up based on the
> event volume.
>
> Clearly, other activemq users are seeing the same issue:
> https://issues.apache.org/activemq/browse/AMQ-1739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel
>
> The ActiveMQ runs on a Linux (CentOS) server. We are using stomp as the
> transport connector and production and consumption are via PHP clients.
>
> Problem Definition
> - When we add thousands of events onto the queue with no consumption, the
> events are registered and things work well.
> - When events are consumed later when no events are being added at the same
> time, things work smoothly.
> - However, when events are being added and consumed at the same time, the
> CLOSE_WAITs shoot up into the thousands (or tens of thousands) based on
> event activity.
>
> We have tried upgrading to ActiveMQ 5.2.0 and having this in the config:
>
> <transportConnector name="stomp"
> uri="stomp://localhost:61613?transport.closeAsync=false"/>
>
> Once the CLOSE_WAITs shoot up into the thousands, the event queue stops
> accepting events.
>
> I've set the ulimit for open files to 32K and we are nowhere close to the
> upper limit.
>
> Any thoughts?