Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 324330

Summary:	Jetty v7.1.6.v20100715 is running out of resources of open files
Product:	[RT] Jetty	Reporter:	KARASZI István <eclipse>
Component:	server	Assignee:	Greg Wilkins <gregw>
Status:	RESOLVED WORKSFORME	QA Contact:
Severity:	normal
Priority:	P3	CC:	eclipse, jesse.mcconnell, jetty-inbox, mgorovoy
Version:	unspecified
Target Milestone:	7.1.x
Hardware:	PC
OS:	Linux
Whiteboard:

Description KARASZI István

2010-09-02 12:53:28 EDT

Build Identifier: 7.1.6.v20100715

If listing the open files with lsof -p PID then it shows a lots of:

java    32262 user 4202u  sock                0,4      0t0 365914626 can't identify protocol

and later it's running out of the max number of open files.

7.1.3.v20100526 is not affected.

Reproducible: Always

Comment 1 Jesse McConnell

2010-09-02 14:07:54 EDT

can you give us your jetty.xml or however your configuring it?

Comment 2 KARASZI István

2010-09-03 05:04:17 EDT

I'm using embedded jetty so I don't have any jetty.xml, but I use the following environment:

BlockingChannelConnector with maxidletime of 5000 (but i've tried with 30000 as well) and reuseAddress is true.

QueuedThreadPool with maxThreads 1024 and maxidletimeMS 4000, setMaxStopTimeMs and setMaxQueued is the default.

Comment 3 Greg Wilkins

2010-09-07 02:33:52 EDT

The BlockingChannelConnector is not well used.  Can you try the SelectChannelConnector and see if that changes anything.

Comment 4 KARASZI István

2010-09-07 12:11:08 EDT

I'll try to benchmark it and reproduce the problem but according to my measures the BlockingChannelConnector is much faster than SelectChannelConnector with the same settings. So for production use I could not use SelectChannelConnector.

Comment 5 Greg Wilkins

2010-09-07 22:33:34 EDT

Often the blocking connectors do benchmark faster than the select channel connector. 

But more often than not, that is because benchmarks are unrealistic in the load they generate.  For example, they often have a few very busy connections and try to run the maximum load over each connection.   In such circumstances, every small increase in request handling latency causes a large drop in throughput.

but in almost all production systems, real load comes from many mostly idle connections, where the latency of a dispatch does not reduce throughput at all.

Have a read of http://blogs.webtide.com/gregw/entry/lies_damned_lies_and_benchmarks

and http://blogs.webtide.com/gregw/entry/cometd_2_throughput_vs_latency


If you can confirm the FD leak is the blocking connector, we will fix it, but I expect that the select channel connector will still be a better choice.

Comment 6 KARASZI István

2010-09-10 09:52:46 EDT

I can't reproduce this with the SelectChannelConnector so the bug is affected only the BlockingChannelConnector.

I'm considering to change to SelectChannelConnector thanks for the articles!

Comment 7 Jesse McConnell

2010-10-06 18:00:58 EDT

I tested this out some with the jetty-server tests while looking through the lsof output and I am not seeing anything off especially.

I am going to close out this issue but if you can reproduce or can give us debug output from running the process that is causing the issue for you please reopen and attach it. A thread dump of the running process would be extremely handy as well.

Reading through the BlockingChannelConnector it seems to me that things should be house-kept correctly but if this is reproducible for you then getting us some more information is what is required at this point.

cheers

Comment 8 KARASZI István

2010-10-06 19:13:40 EDT

It is reproducible in my environment, so if you could write me what kind of debug output you need, I can send you tomorrow (it's 1am here).

Comment 9 KARASZI István

2010-10-25 08:05:55 EDT

you forgot to reply me about the debug information you needed, but the new stable (7.2.0.v20101020) seems like not having this bug.