Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.
Bug 331461 - HTTP/1.0 requests leak SelectChannelEndPoint
Summary: HTTP/1.0 requests leak SelectChannelEndPoint
Status: RESOLVED FIXED
Alias: None
Product: Jetty
Classification: RT
Component: server (show other bugs)
Version: unspecified   Edit
Hardware: PC Linux
: P3 normal (vote)
Target Milestone: 7.1.x   Edit
Assignee: Greg Wilkins CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-11-30 13:50 EST by Jan Prach CLA
Modified: 2010-12-02 15:08 EST (History)
1 user (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jan Prach CLA 2010-11-30 13:50:29 EST
Build Identifier: 7.2.1.v20101111

Hi,

HTTP/1.0 requests leak SelectChannelEndPoint instances! The instances of this class won't get freed, not even after days. Each instance keeps a lot of references within JVM and also system file descriptors / sockets. Java heap and file descriptors disappear really quickly - within minutes if we would put it into production!

I can provide further information. The investigation generated lots of tcpdumps, straces, heapdumps and other stuff :-)


Test environment(s):
  * CentOS with 2.6.18-194.3.1.el5 kernel
  * Ubuntu 10.04 with default 2.6.32 and 2.6.35-22-server kernels
  * JVM 1.6.0_20 and 1_6_22
  * jetty 7.1.6, 7.2.1, 8.0.0.M2 - all of them
  * default settings of all the components (just file descriptor limit increased)



Requests:
POST /helloworld HTTP/1.0
Content-Type: application/json
Content-Length: 105
User-Agent: Syncer/5.00 (unknown)
Host: some.host.com
Connection: Keep-Alive
Pragma: no-cache

{"some json data here": "foo"}



Simplified program that leaks is the simplest hello world as follows:
import java.io.IOException;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import org.eclipse.jetty.server.Request;
import org.eclipse.jetty.server.Server;
import org.eclipse.jetty.server.handler.AbstractHandler;

public class Foo {
  public static class HelloHandler extends AbstractHandler {
    @Override
    public void handle(final String target, final Request baseRequest,
        final HttpServletRequest request, final HttpServletResponse response)
        throws IOException, ServletException {
      response.setStatus(HttpServletResponse.SC_OK);
      baseRequest.setHandled(true);
      response.getWriter().println("Hello World");
    }
  }

  public static void main(final String[] args) throws Exception {
    final Server server = new Server(8088);
    server.setHandler(new HelloHandler());
    server.start();
    server.join();
  }
}


Reproducible: Always
Comment 1 Jan Prach CLA 2010-12-01 08:35:42 EST
wait with this one yet, please

I'll provide better description soon...

the issue is there, it's 100% repeatable but not with the description I provided so far... so don't border with it yet, I'll exact the exact requests causing the problem
Comment 2 Jan Prach CLA 2010-12-01 10:22:15 EST
ok, I got it ... hopefully ;-)

The problem is caused by "Connection: Keep-Alive" in "HTTP/1.0".

Client sends http/1.0 request with keep-alive but it never closes the connection. Jetty will never do so either. Jetty rather consume all file descriptors or the whole heap and die on OutOfMemoryError.

Can jetty by configured in a way that it time-outs these connections just like it does with HTTP/1.1? I believe "connector.setMaxIdleTime(...)" should do so, right? It does not work for http 1.0.
Comment 3 Greg Wilkins CLA 2010-12-02 08:56:28 EST
Ouch! replicated and investigating!!!
Comment 4 Greg Wilkins CLA 2010-12-02 09:38:05 EST
fixed r2574

The issue was that the idle timeout is scheduled in a called to persist(EndPoint), but for HTTP/1.0 if a response was not committed when the handler exited, it had not processed the Connection:keep-alive header and thus didn't know if the connection was persistent.

We are now checking the persistence of the connection after committing the response.
Comment 5 Jan Prach CLA 2010-12-02 15:08:01 EST
perfect, it works in real world with loots of clients ;-)

thanks!