Description Chris Dumoulin 2011-05-04 12:44:55 EDT

Build Identifier: 7.4.0.v20110414

The SelectChannelEndpoint used by the HttpConnection has an idle timeout, which is being set the same as the socket timeout. There is a race condition where if the endpoint hits its idle timeout and closes after the HttpConnection has been selected for an HttpExchange but before the idle timeout gets reset, the HttpExchange will sit around without being serviced, until it eventually times out.

The following code is from org.eclipse.jetty.client.HttpConnection.handle():
{code}
public Connection handle() throws IOException
    {
        if (_exchange != null)
            _exchange.associate(this);

        try
        {
            int no_progress = 0;

            boolean failed = false;
            while (_endp.isBufferingInput() || _endp.isOpen())
...
{code}

In the case where _endp is closed (due to having just hit its idle timeout) we never enter this while loop and the HttpExchange never gets serviced. It eventually expires. I've been able to figure this out by adding some custom log lines to the code and tracing the execution.

I've seen this in my app after setting the timeout to 5 seconds with a call to HttpClient.setTimeout(5000). This is happening with HttpConnections that are being reused exactly 5 seconds after they were last used.

I would expect this race condition to be something that happens very rarely, but it happens quite often; I think it has to do with the fact that the socket timeout and idle timeout are the same. I think something is waiting for 5 seconds for data to come in on the previously used connection, at which point the wait times out and the connection becomes available for use. Then, after the connection is chosen for use with an new HttpExchange the idle timeout occurs.

Based on the logging I got, the endpoint usually times out and closes after the call to AsyncEndpoint.scheduleWrite() and before the call to HttpClient.schedule(_timeout) in HttpConnection.handle().

I see there being two issues here:
1. The race condition whereby a connection is closed due to being idle after it has been chosen to service a request.
2. The fact that the SelectChannelEndpoint's idle timeout is set to the socket timeout, with no way for users of HttpClient to change it.

For issue number 2 above, I'm attaching a patch that sets the SelectChannelEndpoint idle timeout to the same as the HttpClient idle timeout. I'm not sure if this is the best way to go, because I could see you wanting a separate idle timeout for the HttpClient as a whole and the individual connections.

Reproducible: Sometimes

Steps to Reproduce:
1. Call HttpClient.setTimeout(5000)
2. Periodically send requests using HttpClient, always to the same host
3. Look for HttpExchanges that expire/timeout