Bug 363757

Description James Mace 2011-11-14 16:48:14 EST

Build Identifier: 8.0.4.v20111024

When sending large amounts (several hundred KB) of data over a socket connection, it's relatively common for HttpGenerator to be unable to completely flush its buffers of pending data due to the socket buffers becoming saturated.  When chunked transfer encoding is used under these conditions, HttpGenerator can exhibit a number of bugs.  The most common is the omission of one or more chunk headers, though other abnormal behavior is possible.

The incorrectly encoded data stream typically results in a 'bad chunk char' exception on the server.

There appear to be three main problems with the code:

1. The 3-Buffer variant of ChannelEndPoint.flush() will move data between buffers if sufficient space exists.  Doing so may invalidate assumptions made by the chunk encoding logic and the bypass write optimization.  HttpGenerator must take care to not allow buffer coalescing when performing chunk encoding.

2. There are some code paths in HttpGenerator that fail to properly reset the _bypass and _bufferChunked flags when flushes cannot be fully absorbed by the ChannelEndPoint.

3. The _content Buffer will often retain a reference to the Buffer passed to  HttpGenerator.addContent().  Normally, _content is nulled out when it is emptied; however, there exist code paths for which this is not the case.  When this happens, it's possible to place incoming data in an inappropriate location for chunk encoding.

I've attached a reworked version of HttpGenerator which (I believe) addresses these problems.  Additional comments are present in the code.

Additionally, I have attached a test case (testcase.SluggishServer) demonstrating the problem.

Setting severity to 'major' as this interferes with use of chunked transfer encoding for large requests and responses.

Reproducible: Always

Steps to Reproduce:
1. Compile the attached test case (testcase.SluggishServer)
2. Run the test case with the arguments (0, 2000000, 5)
3. If the bug does not manifest, try increasing the request size (second argument to the test case) or altering the READ_DELAY constant in the test case.

The key here is to get the socket buffers into a state in which they are unable to accept all of the pending HttpGenerator data in a single flush.  I'm able to reproduce the problem easily on an Ubuntu 10.04 system; it's somewhat more difficult to trigger on Mac OS.  I've not tried on Windows.