Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 364132

Summary:

Apparent infinite loop when an SSL connection is reset

Product:

[RT] Jetty

Reporter:

Kristoffer Peterhänsel <kristoffer>

Component:

server

Assignee:

Greg Wilkins <gregw>

Status:

RESOLVED FIXED

QA Contact:

Severity:

normal

Priority:

CC:

jetty-inbox

Version:

7.4.5

Target Milestone:

7.5.x

Hardware:

OS:

Windows XP

Whiteboard:

Attachments:

Description	Flags
One of our log files just to show how this ends up eating CPU (and disk space) ;)	none

Description Kristoffer Peterhänsel

2011-11-18 05:12:26 EST

We just got a bug report that a product we make, in which we use Jetty, was taking up 100% CPU. Looking at the thread dump that came along I was a bit puzzled as we had no running threads... Until I looked at the log files and discovered thousands of entries like this:

javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Connection reset
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.checkEOF(Unknown Source)
	at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source)
	at org.eclipse.jetty.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:388)
	at org.eclipse.jetty.io.bio.StreamEndPoint.fill(StreamEndPoint.java:132)
	at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.fill(SocketConnector.java:209)
	at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.fill(SslSocketConnector.java:612)
	at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:289)
	at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:214)
	at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:411)
	at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:241)
	at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.run(SslSocketConnector.java:664)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
	at java.lang.Thread.run(Unknown Source)
Caused by: javax.net.ssl.SSLException: java.net.SocketException: Connection reset
	at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Unknown Source)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(Unknown Source)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(Unknown Source)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.handleException(Unknown Source)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.handleException(Unknown Source)
	... 12 more
Caused by: java.net.SocketException: Connection reset
	at java.net.SocketInputStream.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source)
	at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source)
	at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source)
	... 12 more


How exactly it has been created I have no clue. The server was running on a VMWare server over night and then when crazy when our tester started using it the next day.

Comment 1 Kristoffer Peterhänsel

2011-11-18 05:13:35 EST

Created attachment 207210 [details]
One of our log files just to show how this ends up eating CPU (and disk space) ;)

Comment 2 Greg Wilkins

2011-11-20 23:22:37 EST

Kristoffer,

this is a know problem with the way we have been handling half closed connections for SSL.  We have a temporary fix in the latest 7.5.x release that appears to stop the 100% and we are currently working on a cleaner fix for the 7.6.0 release, which should be out in a few weeks.

How easily can you reproduce?  can you try 7.5.4 or a SNAPSHOT build of 7.6 and see if it avoid this spin?

Comment 3 Kristoffer Peterhänsel

2011-11-21 05:59:18 EST

As far as I know we have only seen this a couple of times in test, so far. So we don't have a simple way to reproduce it at this time.

Anyway. I already moved on to 7.5.4 as a reaction to this. Just in hope something had changed that could fix this. So we will be using that in out next builds.

Comment 4 Kristoffer Peterhänsel

2011-11-22 07:52:14 EST

Turns out it was happening a lot in test with 7.4.5. So they we made a build with 7.5.4 and we'll see if they keep getting it.

Comment 5 Greg Wilkins

2011-11-27 20:01:11 EST

A much better/cleaner fix for this type of issue is in Jetty-7.6 (now head of master branch).   A 7.6.0.RC0 release will be out in a few days.    I will close this issue now, please reopen if you see any reproduction on 7.6.0