Community
Participate
Working Groups
We just got a bug report that a product we make, in which we use Jetty, was taking up 100% CPU. Looking at the thread dump that came along I was a bit puzzled as we had no running threads... Until I looked at the log files and discovered thousands of entries like this: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Connection reset at com.sun.net.ssl.internal.ssl.SSLSocketImpl.checkEOF(Unknown Source) at com.sun.net.ssl.internal.ssl.AppInputStream.read(Unknown Source) at org.eclipse.jetty.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:388) at org.eclipse.jetty.io.bio.StreamEndPoint.fill(StreamEndPoint.java:132) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.fill(SocketConnector.java:209) at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.fill(SslSocketConnector.java:612) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:289) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:214) at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:411) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:241) at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.run(SslSocketConnector.java:664) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) at java.lang.Thread.run(Unknown Source) Caused by: javax.net.ssl.SSLException: java.net.SocketException: Connection reset at com.sun.net.ssl.internal.ssl.Alerts.getSSLException(Unknown Source) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(Unknown Source) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.fatal(Unknown Source) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.handleException(Unknown Source) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.handleException(Unknown Source) ... 12 more Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.readFully(Unknown Source) at com.sun.net.ssl.internal.ssl.InputRecord.read(Unknown Source) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(Unknown Source) at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(Unknown Source) ... 12 more How exactly it has been created I have no clue. The server was running on a VMWare server over night and then when crazy when our tester started using it the next day.
Created attachment 207210 [details] One of our log files just to show how this ends up eating CPU (and disk space) ;)
Kristoffer, this is a know problem with the way we have been handling half closed connections for SSL. We have a temporary fix in the latest 7.5.x release that appears to stop the 100% and we are currently working on a cleaner fix for the 7.6.0 release, which should be out in a few weeks. How easily can you reproduce? can you try 7.5.4 or a SNAPSHOT build of 7.6 and see if it avoid this spin?
As far as I know we have only seen this a couple of times in test, so far. So we don't have a simple way to reproduce it at this time. Anyway. I already moved on to 7.5.4 as a reaction to this. Just in hope something had changed that could fix this. So we will be using that in out next builds.
Turns out it was happening a lot in test with 7.4.5. So they we made a build with 7.5.4 and we'll see if they keep getting it.
A much better/cleaner fix for this type of issue is in Jetty-7.6 (now head of master branch). A 7.6.0.RC0 release will be out in a few days. I will close this issue now, please reopen if you see any reproduction on 7.6.0
