Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 355048

Summary: Stuck threads with SelectChannelConnector
Product: [RT] Jetty Reporter: KARASZI István <eclipse>
Component: serverAssignee: Greg Wilkins <gregw>
Status: CLOSED FIXED QA Contact:
Severity: normal    
Priority: P3 CC: bugs, eclipse, flavio.donze, gregw, gunnar, janb, jetty-inbox, joakim.erdfelt, mgorovoy
Version: unspecified   
Target Milestone: 7.5.x   
Hardware: All   
OS: Linux   
Whiteboard:

Description KARASZI István CLA 2011-08-18 05:39:23 EDT
Build Identifier: 

Since the last week we switched to SelectChannelConnector form BlockingChannelConnector with the newest stable Jetty (7.4.5.v20110725).

Everything looked fine but after half a week on one of my servers the load average jumped instantly +2. We had no clue because the load balancer before the servers balancing the requests equally.

A restart fixed the problem, but another half week passed and the same thing happened. But this time I managed to get a stacktrace dump from it. The stuck processes is the following:

at java.lang.Throwable.fillInStackTrace(Native Method)
at java.lang.Throwable.<init>(Throwable.java:196)
at java.lang.Exception.<init>(Exception.java:41)
at java.io.IOException.<init>(IOException.java:41)
at javax.net.ssl.SSLException.<init>(SSLException.java:33)+
at com.sun.net.ssl.internal.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1267)
at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:65)
at org.eclipse.jetty.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:388)
at org.eclipse.jetty.io.bio.StreamEndPoint.fill(StreamEndPoint.java:132)
at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.fill(SocketConnector.java:209)
at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.fill(SslSocketConnector.java:612)
at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:289)
at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:214)
at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:411)
at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:241)
at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.run(SslSocketConnector.java:664)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
at java.lang.Thread.run(Thread.java:619)

Reproducible: Sometimes
Comment 1 KARASZI István CLA 2011-08-18 05:40:28 EDT
maybe this is related to http://jira.codehaus.org/browse/JETTY-937
Comment 2 KARASZI István CLA 2011-08-18 05:43:04 EDT
My system details:

$ java -version
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)

$ uname -a:
Linux *** 2.6.32-32-server #62-Ubuntu SMP Wed Apr 20 22:07:43 UTC 2011 x86_64 GNU/Linux
Comment 3 Michael Gorovoy CLA 2011-08-24 19:27:33 EDT
> Since the last week we switched to SelectChannelConnector form
> BlockingChannelConnector with the newest stable Jetty (7.4.5.v20110725).

It appears that you are using SslSocketConnector according to the stack trace. Could you please make sure that the configuration is correct and attach your configuration file (or configuration code for embedded usage) to this ticket? 

Also, there have been some fixes that affect this area of the code that had been committed to the repository after 7.4.5 release. Could you please try testing with jetty 7.5.0.RC1 that can be downloaded from http://download.eclipse.org/jetty/?

> java version "1.6.0_20"
As far as we know the JVM bugs that were affecting Jetty had been fixed. Could you please try updating to the latest version of sun-java6-jdk package that is 1.6.0_26 at the time of this writing, and try testing with it? 

Cheers,
Michael
Comment 4 Michael Gorovoy CLA 2011-08-24 19:57:18 EDT
If you would like to attempt to run some diagnostics, you are welcome to try the brand new ThreadMonitor utility that is available as part of 7.5.0.RC1 release. It is part of jetty-monitor module, and you could find the instructions on how to configure this utility in README.txt file that you can be find at http://git.eclipse.org/c/jetty/org.eclipse.jetty.project.git/plain/jetty-monitor/README.txt
Comment 5 KARASZI István CLA 2011-08-25 07:49:06 EDT
We're testing it with the new Java JRE 1.6.0_26, if it works I'll let you know. If not I'll post the detailed information about the configs here.
Comment 6 KARASZI István CLA 2011-09-06 08:12:20 EDT
Unfortunately it happens with the following JVM as well:

$ java -version
java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)
Comment 7 Joakim Erdfelt CLA 2011-09-20 12:31:30 EDT
Bug 357318 seems related.
Comment 8 Gunnar Wagenknecht CLA 2011-10-02 06:15:56 EDT
We are also experiencing this issue using Jetty 7.5.1 and Java 1.6.0_26 on Linux.
Comment 9 Jan Bartel CLA 2011-11-03 00:25:27 EDT
Can we clarify which connector? The bug subject says its the SelectChannelConnector, yet your stack trace is from the SslSocketConnector? I will update the bug subject line as needed.

Also, the most recent jetty release is 7.5.4, might be worthwhile trying that one.

thanks
Jan
Comment 10 Mark A. Ziesemer CLA 2011-11-03 00:53:24 EDT
(In reply to comment #9)
> Can we clarify which connector? The bug subject says its the
> SelectChannelConnector, yet your stack trace is from the SslSocketConnector? I
> will update the bug subject line as needed.
> 
> Also, the most recent jetty release is 7.5.4, might be worthwhile trying that
> one.

Jan - please see bug 357318.  I'm having a very similar issue with SelectChannelConnector (no SSL involved), and against both 7.4.5.v20110725 and 8.0.1.v20110908.  That said, there may be at least 2 different causes here, as my stack traces are completely different than what are shown here.
Comment 11 Greg Wilkins CLA 2012-01-10 21:00:34 EST
I strongly suggest that you try 7.6.0.RC3 or 8.1.0.RC2

We have significantly reworked the NIO layer in these releases to cope with several high cpu spins associated with badly closed connections.

7.5.4 has had some improvements, but the real fixes will be only in >=7.6

regards
Comment 12 Jesse McConnell CLA 2013-01-29 12:17:24 EST
This should be working in the latest releases of jetty, please reopen if you can reproduce on 7.6.8 or higher.