Community
Participate
Working Groups
Build Identifier: Since the last week we switched to SelectChannelConnector form BlockingChannelConnector with the newest stable Jetty (7.4.5.v20110725). Everything looked fine but after half a week on one of my servers the load average jumped instantly +2. We had no clue because the load balancer before the servers balancing the requests equally. A restart fixed the problem, but another half week passed and the same thing happened. But this time I managed to get a stacktrace dump from it. The stuck processes is the following: at java.lang.Throwable.fillInStackTrace(Native Method) at java.lang.Throwable.<init>(Throwable.java:196) at java.lang.Exception.<init>(Exception.java:41) at java.io.IOException.<init>(IOException.java:41) at javax.net.ssl.SSLException.<init>(SSLException.java:33)+ at com.sun.net.ssl.internal.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1267) at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:65) at org.eclipse.jetty.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:388) at org.eclipse.jetty.io.bio.StreamEndPoint.fill(StreamEndPoint.java:132) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.fill(SocketConnector.java:209) at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.fill(SslSocketConnector.java:612) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:289) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:214) at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:411) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:241) at org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.run(SslSocketConnector.java:664) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) at java.lang.Thread.run(Thread.java:619) Reproducible: Sometimes
maybe this is related to http://jira.codehaus.org/browse/JETTY-937
My system details: $ java -version java version "1.6.0_20" Java(TM) SE Runtime Environment (build 1.6.0_20-b02) Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode) $ uname -a: Linux *** 2.6.32-32-server #62-Ubuntu SMP Wed Apr 20 22:07:43 UTC 2011 x86_64 GNU/Linux
> Since the last week we switched to SelectChannelConnector form > BlockingChannelConnector with the newest stable Jetty (7.4.5.v20110725). It appears that you are using SslSocketConnector according to the stack trace. Could you please make sure that the configuration is correct and attach your configuration file (or configuration code for embedded usage) to this ticket? Also, there have been some fixes that affect this area of the code that had been committed to the repository after 7.4.5 release. Could you please try testing with jetty 7.5.0.RC1 that can be downloaded from http://download.eclipse.org/jetty/? > java version "1.6.0_20" As far as we know the JVM bugs that were affecting Jetty had been fixed. Could you please try updating to the latest version of sun-java6-jdk package that is 1.6.0_26 at the time of this writing, and try testing with it? Cheers, Michael
If you would like to attempt to run some diagnostics, you are welcome to try the brand new ThreadMonitor utility that is available as part of 7.5.0.RC1 release. It is part of jetty-monitor module, and you could find the instructions on how to configure this utility in README.txt file that you can be find at http://git.eclipse.org/c/jetty/org.eclipse.jetty.project.git/plain/jetty-monitor/README.txt
We're testing it with the new Java JRE 1.6.0_26, if it works I'll let you know. If not I'll post the detailed information about the configs here.
Unfortunately it happens with the following JVM as well: $ java -version java version "1.6.0_26" Java(TM) SE Runtime Environment (build 1.6.0_26-b03) Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)
Bug 357318 seems related.
We are also experiencing this issue using Jetty 7.5.1 and Java 1.6.0_26 on Linux.
Can we clarify which connector? The bug subject says its the SelectChannelConnector, yet your stack trace is from the SslSocketConnector? I will update the bug subject line as needed. Also, the most recent jetty release is 7.5.4, might be worthwhile trying that one. thanks Jan
(In reply to comment #9) > Can we clarify which connector? The bug subject says its the > SelectChannelConnector, yet your stack trace is from the SslSocketConnector? I > will update the bug subject line as needed. > > Also, the most recent jetty release is 7.5.4, might be worthwhile trying that > one. Jan - please see bug 357318. I'm having a very similar issue with SelectChannelConnector (no SSL involved), and against both 7.4.5.v20110725 and 8.0.1.v20110908. That said, there may be at least 2 different causes here, as my stack traces are completely different than what are shown here.
I strongly suggest that you try 7.6.0.RC3 or 8.1.0.RC2 We have significantly reworked the NIO layer in these releases to cope with several high cpu spins associated with badly closed connections. 7.5.4 has had some improvements, but the real fixes will be only in >=7.6 regards
This should be working in the latest releases of jetty, please reopen if you can reproduce on 7.6.8 or higher.
