Community
Participate
Working Groups
Build Identifier: Helios Jetty 7.4.3 Embedded server. using IO-connector for ssl (SslSocketConnector) after running server for 1-2 days, cpu loads get to 100% & doesnt goes down, independently from number of request. monitoring from jconsole shows that cpu-load is caused by SslSocketConnector (seems like a endless loop of a connector-thread). cacti-report shows that at the time this happens, there's no connected client, memory & network-load normal. trace of thread which cause cpu-load: Name: qtp1939768105-1610 State: RUNNABLE Total blocked: 1 Total waited: 15 Stack trace: org.eclipse.jetty.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:388) org.eclipse.jetty.io.bio.StreamEndPoint.fill(StreamEndPoint.java:132) org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.fill(SocketConnector.java:209) org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.fill(SslSocketConnector.java:612) org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:289) org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:214) org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:411) org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:241) org.eclipse.jetty.server.ssl.SslSocketConnector$SslConnectorEndPoint.run(SslSocketConnector.java:664) org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) java.lang.Thread.run(Thread.java:662) Reproducible: Always Steps to Reproduce: 1.start embedded server with a webapp and Proxyhandler 2.make requests 3.leave server idle & wait, after about 1 day cpu load gets high & will not get down
I am observing the same problem and tracked it down to error handling with the bio SocketConnector: when the read() call on the socket throws an exception, eg due to TCP reset, Jetty enters a loop in which the failing read is repeated over and over. Steps to reproduce (under Linux) in different terminals: 1. Start Jetty with BIO connector and debug output: $ java -jar start.jar etc/jetty-bio.xml DEBUG=true 2. Now create connection to port 8081: $ telnet localhost 8081 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. 3. Now start tcpkill to induce a TCP reset: $ sudo tcpkill -i lo 'tcp port 8081' 4. Now press return in the telnet to send data over the TCP connection. tcpkill will intercept the data and inject a RST. Jetty's read() call with throw an exception and the following exception will be repeated over and over: 2011-07-22 13:51:37.034:DBUG::EXCEPTION java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:113) at org.eclipse.jetty.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:388) at org.eclipse.jetty.io.bio.StreamEndPoint.fill(StreamEndPoint.java:132) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.fill(SocketConnector.java:209) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:289) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:214) at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:411) at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:241) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) at java.lang.Thread.run(Thread.java:662) Bug 351801 is likely a duplicate of this bug.
There is a new Thread Monitor tool in Jetty that has been developed over the past few weeks in attempt to automatically detect spinning and deadlocked threads on a live Jetty instance. This tool is part of the jetty-monitor module that can be added to your maven project as follows. <dependency> <groupId>org.eclipse.jetty</groupId> <artifactId>jetty-monitor</artifactId> <version>7.5.0-SNAPSHOT</version> </dependency> Alternatively, you could download the snapshot directly here: http://goo.gl/cGJo1 For instructions on how to run this tool, please see README.txt file located here: http://goo.gl/UmZ4N It would be great if you could run the Thread Monitor on your instance of Jetty and attach the log file collected after running the tool to this ticket. -Michael
*** Bug 351801 has been marked as a duplicate of this bug. ***
(In reply to comment #1) > I am observing the same problem and tracked it down to error handling with the > bio SocketConnector: when the read() call on the socket throws an exception, eg > due to TCP reset, Jetty enters a loop in which the failing read is repeated > over and over. I've not been able to reproduce the issue using the steps you described when using Jetty 7.5.0-SNAPSHOT. The most recently published snapshot build can be downloaded from here: http://goo.gl/eOQHm. I've compared the code between release 7.4.3 and the current master and noticed that the former contained a couple of calls to the fill() method of a endpoint from outside the fill() method of HttpParser, that contains code that guards from this exact condition, one of them being on line 289 of HttpParser.java. Thus it appears that this change http://goo.gl/znP3J is directly responsible for fixing the current issue. I'm going to mark it resolved pending confirmation from one of you that the problem is fixed.
Awaiting the confirmation that the issue is fixed...
This also happens in 8.0.0.M3 It only happens with the SocketConnector. SelectChannelConnector works fine.
