Community
Participate
Working Groups
Build Identifier: 7.2.2.v20101205 but i tried on other builds as well Hi, We are having a really big problem with Jetty server and continuations, we put a lot of effort on this project and we've encountered a problem that really concerns us, if it is not fixable we might migrate to another Server platform. this bug is reproducible, running stress test crashes the server in an hour, hs_err file summary is: [error occurred during error reporting (printing native stack), id 0xc0000005] R12=0x0000000000000000 0x0000000000000000 is pointing to unknown location Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) J sun.nio.ch.SocketDispatcher.close0(Ljava/io/FileDescriptor;)V J java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel()V J java.nio.channels.spi.AbstractInterruptibleChannel.close()V J org.eclipse.jetty.server.HttpConnection.handle()Lorg/eclipse/jetty/io/Connection; J org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle()V J org.eclipse.jetty.util.thread.QueuedThreadPool$2.run()V j java.lang.Thread.run()V+11 v ~StubRoutines::call_stub another one that happened Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) j sun.nio.ch.SocketDispatcher.close0(Ljava/io/FileDescriptor;)V+0 J java.nio.channels.spi.AbstractSelectableChannel.implCloseChannel()V J org.eclipse.jetty.io.nio.ChannelEndPoint.fill(Lorg/eclipse/jetty/io/Buffer;)I J org.eclipse.jetty.http.HttpParser.parseNext()J J org.eclipse.jetty.server.HttpConnection.handle()Lorg/eclipse/jetty/io/Connection; J org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle()V J org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run()V J org.eclipse.jetty.util.thread.QueuedThreadPool$2.run()V j java.lang.Thread.run()V+11 v ~StubRoutines::call_stub some more info # # A fatal error has been detected by the Java Runtime Environment: # # EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0000000077ab1c30, pid=8808, tid=12772 # # JRE version: 6.0_23-b05 # Java VM: Java HotSpot(TM) 64-Bit Server VM (19.0-b09 mixed mode windows-amd64 compressed oops) # Problematic frame: # C [ntdll.dll+0x51c30] # # An error report file with more information is saved as: # d:\CLFSW\v8\grease\boot\hs_err_pid8808.log [thread 14468 also had an error] # # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp # The crash happened outside the Java Virtual Machine in native code. # See problematic frame for where to report the bug. # [error occurred during error reporting , id 0xc0000005] Windows Event log shows Faulting application name: javaw.exe, version: 6.0.120.4, time stamp: 0x4971aa8a Faulting module name: ntdll.dll, version: 6.1.7600.16559, time stamp: 0x4ba9b802 Exception code: 0xc0000374 Fault offset: 0x00000000000c6df2 Faulting process id: 0x2670 Faulting application start time: 0x01cbbd5bb5eb21e8 Faulting application path: d:\CLFSW\jdk\jdk64\bin\javaw.exe Faulting module path: C:\Windows\SYSTEM32\ntdll.dll Report Id: 440656dd-2954-11e0-8ec2-b8ac6f8bf707 --------------------------------------------------------- we tried to use different versions of Jetty eclipse server and also different JDK versions. we currently using JDK64 6.12 and windows server 2008. our Jetty server is using asynch continuations, 36 threads arriving the Server in order to get jobs. ----------------------- our server configurations are: var server = new Server() var connector = new SelectChannelConnector connector.setPort(Integer.parseInt(port)) connector.setAcceptQueueSize(2048) connector.setMaxIdleTime(300000) connector.setAcceptors(2 * Runtime.getRuntime.availableProcessors) connector.setLowResourceMaxIdleTime(60000) connector.setLowResourcesConnections(25000) connector.setName("workersQQ") server.addConnector(connector) val threadPool = new QueuedThreadPool() threadPool.setMaxThreads(512) threadPool.setMaxIdleTimeMs(300000) server.setThreadPool(threadPool) var contexts = new ContextHandlerCollection {// Client Handler var contextHandler1 = new ContextHandler contextHandler1.setMaxFormContentSize(0) contextHandler1.setContextPath(CommonConstants.SLASH) contextHandler1.setInitParameter("maxInterval", String.valueOf(60000)); contextHandler1.setInitParameter("timeout", String.valueOf(30000)); contextHandler1.setHandler(ClientHandler) contexts.addHandler(contextHandler1) } {// Worker Handler var contextHandler = new ContextHandler contextHandler.setMaxFormContentSize(0) contextHandler.setContextPath(CommonConstants.SLASH) contextHandler.setInitParameter("maxInterval", String.valueOf(60000)); contextHandler.setInitParameter("timeout", String.valueOf(30000)); contextHandler.setHandler(WorkerHandler) contexts.addHandler(contextHandler) } {// Status Handler var contextHandler = new ContextHandler contextHandler.setMaxFormContentSize(0) contextHandler.setContextPath(CommonConstants.SLASH) contextHandler.setInitParameter("maxInterval", String.valueOf(60000)); contextHandler.setInitParameter("timeout", String.valueOf(30000)); contextHandler.setHandler(StatusHandler) contexts.addHandler(contextHandler) } server.setHandler(contexts) server.start; --------------------------------------------------------------------- ---------------------------------ANOTHER BUG------------------------- --------------------------------------------------------------------- we are getting sometimes at the workers side. we are using socket timeout, read timeout ,connection timeout = 0 at the beginning we used to have these parameters configured to 30seconds but we had many timeout exceptions EOF where it wasn’t really timed out. org.eclipse.jetty.io.EofException at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:149) at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:96) at org.eclipse.jetty.util.IO.copy(IO.java:197) at org.eclipse.jetty.util.IO.copy(IO.java:135) at com.clearforest.wd40.grease.GreaseCommonsHanlder.createJob(GreaseCommonHandler.scala:76) at com.clearforest.wd40.grease.ClientHandler$.handle1(ClientHandler.scala:130) at com.clearforest.wd40.grease.ClientHandler$.handle(ClientHandler.scala:24) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:931) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:868) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:247) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:114) at org.eclipse.jetty.server.Server.handle(Server.java:352) at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596) at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:1068) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:805) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:218) at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:426) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:508) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.access$000(SelectChannelEndPoint.java:34) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:40) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:451) at java.lang.Thread.run(Thread.java:619) Reproducible: Always Steps to Reproduce: 1. start server 2. i have 36 clients (workers) that arrive to Server and take jobs, request&response size ~50KB-~300KB 3. 24 other clients that send requests&response to server with size of ~50KB-~300KB. 4. stress test for an hour 5. server crashes and hs_err file show the reason.
Oded, It is unlikely that this is a jetty bug, as java code should not be able to crash the JVM like your are describing. This is almost certainly a JVM or operating system problem. Have you tried submitting the problem to http://java.sun.com/webapps/bugreport/crash.jsp as suggested and/or searching their bug database for something similar? Can you try running your application on a *nix or osX machine to see if it performs differently. If you only have 36 clients, then I am not sure why you are using continuations? Perhaps you could switch to the SocketConnector and try using the ContinuationFilter so that your code runs in blocking mode. This may be more stable on your JVM/OS combination. You should only need select channel connectors and continuations if you expect many thousands of simultaneous clients. If you are just using 36 connections to stress test a system intended to be used by many connections, then have a few busy connections is a very poor simulations of thousands of normal connections. Note also that suggesting that you may migrate to another server does not help resolve the problem. We think that there are compelling reasons to use Jetty, but there will be issues (as there are will all project). Working together in an open source project, most issues will be fixed in time.
Thank you for your reply. *for your question, the server might receive thousands of requests and continuations will be suspended for each one. i only mentioned that currently i tested it with a small amount of continuations. *about the JDK i tried to work with 64 6_23, no changes. some more updates: I tested on 2 different operating systems (Windows server 2008 & 2003). 2008 is crashing, 2003 is not crashing but i received many java.net.BindException: Address already in use ... during stress test so i did the following changes: 1. connector.setReuseAddress(true) on the server side 2. on the client side: manager.getParams().setStaleCheckingEnabled(true) (using httpClient). 3. I enabled keep alive on both sides (it was disabled). ---------------------------------------------------- After that, i tested again on both operating systems: 1. in 2008 it just took more time until it crashed. 2. the BindException vanished but i still got the Jetty EofException that i mentioned before (every few thousands requests). 2 types of these exceptions: --- A. --- org.eclipse.jetty.io.EofException at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:319) at org.eclipse.jetty.http.HttpParser.blockForContent(HttpParser.java:1131) at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:57) at org.eclipse.jetty.util.IO.copy(IO.java:194) at org.eclipse.jetty.util.IO.copy(IO.java:135) at com.clearforest.wd40.grease.WorkerHandler$.sendResult(WorkerHandler.scala:200) at com.clearforest.wd40.grease.WorkerHandler$.handle1(WorkerHandler.scala:109) at com.clearforest.wd40.grease.WorkerHandler$.handle(WorkerHandler.scala:28) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:931) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:868) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:247) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:114) at org.eclipse.jetty.server.Server.handle(Server.java:352) at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596) at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:1068) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:805) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:218) at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:426) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:508) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.access$000(SelectChannelEndPoint.java:34) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:40) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:451) at java.lang.Thread.run(Thread.java:662) Caused by: java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:120) at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:163) at org.eclipse.jetty.io.nio.ChannelEndPoint.fill(ChannelEndPoint.java:149) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:290) ... 23 more --- B. --- org.eclipse.jetty.io.EofException at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:149) at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:96) at org.eclipse.jetty.util.ByteArrayISO8859Writer.writeTo(ByteArrayISO8859Writer.java:102) at org.eclipse.jetty.server.Response.sendError(Response.java:336) at org.eclipse.jetty.server.Response.sendError(Response.java:361) at com.clearforest.wd40.grease.WorkerHandler$.handle1(WorkerHandler.scala:112) at com.clearforest.wd40.grease.WorkerHandler$.handle(WorkerHandler.scala:28) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:931) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:868) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:247) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:114) at org.eclipse.jetty.server.Server.handle(Server.java:352) at org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:596) at org.eclipse.jetty.server.HttpConnection$RequestHandler.content(HttpConnection.java:1068) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:805) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:218) at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:426) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:508) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.access$000(SelectChannelEndPoint.java:34) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:40) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:451) at java.lang.Thread.run(Thread.java:662) Thanks, Oded.
Hi, I have a small update and a question, the configuration that helped me fix the crash is: connector.setUseDirectBuffers(false) when it was true i received: java.lang.OutOfMemoryError: Direct buffer memory at java.nio.Bits.reserveMemory(Bits.java:633) at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:98) at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288) at org.eclipse.jetty.io.nio.DirectNIOBuffer.<init>(DirectNIOBuffer.java: 46) at org.eclipse.jetty.server.nio.AbstractNIOConnector.newRequestBuffer(Ab stractNIOConnector.java:53) at org.eclipse.jetty.http.HttpBuffers$1.newBuffer(HttpBuffers.java:34) at org.eclipse.jetty.io.ThreadLocalBuffers.getBuffer(ThreadLocalBuffers. java:60) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:598) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:212) at org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:42 6) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEn dPoint.java:508) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.access$000(SelectChann elEndPoint.java:34) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEnd Point.java:40) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool .java:451) at java.lang.Thread.run(Thread.java:662) so now the server hasnt crashed yet for the last 36 hours BUT I still get org.eclipse.jetty.io.EofException at org.eclipse.jetty.server.HttpOutput.write(HttpOutput.java:149) every once in a while. are you working on this bug? any idea why this is happening? or how to prevent/workaround it? Do you recommand using Jetty eclipse client instead of HttpClient? Please see my previous reply for full stacktrace. Thanks, Oded.
i guess that there are different bugs in this report. 1. org.eclipse.jetty.io.EofException 2. crash - just happen again right after my last update. All the configurations that i mentioned make the server to be more stable but still the problems exist.
Oded, the EoF exception is not a problem. It is just that a client closed the connection while a response was being written. This is normal and you can let those exceptions got through to the container where they will be silently ignored (unless debug/ignore is turned on). As for the crash itself - It sounds like it is a JVM on windows 2008 problem. As I've said, java should not crash the JVM and I can't even write a java program to deliberately crash the JVM like that.
