Community
Participate
Working Groups
I have an implementation of FilteredItemsSelectionDialog that stopped working. However, I'm not sure what changed so I can't currently confirm if it was a RAP or Jetty update which broke it. The dialog comes up. When the filter triggers, I see my code getting called and the items properly filtered. However, I don't see any items appear in the dialog. The mouse cursor keeps on spinning. Even worse, I can't launch a second RAP application instance in a different browser. Looking at the threads in the VM I found the following two one stuck. It seems that they are somehow waiting on a signal from each other but no one notifies them. Jetty request thread: Name: jetty-server-admin-42 State: WAITING on java.lang.Object@adff2f Total blocked: 21 Total waited: 2.416 Stack trace: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:485) org.eclipse.rwt.internal.lifecycle.UICallBackManager.blockCallBackRequest(UICallBackManager.java:123) org.eclipse.rwt.internal.lifecycle.UICallBackServiceHandler.service(UICallBackServiceHandler.java:87) org.eclipse.rwt.internal.service.ServiceManager$HandlerDispatcher.service(ServiceManager.java:33) org.eclipse.rwt.internal.engine.RWTDelegate.doPost(RWTDelegate.java:46) org.eclipse.rwt.internal.engine.RWTDelegate.doGet(RWTDelegate.java:35) javax.servlet.http.HttpServlet.service(HttpServlet.java:707) javax.servlet.http.HttpServlet.service(HttpServlet.java:820) org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61) org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:126) org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:60) javax.servlet.http.HttpServlet.service(HttpServlet.java:820) org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:538) org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:478) org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223) org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:937) org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:406) org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:183) org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:871) org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117) org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:110) org.eclipse.jetty.server.Server.handle(Server.java:346) org.eclipse.jetty.server.HttpConnection.handleRequest(HttpConnection.java:589) org.eclipse.jetty.server.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:1048) org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:601) org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:214) org.eclipse.jetty.server.HttpConnection.handle(HttpConnection.java:411) org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:535) org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:40) org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) java.lang.Thread.run(Thread.java:662) UI Thread: Name: UIThread [itw62v3tueeu1ohrs9w2vuhjd] State: WAITING on org.eclipse.rwt.internal.lifecycle.UIThread@136d1d7 Total blocked: 34 Total waited: 32 Stack trace: java.lang.Object.wait(Native Method) java.lang.Object.wait(Object.java:485) org.eclipse.rwt.internal.lifecycle.UIThread.switchThread(UIThread.java:60) org.eclipse.rwt.internal.lifecycle.RWTLifeCycle.sleep(RWTLifeCycle.java:263) org.eclipse.swt.widgets.Display.sleep(Display.java:1165) org.eclipse.jface.window.Window.runEventLoop(Window.java:841) org.eclipse.jface.window.Window.open(Window.java:816) org.eclipse.gyrex.admin.ui.p2.internal.AddPackageDialog.addComponentButtonPressed(AddPackageDialog.java:100) org.eclipse.gyrex.admin.ui.p2.internal.AddPackageDialog$1.customButtonPressed(AddPackageDialog.java:62) org.eclipse.gyrex.admin.ui.internal.wizards.dialogfields.ListDialogField.buttonPressed(ListDialogField.java:205) org.eclipse.gyrex.admin.ui.internal.wizards.dialogfields.ListDialogField.doButtonSelected(ListDialogField.java:472) org.eclipse.gyrex.admin.ui.internal.wizards.dialogfields.ListDialogField.access$0(ListDialogField.java:468) org.eclipse.gyrex.admin.ui.internal.wizards.dialogfields.ListDialogField$2.widgetSelected(ListDialogField.java:434) org.eclipse.swt.events.SelectionEvent.dispatchToObserver(SelectionEvent.java:196) org.eclipse.rwt.internal.events.Event.processEvent(Event.java:44) org.eclipse.swt.events.TypedEvent.processEvent(TypedEvent.java:163) org.eclipse.swt.events.TypedEvent.executeNext(TypedEvent.java:203) org.eclipse.swt.widgets.Display.runPendingMessages(Display.java:1134) org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:1124) org.eclipse.jface.window.Window.runEventLoop(Window.java:840) org.eclipse.jface.window.Window.open(Window.java:816) org.eclipse.gyrex.admin.ui.p2.internal.PackagesSection.addButtonPressed(PackagesSection.java:73) org.eclipse.gyrex.admin.ui.p2.internal.PackagesSection$1.widgetSelected(PackagesSection.java:83) org.eclipse.swt.events.SelectionEvent.dispatchToObserver(SelectionEvent.java:196) org.eclipse.rwt.internal.events.Event.processEvent(Event.java:44) org.eclipse.swt.events.TypedEvent.processEvent(TypedEvent.java:163) org.eclipse.swt.events.TypedEvent.executeNext(TypedEvent.java:203) org.eclipse.swt.widgets.Display.runPendingMessages(Display.java:1134) org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:1124) org.eclipse.ui.internal.Workbench.runEventLoop(Workbench.java:2733) org.eclipse.ui.internal.Workbench.runUI(Workbench.java:2694) org.eclipse.ui.internal.Workbench.access$5(Workbench.java:2530) org.eclipse.ui.internal.Workbench$5.run(Workbench.java:702) org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332) org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:685) org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:157) org.eclipse.gyrex.admin.ui.internal.application.AdminApplication.createUI(AdminApplication.java:28) org.eclipse.rwt.internal.lifecycle.EntryPointManager.createUI(EntryPointManager.java:73) org.eclipse.rwt.internal.lifecycle.RWTLifeCycle.createUI(RWTLifeCycle.java:211) org.eclipse.rwt.internal.lifecycle.RWTLifeCycle$UIThreadController.run(RWTLifeCycle.java:88) java.lang.Thread.run(Thread.java:662) org.eclipse.rwt.internal.lifecycle.UIThread.run(UIThread.java:102)
So that's interesting, it has something to do with the number of acceptors on the Jetty NIO SelectChannelConnector. (cc Greg) This one works: final SelectChannelConnector connector = new SelectChannelConnector(); connector.setPort(DEFAULT_ADMIN_PORT); connector.setMaxIdleTime(30000); connector.setLowResourcesConnections(20000); connector.setLowResourcesMaxIdleTime(5000); connector.setForwarded(true); server.addConnector(connector); However, as soon as I add: connector.setAcceptors(2); I get the "deadlock". Thus, it could be something in RAP that is causing this in Jetty. For reference, here are the Jetty acceptor thread dumps when it hangs: Name: jetty-server-admin-40 Acceptor1 SelectChannelConnector@0.0.0.0:3110 STARTED State: BLOCKED on java.lang.Object@1254e47 owned by: jetty-server-admin-41 Acceptor0 SelectChannelConnector@0.0.0.0:3110 STARTED Total blocked: 1 Total waited: 1 Stack trace: sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:136) org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:92) org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:830) org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) java.lang.Thread.run(Thread.java:662) Name: jetty-server-admin-41 Acceptor0 SelectChannelConnector@0.0.0.0:3110 STARTED State: RUNNABLE Total blocked: 0 Total waited: 1 Stack trace: sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152) - locked java.lang.Object@1254e47 org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:92) org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:830) org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) java.lang.Thread.run(Thread.java:662) Name: jetty-server-admin-42 Selector1 SelectChannelConnector@0.0.0.0:3110 STARTED State: RUNNABLE Total blocked: 32 Total waited: 5 Stack trace: sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:273) sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:255) sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:136) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked sun.nio.ch.Util$2@18851ee - locked java.util.Collections$UnmodifiableSet@bb21ab - locked sun.nio.ch.WindowsSelectorImpl@19f89ee sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:504) org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:228) org.eclipse.jetty.server.nio.SelectChannelConnector$1.run(SelectChannelConnector.java:268) org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) java.lang.Thread.run(Thread.java:662) Name: jetty-server-admin-43 Selector0 SelectChannelConnector@0.0.0.0:3110 STARTED State: RUNNABLE Total blocked: 42 Total waited: 6 Stack trace: sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method) sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:273) sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:255) sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:136) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked sun.nio.ch.Util$2@1399671 - locked java.util.Collections$UnmodifiableSet@1c3fd59 - locked sun.nio.ch.WindowsSelectorImpl@2e05f1 sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:504) org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:228) org.eclipse.jetty.server.nio.SelectChannelConnector$1.run(SelectChannelConnector.java:268) org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529) java.lang.Thread.run(Thread.java:662)
I suppose that this is not exactly a deadlock since the lock that the "jetty-server-admin-42" Thread is waiting on cannot be acquired by the UIThread. I'm not familiar with the Jetty internals and don't know what those acceptor treads exactly are. I guess this setting represents the maximum number of request threads, but maybe the UIThread which is created for the user session by the request thread is also affected by this limit. Since the UICallback mechanism (which is active in your example) blocks one thread for the standing ui callback request, restricting Jetty to 2 threads might just be too aggressive. Does the actual problem go away when setting setAcceptors() to 3 or 4? I would recommend a much higher limit for production, but this test could be at least an indicator.
No, the problems goes away when I set acceptors to exactly 1 (one!). From my understanding, the acceptors (in the NIO world) are just listening for new connection events and spawn an actual request thread when necessary. FWIW, the tread pool for request processing is way larger than the number of acceptors.
(In reply to comment #3) > No, the problems goes away when I set acceptors to exactly 1 (one!). Well, I don't set it explicitly. One (1) is the default.
Maybe one is a special case that is handled differently? I think it would still be interesting what happens with 3 or 4. Looking on your thread dumps again, I think they look completely healthy. They are not waiting for each other. The first one is a standing UICallBack request, kept waiting on the server until it needs to ping pack the client. The second one is the running session's UIThread that sleeps until the next request arrives. In UIThread#switchThread, the request thread should be released and return to the client, which obviously doesn't happen for whatever reason. BTW, we have redesigned the UICallBack in 1.5M1 (bug 344989) and are currently working on one regression (bug 353891). Could you check if you can reproduce it with 1.5M1 or the latest nightly in a couple of days? If it works with 1.5, I would tend to not start digging into this issue, given that it doesn't happen with the default Jetty configuration.
Could this be related to bug 357318?
Gunnar, is this bug current? Do you face the problem with RAP 2.3?
I never tried to reproduce it again. It works with the current Jetty configuration.
(In reply to comment #8) > I never tried to reproduce it again. It works with the current Jetty > configuration. Than, I'll close it as WORKSFORME. Please reopen if you disagree.
