Bugzilla – Full Text Bug Listing
| Summary: | 100% CPU used by org.eclipse.jetty.io.SelectorManager Thread | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | [RT] Jetty | Reporter: | Damien Solimando <dsolimando> | ||||||
| Component: | servlet-3.1 | Assignee: | Greg Wilkins <gregw> | ||||||
| Status: | CLOSED INVALID | QA Contact: | |||||||
| Severity: | normal | ||||||||
| Priority: | P3 | CC: | fabiofz, gregw, jetty-inbox | ||||||
| Version: | 9.2.6 | ||||||||
| Target Milestone: | 9.2.x | ||||||||
| Hardware: | PC | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | |||||||||
| Attachments: |
|
||||||||
Damien, we will need a lot more information than that to diagnose. Can you get the output of server.dump()? This can be called by JMX or by setting dump before stop and killing the server. A few thread dumps a few seconds apart would also be useful. Can you tell us more about what your async servlet is doing? async reads? async writes? async dispatch? async complete? large requests? large responses? Also 9.2.8 has just been release, so it would be good to verify if that has the same problem. Hello Greg, The app servlets are doing async reads (readStream), async writes(writeStream), and async completes. The requests are not that large (max 500 KB in case of image uploads and downloads) All the code is non blocking Somme infos about the runtime: all HTTP requests are using the new async servlet 3.1 API's. These requests are submitted to a single thread pool in charge of executing the server code. Server code use Netty async client for external http requests and blocking database calls are done with a dedicated thread pool. About server dump, may a complete Jstack output be enough ? No JMX is activated and activate it may be a bit touchy. Created attachment 250999 [details]
Jstack of VM when CPU is running at 100%
Netty experimented the same issue. https://github.com/netty/netty/issues/327 Can you post the actual async code that is causing this problem? Note also, you may want to try 9.2.9 or 9.3.0-SNAPSHOT as we have fixed several issues with asyncIO in both those release... specially 9.3 I can see this same issue in 9.2.5 on Eclipse. Java: JDK 8 / Windows 8.1 / Eclipse 4.5m5. This only happens after having Eclipse open for some time... it seems like Java 8 is optimizing some path and then all of the sudden jetty starts consuming a lot of CPU. I'm attaching a snapshot from yourkit which shows the information... note that 33% is a full CPU processor, so, you can see that the call from org.eclipse.jetty.io.SelectorManager$ManagedSelector.select() is using CPU 100%, and the Log.debug call (org.eclipse.jetty.util.log.Slf4jLog.debug(String, Object[])) is 60% of that time. Created attachment 251841 [details]
Yourkit profile dump
Fabio, Damien sorry but we don't have resources/time to look at such errors on older releases like 9.2.5 unless you have a really good reproduction that we can look at. We also would need to know what your servlet is doing, eg what async IO operations and more details of how your are scheduling them, if you are doing lots of input or lots of output. I'm not saying that you have not found a problem, I'm just saying that: a) you probably have found different problems b) they may have been fixed in a more recent release c) if they are not fixed, you need to provide a lot more information about what you are doing that triggers the problem. It is not sufficient just to demonstrate the looping. If you can reproduce on the current release, or can create a reproducable test case on 9.2.5, then please re-open. Ok... As a note, I've raised the issue to https://bugs.eclipse.org/bugs/show_bug.cgi?id=453910 (as I've observed the issue by using Eclipse and only found the culprit using a profiler). Can you add a comment there saying which would be the recommended version which would be supported so that this is hopefully fixed? (as a note, I had this issue in the 8.x series too: https://bugs.eclipse.org/bugs/show_bug.cgi?id=457288). I switched to 9.2.7 and problem seems resolved |
The CPU goes 100% without a reason. The deployed web app intensively use Async Servlets Please find the stack trace of the related thread "qtp280134559-10-selector-ServerConnectorManager@2badc7f/0" prio=10 tid=0x0000000001ef7000 nid=0x1a73 runnable [0x00007fa57f553000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:70) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <0x00000000e9edb150> (a sun.nio.ch.Util$2) - locked <0x00000000e9edb160> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000e9edb0f8> (a sun.nio.ch.PollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.select(SelectorManager.java:596) at org.eclipse.jetty.io.SelectorManager$ManagedSelector.run(SelectorManager.java:545) at org.eclipse.jetty.util.thread.NonBlockingThread.run(NonBlockingThread.java:52) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:620) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:540) at java.lang.Thread.run(Thread.java:745)