Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 354407

Summary: Jetty run away CPU usage with BlockingChannelConnector above 7.3.1
Product: [RT] Jetty Reporter: KARASZI István <eclipse>
Component: serverAssignee: Thomas Becker <tbecker>
Status: CLOSED FIXED QA Contact:
Severity: normal    
Priority: P3 CC: eclipse, gregw, janb, jesse.mcconnell, jetty-inbox, mgorovoy
Version: unspecified   
Target Milestone: 7.5.x   
Hardware: All   
OS: All   
Whiteboard:

Description KARASZI István CLA 2011-08-10 12:00:38 EDT
Build Identifier: 

I'm experiencing problems with versions newer than Jetty 7.3.1. If I'm using the BlockingChannelConnector (I know-I know, I should use SelectChannelConnector instead).

I was trying to get more debugging information but it was really hard. I was trying to get the common stack traces, the run away CPU threads without luck. So I'm reporting this and I hope this information would enlighten something in your head about the problem.

If you need any further information feel free to contact me, I would more than happy to help you with this.

Reproducible: Always
Comment 1 KARASZI István CLA 2011-08-11 04:58:27 EDT
I forgot to add yesterday the symptoms.

If I start my embedded Jetty server with BlockingChannelConnector, first everything runs okay. But after about 10 minutes (high concurrent request numbers) the load starts to grow.

The average load on the same codebase with Jetty 7.3.1 is around 2.5, but with the new versions after 10 minutes it's around 20 and increasing.
Comment 2 KARASZI István CLA 2011-08-15 10:13:25 EDT
We're experiencing load increasing with the new version as well, but not that much as with the BlockingChannelConnector.

The interesting thing is that the 'TLB shootdowns' are also gets high when the load goes up.
Comment 3 Michael Gorovoy CLA 2011-08-24 13:16:09 EDT
It would be great if you could provide us with a test application that we can use to recreate this issue.

In absence of that please give us some idea about your request characteristics and patterns, such as size, frequency, number of concurrent, processing time, filters, frameworks, etc. as well as whether any back end connections or something similar are involved.

Cheers,
Michael
Comment 4 Thomas Becker CLA 2011-09-22 05:55:34 EDT
Hi Istvan,

any updates on this? With the given information only we'll have a hard time to reproduce this issue (if we can at all). 

Cheers,
Thomas
Comment 5 KARASZI István CLA 2011-09-22 06:17:14 EDT
Dear Thomas!

I can reproduce this bug easily on the production system but I'm unable to do it on the development or on the test (CI) systems.

I was trying to benchmark it with multiple different requests without luck. It happens after 30 minutes on the production system, so I can give you access to it.
Comment 6 Jesse McConnell CLA 2011-09-28 11:54:28 EDT
Have you seen this with 7.5.1?

Joakim was chasing down an issue with opennms that this sounds a little bit like.  In that issue ie8 and ie9 were able to cause 100% cpu issues in some older releases of jetty when they closed the browser.  That seems to have been resolved in 7.5.1 with that blocking connector.

As for access to your production machines, we really can't do that without some sort of outside arrangement, the 'what if' game it too scary 'ie what if we were to accidently kill a process we shouldn't, etc'
Comment 7 Thomas Becker CLA 2011-10-12 08:40:17 EDT
Hi István,

can you confirm that the problem still exists with the latest jetty version (7.5.3)? 

Cheers,
Thomas
Comment 8 Jesse McConnell CLA 2011-10-31 08:52:09 EDT
closing this issue since there have been fixes in the area and we haven't heard an update from the author in over a month

if this is still an issue on recent  releases, please reopen the issue with an update
Comment 9 Jan Bartel CLA 2011-10-31 17:49:42 EDT
(In reply to comment #8)
> closing this issue since there have been fixes in the area and we haven't heard
> an update from the author in over a month
> 
> if this is still an issue on recent  releases, please reopen the issue with an
> update

Just FYI, I've done tests on jetty-8.0.3, 8.0.4, 8.0.5 recently using siege with 512 simultaneous simulated users in benchmarking mode and not seen any runaway CPU.

Jan