Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 318689

Summary: StressTest failure, possibly intermittently
Product: [RT] Jetty Reporter: Glyn Normington <glyn.normington>
Component: buildAssignee: Jesse McConnell <jesse.mcconnell>
Status: CLOSED FIXED QA Contact:
Severity: trivial    
Priority: P3 CC: gregw, jetty-inbox
Version: unspecified   
Target Milestone: 7.1.x   
Hardware: Macintosh   
OS: Mac OS X - Carbon (unsup.)   
Whiteboard:

Description Glyn Normington CLA 2010-07-02 06:16:34 EDT
This occurred while building Jetty 7 at svn revision 2061 using mvn clean install.

Symptoms::

java.lang.AssertionError: T7-0 expected:<16> but was:<15>
	at org.junit.Assert.fail(Assert.java:91)
	at org.junit.Assert.failNotEquals(Assert.java:645)
	at org.junit.Assert.assertEquals(Assert.java:126)
	at org.junit.Assert.assertEquals(Assert.java:470)
	at org.eclipse.jetty.server.StressTest.doPaths(StressTest.java:347)
	at org.eclipse.jetty.server.StressTest.doLoops(StressTest.java:296)
	at org.eclipse.jetty.server.StressTest.access$100(StressTest.java:40)
	at org.eclipse.jetty.server.StressTest$1.run(StressTest.java:158)
2010-07-02 10:00:47.135:INFO::qtp818133847{8<=14<=15/500,0} min/ave/max/target=0/18/22/40 errors/finished/loops=0/0/20 idle/threads=14/15
2010-07-02 10:00:48.136:INFO::qtp818133847{8<=15<=16/500,0} min/ave/max/target=0/37/40/40 errors/finished/loops=0/16/20 idle/threads=15/16
2010-07-02 10:00:49.136:INFO::qtp818133847{8<=15<=16/500,0} min/ave/max/target=0/38/40/40 errors/finished/loops=0/19/20 idle/threads=15/16
2010-07-02 10:00:50.136:INFO::qtp818133847{8<=15<=16/500,0} min/ave/max/target=0/38/40/40 errors/finished/loops=0/19/20 idle/threads=15/16
2010-07-02 10:00:51.136:INFO::qtp818133847{8<=15<=16/500,0} min/ave/max/target=0/38/40/40 errors/finished/loops=0/19/20 idle/threads=15/16
2010-07-02 10:00:52.136:INFO::qtp818133847{8<=15<=16/500,0} min/ave/max/target=0/38/40/40 errors/finished/loops=0/19/20 idle/threads=15/16
2010-07-02 10:00:53.137:INFO::qtp818133847{8<=15<=16/500,0} min/ave/max/target=0/38/40/40 errors/finished/loops=0/19/20 idle/threads=15/16
2010-07-02 10:00:54.137:INFO::qtp818133847{8<=15<=16/500,0} min/ave/max/target=0/38/40/40 errors/finished/loops=0/19/20 idle/threads=15/16
2010-07-02 10:00:55.137:INFO::qtp818133847{8<=15<=16/500,0} min/ave/max/target=0/38/40/40 errors/finished/loops=0/19/20 idle/threads=15/16
STALLED!!!
qtp818133847{8<=15<=16/500,0}
2010-07-02 10:00:56.137:INFO::channel sun.nio.ch.ServerSocketChannelImpl[/0.0.0.0:59427] is open
2010-07-02 10:00:56.138:INFO::SelectSet 0 : Thread[qtp818133847-599 - Acceptor0 SelectChannelConnector@0.0.0.0:59427,5,main]
2010-07-02 10:00:56.138:INFO::?at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
2010-07-02 10:00:56.138:INFO::?at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:136)
2010-07-02 10:00:56.138:INFO::?at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:69)
2010-07-02 10:00:56.138:INFO::?at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
2010-07-02 10:00:56.138:INFO::?at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
2010-07-02 10:00:56.138:INFO::?at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:482)
2010-07-02 10:00:56.138:INFO::?at org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:195)
2010-07-02 10:00:56.138:INFO::?at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:134)
2010-07-02 10:00:56.138:INFO::?at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:793)
2010-07-02 10:00:56.138:INFO::?at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
2010-07-02 10:00:56.138:INFO::?at java.lang.Thread.run(Thread.java:637)
2010-07-02 10:00:56.240:INFO::SelectSet 0 1
2010-07-02 10:00:56.241:INFO::sun.nio.ch.ServerSocketChannelImpl[/0.0.0.0:59427] 16 16 null
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD FAILURE

Environment::

Hardware:
  Model Name:    Mac Pro
  Model Identifier:    MacPro1,1
  Processor Name:    Dual-Core Intel Xeon
  Processor Speed:    2.66 GHz
  Number Of Processors:    2
  Total Number Of Cores:    4
  L2 Cache (per processor):    4 MB
  Memory:    12 GB

OS:
  System Version:    Mac OS X 10.6.4 (10F569)
  Kernel Version:    Darwin 10.4.0

Java:
  java version "1.6.0_20"
 Java(TM) SE Runtime Environment (build 1.6.0_20-b02-279-10M3065)
 Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01-279, mixed mode)

Maven:
 Apache Maven 2.2.0 (r788681; 2009-06-26 14:04:01+0100)
Comment 1 Greg Wilkins CLA 2010-07-02 09:22:10 EDT
Glynn,
do these failures occur for you if you just run the individual tests, or just do a mvn clean install in a lower level module?

We have had some problems with macOS where they do not recycle file descriptors fast enough, so if all tests are run together then there are delays waiting for available FDs... and many of our tests are a little time sensitive (eg wait 250ms and then test result).

Can you try increasing the ulimit on your machine/account for file descriptors to at least 10000.
Comment 2 Glyn Normington CLA 2010-07-02 10:03:57 EDT
(In reply to comment #1)
> Glynn,

Glyn, for future reference ;-)

> do these failures occur for you if you just run the individual tests, or just
> do a mvn clean install in a lower level module?

Haven't tried that. I'm only trying top-level builds now.

> 
> We have had some problems with macOS where they do not recycle file descriptors
> fast enough, so if all tests are run together then there are delays waiting for
> available FDs... and many of our tests are a little time sensitive (eg wait
> 250ms and then test result).
> 
> Can you try increasing the ulimit on your machine/account for file descriptors
> to at least 10000.

Perfect. BUILD SUCCESSFUL. Thanks!

I'll close off the other bugs.

(In fact this *seems* to help with at least one other problem we were struggling with where our documentation build was left hanging trying to stop the Apache fop renderer.)
Comment 3 Glyn Normington CLA 2010-07-02 10:06:00 EDT
I needed to set ulimit -n 10000.
Comment 4 Greg Wilkins CLA 2010-07-02 11:02:06 EDT
Glyn,

glad that helped.... although it is still a bit of a worry that macOs appears to need that (and that we have quiet a few timing sensitive tests).

cheers
Comment 5 Glyn Normington CLA 2010-07-02 12:03:16 EDT
(In reply to comment #4)
> Glyn,
> 
> glad that helped.... although it is still a bit of a worry that macOs appears
> to need that (and that we have quiet a few timing sensitive tests).
> 
> cheers

I wouldn't worry too much. Sometimes test environments can be a bit peculiar compared to production environments.

I captured the ulimit stuff in some machine setup instructions [1] on the Virgo wiki. Maybe it would be worth doing that in the Jetty wiki too? Just a thought.

Glyn
[1] http://wiki.eclipse.org/Virgo/Committers#Machine_Setup