Bugzilla – Full Text Bug Listing
| Summary: | Performance tests for bug 543480 fluctuate in CI | ||
|---|---|---|---|
| Product: | [Eclipse Project] JDT | Reporter: | Sebastian Lohmeier <sebastian> |
| Component: | Core | Assignee: | Sebastian Lohmeier <sebastian> |
| Status: | VERIFIED FIXED | QA Contact: | Stephan Herrmann <stephan.herrmann> |
| Severity: | normal | ||
| Priority: | P3 | CC: | jarthana, sarika.sinha, stephan.herrmann |
| Version: | 4.14 | ||
| Target Milestone: | 4.14 M3 | ||
| Hardware: | PC | ||
| OS: | Linux | ||
| See Also: |
https://git.eclipse.org/r/150662 https://git.eclipse.org/c/jdt/eclipse.jdt.core.git/commit/?id=ea2cf5819a5be4907255ff9da714a96100bb270f https://bugs.eclipse.org/bugs/show_bug.cgi?id=572413 |
||
| Whiteboard: | |||
|
Description
Sebastian Lohmeier
This occured again: https://download.eclipse.org/eclipse/downloads/drops4/I20191009-1800/testresults/html/org.eclipse.jdt.core.tests.compiler_ep414I-unit-cen64-gtk3-java8_linux.gtk.x86_64_8.0.html The attached Gerrit patch is ready to be reviewed :-) Gerrit change https://git.eclipse.org/r/150662 was merged to [master]. Commit: http://git.eclipse.org/c/jdt/eclipse.jdt.core.git/commit/?id=ea2cf5819a5be4907255ff9da714a96100bb270f (In reply to Eclipse Genie from comment #3) > Gerrit change https://git.eclipse.org/r/150662 was merged to [master]. > Commit: > http://git.eclipse.org/c/jdt/eclipse.jdt.core.git/commit/ > ?id=ea2cf5819a5be4907255ff9da714a96100bb270f Changes look solid to me, released for 4.14 M3 Thanks, Sebastian! Things are quiet now. Verified for 4.14 M3 using build I20191119-1250 We are seeing this again in gerrit now: https://ci.eclipse.org/jdt/job/eclipse.jdt.core-Gerrit-BETA/657/testReport/junit/org.eclipse.jdt.core.tests.compiler.regression/GenericTypeTest/testBug543480BasedOnTest2FromComment4ToSameSameOptimization___1_8/~ Stacktrace junit.framework.AssertionFailedError: Potential fluctuation of a performance test: average long compile time 506.8333333333333ms should be less than 10.0x the average short compile time 7.666666666666667ms long compile times: [#7 71ms (excluded), #6 158ms (excluded), #8 218ms, #5 241ms, #2 407ms, #3 541ms, #9 804ms, #4 830ms, #10 1277ms (excluded), #1 2290ms (excluded)] short compile times: [#8 6ms (excluded), #9 6ms (excluded), #10 6ms, #4 7ms, #6 7ms, #5 8ms, #2 9ms, #3 9ms, #1 12ms (excluded), #7 17ms (excluded)] at org.eclipse.jdt.core.tests.compiler.regression.GenericTypeTest.assertCompileTimes(GenericTypeTest.java:52752) Thanks for reporting this, Jay! So the long compile times fluctuate a lot. Looking at them in the order of appearance (not sorted by duration as in the output): #1 2290ms (excluded) #2 407ms #3 541ms #4 830ms #5 241ms #6 158ms (excluded) #7 71ms (excluded) #8 218ms #9 804ms #10 1277ms (excluded) I was not able to reproduce these numbers when running only GenericTypeTest locally in the IDE. Could you trigger then Jenkins jobs for the Gerrit patch again? I would expect that the numbers are not reproducible. Since the tests run on build machines, is it possible that other tests / Jenkins jobs run in parallel while the test is executing? (In reply to Sebastian Lohmeier from comment #7) > Could you trigger then Jenkins jobs for the Gerrit patch again? I would > expect that the numbers are not reproducible. I think so too. We had several builds succeeding this week. Just to be sure, Sarika can you please re-trigger it? > Since the tests run on build machines, is it possible that other tests / > Jenkins jobs run in parallel while the test is executing? Yes, very likely too. |