Bug 370649

Comment 1 Dani Megert 2012-02-15 02:03:41 EST

ping!

Comment 2 Serge Beauchamp 2012-02-15 08:52:33 EST

Hi, those links do not seem to work, do they work only inside the eclipse domain?

I checked the latest drop, but there's no errors reported.  Do you have another link to the failure report?

Comment 3 Dani Megert 2012-02-15 08:57:12 EST

(In reply to comment #2)
> Hi, those links do not seem to work, do they work only inside the eclipse
> domain?
No they used to work just fine but because the bug was not looked at, the N-build has vanished in the meantime (they are deleted after a certain period).


> I checked the latest drop, but there's no errors reported.  Do you have 
> another link to the failure report?

http://download.eclipse.org/eclipse/downloads/drops/N20120212-2000/testresults/html/org.eclipse.core.tests.resources_win32.win32.x86_7.0.html

http://download.eclipse.org/eclipse/downloads/drops/N20120213-2000/testresults/html/org.eclipse.core.tests.resources_win32.win32.x86_7.0.html

Comment 4 Grzegorz Abramczyk 2012-02-15 09:14:55 EST

Tests from org.eclipse.core.tests.resources.regression.Bug_303517 (failure in first and second test) behave nondeterministically also on JRE6. I'm trying to investigate why. 

The test reports can be retrieved using google cache.

Comment 5 Grzegorz Abramczyk 2012-02-17 05:52:17 EST

I think I have found source of the problem. Some tests (for example from Bug_303517) depends on order of execution of test methods. And on JVM 7 (http://bugs.sun.com/bugdatabase/view_bug.do;jsessionid=8ae3b46cb2a96ffffffffef2a88f5e5eede3?bug_id=7023180) order may vary from execution to execution. I think the unit tests have to be fixed because:
* I don't think dependencies between test cases are not a good thing.
* From JVM bug tickets I conclude It would not be fixed. And consistent ordering was never guaranteed. 
* If we want to fix ordering (to at least have consistent ordering if not in order of declaration in class file) JUnit classes have to be changed.

Inconsistent behavior on JVM 6 was my mistake, I have had run tests in UI thread.

Comment 6 Dani Megert 2012-02-17 05:55:39 EST

(In reply to comment #5)
> I think I have found source of the problem. Some tests (for example from
> Bug_303517) depends on order of execution of test methods. And on JVM 7
> (http://bugs.sun.com/bugdatabase/view_bug.do;jsessionid=8ae3b46cb2a96ffffffffef2a88f5e5eede3?bug_id=7023180)
> order may vary from execution to execution. I think the unit tests have to be
> fixed because:
> * I don't think dependencies between test cases are not a good thing.
It can make sense in some cases, but then it must be guaranteed by the test itself, e.g. by using 'org.eclipse.test.OrderedTestSuite' (in the '/org.eclipse.test.performance' bundle).


> * From JVM bug tickets I conclude It would not be fixed. And consistent
> ordering was never guaranteed. 
Correct.

Comment 7 Grzegorz Abramczyk 2012-02-17 06:13:29 EST

Created attachment 211169 [details]
tearDown method was calling super.setUp() instead of super.tearDown()

I have found bug in Bug_303517.
After fix it works few times in a row.

I now will check failures in other tests.

Comment 8 Szymon Brandys 2012-04-02 09:14:03 EDT

Grzesiek, I applied the patch but the tests still seem to fail from time to time. Are you still on it?

Comment 9 Grzegorz Abramczyk 2012-04-10 05:29:54 EDT

Created attachment 213789 [details]
Fix for dependency between testAssociationInheritance and testPreferences

I have found negative dependency between testAssociationInheritance and testPreferences methods so testPreferences  failed if executed after testAssociationInheritance.

I have prepared crude fix using reflection to clean up PreferencesService and ContentTypeManager but there may be a better way to do this.

Comment 10 Grzegorz Abramczyk 2012-04-10 06:04:52 EDT

I have found that failure of IContentTypeManagerTest.testPreferences may cause failure of SpecificContextTest.testIsAssociatedWith. My last fix helps also in this case.

Comment 11 Dani Megert 2012-06-07 05:16:00 EDT

We already have an RC4 candidate. This bug does not justify a rebuild.

Comment 12 Dani Megert 2013-01-30 07:49:42 EST

(In reply to comment #10)
> I have found that failure of IContentTypeManagerTest.testPreferences may
> cause failure of SpecificContextTest.testIsAssociatedWith. My last fix helps
> also in this case.

They both failed again in 
http://download.eclipse.org/eclipse/downloads/drops4/I20130129-0800/testresults/html/org.eclipse.core.tests.resources_macosx.cocoa.x86_5.0.html

Comment 13 Szymon Brandys 2013-01-30 08:40:09 EST

At this point I can just comment them out and try to investigate during M6.

Comment 14 Szymon Brandys 2013-01-30 09:01:22 EST

(In reply to comment #13)
> At this point I can just comment them out and try to investigate during M6.

I have disabled the test for now on Mac but it still needs investigation on Mac to see what is going on.

Comment 15 Dani Megert 2013-01-30 09:43:52 EST

(In reply to comment #14)
> (In reply to comment #13)
> > At this point I can just comment them out and try to investigate during M6.
> 
> I have disabled the test for now on Mac but it still needs investigation on
> Mac to see what is going on.

Looking at the test results, I see that the order is different between the case where it works and where it fails, so, I suggest you simply use org.eclipse.test.OrderedTestSuite#OrderedTestSuite(Class) and hope for the best.

Comment 16 Szymon Brandys 2013-01-30 10:14:20 EST

(In reply to comment #15)
> Looking at the test results, I see that the order is different between the
> case where it works and where it fails, so, I suggest you simply use
> org.eclipse.test.OrderedTestSuite#OrderedTestSuite(Class) and hope for the
> best.

I'm applying the fix and will wait for the next build(s) to verify.

Comment 17 Szymon Brandys 2013-01-30 10:50:33 EST

Marking fixed.

Comment 18 Dani Megert 2013-01-31 06:41:40 EST

Looks good in I20130130-2000.