Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 360580

Summary: Many tests are not Java 7 friendly
Product: [Eclipse Project] PDE Reporter: Michael Rennie <Michael_Rennie>
Component: API ToolsAssignee: Michael Rennie <Michael_Rennie>
Status: VERIFIED FIXED QA Contact:
Severity: normal    
Priority: P3 CC: ankur_sharma, curtis.windatt.public, daniel_megert
Version: 3.8Flags: curtis.windatt.public: review+
Target Milestone: 3.8 M4   
Hardware: PC   
OS: Windows 7   
Whiteboard:
Attachments:
Description Flags
API Tools test failures
none
Patch none

Description Michael Rennie CLA 2011-10-11 15:16:06 EDT 
Version: 3.8.0
Build id: I20111005-0925

code from master.

While running a bunch of tests using Java 7 I was getting some odd and inconsistent failures. One was an NPE in ClassFileScannerTests#testScanInnerStaticInnerClass() - caused because the test relies on an earlier test to set up the environment.

We should do a full pass of the test suite to find other tests that could fail in the same manner.
Comment 1 Michael Rennie CLA 2011-10-11 15:17:37 EDT 
Just got another failure in ApiBaselineManagerTests#testCleanupManager() because the test it relies on to fill the manager had not been run yet.
Comment 2 Michael Rennie CLA 2011-10-12 12:48:44 EDT 
I did a pass over the tests for API tools and I have them all passing on my windows and linux machines. I pushed the changes to master - Curtis, Dani, Ankur, please run the test suites using Java 7 to see if there is anything I missed.
Comment 3 Ankur Sharma CLA 2011-10-13 06:39:25 EDT 
Created attachment 205113 [details]
API Tools test failures

I encountered 6 failures. On first look, they look genuine. So will investigate more and post the findings.


API Tools Plugin Test Suite
org.eclipse.pde.api.tools.tests.ApiToolsPluginTestSuite
org.eclipse.pde.api.tools.builder.tests.ApiBuilderTest
org.eclipse.pde.api.tools.builder.tests.usage.UsageTest
org.eclipse.pde.api.tools.builder.tests.usage.Java7ClassUsageTests
testTryWithF(org.eclipse.pde.api.tools.builder.tests.usage.Java7ClassUsageTests)
junit.framework.AssertionFailedError: Should not be a JDT error:TryWithResourcesClass cannot be resolved to a type

testTryWithI(org.eclipse.pde.api.tools.builder.tests.usage.Java7ClassUsageTests)
junit.framework.AssertionFailedError: Should not be a JDT error:TryWithResourcesClass cannot be resolved to a type


org.eclipse.pde.api.tools.anttasks.tests.ApiToolsAntTasksTestSuite
org.eclipse.pde.api.tools.anttasks.tests.ApiToolingAnalysisAntTaskTests
test1(org.eclipse.pde.api.tools.anttasks.tests.ApiToolingAnalysisAntTaskTests)
junit.framework.AssertionFailedError: deltatest folder must exist

test5(org.eclipse.pde.api.tools.anttasks.tests.ApiToolingAnalysisAntTaskTests)
junit.framework.AssertionFailedError: deltatest folder must exist

test6(org.eclipse.pde.api.tools.anttasks.tests.ApiToolingAnalysisAntTaskTests)
junit.framework.AssertionFailedError: deltatest2 folder must exist

test7(org.eclipse.pde.api.tools.anttasks.tests.ApiToolingAnalysisAntTaskTests)
junit.framework.AssertionFailedError: deltatest folder must exist
Comment 4 Michael Rennie CLA 2011-10-13 10:15:10 EDT 
(In reply to comment #3)
> Created attachment 205113 [details]
> API Tools test failures
> 
> I encountered 6 failures. On first look, they look genuine. So will investigate
> more and post the findings.

Looks like the builder tests failed to build the class files and the Ant tests were failing to create resources. Ankur, was there anything written out to the console? Any ResourceExceptions? Which Java 7 JDK were you using?
Comment 5 Curtis Windatt CLA 2011-10-17 12:59:26 EDT 
(In reply to comment #4)
> (In reply to comment #3)
> > Created attachment 205113 [details] [details]
> > API Tools test failures
> > 
> > I encountered 6 failures. On first look, they look genuine. So will investigate
> > more and post the findings.
> 
> Looks like the builder tests failed to build the class files and the Ant tests
> were failing to create resources. Ankur, was there anything written out to the
> console? Any ResourceExceptions? Which Java 7 JDK were you using?

I can reproduce the test failures that Ankur is describing (they are the only test failures I get).  Related output in the console:

org.eclipse.core.internal.resources.ResourceException: Resource '/APIPluginTests/.settings' does not exist.
	at org.eclipse.core.internal.resources.Resource.checkExists(Resource.java:341)
	at org.eclipse.core.internal.resources.Resource.checkAccessible(Resource.java:215)
	at org.eclipse.core.internal.resources.Container.members(Container.java:266)
	at org.eclipse.core.internal.resources.Container.members(Container.java:249)
	at org.eclipse.pde.api.tools.internal.ApiFilterStore$1.runInWorkspace(ApiFilterStore.java:150)
	at org.eclipse.core.internal.resources.InternalWorkspaceJob.run(InternalWorkspaceJob.java:38)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)
!SUBENTRY 1 org.eclipse.core.resources 4 368 2011-10-17 11:38:09.025
!MESSAGE Resource '/APIPluginTests/.settings' does not exist.
Unexpected JDT Marker
TryWithResourcesClass cannot be resolved to a type
Comment 6 Dani Megert CLA 2011-10-26 10:40:36 EDT 
With JRE 7 I also always get the following two failures:
- Java7ClassUsageTests.testTryWithF()
- Java7ClassUsageTests.testTryWithI()

BUT, also the other tests are definitely not yet stable. I get (different) failures when running JRE 6 and JRE 7 and often on a second run the tests showed different failures or were green again.

For example using JRE 1.7.0 IBM J9 2.6 Windows 7 x86-32 20110810_88604 I first got this in addition to the 2 mentioned above:
- InterfaceCompatibilityTypeParameterTests.testAddSecondaryTypeParameterF()

Then on the second run I got this in addition to the 2 mentioned above:
- FieldCompatibilityModifierTests.testProtectedToPrivateF()

And with Oracle JRE 7 Update 1 I got 
- ClassCompatibilityFieldTests.testRemoveProtectedFieldNoInstantiateF()

And when running on Oracle JRE 6 I got:
- ClassCompatibilityMemberTypeTests.testProtectedToPrivateF()


I always ran 'ApiToolsPluginTestSuite'.


Note: No problems running 'ApiToolsTestSuite'.
Comment 7 Curtis Windatt CLA 2011-11-03 12:48:51 EDT 
Ankur said he had made some progress here on the status call.
Comment 8 Ankur Sharma CLA 2011-11-04 05:07:00 EDT 
Created attachment 206452 [details]
Patch

Curtis, let me know when its ok to apply it
Comment 9 Curtis Windatt CLA 2011-11-04 09:27:11 EDT 
(In reply to comment #8)
> Created attachment 206452 [details]
> Patch
> 
> Curtis, let me know when its ok to apply it

If the patch fixes the tests, you should commit it. I'll try to leave some time today to test it.
Comment 10 Michael Rennie CLA 2011-11-09 15:40:31 EST 
(In reply to comment #9)
>  I'll try to leave some time today to test it.

looks like Ankur committed the patch: http://git.eclipse.org/c/pde/eclipse.pde.ui.git/commit/?id=48b6a2d8ed2eb56ca0539b1484ef7b1c4c961d32
Comment 12 Michael Rennie CLA 2011-11-15 12:26:47 EST 
Curtis found an interesting relationship in the tests that caused some of the Java 7-specific ones to fail. I pushed a fix to: http://git.eclipse.org/c/pde/eclipse.pde.ui.git/commit/?id=780d4ddb0e96b578e4b4e4ede271b6eff255981d
Comment 13 Curtis Windatt CLA 2011-11-15 14:48:56 EST 
Verified that the tests now all pass for me.
Comment 14 Curtis Windatt CLA 2011-11-15 14:49:19 EST 
Verified
Comment 15 Dani Megert CLA 2011-11-23 09:41:56 EST 
It is much more stable now. I only got one transient failure during my testing:

With JRE 1.7.0 IBM J9 2.6 Windows 7 x86-32 20110810_88604 (JIT enabled, AOT enabled) the following failed once:

org.eclipse.pde.api.tools.builder.tests.compatibility.InvalidSinceTagTests.testInvalidMethodF()