Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 469953

Summary:	[Releng] Update the build to support a global POM
Product:	[Modeling] Papyrus	Reporter:	Camille Letavernier <cletavernier>
Component:	Others	Assignee:	Camille Letavernier <cletavernier>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	P3
Version:	1.1.0
Target Milestone:	---
Hardware:	All
OS:	All
Whiteboard:

Description Camille Letavernier

2015-06-11 10:33:27 EDT

The idea is to provide a main pom (Which we currently don’t have) to build all components of Papyrus at the same time (Especially useful for Gerrit and Sonar)
 
However, we made several bad choices in the current build implementation, which means that simply adding a top-level pom is not sufficient. An important build refactoring is required
 
-	Top poms will be moved to a different location (Because we can’t have several Poms in the same folder)
-	Top poms will be renamed (Because they all had the same artifact ID)
-	Shared artifacts (Especially the test framework) will have to be built/deployed separately
 
Additionally, some changes are required in the test framework, to be able to run tests for each test plug-in separately. We also need to distinguish plug-ins which provide helpers for the tests (diagram.tests, junit.framework, ...), and actual test plug-ins
 
These changes will be more disruptive than I expected, so i’d like to push them on both Mars maintenance and Neon, to keep builds and Hudson jobs as consistent as possible between branches. Feature branches and existing Gerrit contributions affecting POMs/Releng will need to be updated accordingly.

Comment 1 Camille Letavernier

2015-06-11 11:24:22 EDT

Initial build refactoring pushed to master. Tests are not fixed yet

The Mars and Gerrit builds have been updated to match these changes. Gerrit builds will now fail for commits which are not rebased on these latest commits, meaning that Gerrit won't work for e.g. Luna builds anymore

Comment 2 Camille Letavernier

2015-06-11 11:43:50 EDT

TODO:

- Fix tests/test framework/shared test plug-ins
- Fix update site repositories, which are not defined/used when building everything (Since we don't rely on update sites from a previous build). Might need a specific profile

Comment 3 Camille Letavernier

2015-06-12 08:24:58 EDT

> - Fix tests/test framework/shared test plug-ins

The tests are now fixed. To avoid running each test plug-in separately, a new property has been added: ${skipInnerTests}. When set to true, tests will not be executed. By default, it is always true, except for Papyrus-AllTests

Shared plug-ins (oep.junit.framework, oep.junit.utils and oep.bundle.tests) are now built (compiled) separately, in the Developer build, then reused by test builds (Main and Extra). This was required because a Maven artifact can only have one parent and should be built only once. This wasn't previously the case because we (accidentally) tricked Maven by having two parents with the same ID

Main-Tests and Extra-Tests now depend on the Developer job, but this shouldn't be a big issue in practice because these plug-ins don't evolve much. When updating Junit Utils and Junit Framework, we may need to manually trigger the Papyrus-X-Developer build before using the new version

Comment 4 Camille Letavernier

2015-06-12 09:00:18 EDT

The new developer update site:

https://hudson.eclipse.org/papyrus/job/Papyrus-Master-Developer/lastSuccessfulBuild/artifact/repository

Comment 5 Camille Letavernier

2015-06-12 09:54:52 EDT

Tycho 0.23 doesn't seem to fix the Surefire regression introduced in 0.22, that produces an invalid XML Junit report:

ERROR: Failed to archive test reports
hudson.util.IOException2: Failed to read /home/hudson/genie.papyrus/.hudson/jobs/Papyrus-Master-Tests-Failures/workspace/tests/junit/plugins/core/org.eclipse.papyrus.tests/target/surefire-reports/TEST-org.eclipse.papyrus.tests.AllTests.xml
	at hudson.tasks.junit.TestResult.parse(TestResult.java:203)
	at hudson.tasks.junit.TestResult.parse(TestResult.java:141)
	at hudson.tasks.junit.TestResult.<init>(TestResult.java:104)
	at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:111)
	at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:83)
	at hudson.FilePath.act(FilePath.java:793)
	at hudson.FilePath.act(FilePath.java:775)
	at hudson.tasks.junit.JUnitParser.parse(JUnitParser.java:79)
	at hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:125)
	at hudson.tasks.junit.JUnitResultArchiver.perform(JUnitResultArchiver.java:137)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:34)
	at hudson.model.AbstractBuild$AbstractRunner.perform(AbstractBuild.java:736)
	at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:714)
	at hudson.model.AbstractBuild$AbstractRunner.performAllBuildSteps(AbstractBuild.java:690)
	at hudson.model.Build$RunnerImpl.post2(Build.java:163)
	at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:652)
	at hudson.model.Run.run(Run.java:1517)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)
	at hudson.model.ResourceController.execute(ResourceController.java:82)
	at hudson.model.Executor.run(Executor.java:137)
Caused by: org.dom4j.DocumentException: Error on line 33023 of document file:///home/hudson/genie.papyrus/.hudson/jobs/Papyrus-Master-Tests-Failures/workspace/tests/junit/plugins/core/org.eclipse.papyrus.tests/target/surefire-reports/TEST-org.eclipse.papyrus.tests.AllTests.xml : XML document structures must start and end within the same entity. Nested exception: XML document structures must start and end within the same entity.
	at org.dom4j.io.SAXReader.read(SAXReader.java:482)
	at org.dom4j.io.SAXReader.read(SAXReader.java:264)
	at hudson.tasks.junit.SuiteResult.parse(SuiteResult.java:93)
	at hudson.tasks.junit.TestResult.parse(TestResult.java:187)
	... 19 more
Caused by: org.xml.sax.SAXParseException; systemId: file:///home/hudson/genie.papyrus/.hudson/jobs/Papyrus-Master-Tests-Failures/workspace/tests/junit/plugins/core/org.eclipse.papyrus.tests/target/surefire-reports/TEST-org.eclipse.papyrus.tests.AllTests.xml; lineNumber: 33023; columnNumber: 1; XML document structures must start and end within the same entity.
	at org.apache.xerces.util.ErrorHandlerWrapper.createSAXParseException(Unknown Source)
	at org.apache.xerces.util.ErrorHandlerWrapper.fatalError(Unknown Source)
	at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
	at org.apache.xerces.impl.XMLErrorReporter.reportError(Unknown Source)
	at org.apache.xerces.impl.XMLScanner.reportFatalError(Unknown Source)
	at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.endEntity(Unknown Source)
	at org.apache.xerces.impl.XMLDocumentScannerImpl.endEntity(Unknown Source)
	at org.apache.xerces.impl.XMLEntityManager.endEntity(Unknown Source)
	at org.apache.xerces.impl.XMLEntityScanner.load(Unknown Source)
	at org.apache.xerces.impl.XMLEntityScanner.scanContent(Unknown Source)
	at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanContent(Unknown Source)
	at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
	at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
	at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
	at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
	at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
	at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
	at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source)
	at org.dom4j.io.SAXReader.read(SAXReader.java:465)
	... 22 more

	
Once the tests are fixed, I'll revert back to Tycho 0.21 (The build should still be compatible in that direction)

Comment 6 Camille Letavernier

2015-06-12 11:46:07 EDT

The Hudson server is failing on Extra-Tests (Which works for me locally). I've no idea why. The tests are properly compiled, but starting an Eclipse Runtime fails on the following exception:

!SESSION Fri Jun 12 10:09:05 EDT 2015 ------------------------------------------
!ENTRY org.eclipse.equinox.launcher 4 0 2015-06-12 10:09:05.837
!MESSAGE Exception launching the Eclipse Platform:
!STACK
java.lang.SecurityException: SHA-256 digest error for org/eclipse/osgi/framework/log/FrameworkLogEntry.class
        at sun.security.util.ManifestEntryVerifier.verify(ManifestEntryVerifier.java:220)
        at java.util.jar.JarVerifier.processEntry(JarVerifier.java:241)
        at java.util.jar.JarVerifier.update(JarVerifier.java:228)
        at java.util.jar.JarVerifier$VerifierStream.read(JarVerifier.java:483)
        at sun.misc.Resource.getBytes(Resource.java:124)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:444)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
        at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:241)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:669)
        at org.eclipse.equinox.launcher.Main.basicRun(Main.java:608)
        at org.eclipse.equinox.launcher.Main.run(Main.java:1515)
        at org.eclipse.equinox.launcher.Main.main(Main.java:1488)

Comment 7 Camille Letavernier

2015-06-15 07:00:56 EDT

Two outstanding issues:

- The dependency resolution seems a little bit different when using a global POM (Because update sites/features are produced during a single build). It seems that we need to reference the Features directly, rather than the "Feature group" (which can't be found). I'm not sure why. Referencing the Feature seems to work
- The Extra Tests still fail on Hudson; Eclipse won't start (Comment 6)

I may have a solution for the first issue; I'm currently running it locally. Regarding the second one, that's more difficult to test; maybe the new platform configuration will solve this issue as well. Will have to be tested on Hudson

Comment 8 Camille Letavernier

2015-06-15 08:07:07 EDT

- Master-Tests is back to normal
- Master-Full seems OK (Still running)
- Gerrit fails because it tries to execute the tests, although the skipTests parameter is set to true

I suspect the skipTests=false defined in the POM has a higher priority than the one passed as job parameter, so we may need a specific Gerrit profile

I will start with "clean compile" instead of "clean verify", which should skip the test phase entirely (I don't remember why we used "clean verify" for Gerrit...)

> Regarding the second one, that's more difficult to test; maybe the new platform configuration will solve this issue as well. Will have to be tested on Hudson

- Extra-Tests still fails to start Eclipse; nothing new :(

Comment 9 Camille Letavernier

2015-06-15 09:33:02 EDT

> - Extra-Tests still fails to start Eclipse; nothing new :(

Interestingly, the same tests work fine in Papyrus-Full (using the global POM.xml)....

Comment 10 Camille Letavernier

2015-06-16 08:22:36 EDT

> Tycho 0.23 doesn't seem to fix the Surefire regression introduced in 0.22, that produces an invalid XML Junit report:

We now use Tycho 0.21 exclusively for the Papyrus-X-Tests-Failures job, via Hudson properties:

tycho-version=0.21.0
testSuite=org.eclipse.papyrus.tests

The testSuite needs to be specified in the build configuration because this parameter doesn't exist in 0.23 anymore

Comment 11 Camille Letavernier

2015-06-17 08:30:58 EDT

The releng launch configs have been renamed to be prefixed with "Papyrus - "

The extra tests standalone build is still broken, but this task is now complete