Community
Participate
Working Groups
executing tycho integration tests takes 20 or more minutes which makes the development turnaround cycle for tycho too slow. Some options we could consider to speed up execution time: - don't fork VM for maven calls. This should be possible using the most recent version of maven verifier (have to make sure we clean all caches/static state though) - make sure all tests which reach out to the network use a local file:// p2 repo instead
example for in-process maven calls ( see profile "embedded"): http://svn.apache.org/repos/asf/maven/core-integration-testing/trunk/core-it-suite/pom.xml
I tried running ITs (test classes, not methods) in parallel with surefire 2.9: <parallel>classes</parallel> <threadCount>2</threadCount> on my machine, this speeds up total IT test execution time by a factor of 4 to 5. (down from ~20 min to ~4-5 min) What we would need to do for parallel test class execution to work reliably: - tests need to be truly independent of each other => separate a few test projects which are shared between several test classes => check for statics/singletons used from concurrent tests - local repo needs to be pre-filled with p2 caches (and any needed maven deps) (For p2 I have the idea of using the p2 mirror mojo before running ITs to fill a local p2 repo with minimal content, say org.eclipse.rcp feature. All ITs would then use the local p2 repo and no longer go on the network) - concurrent access to tycho index files in local repo should be protected (bug 347963 ) Especially the last point is some effort but will pay off in the long run IMHO.
(In reply to comment #2) > => check for statics/singletons used from concurrent tests > > - local repo needs to be pre-filled with p2 caches (and any needed maven deps) > (For p2 I have the idea of using the p2 mirror mojo before running ITs to > fill a local p2 repo with minimal content, say org.eclipse.rcp feature. > All ITs would then use the local p2 repo and no longer go on the network) > > - concurrent access to tycho index files in local repo should be protected (bug > 347963 ) > > Especially the last point is some effort but will pay off in the long run IMHO. What is the point in "pre-filling" p2 local cache? If the cache is not thread/process -safe, I think it makes far more sense to fix the cache than to find ways to pre-populate it. After all, running multiple builds concurrently _is_ expected to work.
(In reply to comment #3) > What is the point in "pre-filling" p2 local cache? If the cache is not > thread/process -safe, I think it makes far more sense to fix the cache than to > find ways to pre-populate it. After all, running multiple builds concurrently > _is_ expected to work. sure. I mixed up 2 things here: For my local experiment I pre-filled the local repo so I would run into concurrency problems with less probability. The point here was just to find out the potential speed-up. You are right that fixing the concurrency issue in bug 347963 would make this obsolete. For maven core, I understood that the multi-process safe local repo will only come with Tesla so up to then we need to make sure maven (plugin) dependencies of test projects are pre-filled before ITs run.
http://git.eclipse.org/c/tycho/org.eclipse.tycho.git/commit/?id=cc0a1231c5ea46dec13541a43a31a0f72363381d adds experimental support for running ITs in parallel (factor 4-5 speedup on my machine)
Do I need to do anything special to run ITs beyond adding -Pparallel? I am getting a few IT failures on my macbookpro Tests in error: testP2DataGeneratedForPomDependency(org.eclipse.tycho.test.pomDependencyConsider.p2Data.generate.PomDependencyOnNonTychoArtifactTest): Exit code was non-zero: 1; command line and log = testSigning(org.eclipse.tycho.test.jarsigning.JarSigningTest): Exit code was non-zero: 1; command line and log = eeFromBREE(org.eclipse.tycho.test.bug364095_dependencyResolverBREE.DependencyResolverEETest): Exit code was non-zero: 1; command line and log = test(org.eclipse.tycho.test.bug363331_extraTargetPlatformRequirements.ExtraTargetPlatformRequirementsTest): Exit code was non-zero: 1; command line and log = testFeaturePatch(org.eclipse.tycho.test.featurePatch.build.FeaturePatchTest): Exit code was non-zero: 1; command line and log =
(In reply to comment #6) > Do I need to do anything special to run ITs beyond adding -Pparallel? I am > getting a few IT failures on my macbookpro > > > > > Tests in error: > > testP2DataGeneratedForPomDependency(org.eclipse.tycho.test.pomDependencyConsider.p2Data.generate.PomDependencyOnNonTychoArtifactTest): > Exit code was non-zero: 1; command line and log = > testSigning(org.eclipse.tycho.test.jarsigning.JarSigningTest): Exit code was > non-zero: 1; command line and log = > > eeFromBREE(org.eclipse.tycho.test.bug364095_dependencyResolverBREE.DependencyResolverEETest): > Exit code was non-zero: 1; command line and log = > > test(org.eclipse.tycho.test.bug363331_extraTargetPlatformRequirements.ExtraTargetPlatformRequirementsTest): > Exit code was non-zero: 1; command line and log = > testFeaturePatch(org.eclipse.tycho.test.featurePatch.build.FeaturePatchTest): > Exit code was non-zero: 1; command line and log = It's experimental because I suspect issues e.g. with tycho-p2-runtime when used by several concurrent processes (configuration/ folder cannot be shared by several processes I think) Are your errors reproducible every time? If so please attach detailed surefire reports. Errors I am seeing are rare and not reproducible.
Even though I can imagine more changes to the integration tests to speed up execution time, parallel test execution with bug 365012 fixed solves this problem for me.
FYI, I am having problems running -Pparallel with p2 juno M4 build. At least on macosx. Did not have a chance to try other platforms yet.
Please disregard my last comment, I was not able to reproduce failures with -Pparallel once I fully updated to juno m4+ embedded runtime.
