Community
Participate
Working Groups
Most of Tycho local repository access access is now guarded against concurrent modifications (see bug 347963), so it makes sense to go through rest of Tycho code and makes sure we properly local filesystem access everywhere. One obvious place that needs to be fixed is TychoP2RuntimeLocator.resolveRuntimeArtifact which unzipes tycho osgi runtime in Maven local repository. We need to make sure only one process can do this at any given time, and one process cannot delete/change unzipped tycho osgi runtime while another is running (one implication of this, locking service implementation can't be part of tycho osgi runtime). What other places still need synchronization?
(In reply to comment #0) > Most of Tycho local repository access access is now guarded against concurrent > modifications (see bug 347963), so it makes sense to go through rest of Tycho > code and makes sure we properly local filesystem access everywhere. > > One obvious place that needs to be fixed is > TychoP2RuntimeLocator.resolveRuntimeArtifact which unzipes tycho osgi runtime > in Maven local repository. I tried to fix this, but this is before the embedded equinox is up which provides the file locking service. >We need to make sure only one process can do this at > any given time, and one process cannot delete/change unzipped tycho osgi > runtime while another is running (one implication of this, locking service > implementation can't be part of tycho osgi runtime). Either refactor file locking to make this possible (not sure how to do this) or use low-level locking just for this case. > What other places still need synchronization? - there are synchronization issues with several processes runinng the embedded equinox. E.g. all processes use the same configuration/ folder e.g. to store .prefs files. I have also seen "could not acquire lock for configuration" or similar error messages from equinox. Maybe use a separate temporary configuration/ folder for each process? - we need to investigate if the p2 cache in ~/.m2/repository/.cache/tycho is multi-process safe (does p2 take care of locking already?)
(In reply to comment #1) > - there are synchronization issues with several processes runinng the embedded > equinox. E.g. all processes use the same configuration/ folder e.g. to store > .prefs files. I have also seen "could not acquire lock for configuration" or > similar error messages from equinox. Maybe use a separate temporary > configuration/ folder for each process? > I think the same problems applies both for embedded equinox and when running tycho osgi apps in external jvm. I had to deal with similar problem in the past and equinox writes some cached state during shutdown... which messes up the state if two or more instances run in parallel. The only solution I was able to find is to use separate configuration directory for each equinox instance.
(In reply to comment #2) a temporary configuration/ folder seems to fix the problem. While trying this out I saw that there is quite some garbage left over in this folder from each run so we need to clean up (~200 kB). Disposable.dispose() seems like a good time to do this but it seems the dispose method in http://git.eclipse.org/c/tycho/org.eclipse.tycho.git/tree/tycho-equinox/src/main/java/org/eclipse/tycho/equinox/embedder/DefaultEquinoxEmbedder.java is never called :( Otherwise we have to fall back to maven ExecutionListener.sessionEnded() or something similar. Any hints?
Created attachment 207674 [details] sample test failure log I can't successfully run tests with -Pparallel neither on my macbookpro (4-core/8-thread) nor on much older linux box (2-core/2-thread). I do not have physical windows box to see if -Pparallel works more reliably there. From the failure logs I checked, the problem seems to be related to embedded osgi runtime, see attached sample test failure log. Disposable.dispose is not called due to a bug in Maven core, so feel free to open a bug report against maven core. We could use ExecutionEvent.SessionEnded to shutdown embedder from our code, but Maven 3.0 events delivery infrastructure is not very good (one global event listener), so I am not sure if we want to use that. Maven 3.0.3 has better event listener APIs, so we can implement proper cleanup for 3.0.3+ only.
(In reply to comment #4) > Created attachment 207674 [details] > sample test failure log > > I can't successfully run tests with -Pparallel neither on my macbookpro > (4-core/8-thread) nor on much older linux box (2-core/2-thread). I do not have > physical windows box to see if -Pparallel works more reliably there. From the > failure logs I checked, the problem seems to be related to embedded osgi > runtime, see attached sample test failure log. yes I am also seeing failures on windows. After hacking at the two problems: 1. file locking while extracting tycho-p2-runtime 2. use a temporary configurationarea for each embedded osgi I have not seen problems with parallel tests anymore ( that doesn't prove they are gone ...). > Disposable.dispose is not called due to a bug in Maven core, so feel free to > open a bug report against maven core. > > We could use ExecutionEvent.SessionEnded to shutdown embedder from our code, > but Maven 3.0 events delivery infrastructure is not very good (one global event > listener), so I am not sure if we want to use that. Maven 3.0.3 has better > event listener APIs, so we can implement proper cleanup for 3.0.3+ only. we will have to support maven 3.0 for quite some time so 3.0.3+ is not really an option I think. I have tried with an execution listener which piggy-backs on the one global execution listener so this could be an option. What I am more concerned about is the ~200kB payload of caches which are initialzed every time when using a temporary configuration area. The only idea I could come up with after some googling is to use the equinox multi-user mode [1] to work around this. Idea: 1. We (lock and) prime the shared configuration area once using the -initialize CLI option [2] 2. We put a marker file ".initialized" or similar into the primed configuration area 3. All subsequent calls will check for the marker file and make the shared configuration area readonly using osgi.sharedConfiguration.area.readOnly=true and point osgi.configuration.area to a temporary dir This way the bulk of caches/state stays in the shared configuration area and the amount of garbage in the temp dir should be much less. [1] http://help.eclipse.org/indigo/topic/org.eclipse.platform.doc.isv/reference/misc/multi_user_installs.html [2] http://help.eclipse.org/indigo/index.jsp?topic=%2Forg.eclipse.platform.doc.isv%2Freference%2Fmisc%2Fruntime-options.html
I honestly think it is okay to provide degraded support for 3.0-3.0.2, i.e. Tycho will still work but will leave garbage files behind. If people complain, we just tell them to upgrade to Maven 3.0.3+. I agree that primed read-only configuration area is the best solution, but I am not sure if it's worth the hassle -- from what I remember, these cache files contain resolved OSGi state and equinox extension registry, both should be quite fast to recreate given small number of bundles in Tycho embedded osgi runtime. Also, I wonder if we actually need instance-specific read/write equinox configuration area at all. Tycho embedded osgi runtime is static, so I don't see why Equinox would need to write anything instance-specfic. At the same time, we most likely need instance specific p2 data area, so we may just manage everything under the same temporary directory.
(In reply to comment #6) > I honestly think it is okay to provide degraded support for 3.0-3.0.2, i.e. > Tycho will still work but will leave garbage files behind. agreed if it still works with maven 3.0 Where are the new listener APIs? > I agree that primed read-only configuration area is the best solution, but I am > not sure if it's worth the hassle -- from what I remember, these cache files > contain resolved OSGi state and equinox extension registry, both should be > quite fast to recreate given small number of bundles in Tycho embedded osgi > runtime. Probably premature optimization on my part. Let's do the simple solution first (new temporary configuation dir for each instance and remove garbage if possible)
(In reply to comment #4) > Disposable.dispose is not called due to a bug in Maven core, so feel free to > open a bug report against maven core. See discussion on sisu-dev http://dev.eclipse.org/mhonarc/lists/sisu-dev/msg00001.html
(In reply to comment #8) created maven core bug http://jira.codehaus.org/browse/MNG-5206
(In reply to comment #7) > (In reply to comment #6) > > I honestly think it is okay to provide degraded support for 3.0-3.0.2, i.e. > > Tycho will still work but will leave garbage files behind. > > agreed if it still works with maven 3.0 > Where are the new listener APIs? > I was thinking about org.apache.maven.eventspy.EventSpy... unfortunately, it looks like implementation does not support dynamic contribution of event spies by extensions plugins, see https://github.com/apache/maven-3/blob/trunk/maven-core/src/main/java/org/apache/maven/eventspy/internal/EventSpyDispatcher.java#L45 . I think we can still use EventSpyDispatcher.chainListener from TychoMavenLifecycleParticipant, but this is not as pretty.
(In reply to comment #10) > I was thinking about org.apache.maven.eventspy.EventSpy... unfortunately, it > looks like implementation does not support dynamic contribution of event spies > by extensions plugins, see I did not find a way to use the 3.0.3 listener APIs with maven 3.0 compatibility preserved so I used the 3.0 listener API. Anyway, after pushing commits 7f78257 365012 protect p2 runtime extraction with lock d5335c0 365012 Use a temp folder for osgi.configuration.area I can now repeatedly run tycho-its with -Pparallel (on Windows 7) and don't get test failures anymore. @Igor, Tobias: if you can confirm this I propose to close this bug.
Still fails on macosx using commit 7f78257fff org.apache.maven.InternalErrorException: Internal error: java.lang.RuntimeException: Could not instantiate required component at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:163) at org.apache.maven.cli.MavenCli.execute(MavenCli.java:445) at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:168) at org.apache.maven.cli.MavenCli.main(MavenCli.java:132) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:290) at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:230) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409) at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352) Caused by: java.lang.RuntimeException: Could not instantiate required component at org.eclipse.tycho.core.resolver.DefaultTargetPlatformResolverFactory.lookupPlatformResolver(DefaultTargetPlatformResolverFactory.java:76) at org.eclipse.tycho.core.resolver.DefaultTychoDependencyResolver.setupProject(DefaultTychoDependencyResolver.java:71) at org.eclipse.tycho.core.maven.TychoMavenLifecycleParticipant.afterProjectsRead(TychoMavenLifecycleParticipant.java:56) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:268) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:151) ... 11 more Caused by: org.codehaus.plexus.component.repository.exception.ComponentLookupException: com.google.inject.ProvisionException: Guice provision errors: 1) Error injecting: org.eclipse.tycho.p2.resolver.P2TargetPlatformResolver at ClassRealm[extension>org.eclipse.tycho:tycho-maven-plugin:0.14.0-SNAPSHOT, parent: ClassRealm[maven.api, parent: null]] while locating org.eclipse.tycho.core.TargetPlatformResolver annotated with @com.google.inject.name.Named(value=p2) 1 error role: org.eclipse.tycho.core.TargetPlatformResolver roleHint: p2 at org.codehaus.plexus.DefaultPlexusContainer.lookup(DefaultPlexusContainer.java:253) at org.codehaus.plexus.DefaultPlexusContainer.lookup(DefaultPlexusContainer.java:241) at org.eclipse.tycho.core.resolver.DefaultTargetPlatformResolverFactory.lookupPlatformResolver(DefaultTargetPlatformResolverFactory.java:74) ... 15 more Caused by: com.google.inject.ProvisionException: Guice provision errors: 1) Error injecting: org.eclipse.tycho.p2.resolver.P2TargetPlatformResolver at ClassRealm[extension>org.eclipse.tycho:tycho-maven-plugin:0.14.0-SNAPSHOT, parent: ClassRealm[maven.api, parent: null]] while locating org.eclipse.tycho.core.TargetPlatformResolver annotated with @com.google.inject.name.Named(value=p2) 1 error at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:957) at com.google.inject.Scopes$1$1.get(Scopes.java:63) at org.sonatype.guice.bean.locators.LazyQualifiedBean.getValue(LazyQualifiedBean.java:66) at org.sonatype.guice.plexus.locators.LazyPlexusBean.getValue(LazyPlexusBean.java:54) at org.codehaus.plexus.DefaultPlexusContainer.lookup(DefaultPlexusContainer.java:249) ... 17 more Caused by: com.google.inject.ProvisionException: Guice provision errors: 1) Error notifying InjectionListener org.sonatype.guice.plexus.binders.PlexusBeanBinder@20e5ace9 of org.eclipse.tycho.p2.resolver.P2TargetPlatformResolver. Reason: java.lang.RuntimeException: java.io.IOException: File /tmp/tycho-bootstrap.localrepo/org/eclipse/tycho/tycho-bundles-external/0.14.0-SNAPSHOT/eclipse/configuration/config.ini does not exist while locating org.eclipse.tycho.p2.resolver.P2TargetPlatformResolver 1 error at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:957) at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:983) at org.sonatype.guice.bean.reflect.AbstractDeferredClass.get(AbstractDeferredClass.java:48) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40) at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:948) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:994) at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:944) ... 21 more Caused by: java.lang.RuntimeException: java.io.IOException: File /tmp/tycho-bootstrap.localrepo/org/eclipse/tycho/tycho-bundles-external/0.14.0-SNAPSHOT/eclipse/configuration/config.ini does not exist at org.eclipse.sisu.equinox.embedder.internal.DefaultEquinoxEmbedder.checkStarted(DefaultEquinoxEmbedder.java:250) at org.eclipse.sisu.equinox.embedder.internal.DefaultEquinoxEmbedder.getService(DefaultEquinoxEmbedder.java:224) at org.eclipse.sisu.equinox.embedder.internal.DefaultEquinoxEmbedder.getService(DefaultEquinoxEmbedder.java:220) at org.eclipse.tycho.p2.resolver.P2TargetPlatformResolver.initialize(P2TargetPlatformResolver.java:439) at org.codehaus.plexus.PlexusLifecycleManager.initialize(PlexusLifecycleManager.java:227) at org.codehaus.plexus.PlexusLifecycleManager.manage(PlexusLifecycleManager.java:151) at org.sonatype.guice.plexus.binders.PlexusBeanBinder.afterInjection(PlexusBeanBinder.java:79) at com.google.inject.internal.MembersInjectorImpl.notifyListeners(MembersInjectorImpl.java:97) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:95) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:253) at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:948) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1001) at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:944) ... 27 more Caused by: java.io.IOException: File /tmp/tycho-bootstrap.localrepo/org/eclipse/tycho/tycho-bundles-external/0.14.0-SNAPSHOT/eclipse/configuration/config.ini does not exist at org.codehaus.plexus.util.FileUtils.copyFile(FileUtils.java:1003) at org.codehaus.plexus.util.FileUtils.copyFileToDirectory(FileUtils.java:955) at org.eclipse.sisu.equinox.embedder.internal.DefaultEquinoxEmbedder.copyToTempFolder(DefaultEquinoxEmbedder.java:143) at org.eclipse.sisu.equinox.embedder.internal.DefaultEquinoxEmbedder.doStart(DefaultEquinoxEmbedder.java:86) at org.eclipse.sisu.equinox.embedder.internal.DefaultEquinoxEmbedder.start(DefaultEquinoxEmbedder.java:65) at org.eclipse.sisu.equinox.embedder.internal.DefaultEquinoxEmbedder.checkStarted(DefaultEquinoxEmbedder.java:248) ... 39 more
(In reply to comment #12) the artifact org/eclipse/tycho/tycho-bundles-external/0.14.0-SNAPSHOT/ sounds fishy to me.
(In reply to comment #13) > (In reply to comment #12) > the artifact > > org/eclipse/tycho/tycho-bundles-external/0.14.0-SNAPSHOT/ > > sounds fishy to me. forget it, Tobias renamed it...
(In reply to comment #12) > Still fails on macosx using commit 7f78257fff http://git.eclipse.org/c/tycho/org.eclipse.tycho.git/commit/?id=444f76683b24f776fecd6e4d30548ecab850b4a5 hopefully fixes it. Unfortunately I can't reproduce the problem here but there was definitely a bug in the code which was supposed to make the runtime zip extraction multi-process safe.
I ran the build several times both on the macbook and on a linux desktop and I can confirm the tests all pass now with -Pparallel.
Today's master also works on my Win7 PC. This is nice! Finally something to do for all these cores ;-)
I tested on Windows 7 and Linux. I qualify this as "good enough" for now.
