Community
Participate
Working Groups
Created attachment 285883 [details] Single plugin with platform-specific fragments with native-libs The current 2.3.0-SNAPSHOT of Tycho cannot resolve the classpath of bundles that use a Bundle-NativeCode header with attributes in their MANIFEST.MF. In my scenario I have one host-plugin and two platform-specific fragments with native libraries. One for Windows and one for Linux. The attached minimal example demonstrates it. The META-INF/MANIFEST.MF of the Windows fragment contains (besides others) the following headers: """ Eclipse-PlatformFilter: (& (osgi.os=win32) (osgi.ws=win32) (osgi.arch=x86_64)) Bundle-NativeCode: /lib/libFile.dll; osname=win32; processor=x86_64 """ The META-INF/MANIFEST.MF of the Linux fragment contains (besides others) the following headers: """ Eclipse-PlatformFilter: (& (osgi.os=linux) (osgi.ws=gtk) (osgi.arch=x86_64)) Bundle-NativeCode: /lib/libFile.so; osname=linux; processor=x86_64 """ When the attached example is excuted with the default command "mvn clean verify", it fails with the following errors while resolving all projects (I've run it on a Windows computer): [INFO] Resolving class path of MavenProject: de.foo:de.foo.bar.linux:1.0.0-SNAPSHOT @ C:\Users\Hannes\Desktop\minimalExample\de.foo.bar.linux\.polyglot.META-INF [ERROR] Internal error: java.lang.RuntimeException: org.osgi.framework.BundleException: Bundle de.foo.bar.linux cannot be resolved:de.foo.bar.linux [2] [ERROR] Unresolved requirement: Require-Capability: osgi.native; native.paths:List<String>="/lib/dummyLib.so"; filter:="(&(osgi.native.osname~=linux)(osgi.native.processor~=x86_64))" [ERROR] -> [Help 1] org.apache.maven.InternalErrorException: Internal error: java.lang.RuntimeException: org.osgi.framework.BundleException: Bundle de.foo.bar.linux cannot be resolved:de.foo.bar.linux [2] Unresolved requirement: Require-Capability: osgi.native; native.paths:List<String>="/lib/dummyLib.so"; filter:="(&(osgi.native.osname~=linux)(osgi.native.processor~=x86_64))" at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:120) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289) at org.apache.maven.cli.MavenCli.main (MavenCli.java:193) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:566) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347) Caused by: java.lang.RuntimeException: org.osgi.framework.BundleException: Bundle de.foo.bar.linux cannot be resolved:de.foo.bar.linux [2] Unresolved requirement: Require-Capability: osgi.native; native.paths:List<String>="/lib/dummyLib.so"; filter:="(&(osgi.native.osname~=linux)(osgi.native.processor~=x86_64))" at org.eclipse.tycho.core.osgitools.OsgiBundleProject.getResolverState (OsgiBundleProject.java:282) at org.eclipse.tycho.core.osgitools.OsgiBundleProject.resolveClassPath (OsgiBundleProject.java:175) at org.eclipse.tycho.core.resolver.DefaultTychoResolver.resolveProject (DefaultTychoResolver.java:142) at org.eclipse.tycho.core.maven.TychoMavenLifecycleParticipant.lambda$resolveProjects$0 (TychoMavenLifecycleParticipant.java:127) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept (ForEachOps.java:183) at java.util.stream.WhileOps$1$1.accept (WhileOps.java:99) at java.util.ArrayList$ArrayListSpliterator.tryAdvance (ArrayList.java:1632) at java.util.stream.ReferencePipeline.forEachWithCancel (ReferencePipeline.java:127) at java.util.stream.AbstractPipeline.copyIntoWithCancel (AbstractPipeline.java:502) at java.util.stream.AbstractPipeline.copyInto (AbstractPipeline.java:488) at java.util.stream.AbstractPipeline.wrapAndCopyInto (AbstractPipeline.java:474) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential (ForEachOps.java:150) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential (ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate (AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.forEach (ReferencePipeline.java:497) at org.eclipse.tycho.core.maven.TychoMavenLifecycleParticipant.resolveProjects (TychoMavenLifecycleParticipant.java:158) at org.eclipse.tycho.core.maven.TychoMavenLifecycleParticipant.afterProjectsRead (TychoMavenLifecycleParticipant.java:102) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:264) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289) at org.apache.maven.cli.MavenCli.main (MavenCli.java:193) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:566) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347) Caused by: org.osgi.framework.BundleException: Bundle de.foo.bar.linux cannot be resolved:de.foo.bar.linux [2] Unresolved requirement: Require-Capability: osgi.native; native.paths:List<String>="/lib/dummyLib.so"; filter:="(&(osgi.native.osname~=linux)(osgi.native.processor~=x86_64))" at org.eclipse.tycho.core.osgitools.EquinoxResolver.assertResolved (EquinoxResolver.java:384) at org.eclipse.tycho.core.osgitools.EquinoxResolver.newResolvedState (EquinoxResolver.java:127) at org.eclipse.tycho.core.osgitools.OsgiBundleProject.getResolverState (OsgiBundleProject.java:279) at org.eclipse.tycho.core.osgitools.OsgiBundleProject.resolveClassPath (OsgiBundleProject.java:175) at org.eclipse.tycho.core.resolver.DefaultTychoResolver.resolveProject (DefaultTychoResolver.java:142) at org.eclipse.tycho.core.maven.TychoMavenLifecycleParticipant.lambda$resolveProjects$0 (TychoMavenLifecycleParticipant.java:127) at java.util.stream.ForEachOps$ForEachOp$OfRef.accept (ForEachOps.java:183) at java.util.stream.WhileOps$1$1.accept (WhileOps.java:99) at java.util.ArrayList$ArrayListSpliterator.tryAdvance (ArrayList.java:1632) at java.util.stream.ReferencePipeline.forEachWithCancel (ReferencePipeline.java:127) at java.util.stream.AbstractPipeline.copyIntoWithCancel (AbstractPipeline.java:502) at java.util.stream.AbstractPipeline.copyInto (AbstractPipeline.java:488) at java.util.stream.AbstractPipeline.wrapAndCopyInto (AbstractPipeline.java:474) at java.util.stream.ForEachOps$ForEachOp.evaluateSequential (ForEachOps.java:150) at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential (ForEachOps.java:173) at java.util.stream.AbstractPipeline.evaluate (AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.forEach (ReferencePipeline.java:497) at org.eclipse.tycho.core.maven.TychoMavenLifecycleParticipant.resolveProjects (TychoMavenLifecycleParticipant.java:158) at org.eclipse.tycho.core.maven.TychoMavenLifecycleParticipant.afterProjectsRead (TychoMavenLifecycleParticipant.java:102) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:264) at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192) at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105) at org.apache.maven.cli.MavenCli.execute (MavenCli.java:957) at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:289) at org.apache.maven.cli.MavenCli.main (MavenCli.java:193) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at jdk.internal.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:566) at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:282) at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:225) at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:406) at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:347) When Tycho 2.2.0 is used (the value of the variable "tycho.version" is adjusted accordingly) everything works fine. Also if I remove the osname and processor attributes in both Manifests everything works fine too. What's interesting is, that it fails even after I removed the osname attributes in both fragments MANIFESTs or if I remove the osname attributes in the fragment of the not running platform. So it always fails with an equivalent error message, even tough the current platform matches the environment requirements. So contrary to my Mailing-List answer (https://www.eclipse.org/lists/tycho-user/msg08995.html), I assume that in this phase the capabilities required by the Bundle-NativeCode Header are not provided correctly at all?
Created attachment 285884 [details] Single plugin with native-libs I tested a second scenario with a single plugin that has a Bundle-NativeCode Header with two clauses one for Linux and one for Windows in its MANIFEST. And I observed the same problem. I attached a second example that demonstrates this scenario, too.
For the sake of completeness (again) the reference to the part of the OSGi spec that might be interesting for this issue: https://docs.osgi.org/specification/osgi.core/8.0.0/framework.module.html#framework.module-loading.native.code.libraries I'm not familiar with the code of the resolver, but maybe a solution to make it work without removing the environment-attributes could be to append the optional-clause virtually when resolving the bundles at build-time. The specification says about the optional "*" in conjunction with using the Wiring-API: "If the optional '*' is specified at the end of the Bundle-NativeCode manifest header, then the native code for the bundle is considered to be optional. When the Framework converts a Bundle-NativeCode header into an osgi.native requirement which is designated as optional then the requirement resolution directive must be set to optional" Maybe this could be part of the solution.
Mickael should know more about the new resolver used. In general I think it is correct that tycho does not handle this as optional, that's what the environment configuration is for and it obviously should error if the bundle does not match but is requested to be installed.
org.eclipse.tycho/tycho-core/src/main/java/org/eclipse/tycho/core/osgitools/EquinoxResolver.java line 300 is probably where the magic to add the capability is missing. @Hannes: are you willing to give a try to a patch? I think there are some tests for Eclipse-PlatformFilter in the tycho-core module you may be able to copy and adapt for the case of Bundle-NativeCode; then from a failed tests it shouldn't be too hard to add the right capabilities in EquinoxResolver. If you're not available to play with this piece of code, feel free to say so; I can spend some time on it.
Yes I can try to fix this issue. Thanks the hints, they are a good starting point. If I don't succeed after the weekend I'll let you in order to not delay the release too much. The plan is to release wednesday, isn't it?
(In reply to Hannes Wellmann from comment #5) > If I don't succeed after the weekend I'll let you in order to not delay the > release too much. The plan is to release wednesday, isn't it? That plan is flexible, we'll see ;)
New Gerrit change created: https://git.eclipse.org/r/c/tycho/org.eclipse.tycho/+/178257
After some debugging it turned out, that the required capabilities "osgi.native.osname" and "osgi.native.processor" are actually there. The actual problem in the attached examples was that an alias is used for the required 'processor' attribute. The Bundle-NativeCode header in the Manifests require 'processor=x86_64', while the system-bundle provides only osgi.native.processor=x86-64 (not underscore vs. hyphen). Even not recommended "x86_64" is a valid alias for the value "x86-64" of the "osgi.native.processor" capability, according to this overview (or the OSGI-spec): https://docs.osgi.org/reference/osnames.html After some search I found that all valid aliases for osname and processor are provided by the org.eclipse.osgi.internal.framework.AliasMapper. After some more search and debugging I found that, in a 'real' Equinox OSGi runtime the aliasMapper is used by org.eclipse.osgi.storage.Storage#getSystemExtraCapabilities() in order to setup all capabilities including their aliases. So in order to fix this issue I replaced in EquinoxResolver the StringBuilder to build the platformCapability by an adapted version of Storage#getSystemExtraCapabilities(). The only thing that changed in the end is, that the aliases for osname and processor are added. Unfortunately I could not yet found a proper place for the tests yet. My hottest candidates are "EquinoxResolverTest" or "TychoTest". Is on of those a suitable place? The question is also how should it be tested? Based on minimal Manifests that should resolve or should I simply check that the aliases are also provided? In the first case I assume "TychoTest" is the way to go, in the latter I think it's "EquinoxResolverTest"? Additionally I noticed the capabilities 'osversion', 'language' are not supported at all at the moment. But this would probably require to extend the TargetEnvironment, wouldn't it? I don't need that but, maybe it is a valuable extension for others?
Great work Hannes! Yes, having a test in EquinoxResolverTest with a minimal pom file that reproduce the initial issue could be sufficient to prevent from further regressions on that aspect. If you prefer, you can add a case to tycho-its for that, but it's usually a bit more complex to setup and execution is much slower; however it does test Tycho in a workflow that's much closer for real usage. If you're unable to write the test soon, we can merge the patch as it, assuming it does work for you and all existing tests remain green. Then you can either write the test later (even after 2.3 release) or not write the test at all (but regression would then be more probable). Your call.
Thank you Mickael. Alright then I will create test(s) in EquinoxResolverTest. I think I will be able to write the tests within the next one or two days (I'll try my best but cannot promise it). So if this is ok for the 2.3-release I'd prefer to postpone the merge of this patch until the tests are completed. I tested my patch with my companies build already and it worked well without any errors. As a site-note: I noticed that the build at my machine takes longer with the 2.3-SNAPSHOT (8min 50sec) than with Tycho 2.2 (6min 26sec). I haven't investigated it further, but could it be possible that the resolution takes longer now (the times of the single modules didn't change significantly)? But maybe it's just because of the local SNAPSHOT-build.
(In reply to Hannes Wellmann from comment #10) > I haven't investigated it > further, but could it be possible that the resolution takes longer now (the > times of the single modules didn't change significantly)? But maybe it's > just because of the local SNAPSHOT-build. I'm not sure. It could be, but IMO the OSGi resolution itself is not significantly longer to explain such a difference. Where do you see those extra 84 seconds spend? During resolution before it prints the build order, or later during some specific mojo execution? What's interesting usually is to get the relative time of each log line compared to process start so a simple text comparison can highlight were time was spent.
Gerrit change https://git.eclipse.org/r/c/tycho/org.eclipse.tycho/+/178257 was merged to [master]. Commit: http://git.eclipse.org/c/tycho/org.eclipse.tycho.git/commit/?id=553e93ae0bf14dda4d40475f605f57e8675a9987
Thanks Hannes for the report, fix, test! Great contribution on a difficult and critical piece of code!
