Bugzilla – Full Text Bug Listing
| Summary: | [maven] Aggregator's Maven result produces incompatible version numbers/ranges | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Technology] CBI | Reporter: | Karl M. Davis <karl> | ||||||||||||||||||
| Component: | CBI p2 Repository Aggregator | Assignee: | Project Inbox <b3.aggregator-inbox> | ||||||||||||||||||
| Status: | RESOLVED FIXED | QA Contact: | |||||||||||||||||||
| Severity: | major | ||||||||||||||||||||
| Priority: | P3 | CC: | email, shahim, stephan.herrmann, thomas | ||||||||||||||||||
| Version: | unspecified | ||||||||||||||||||||
| Target Milestone: | --- | ||||||||||||||||||||
| Hardware: | PC | ||||||||||||||||||||
| OS: | Linux | ||||||||||||||||||||
| Whiteboard: | |||||||||||||||||||||
| Attachments: |
|
||||||||||||||||||||
|
Description
Karl M. Davis
The same issue just killed our build (which was working until a variant with dash-separated version was uploaded by s.o. to maven central, see also http://www.eclipse.org/forums/index.php/t/585108 ) To get our build working again I had to hack b3 as to produce legal maven versions. So, it works for us, but it isn't yet polished to be robust for general use. Would b3 be interested in that patch? BTW, if a repo should indeed be legal for both p2 and maven, it seems we need to advertise the same artefact by different file names, for two reasons: - different version (dot vs. dash, this issue) - different short artefact names (see bug 419404) Would it make sense to let b3 create hardlinks to combine compatibility with space efficiency? (In reply to Stephan Herrmann from comment #1) > The same issue just killed our build (which was working until a variant with > dash-separated version was uploaded by s.o. to maven central, see also > http://www.eclipse.org/forums/index.php/t/585108 ) Correct link is http://www.eclipse.org/forums/index.php/t/585108/ Created attachment 237334 [details]
preparation 1
This patch contains no substantial change, I just let the save action apply the projects preferred formatting (separate patch to noise in subsequent patches).
Created attachment 237335 [details]
minimal patch to mavenize versions
This patch just hooks a basic translation of versions into the aggregator, more still to come ...
Created attachment 237336 [details] improve feedback of command line invocations Corrected version of bug 316401 comment 5: trivial change to make errors visible at the end of a command line run. Created attachment 237337 [details]
preparation 2
In preparation of a small model change I simply regenerated model and edit code and watched endless white space changes. Separate patch to reduce noise in the real patches.
Created attachment 237338 [details]
add one attribute to the model
Here I added an attribute "strictMavenVersions" to the model, to make the new behavior configurable (off by default).
Created attachment 237339 [details]
connect new behavior to the new model attribute
This patch connects the previous patches: use strict maven versions only when the attribute "strictMavenVersions" is set to true.
Created attachment 237340 [details] support two filenames per artefact This is the most involved patch to ensure that each artefact can be accessed using both its osgi name and its mavenized name (see comment 2). For this purpose several methods need to know if we are interested in an osgi artefact or a maven artefact, hence several signatures had to be extended with a flag. For the Maven view the artefact name on disk is corrected (was: the fully qualified name). During mirroring it is not trivial to find the InstallableUnitMapping, solved by - MirrorGenerator.run() local map keyedIUs IArtifactKey -> IInstallableUnit - InstallableUnitMapping.findUnit(IInstallableUnit) Also, during mirroring we need the result File in order to create a link using its second name. - From a CanonicalizeRequest we retrieve the File from a new field initialized in perform() - Inner calls to MirrorGenerator.mirror(..) only return an IArtifactDescriptor, from which I reconstruct the File using the internal class SimpleArtifactRepository (p2) and its method createLocation(), not sure if there's a better approach. Creating the actual hardlink would be so nice using java.nio.file (see the commented version of MirrorGenerator.createLink()), since we don't have it at Java 6 I made a barebone implementation that works only on *nix. Finally, I noticed that this implementation accidentally creates a *.jar link to extension-less files (from binary/ part of a repo). This doesn't directly harm but should still be avoided (should be fixed in createMavenJarLink()). The patches contributed in comment 4 - comment 10 comply with http://www.eclipse.org/legal/CoO.php Created attachment 237341 [details]
test aggregation
For a minimal test here's a b3aggr file that demonstrates:
org.eclipse.emf.ecore, e.g., will be accessible inside org/eclipse/emf.ecore/2.9.1-v20130827-0309/ as
p2:
org.eclipse.emf.ecore-2.9.1.v20130827-0309.jar
maven:
emf.ecore-2.9.1-v20130827-0309.jar
Note the differences in name (due to a mavenMapping) and version (due to "strictMavenVersions")
All maven-metadata.xml and *.pom use the mavenized versions (using '-').
BTW: the aggregation from comment 12 needs the fix in bug 421396 :) Stephan, can you please write a short summary of what all changes amounts to. Also, it's unclear to me if I must apply all patches or if some patches includes others. (In reply to Thomas Hallgren from comment #14) > Stephan, can you please write a short summary of what all changes amounts > to. Also, it's unclear to me if I must apply all patches or if some patches > includes others. The *real* short version would have been comment 12, so explaining a bit: Versions are mapped OSGi->Maven by replacing the third dot (if any) with a dash -> fixes the problem of versions which maven cannot parse e.g.: 10.0.1.v201203051515 -> 10.0.1-v201203051515 New format used throughout: pom, maven-metadata, filenames ... Maven file names use the artifactId instead of the fully qualified osgi name -> fixes wrong file names when applying maven mappings that create the artifactId from just a portion of the FQN e.g.: when mapping com.google.guava to G=com.google A=guava the file name still used to be com.google.guava_10.0.1.v201203051515.jar is now: guava_10.0.1-v201203051515.jar Directory structure is not affected by this change. To share artifact bits (and thus prevent the repo from doubling in disk space) I try to create hardlinks, which is implemented only for *unix, on windows this fails with an error. e.g.: these two are the same file (maven/p2): -rw-r--r-- 2 sherrmann www 1593087 Nov 7 13:33 guava-10.0.1-v201203051515.jar -rw-r--r-- 2 sherrmann www 1593087 Nov 7 13:33 com.google.guava-10.0.1.v201203051515.jar All this is controlled by a new attribute Aggregation.strictMavenVersions, edit code has been regenerated to support this flag in the editor, too. --- And: I intentionally uploaded incremental patches, which all have to be applied in order, partly because the configured formatter profile caused large amounts of white space changes, which would have obscured the (comparatively small) payload. I created a new branch 'maven-versions' for continued work on this where I managed to get rid of most of the white-space noise for the generated EMF code. It's done by right-clicking the project and then select 'Source' -> 'Clean-up...' Your patches (including the one in bug 421396) went in with this commit: http://git.eclipse.org/c/b3/b3.git/commit/?id=fbc035bb28ee701041e1dee7f53223ce9af111c9 There is no need for links. The aggregator creates a special artifacts.xml that a rule for each artifact jar file. p2 will never use these names beyond for just opening the actual file. All meta-data used by p2 stems from the metadata.xml/jar file so the name of the file has no impact. The p2 API never exposes it. The rules looks like this: <rule filter="(& (classifier=osgi.bundle)(id=org.eclipse.core.jobs)(version=3.5.300.v20130429-1813))" output="${repoUrl}/org/eclipse/core/org.eclipse.core.jobs/3.5.300.v20130429-1813/org.eclipse.core.jobs-3.5.300.v20130429-1813.jar"/> and they are added by MavenManager.addMappingRule(). *** Bug 421396 has been marked as a duplicate of this bug. *** (In reply to Thomas Hallgren from comment #16) > I created a new branch 'maven-versions' for continued work on this where I > managed to get rid of most of the white-space noise for the generated EMF > code. It's done by right-clicking the project and then select 'Source' -> > 'Clean-up...' > > Your patches (including the one in bug 421396) went in with this commit: > http://git.eclipse.org/c/b3/b3.git/commit/ > ?id=fbc035bb28ee701041e1dee7f53223ce9af111c9 Cool, thanks. Let me know if anything in the code is unclear or bogus. > There is no need for links. The aggregator creates a special artifacts.xml > that a rule for each artifact jar file. p2 will never use these names beyond > for just opening the actual file. All meta-data used by p2 stems from the > metadata.xml/jar file so the name of the file has no impact. The p2 API > never exposes it. > > The rules looks like this: > > <rule filter="(& > (classifier=osgi.bundle)(id=org.eclipse.core.jobs)(version=3.5.300.v20130429- > 1813))" > output="${repoUrl}/org/eclipse/core/org.eclipse.core.jobs/3.5.300.v20130429- > 1813/org.eclipse.core.jobs-3.5.300.v20130429-1813.jar"/> > > and they are added by MavenManager.addMappingRule(). This sounds very good. So, p2 could actually use a file guava-10.0.1-v201203051515.jar for installing com.google.guava into equinox? In this case we might want to revert my last patch (attachment 237340 [details]) and replace it with a variant that - uses only maven filenames (if strictMavenVersions is set?) - creates the suitable mapping rules to connect the osgi view to these files ? (In reply to Stephan Herrmann from comment #18) > This sounds very good. So, p2 could actually use a file > guava-10.0.1-v201203051515.jar for installing com.google.guava into equinox? > Yes. > In this case we might want to revert my last patch (attachment 237340 [details] > [details]) > and replace it with a variant that > - uses only maven filenames (if strictMavenVersions is set?) > - creates the suitable mapping rules to connect the osgi view to these files > ? Yes. But since that all patches went in as one commit it's now hard to revert the last one. So what's easier for you? Do you want me to replace the current commit with a series of commits where the last one is excluded or can you revert what needs to be reverted manually? Heh, just learned that individual patches can be reverted by just checking the 'revert' box in the apply patch dialog :-) I pushed a new commit where the last patch has been reverted. Sorry, Java8 work in JDT/Core had higher priority over this one. I'll try to find the time to polish my patch in the new year. What are the deadlines for inclusion in the next version of b3? (In reply to Stephan Herrmann from comment #21) > What are the deadlines for inclusion in the next version of b3? There are no current deadlines. We publish when we feel that there's something significant that our users will benefit from. I consider this patch to be such a change. AFAICT, there's no additional changes needed in order for this to work. The MavenManager already takes the new strictMavenVersion into consideration when creating the artifact. I'm merging this into master now and marking this as resolved. Thanks a lot for the contribution. (sorry for the silence, as mentioned: Java 8 eats all my time ...) (In reply to Thomas Hallgren from comment #23) > AFAICT, there's no additional changes needed in order for this to work. The > MavenManager already takes the new strictMavenVersion into consideration > when creating the artifact. I'm merging this into master now and marking > this as resolved. Sounds great! Thanks. When / how will this be made available? It's now available from our regular update sites: http://download.eclipse.org/modeling/emft/b3/headless-4.3 http://download.eclipse.org/modeling/emft/b3/updates-4.3 The builds were based on Kepler-SR2 [Bookkeeping change only. Moving bugs to the new "home" of aggregator, CBI. No change to assignee for resolved and verified bugs.] Hi, I am trying to use this feature to create a maven repo from a p2 repo. I have set Strict Maven Versions to true and I can see that for example for org.eclipse.core.runtime the version string generated is 3.9.100-v20131218-1515. The directory beneath the artifact ID contains this version number, also the pom file. Unfortunately, the maven-metadata.xml looks like this: <?xml version="1.0" encoding="ASCII"?> <metadata xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://maven.apache.org/METADATA/1.0.0"> <groupId>eip-target</groupId> <artifactId>org.eclipse.core.runtime</artifactId> <version>1</version> <versioning> <release>3.9.100.v20131218-1515</release> <versions> <version>3.9.100.v20131218-1515</version> </versions> <lastUpdated>20161129220510</lastUpdated> </versioning> </metadata> It is still using the . (dot) instead of the - (dash) to separate the qualifier. I am using version 0.3.0.v20140928-0617 of the b3 aggregator from within Neon. Is there anything that I could have configured wrong? When I manually fix the maven-metadata.xml to use the dash-style version dependency resolution in my build works. (In reply to Thomas Hofmann from comment #27) > Hi, I am trying to use this feature to create a maven repo from a p2 repo. > I have set Strict Maven Versions to true and I can see that for example for > org.eclipse.core.runtime the version string generated is > 3.9.100-v20131218-1515. > The directory beneath the artifact ID contains this version number, also the > pom file. > Unfortunately, the maven-metadata.xml looks like this: > > <?xml version="1.0" encoding="ASCII"?> > <metadata xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" > xsi:noNamespaceSchemaLocation="http://maven.apache.org/METADATA/1.0.0"> > <groupId>eip-target</groupId> > <artifactId>org.eclipse.core.runtime</artifactId> > <version>1</version> > <versioning> > <release>3.9.100.v20131218-1515</release> > <versions> > <version>3.9.100.v20131218-1515</version> > </versions> > <lastUpdated>20161129220510</lastUpdated> > </versioning> > </metadata> > > It is still using the . (dot) instead of the - (dash) to separate the > qualifier. I am using version 0.3.0.v20140928-0617 of the b3 aggregator from > within Neon. > > Is there anything that I could have configured wrong? > > When I manually fix the maven-metadata.xml to use the dash-style version > dependency resolution in my build works. I see the same and filed bug 508412 |