Bugzilla – Full Text Bug Listing
| Summary: | Update all parent versions to match our build stream | ||
|---|---|---|---|
| Product: | [Eclipse Project] Platform | Reporter: | Paul Webster <pwebster> |
| Component: | Releng | Assignee: | Platform-Releng-Inbox <platform-releng-inbox> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | P3 | CC: | daniel_megert, david_williams, john.arthorne, markus.kell.r, Michael_Rennie, mistria, Silenio_Quarti, sptaszkiewicz, t-oberlies, thanh.ha, tparker |
| Version: | 4.3 | ||
| Target Milestone: | 4.3 M7 | ||
| Hardware: | PC | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Bug Depends on: | 403958, 403959, 403960, 403961, 403962, 403963, 403964, 403965, 403966, 403967, 403968, 403969, 403970, 403971, 404908, 404909, 404910 | ||
| Bug Blocks: | 404148 | ||
|
Description
Paul Webster
I'd provide patches to each component in this case (lots of mechanical work). example script: http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/tree/scripts/xsl/xparentVersions.sh PW I think we should do this next week in M7. re: 4.2.2+ We can do the same thing in 4.2.2+ (with 4.2.2-SNAPSHOT) but in that case I'd provide the 4.2.2+ patches before we do the 4.3 patches. PW Maybe I don't understand the benefits of doing this, but sounds like a lot of churn for our builds, which have already been through a lot of churn, and a lot of other high priority items left to do. My main concern is adding to future complications as well ... such as you say, "3) our CBI build timestamp provider is set to ignore pom.xml changes when calculating the timestamp" but there are pom changes that can effect the end-result, so that seems like its adding fragility to the system? It seems that "publishing the build-related poms (all those with packaging type of pom) to repo.eclipse.org as snapshots." is an extra requirement, which is coming in pretty late. Can't those other builds use our p2 repos? What am I missing? (In reply to comment #3) > My main concern is adding to future complications as well ... such as you > say, "3) our CBI build timestamp provider is set to ignore pom.xml changes > when calculating the timestamp" but there are pom changes that can effect > the end-result, so that seems like its adding fragility to the system? There were always builder related changes that are few and far between that have caused us to manually up the qualifier ... that's what the forceQualifierUpdate.txt file can be used for (make it explicit). > > It seems that "publishing the build-related poms (all those with packaging > type of pom) to repo.eclipse.org as snapshots." is an extra requirement, > which is coming in pretty late. Can't those other builds use our p2 repos? Actually, the requirement that we (eclipse) start publishing our bundles to maven is an old requirement that we've ignore for a long, long time. The way we've got things configured now we can't do it using maven's own snapshot-deploy process, and we won't be able to do it in the future either. These are parent poms, so they're not visible or used in p2 at all. The way our parent poms are right now is a side-effect of them simply being kept from the proto-type, not really because that's a good idea. If we can follow a standard practice (they mirror the stream we're building) and prevent that from forcing qualifier changes and then not interfere with how everybody else in the world uses maven, I think it's worth the effort. PW Proposal: 4.2.2+ Provide patches for 4.2.2+ that set all of the pom-package module versions to 4.2.2-SNAPSHOT Add the maven https://repo.eclipse.org/content/groups/eclipse to each git repo root pom file 3.8.2+ Set the eclipse-platform-parent version to 4.2.2-SNAPSHOT. I've confirmed the difference in the parent is one extra repo and a different comparator baseline. Both are properties, and can be set on a 3.8.2+ build so it's OK to use the 4.2.2+ eclipse-platform-parent 4.3 Provide the patches for master that set all of the pom-package module versions to 4.3.0-SNAPSHOT Add the maven https://repo.eclipse.org/content/groups/eclipse to each git repo root pom file Patches should be applied starting after 20:00 EDT tonight but before 20:00 EDT Thursday night. PW One minor comment, it is good practice for Nexus to use the repositories URL instead of a groups URL [1]. So instead of using:
https://repo.eclipse.org/content/groups/eclipse
use:
https://repo.eclipse.org/content/repositories/eclipse
Unfortunately for groups Nexus advertises the groups URL.
[1] http://dev.eclipse.org/mhonarc/lists/tycho-user/msg04054.html
Some ideas when comparing it to JBoss Tools. Background: JBoss Tools is basically building multiple interlinked components independently (for faster CI feedback and better team productivity). Output of those builsa are p2 repositories. Then we aggregated them into our repository, which is the JBoss Tools distribution * Publishing parent pom to Nexus is definitely a good idea if you want to split your build into several sub-builds, and aggregate them. We've done this a few months ago and have noticed clear improvements on build process and local builds, since it's easier to consume parent from Nexus than it is to rebuild it. In case you're always building all-at-once and don't plan to allow more modularity, I don't see any benefit in publishing parent pom. * If you go for versioning parent pom, take care that in order to be able to reproduce builds, you'll need to be able to always find the matching revision of parent pom that was used in order to build your code. In our case, we only focus on reproducible milestones and releases. To do so, we simply only allow using RELEASES (and no-SNAPSHOTs) of parent pom in the revision of code that will be released. This parent pom release is tagged on SCM so you can associate a release with a revision for the parent pom in order to reproduce build. SNAPSHOT does not give this information. * We figured out that a parent pom can have a longer life (without being changed) than the components that use it, so we are moving towards putting the project version/build stream (JBT 4.1) out of the parent pom to allow 1 parent <-> multiple stream mapping. This will allow us to use the same version of parent pom for multiple streams if we want. Each component is responsible of choosing a parent pom that fits his need, and is free to do it as long as it's able to provide a p2 site as we want it. The project version/build stream is stored in the highest-level component (the repository we ship to end-users) Thanks for the comments Mickael. (In reply to comment #7) > > * Publishing parent pom to Nexus is definitely a good idea if you want to > split your build into several sub-builds, and aggregate them. We've done > this a few months ago and have noticed clear improvements on build process > and local builds, since it's easier to consume parent from Nexus than it is > to rebuild it. > In case you're always building all-at-once and don't plan to allow more > modularity, I don't see any benefit in publishing parent pom. This is the primary usecase we want. Our build is still the full aggregator build, but there are many consumers that want to build out of one of the component repos and they have to clone the aggregator just to install the eclipse-platform-parent. > * If you go for versioning parent pom, take care that in order to be able to > reproduce builds, you'll need to be able to always find the matching > revision of parent pom that was used in order to build your code. In our > case, we only focus on reproducible milestones and releases. To do so, we > simply only allow using RELEASES (and no-SNAPSHOTs) of parent pom in the > revision of code that will be released. Our builds will be monolithic for a while, and the parent pom used in our builds is tagged and versioned along with the rest of the repositories. If we looked at breaking things up to allow components to build and contribute on their own, we would definitely have to make sure that they all used the same parent pom when contributing to one build. > * We figured out that a parent pom can have a longer life (without being > changed) than the components that use it, so we are moving towards putting > the project version/build stream (JBT 4.1) out of the parent pom to allow 1 > parent <-> multiple stream mapping. This will allow us to use the same > version of parent pom for multiple streams if we want. Each component is > responsible of choosing a parent pom that fits his need, and is free to do > it as long as it's able to provide a p2 site as we want it. The project > version/build stream is stored in the highest-level component (the > repository we ship to end-users) This might be something to consider long term (after Luna) when we find out what the main pain points will be using the CBI build. PW We've decided to delay this until immediately after next week's I build to give anybody that wishes a chance to comment. PW (In reply to comment #9) > We've decided to delay this until immediately after next week's I build to > give anybody that wishes a chance to comment. > > PW I do not understand why ALL bundles need to get a 'forceQualifierUpdate.txt'. If really all need it, then this is definitely something that belongs into Tycho (or whoever specified the semantics of that file) and not into every single bundle in the SDK. I can understand that this is needed for e.g. features, which want to force a qualifier update, because their content might have changed without any changes in the feature bundle itself. For now I suggest to wait with that part of the changes. (In reply to comment #10) > I do not understand why ALL bundles need to get a > 'forceQualifierUpdate.txt'. If really all need it, then this is definitely > something that belongs into Tycho (or whoever specified the semantics of > that file) and not into every single bundle in the SDK. I can understand > that this is needed for e.g. features, which want to force a qualifier > update, because their content might have changed without any changes in the > feature bundle itself. > > For now I suggest to wait with that part of the changes. They're necessary for all the bundles. Otherwise the qualifiers on bundles that aren't under active development will revert to much earlier versions (as the only changes we've been making would be pom changes, and they don't count towards the qualifier calculation). If a component doesn't want them, they can skip them and it's up to that component to make sure their qualifiers don't regress (which will involve adding them back in those cases :-) PW (In reply to comment #11) > (In reply to comment #10) > > I do not understand why ALL bundles need to get a > > 'forceQualifierUpdate.txt'. If really all need it, then this is definitely > > something that belongs into Tycho (or whoever specified the semantics of > > that file) and not into every single bundle in the SDK. I can understand > > that this is needed for e.g. features, which want to force a qualifier > > update, because their content might have changed without any changes in the > > feature bundle itself. > > > > For now I suggest to wait with that part of the changes. > > They're necessary for all the bundles. Otherwise the qualifiers on bundles > that aren't under active development will revert to much earlier versions > (as the only changes we've been making would be pom changes, and they don't > count towards the qualifier calculation). > > If a component doesn't want them, they can skip them and it's up to that > component to make sure their qualifiers don't regress (which will involve > adding them back in those cases :-) > > PW Is this file just to ensure a one-time touch? Or does it have a special meaning to always force? If the latter, this would be wrong, since we do not want qualifier updates on an I-build without changes in a bundle If the former, we should try to just do one build where we set a global 'force', so that all qualifiers get a one-time update. > > They're necessary for all the bundles. Otherwise the qualifiers on bundles
> > that aren't under active development will revert to much earlier versions
> > (as the only changes we've been making would be pom changes, and they don't
> > count towards the qualifier calculation).
Why would the qualifier decrease?
(In reply to comment #12) > > Is this file just to ensure a one-time touch? Or does it have a special > meaning to always force? If the latter, this would be wrong, since we do not > want qualifier updates on an I-build without changes in a bundle If the > former, we should try to just do one build where we set a global 'force', so > that all qualifiers get a one-time update. It has no special meaning outside of the fact that it's a text file that can be touched and included in a commit. It won't effect anything on an ongoing basis unless it's touched and committed. (In reply to comment #13) > > Why would the qualifier decrease? A while back we noticed some really old qualifiers because we were excluding the pom.xml from qualifier calculations and those were the only files that changed in some projects. So we stopped excluding them. But for us to match our build pom versions to the streams to follow the maven convention that means after every release we end up touching every pom, so we turn back on "ignore poms" so when we start Luna those pom changes won't up the qualifiers of bundles that haven't changed. But ... then we're back to the original problem, where some bundles that only contain pom changes will have their qualifier drop backwards in time. PW (In reply to comment #14) > (In reply to comment #12) > > > > Is this file just to ensure a one-time touch? Or does it have a special > > meaning to always force? If the latter, this would be wrong, since we do not > > want qualifier updates on an I-build without changes in a bundle If the > > former, we should try to just do one build where we set a global 'force', so > > that all qualifiers get a one-time update. > > It has no special meaning outside of the fact that it's a text file that can > be touched and included in a commit. It won't effect anything on an ongoing > basis unless it's touched and committed. OK, got that. Good. > (In reply to comment #13) > > > > Why would the qualifier decrease? > > A while back we noticed some really old qualifiers because we were excluding > the pom.xml from qualifier calculations and those were the only files that > changed in some projects. So we stopped excluding them. Why? The pom file is not part of the built bundle, so not updating the qualifier shouldn't be an issue, except that it "looks" old. > But for us to match our build pom versions to the streams to follow the > maven convention that means after every release we end up touching every > pom, so we turn back on "ignore poms" so when we start Luna those pom > changes won't up the qualifiers of bundles that haven't changed. But ... > then we're back to the original problem, where some bundles that only > contain pom changes will have their qualifier drop backwards in time. I still do not understand how the qualifier would ever decrease. And once we have restored the version compare test, we should be fine anyway (see bug 391499), i.e. we can touch those bundles that actually deserve it. (In reply to comment #15) > Why? The pom file is not part of the built bundle, so not updating the > qualifier shouldn't be an issue, except that it "looks" old. It's more of a one-time problem than an ongoing one. > I still do not understand how the qualifier would ever decrease. It's a one time thing, because of the fix we put in for Bug 400519. This requirement undoes that one. Once we've stabilized the poms (presumably for Kepler) it wouldn't be an issue going forward, but it would make M7 contain bundles with qualifiers less then M6. p2 is fine with that (for a specific build), but it can cause problems in certain slicing operations. PW (In reply to comment #17) > (In reply to comment #15) > > Why? The pom file is not part of the built bundle, so not updating the > > qualifier shouldn't be an issue, except that it "looks" old. > > It's more of a one-time problem than an ongoing one. > > > I still do not understand how the qualifier would ever decrease. > > It's a one time thing, because of the fix we put in for Bug 400519. This > requirement undoes that one. Once we've stabilized the poms (presumably for > Kepler) it wouldn't be an issue going forward, but it would make M7 contain > bundles with qualifiers less then M6. p2 is fine with that (for a specific > build), but it can cause problems in certain slicing operations. > > PW Sorry, I'm confused. For normal bundles we do use a fixed timestamp. No calculation happens here. So, how can this decrease? (In reply to comment #18) > (In reply to comment #17) > > (In reply to comment #15) > > > Why? The pom file is not part of the built bundle, so not updating the > > > qualifier shouldn't be an issue, except that it "looks" old. > > > > It's more of a one-time problem than an ongoing one. > > > > > I still do not understand how the qualifier would ever decrease. > > > > It's a one time thing, because of the fix we put in for Bug 400519. This > > requirement undoes that one. Once we've stabilized the poms (presumably for > > Kepler) it wouldn't be an issue going forward, but it would make M7 contain > > bundles with qualifiers less then M6. p2 is fine with that (for a specific > > build), but it can cause problems in certain slicing operations. > > > > PW > > > Sorry, I'm confused. For normal bundles we do use a fixed timestamp. No > calculation happens here. So, how can this decrease? To be specific: a "normal" bundle built (if built), will always have a newer qualifier, so how can it decrease? All of our CBI builds calculate the qualifier for each bundle based on the state of the git repo. They're not written down anywhere like they used to be for map files. We tag our build input so that the build is reproducible, as the same set of git commits will always produce the same qualifier. In general, we don't want the up-version of build poms to update all of our qualifiers (so we want to ignore pom.xml while calculating the changes). We don't want our current qualifiers to go down (based on bug 401797) so the text file gets added to make sure they don't. PW (In reply to comment #20) > All of our CBI builds calculate the qualifier for each bundle based on the > state of the git repo. They're not written down anywhere like they used to > be for map files. Well, aren't they written into the manifest and are part of the JAR file name? > In general, we don't want the up-version of build poms to update all of our > qualifiers (so we want to ignore pom.xml while calculating the changes). We > don't want our current qualifiers to go down (based on bug 401797) so the > text file gets added to make sure they don't. Sorry, but I still don't see how a normal bundle (not a feature) could ever be *built* and published with a smaller qualifier / timestamp than in a previous build. (In reply to comment #21) > Sorry, but I still don't see how a normal bundle (not a feature) could ever > be *built* and published with a smaller qualifier / timestamp than in a > previous build. for example, o.e.core.commands current qualifier is based on: commit cb9be955e394d7bd32348d93d62a221703922cab Author: Paul Webster <pwebster@ca.ibm.com> Date: Wed Feb 20 16:51:21 2013 -0500 so it gets a qualifier v20130220-2151 After this change we start ignore commits that only touch pom.xml, so the next build will base the qualifier on: commit cfdcc74b6f280106c07da89682777d16f9241deb Author: Paul Webster <pwebster@ca.ibm.com> Date: Thu Jan 24 09:55:47 2013 -0500 because that's the first commit that contains a non-pom.xml file change in the commit, so the new qualifier will be v20130124-1455. PW (In reply to comment #22) > (In reply to comment #21) > > Sorry, but I still don't see how a normal bundle (not a feature) could ever > > be *built* and published with a smaller qualifier / timestamp than in a > > previous build. > > for example, o.e.core.commands current qualifier is based on: > commit cb9be955e394d7bd32348d93d62a221703922cab > Author: Paul Webster <pwebster@ca.ibm.com> > Date: Wed Feb 20 16:51:21 2013 -0500 > > so it gets a qualifier v20130220-2151 > > After this change we start ignore commits that only touch pom.xml, so the > next build will base the qualifier on: > commit cfdcc74b6f280106c07da89682777d16f9241deb > Author: Paul Webster <pwebster@ca.ibm.com> > Date: Thu Jan 24 09:55:47 2013 -0500 > > because that's the first commit that contains a non-pom.xml file change in > the commit, so the new qualifier will be v20130124-1455. > > PW That sounds horrible! If the builder was changed in that manner, then it should also be capable to do a one-time forced rebuild. Plus, I was told that the previous map file info is now stored in the aggregator bundle. Hence, I would expect, that the builder itself can detect that an older commit / timestamp would be used and just ignore it. David? Just to confirm: with bug 400519 being fixed, a pom change will be included and hence trigger a new qualifier. At least that's what I conclude from reading your bug 400519 comment 4: "Include the pom.xml in qualifier calculations". (In reply to comment #24) > Just to confirm: with bug 400519 being fixed, a pom change will be included > and hence trigger a new qualifier. At least that's what I conclude from > reading your bug 400519 comment 4: "Include the pom.xml in qualifier > calculations". This fix undoes that one. It's a trade off. Option 1: we ignore pom file changes and use the text file to force qualifier updates for that bundle. Then when we release Kepler and start Luna, up-versioning the build poms will not effect the qualifiers of the bundles. Option 2: we include pom file changes in qualifier calculations. As in bug 400519 that means that changes to the pom that *might* effect building the bundle will up the qualifier, and extra text files are not necessary. But it also means that all bundles will change qualifiers when we start up Luna, and Kepler SR1. This is all controlled from the parent pom, so if it's decided to remove the pom ignore it is easy to do. PW (In reply to comment #25) > (In reply to comment #24) > > Just to confirm: with bug 400519 being fixed, a pom change will be included > > and hence trigger a new qualifier. At least that's what I conclude from > > reading your bug 400519 comment 4: "Include the pom.xml in qualifier > > calculations". > > This fix undoes that one. It's a trade off. > > Option 1: we ignore pom file changes and use the text file to force > qualifier updates for that bundle. Then when we release Kepler and start > Luna, up-versioning the build poms will not effect the qualifiers of the > bundles. > > Option 2: we include pom file changes in qualifier calculations. As in bug > 400519 that means that changes to the pom that *might* effect building the > bundle will up the qualifier, and extra text files are not necessary. But > it also means that all bundles will change qualifiers when we start up Luna, > and Kepler SR1. With "start up" I guess you mean building. And there, you refer to changing the parent <version>4.3.0-SNAPSHOT</version> to <version>4.4.0-SNAPSHOT</version> in all bundles, right? If so, my next question is: is this really needed i.e. can't we keep the old version number in bundles that aren't touched or, remove the parent version entirely from the bundle's pom and maintain this in the root pom? Personally, I'd prefer option 2 plus not being forced to touch the pom file for bundles that have no change in SR1 or Luna. (In reply to comment #23) > (In reply to comment #22) > > (In reply to comment #21) > > > Sorry, but I still don't see how a normal bundle (not a feature) could ever > > > be *built* and published with a smaller qualifier / timestamp than in a > > > previous build. > > > > for example, o.e.core.commands current qualifier is based on: > > commit cb9be955e394d7bd32348d93d62a221703922cab > > Author: Paul Webster <pwebster@ca.ibm.com> > > Date: Wed Feb 20 16:51:21 2013 -0500 > > > > so it gets a qualifier v20130220-2151 > > > > After this change we start ignore commits that only touch pom.xml, so the > > next build will base the qualifier on: > > commit cfdcc74b6f280106c07da89682777d16f9241deb > > Author: Paul Webster <pwebster@ca.ibm.com> > > Date: Thu Jan 24 09:55:47 2013 -0500 > > > > because that's the first commit that contains a non-pom.xml file change in > > the commit, so the new qualifier will be v20130124-1455. > > > > PW > > That sounds horrible! If the builder was changed in that manner, then it > should also be capable to do a one-time forced rebuild. Plus, I was told > that the previous map file info is now stored in the aggregator bundle. > Hence, I would expect, that the builder itself can detect that an older > commit / timestamp would be used and just ignore it. David? Don't ask me ... this is Paul's baby. :) But you are well capturing the "churn" I was afraid of. The trade-off of including the pom or not including it seems to over complicate things, along with special files that are not really part of the build that sometimes have to be touched, sometimes not, to me, and makes our build "special". Is it really a well accepted best practice? Do we really know that most POM changes would not effect the bundle being built? Seems like something we'll move back and forth on a number of times, yet ... and maybe we should just go back to having map files to say what we really want (ha ha, just kidding .. really! ... well, mostly ... sounds like we have an "in-between" case). It does seem reasonable to improve the CBI/Tycho "versioner" that it should never "go lower" than the starting point (and it could know, in above example, that the starting point was the date for cb9be955e394d7bd32348d93d62a221703922cab and not the current, pom-less point of cfdcc74b6f280106c07da89682777d16f9241deb and its date. But ... that sounds pretty complicated to implement ... I just mean "in theory" it might be a good feature request. I think this particular case is triple complicated by the case of our Juno maintenance. If I'm not mistaken, for "maven users" we'll now have some bundles clearly identified as "from the 3.8 stream" and some "from the 4.2 stream" when in reality, nearly all of them come from the same stream? And hence nearly all of them should have same version and qualifier. The fact that we have to "ignore the POM" to accomplish that tells me something is wrong ... well, we already know that that is ... that we have a monolithic build instead of building in stream specific pieces. So, in conclusion?, I do not know of a better solution that possible right now ... just hope everyone understands the big picture of why this is important to do right now better than I do! (In reply to comment #26) > > With "start up" I guess you mean building. And there, you refer to changing > the parent <version>4.3.0-SNAPSHOT</version> to > <version>4.4.0-SNAPSHOT</version> in all bundles, right? Yes, that's correct. > If so, my next > question is: is this really needed i.e. can't we keep the old version number > in bundles that aren't touched or, remove the parent version entirely from > the bundle's pom and maintain this in the root pom? We need to up the parent version as our parent pom changes when we move to a new stream (new URLs for our consumable repos, like orbit, emf, ecf, as well as our comparator baseline changes (like 4.3-I-builds to 4.4-I-builds), etc. I tried removing it (that didn't work) and even parameterizing it with a property like some other things we consume (that didn't work, maven specifically requires that elements artifactId, groupId, and version be specifics and not properties). So we really do need to up-version the parents and that will change all the poms. > Personally, I'd prefer option 2 plus not being forced to touch the pom file > for bundles that have no change in SR1 or Luna. If we pick option 2 then starting a new stream like Luna will push up the qualifiers and it means that all of our bundles will look like they've changed to p2. That's why I chose Option 1 (to ignore the pom files from changes) and add the text file. Then when we move to Luna and Kepler SR1 both of those changes will work as the maven consumers expect, will have us aligned for building smaller subsets of git repos without having to build the entire aggregator, and won't break all of our qualifiers w.r.t. p2 re-downloading every plugin. PW > So we really do need to up-version the parents and that will change all the
> poms.
That shows that pure Maven is not suitable as a build tool for OSGi projects. Why don't we get rid of all these pom files and let Tycho generate them on-the-fly during the build?
It seems clear now that we won't ever have a clean story with Maven alone, so we should just require Tycho for CBI builds.
(In reply to comment #29) > That shows that pure Maven is not suitable as a build tool for OSGi > projects. Maven has a well defined process that works reliably and can be applied anywhere. But I don't find it scales well in our case (or in any project where you'd have more than 10 bundles). > Why don't we get rid of all these pom files and let Tycho generate > them on-the-fly during the build? Tycho can generate initial pom.xmls for plugins and features http://wiki.eclipse.org/Tycho/Reference_Card#Generating_POM_files But that functionality expects you to go edit the files afterwards. It doesn't even generate in the parent information and only allows one groupId per run (which searches downwards starting in the current directory and creates POMs for any manifests that it finds). PW (In reply to comment #30) > (In reply to comment #29) > > That shows that pure Maven is not suitable as a build tool for OSGi > > projects. > > Maven has a well defined process that works reliably and can be applied > anywhere. But I don't find it scales well in our case (or in any project > where you'd have more than 10 bundles). > > > Why don't we get rid of all these pom files and let Tycho generate > > them on-the-fly during the build? > > Tycho can generate initial pom.xmls for plugins and features > http://wiki.eclipse.org/Tycho/Reference_Card#Generating_POM_files > > But that functionality expects you to go edit the files afterwards. It > doesn't even generate in the parent information and only allows one groupId > per run (which searches downwards starting in the current directory and > creates POMs for any manifests that it finds). > > PW I do not have the details, but colleagues working on Jazz builds told me, that Tycho (or was it Maven?), can generate the pom files, and that's what they do. Eventually bug 386114 and bug 387802 have to be fixed. It's clear that the current approach does not scale. The issue described (with qualifier changing with parent pom and more generally qualifier changing when there is no change in output bundle) would be resolved by creating a mechanism that would allow comparison and replacement of bundles with a baseline *independently of qualifier*. Allowing Tycho to compare x.y.z.t+1 with x.y.z.t and replace x.y.z.t+1 in output by x.y.z.t if no file changed (except MANIFEST.MF where we'd allow qualifier difference) would do the trick. For JBoss Tools, we currently rebuild more often than necessary because of the same problem (check compatibility with newer stuff), and we don't have reproducible qualifiers because it has pitfalls. And it seems like the "compare and replace ignoring qualifier" mechanism would also be ok for our use-case. May make build longer, but there is a big added-value in it. Thanks Paul and others for answering all my questions. Let me summarize what I captured so far: 1) Looks like it was not necessary to coordinate those changes, at least not for "our" build. N20130326-2000 built fine and works for me. 2) We all agree that the current solution is not nice and requires tweaks and trade-offs to be made. It is a workaround to get the builds rolling for now. 3) The final solution must not require the Platform team to massage the pom file each time a version changes or a new release starts. See bug 386114 and bug 387802. If it means, that those who want to build Eclipse, need to install Tycho or some other add-ons, then this is an acceptable trade-off as it removes redundant information, unnecessary changes to bundles that don't change and load from the developers. Also note, that our main focus is to enable CBI and Tycho is part of the CBI approach. 4) With the fix for this bug, pom files won't be considered anymore for computing the times stamp. After reading all comments in the various bugs and given we probably won't have 3) until Kepler ships, I have to agree with Paul, that this option is better than having all qualifiers changed for Luna and Kepler SR1. 5) 'forceQualifierUpdate.txt' has no real meaning, except that it's a one time trigger to update the qualifier, and in the future it can be used when a bundle needs to be "touched" (i.e. make a dummy change to that file). Components, which do not want this file and use other means to "touch" their bundle, can exclude this change from the patch, BUT: they must verify in the next 3.8.2+, 4.2.2+ and master builds, that their bundle versions does not decrease. Hopefully, David will restore the version comparator (bug 391499) soon to help us with that ;-). 6) Repositories, who provide their identical 3.x code to both 3.x and 4.x, i.e. who do not have a R4_2_maintenance branch, might better use "3.8.2" instead of "4.2.2" as parent version (see bug 403969 comment 8 for details). 7) The reordering of xmlns and xsi in the patches is not semantically relevant and can either be applied or excluded. My conclusion is, that at this point, to go forward, we should add Paul's patches with optional modifications by the components, see 4-7). Thank you Dani for that excellent summary, I believe you've captured everything. (In reply to comment #33) > 1) Looks like it was not necessary to coordinate those changes, at least not > for "our" build. N20130326-2000 built fine and works for me. Many of the other downstream builders are broken, and our builds work temporarily for David :-) He still has a copy of the 1.0.0-SNAPSHOT parent in his local maven repo on build.eclipse.org, but if his local maven repo changes or he has to delete it our build will break as well. PW (In reply to comment #34) > Thank you Dani for that excellent summary, I believe you've captured > everything. > > (In reply to comment #33) > > 1) Looks like it was not necessary to coordinate those changes, at least not > > for "our" build. N20130326-2000 built fine and works for me. > > Many of the other downstream builders are broken, and our builds work > temporarily for David :-) He still has a copy of the 1.0.0-SNAPSHOT parent > in his local maven repo on build.eclipse.org, but if his local maven repo > changes or he has to delete it our build will break as well. > > PW I too would like to thank Dani for the nice summary. Two questions remain, for me. 1) It sounds like to be sure we get accurate builds, I should remove the "local maven" repo every build and start fresh. Is that a "feature" of Maven/Tycho or are there bugs open to improve on that? (With the old PDE builds, we always "cleaned more than we had to" just to be safe, but it was fairly well understood why and what would have to be fixed to avoid that ... if anyone ever wanted to invest the time in it). So, I just want to be explicit. Clean each build and suffer the extra 15/20 minutes even when not needed? 2) What are the advantages to introducing a new file, forceQualifierUpdate.txt, to "force" a qualifier update? I was under the impression most teams that had to do this, in the past, would simply add a comment to "build.properties". So, I'd like to understand better the pros/cons. I see the "pro" as then there is one file unrelated to anything else used for this special purpose. The "con" is it is an extra file that is not used for anything else :) and presumably should not end up in built-bundles, right? (In reply to comment #35) > 2) What are the advantages to introducing a new file, > forceQualifierUpdate.txt, to "force" a qualifier update? I was under the > impression most teams that had to do this, in the past, would simply add a > comment to "build.properties". So, I'd like to understand better the > pros/cons. I see the "pro" as then there is one file unrelated to anything > else used for this special purpose. The "con" is it is an extra file that is > not used for anything else :) and presumably should not end up in > built-bundles, right? That's all correct. See also 5) from comment 33 for pros and cons. (In reply to comment #35) > 1) It sounds like to be sure we get accurate builds, I should remove the > "local maven" repo every build and start fresh. Is that a "feature" of > Maven/Tycho or are there bugs open to improve on that? (With the old PDE > builds, we always "cleaned more than we had to" just to be safe, but it was > fairly well understood why and what would have to be fixed to avoid that ... > if anyone ever wanted to invest the time in it). So, I just want to be > explicit. Clean each build and suffer the extra 15/20 minutes even when not > needed? I'm not sure its absolutely necessary to remove the local repo on every build. It's the trade off (maven downloads half the internet and p2 the other half :-) vs we don't want to accumulate too much over time. I would recommend cleaning the repo once a week. > 2) What are the advantages to introducing a new file, > forceQualifierUpdate.txt, to "force" a qualifier update? I was under the > impression most teams that had to do this, in the past, would simply add a > comment to "build.properties". So, I'd like to understand better the > pros/cons. I see the "pro" as then there is one file unrelated to anything > else used for this special purpose. The "con" is it is an extra file that is > not used for anything else :) and presumably should not end up in > built-bundles, right? Right, it definitely won't end up in the built bundles. PW I'm having build issues when trying to build R4_2_maintenance and R3_8_maintenance branch. Looks like some poms still need to be updated. [ERROR] The build could not read 13 projects -> [Help 1] [ERROR] [ERROR] The project org.eclipse.jdt:org.eclipse.jdt.core.tests.binaries:1.0.1-SNAPSHOT (/aggregator/eclipse.jdt.core.binaries/org.eclipse.jdt.core.tests.binaries/pom.xml) has 1 error [ERROR] Non-resolvable parent POM: Could not find artifact eclipse.jdt.core.binaries:eclipse.jdt.core.binaries:pom:4.2.2-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 14, column 11 -> [Help 2] [ERROR] [ERROR] The project org.eclipse.pde:org.eclipse.pde:3.8.0-SNAPSHOT (/aggregator/eclipse.pde.ui/ui/org.eclipse.pde/pom.xml) has 1 error [ERROR] Non-resolvable parent POM: Could not find artifact eclipse.pde.ui:eclipse.pde.ui:pom:3.8.0-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 16, column 11 -> [Help 2] [ERROR] [ERROR] The project org.eclipse.pde:org.eclipse.pde.core:3.8.2-SNAPSHOT (/aggregator/eclipse.pde.ui/ui/org.eclipse.pde.core/pom.xml) has 1 error [ERROR] Non-resolvable parent POM: Could not find artifact eclipse.pde.ui:eclipse.pde.ui:pom:3.8.0-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 16, column 11 -> [Help 2] [ERROR] [ERROR] The project org.eclipse.update:org.eclipse.update.configurator:3.3.200-SNAPSHOT (/aggregator/eclipse.platform/update/org.eclipse.update.configurator/pom.xml) has 1 error [ERROR] Non-resolvable parent POM: Could not find artifact org.eclipse.platform:eclipse.platform:pom:4.2.2-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 14, column 11 -> [Help 2] [ERROR] [ERROR] The project org.eclipse.update:org.eclipse.update.core:3.2.600-SNAPSHOT (/aggregator/eclipse.platform/update/org.eclipse.update.core/pom.xml) has 1 error [ERROR] Non-resolvable parent POM: Could not find artifact org.eclipse.platform:eclipse.platform:pom:4.2.2-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 14, column 11 -> [Help 2] [ERROR] [ERROR] The project org.eclipse.ant:org.eclipse.ant.core:3.2.401-SNAPSHOT (/aggregator/eclipse.platform/ant/org.eclipse.ant.core/pom.xml) has 1 error [ERROR] Non-resolvable parent POM: Could not find artifact org.eclipse.platform:eclipse.platform:pom:4.2.2-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 14, column 11 -> [Help 2] [ERROR] [ERROR] The project org.eclipse.ant:org.eclipse.ant.launching:1.0.200-SNAPSHOT (/aggregator/eclipse.platform/ant/org.eclipse.ant.launching/pom.xml) has 1 error [ERROR] Non-resolvable parent POM: Could not find artifact org.eclipse.platform:eclipse.platform:pom:4.2.2-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 14, column 11 -> [Help 2] [ERROR] [ERROR] The project org.eclipse.ant:org.eclipse.ant.ui:3.5.301-SNAPSHOT (/aggregator/eclipse.platform/ant/org.eclipse.ant.ui/pom.xml) has 1 error [ERROR] Non-resolvable parent POM: Could not find artifact org.eclipse.platform:eclipse.platform:pom:4.2.2-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 14, column 11 -> [Help 2] [ERROR] [ERROR] The project org.eclipse.ant:org.eclipse.ant.tests.core:3.3.0-SNAPSHOT (/aggregator/eclipse.platform/ant/org.eclipse.ant.tests.core/pom.xml) has 1 error [ERROR] Non-resolvable parent POM: Could not find artifact org.eclipse.platform:eclipse.platform:pom:4.2.2-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 14, column 11 -> [Help 2] [ERROR] [ERROR] The project org.eclipse.ant:org.eclipse.ant.tests.ui:3.8.1-SNAPSHOT (/aggregator/eclipse.platform/ant/org.eclipse.ant.tests.ui/pom.xml) has 1 error [ERROR] Non-resolvable parent POM: Could not find artifact org.eclipse.platform:eclipse.platform:pom:4.2.2-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 14, column 11 -> [Help 2] [ERROR] [ERROR] The project org.eclipse.platform:org.eclipse.platform:4.2.2-SNAPSHOT (/aggregator/eclipse.platform/platform/org.eclipse.platform/pom.xml) has 1 error [ERROR] Non-resolvable parent POM: Could not find artifact org.eclipse.platform:eclipse.platform:pom:4.2.2-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 14, column 11 -> [Help 2] [ERROR] [ERROR] The project org.eclipse.sdk:org.eclipse.sdk:4.2.2-SNAPSHOT (/aggregator/eclipse.platform/platform/org.eclipse.sdk/pom.xml) has 1 error [ERROR] Non-resolvable parent POM: Could not find artifact org.eclipse.platform:eclipse.platform:pom:4.2.2-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 14, column 11 -> [Help 2] [ERROR] [ERROR] The project org.eclipse.platform:rt.equinox.p2:3.8.0-SNAPSHOT (/aggregator/rt.equinox.p2/pom.xml) has 1 error [ERROR] Non-resolvable parent POM: Could not find artifact org.eclipse:eclipse-platform-parent:pom:1.0.0-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 18, column 11 -> [Help 2] [ERROR] (In reply to comment #37) > I'm not sure its absolutely necessary to remove the local repo on every > build. It's the trade off (maven downloads half the internet and p2 the > other half :-) vs we don't want to accumulate too much over time. I would > recommend cleaning the repo once a week. A general Maven recommendation is to not call "mvn clean install" but instead call "mvn clean verify". The latter skips the install step, which puts locally built artifacts into the local Maven repository. If you have to use "mvn clean install" for some reason, there is a Tycho option to ignore locally built Tycho artifacts in the local Maven repository. By default, locally built Tycho artifacts are implicitly added to the target platform [1] (which mimics the behaviour of "install" in a plain Maven build). You can disable this with the command line option -Dtycho.localArtifacts=ignore. This is usually what you want in a CI build. [1] http://wiki.eclipse.org/Tycho/Target_Platform#Locally_built_artifacts (In reply to comment #39) > (In reply to comment #37) > > I'm not sure its absolutely necessary to remove the local repo on every > > build. It's the trade off (maven downloads half the internet and p2 the > > other half :-) vs we don't want to accumulate too much over time. I would > > recommend cleaning the repo once a week. > > A general Maven recommendation is to not call "mvn clean install" but > instead call "mvn clean verify". The latter skips the install step, which > puts locally built artifacts into the local Maven repository. > > If you have to use "mvn clean install" for some reason, there is a Tycho > option to ignore locally built Tycho artifacts in the local Maven > repository. By default, locally built Tycho artifacts are implicitly added > to the target platform [1] (which mimics the behaviour of "install" in a > plain Maven build). You can disable this with the command line option > -Dtycho.localArtifacts=ignore. This is usually what you want in a CI build. > > [1] http://wiki.eclipse.org/Tycho/Target_Platform#Locally_built_artifacts It is hard for me to express how much this comment clears things up for me! :) We do use -Dtycho.localArtifacts=ignore for our production builds, but I've always wonder what the "install" meant, and why we'd want to do it. Sound like for our production builds, we should use "mvn clean verify" for our production builds? It would still produce the same /targets/... (which is what we collect for our download pages, etc.) If/when we have "build on demand" (to rebuild just a few repos) that might change, but sound like is "mvn clean verify" better for production builds. Let me know if I'm misunderstanding. Thanks, Build seems to work now after all the patches were pulled in. (In reply to comment #40) > ... but sound like is "mvn clean verify" better for > production builds. Let me know if I'm misunderstanding. This is correct. In fact, "mvn clean verify" should always be used if you don't explicitly want to make use of the Maven feature that are enabled by calling "install". (And if you don't know what these features are, you probably don't want to use them ;-) I think that "mvn clean package" is also in wide use in the Maven community. In a plain Maven build, this means that unit tests are executed (bound to the "test" phase which is before "package"), but integration tests aren't (because they are bound to the "integration-test" phase, which comes after "package"). However, on a normal Tycho project it doesn't make much sense to call "mvn clean package" because all of Tycho's tests are technically integration tests. (They run on the packaged build result and are hence bound to integration-test.). So in short: Where people would use "mvn clean install" on a plain Maven project, they should use "mvn clean verify" on a Tycho project. (In reply to comment #42) > (In reply to comment #40) > > I think that "mvn clean package" is also in wide use in the Maven community. > In a plain Maven build, this means that unit tests are executed (bound to > the "test" phase which is before "package"), but integration tests aren't > (because they are bound to the "integration-test" phase, which comes after > "package"). However, on a normal Tycho project it doesn't make much sense to > call "mvn clean package" because all of Tycho's tests are technically > integration tests. (They run on the packaged build result and are hence > bound to integration-test.). So in short: Where people would use "mvn clean > install" on a plain Maven project, they should use "mvn clean verify" on a > Tycho project. We in eclipse currently run with -Dmaven.test.skip=true so I'm wondering if that means mvn clean package would work for us? But, I'll try mvn clean verify and see if that still works. (I think at one point, we had to run with mvn clean install, because not all "parents" were defined correctly ... guess that show up better if we used mvn clean verify or mvn clean package). I hate to ask, because you may have already pointed the way elsewhere, but is there a handy doc that shows all the possible phases that Maven, and Tycho use? @David -Dmaven.test.skip will ignore "eclipse-test-plugins" artifacts so you may not be able to resolve some bundles when using it. If you only want to skip test _executions_, you should use -DskipTests=true ( ). So the recommended command line for you would be "mvn clean verify -DskipTests=true". HTH. (In reply to comment #44) > @David > -Dmaven.test.skip will ignore "eclipse-test-plugins" artifacts so you may > not be able to resolve some bundles when using it. If you only want to skip > test _executions_, you should use -DskipTests=true ( ). > > So the recommended command line for you would be "mvn clean verify > -DskipTests=true". > > HTH. After trying mvn clean verify (instead of mvn clean install) I was getting an error in our scripts. I've documented in bug 405224. Not sure if it is the "verify" vs. "install" difference per se, or merely and indication that something (else) is not defined correctly ... or ... could be completely unrelated and due to some other error buried deeper in the logs. (In reply to comment #43) > I hate to ask, because you may have already pointed the way elsewhere, but is > there a handy doc that shows all the possible phases that Maven, and Tycho use? There is a good introduction to the lifecycle in Maven in general [1], but unfortunately there is no explicit documentation about what Tycho does in which phase. I could point you to the sources [2], but I'm not sure if this is the answer you wanted to hear... [1] http://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html [2] http://git.eclipse.org/c/tycho/org.eclipse.tycho.git/tree/tycho-maven-plugin/src/main/resources/META-INF/plexus/components.xml Quick question. Sounds like "mvn clean verify" is the way to go. Would there be any issues if I updated the wiki [1] to recommend using this instead of "install"? I've been using "verify" in all my own builds and haven't noticed any issues on my end. [1] http://wiki.eclipse.org/Platform-releng/Platform_Build (In reply to comment #47) > Quick question. Sounds like "mvn clean verify" is the way to go. Would there > be any issues if I updated the wiki [1] to recommend using this instead of > "install"? > > I've been using "verify" in all my own builds and haven't noticed any issues > on my end. > > > [1] http://wiki.eclipse.org/Platform-releng/Platform_Build I'd update it to say mvn clean verify. We've been using that in production builds for a few weeks, with no issues. As I documented in bug 405224, the "localRepo" that's created is LOTS smaller (an order of magnitude or two) so that alone has got to be worth something! Thanks David, I updated the wiki to use verify instead of install. |