Community
Participate
Working Groups
Currently, many projects auto-register an updates repository using the referenced repositories feature or p2 hooks. This a work-around for the fact that simultaneous release process currently does not provide a way to deliver fixes between the coordinated releases. This approach has unintended consequence of baking in an update policy into the project's release artifacts. This makes it difficult for third-parties to create distributions with a different update policy and is one of the things that is standing in the way of Eclipse having multiple integration streams, as practiced elsewhere. This is a proposal to make a small change to the current simultaneous release process to provide a mechanism for projects that are part of the simultaneous release to deliver updates. Implementing this proposal will remove the need for the current work-around. 1. Start with the current simultaneous release process 2. Add a reference repository URL to the simultaneous release repository, such as http://download.eclipse.org/releases/mars/updates 3. The updates repository will be a composite to which projects can contribute references to releases that happen in between coordinated releases 4. The updates composite is managed by the "updates administrator" (I will volunteer or Planning Council can appoint someone else) 5. The burden is on the project to test compatibility. If a project contributes an update in this manner and a cross-project issue crops up, once the issue is validated, the updates administrator immediately drops the project's repository from the updates composite, thus returning us to a known good state. Then it's up to the project to rectify the issue with a new release before being re-added. In some cases, it might mean that the project has to wait for another project to update first or work with them at our coordinated release points. There is no new risk in this approach. Currently, if the user doesn't wish to receive out-of-band updates, they need to disable all the repositories added by various projects. Similarly, after this proposal is implemented, the user would need to disable the simrel updates repository for the same effect. Implementing this proposal will gain us additional tools in dealing with cross-project issues (seen #5) and would decouple project release artifacts from the update policy.
*** Bug 483320 has been marked as a duplicate of this bug. ***
*** Bug 483321 has been marked as a duplicate of this bug. ***
+1
I am sympathetic to the problem but have some comments and questions about this initial proposal. It would be best if the issue, and bug title, were stated in terms of "the problem", instead of a solution. It is not just the "opposite" of the title, in this case, since projects can contribute updates outside the coordinated release, but you are saying you don't like the method they do so. So, is the problem "too many reference repositories added" (i.e. just the number of them?) or more "reference repositories added" (i.e. that they are added, at all?) If too many, I'd ask, "how many are there"? You suggest users can disable the sim. rel. repo if they do not want "off-cycle" updates, which seems like a pretty big disadvantage since then users don't get the normal updates either. And it is expecting a lot for users to know when to re-enable it. And, remember, we do sometimes have unplanned "Sim. Release" updates, such as to fix security issues, and similar. So imagine we had a separate repository, such as "neon/updates" that users could enable/disable independently of the main one. Would that "fix" the problem? In either case, I'd have a remaining issue I know less how to solve. It seems if we had an "updates" repository, then every time there was a "Sim. Release" the "updates" repository should be "reset to nothing". But this violates the "immutable principle" of p2 repositories. But if not reset to zero, then how would we enforce that whatever was in "updates" actually made its way into the main "Sim. Release" stream? If we didn't enforce it, I suspect there would eventually be cases where people did not bother. I don't mean any of these comments or questions as criticisms, and not saying the problem can not be improved ... I am just trying to be clearer what the problem is and makes sure the solution is not worse than the problem. Thanks for opening this bug.
Setting myself as owner so notices not sent to everyone for every comment. If you want to follow this issue, please CC yourself.
Good questions. Thanks for the response. > It would be best if the issue, and bug title, were stated in terms of "the > problem", instead of a solution. It is not just the "opposite" of the title, > in this case, since projects can contribute updates outside the coordinated > release, but you are saying you don't like the method they do so. The root problem is that we currently lack separation of project release artifacts from update policy. This means that project repositories should not include reference repositories. However, to solve this problem, we need another means for projects to deliver updates to simultaneous release consumers in between coordinated releases. This bug is phrased the way it is, because it's a step one of a two step solution. The second step would be for Architecture Council to work with projects to actually remove reference repositories from project release repositories. Note that increased number of coordinated releases is not a solution that will enable us to resolve the root issue. Consider the scenario of a critical bug being found at the last minute, when it's too late to get the fix into the coordinated release. The projects need to retain ability deliver the fix so that all the user has to do is check for updates. > So, is the problem "too many reference repositories added" (i.e. just the > number of them?) or more "reference repositories added" (i.e. that they are > added, at all?) The issue is that projects are adding reference repositories from their release repositories. Reducing the number of repositories registered in user's workspace is not the focus, but would be a nice side benefit. > You suggest users can disable the sim. rel. repo if they do not want > "off-cycle" updates, which seems like a pretty big disadvantage since then > users don't get the normal updates either. And it is expecting a lot for users > to know when to re-enable it. The comment you are referring to is meant to illustrate that there is no additional risk from implementing the proposal. Users can disable repositories now that deliver off-cycle updates. They will still be able to do that. > In either case, I'd have a remaining issue I know less how to solve. It seems > if we had an "updates" repository, then every time there was a "Sim. Release" > the "updates" repository should be "reset to nothing". But this violates the > "immutable principle" of p2 repositories. But if not reset to zero, then how > would we enforce that whatever was in "updates" actually made its way into the > main "Sim. Release" stream? If we didn't enforce it, I suspect there would > eventually be cases where people did not bother. My assumption is that we do clear the updates repository with every coordinated release. I am not aware of the immutable principle for p2 repositories. It's a common practice to have repositories that represent a moving target. In any case, the updates repository is meant to be used in conjunction with the main simrel repository and the union of those two repositories can be considered immutable, at least in as far as artifacts are never removed, but they do move from one repository to the other.
I prefer projects manage their own update policies. Self registering their update repos is the best mechanism for that and why that mechanism is there to begin with.
> I prefer projects manage their own update policies. Self registering their > update repos is the best mechanism for that and why that mechanism is there > to begin with. It sounds like you want to assert control over the update policy for your projects in all contexts, such as in a third-party distribution, regardless of the wishes of the maintainers of that distribution stream. That's a pretty inflexible position and strikes me as at odds with the core principles of Eclipse Foundation to provide re-usable blocks for others to consume.
(In reply to Konstantin Komissarchik from comment #8) > > I prefer projects manage their own update policies. Self registering their > > update repos is the best mechanism for that and why that mechanism is there > > to begin with. > > It sounds like you want to assert control over the update policy for your > projects in all contexts, such as in a third-party distribution, regardless > of the wishes of the maintainers of that distribution stream. > > That's a pretty inflexible position and strikes me as at odds with the core > principles of Eclipse Foundation to provide re-usable blocks for others to > consume. I guess, yes, especially distributions that consume and don't contribute. I have little time for them if they disagree with what contributors are doing. Especially if we're trying to make user's lives better by getting timely updates to them. Having said that, every build gets a p2 site. Distributors can pick and choose which one they want to use.
I'll make some other observations, on both sides of the issue, and then let this rest for a while, for my own thoughts and others comments. One "solution" that some have used, instead of adding "reference repos" during feature installation, is to provide update repositories in some EPP packages. Last I looked, both webtools and mylin did this in JEE package. That seems better than doing it in a feature's p2 file operations. It "fits in" with the "EPP is our product" theme ... and ours to do with as we see fit, whereas features installed in isolation are, in my view, intended more to be building blocks, and not "take over" the Eclipse installation. To state it, in the extreme. :) On the other side, whatever we come up with for the Sim. Release does not really solve the issue "for everyone" since some might install features that are not part of Sim. Release, or perhaps not even part of the Eclipse Foundation. Those features could tweak the reference repositories outside of our control. I think the use case that is most desired to be solved is the "for everyone" case, and not sure how to do that, except to get p2 to "remove" the reference repository concept? And *this* might be "anti-adopter" since I think some adopters like to do this, when someone installs one of their special features, such as a server or something. Just stating a couple of points, not arguing. :) I can see both sides, but would like to continue to discuss and at least eventually figure out a solution of some sort -- or to gain a clear consensus that a solution is not desired -- that it is working as it should (as Doug as said). Thanks for reading.
> One "solution" that some have used, instead of adding "reference repos" during > feature installation, is to provide update repositories in some EPP packages. > Last I looked, both webtools and mylin did this in JEE package. That seems > better than doing it in a feature's p2 file operations. It "fits in" with the > "EPP is our product" I think that's less ideal than solving this problem at simrel repository level. Adding reference repositories at EPP level means more work for projects (adding it separately to every relevant package) and would likely result in fragmentation. There is also the issue of long-term relevance of EPP packages with advent of the network installer (Oomph). If we treat the simrel repository as an update stream and project release repositories as frozen points in time, then everyone can achieve their goals. > On the other side, whatever we come up with for the Sim. Release does not > really solve the issue "for everyone" since some might install features that > are not part of Sim. Release, The goal is to discourage the use of reference repositories for all projects at Eclipse Foundation in alignment with our mission of building re-usable blocks. Simrel enters into the picture as a provider of an important update stream. Since it's a key distribution mechanism for many projects, by providing a better update delivery mechanism, we remove all of the justifiable need for projects to self-register update repositories. I consider the desire to control the update policy of third-party distribution streams as not in alignment with Eclipse Foundation's mission and therefore unjustifiable. It's not a goal to take away ability to add reference repositories for third-party projects or products. Their providers have different missions and business concerns. The provider of an update stream can take up any issues with them directly.
> Having said that, every build gets a p2 site. Distributors can pick and choose > which one they want to use. That's not sufficient if the project's p2 site for a release auto-registers an update repository and interferes with the distribution provider's efforts to set an update policy that makes sense for them and their users. The situation we have at Eclipse Foundation right now is akin to Linux kernel including a self-updater that applies whatever patches the kernel team feels everyone should be using instead of allowing the distro providers to decide when to move up to a particular kernel version.
I've gone through the discussion, and I fully agree with the goal of allowing projects to deliver fixes (maintenance releases) between SimRel releases. It's basically the purpose of continuous (maintenance) release. However, I believe the updates/ proposal is a workaround for an issue that actually stands in the main release train. What makes the value of this updates/ site proposal is mostly the fact that contributions are *not permissive*, ie there would be people in charge of monitoring that the contribution is acceptable for the release train. The definition of "acceptable" would for example include the fact that no feature contributes/enables additional p2 repository. Why not starting this on the actual release train? The checks that would be done for "updates/" would be done for any contribution. That way, we know that whatever is at the HEAD of the release train is in a good quality to be built and distributed. The main issue about the release train is that it has dozens of committers, whose concerns about overall Eclipse quality can vary because of their needs, background or experience. So we cannot really trust it to be of good quality to respin a release whenever we want. If we imagine a short set of committers, reviewing incoming changes with strict rules, then we could safely release a maintenance simultaneous release whenever we want to. And if we can accept changes and release them whenever we want, then we have this problem solved without a new URL, and we get continuous delivery by the way.
Here's a proposal for what we could do to split artifact publishing from artifact update policy, with little effort to maintain (simrel side) and little effort to set up (individual projects side): https://wiki.eclipse.org/Planning_Council/January_06_2016/bug483322
> Here's a proposal for what we could do to split artifact publishing from > artifact update policy, with little effort to maintain (simrel side) and little > effort to set up (individual projects side): > > https://wiki.eclipse.org/Planning_Council/January_06_2016/bug483322 This is very similar to the proposal that I put forward. The difference is two streams instead of one and the lack of a link to the maintenance stream from the main simrel repository. I have a reservation over that second aspect as it would cause installs originating from Oomph and other similar solutions to install original simrel bits when updated versions of various components are available in the maintenance repository. In terms of effort required, the two proposals are equivalent.
> the lack of a link to the maintenance stream from the main simrel repository. > I have a reservation over that second aspect as it would cause installs > originating from Oomph and other similar solutions to install original simrel > bits when updated versions of various components are available in the > maintenance repository. This can be fixed by adding /maintenance and /minor URLs as referenced repositories in the main simrel repository as opposed to adding these in the EPP packages.
Does anyone already thought about the criteria that would allow a project to contribute an "update" to all Eclipse IDE users out of the Simultaneous Release? Many people here are interested by providing ability for project to release updates between SimRel builds, but we need to first agree on the criteria to allow that... For example, our vision for JBoss Tools is that this Eclipse "rolling update" site should contain only maintenance fix, and no API change; to guarantee it wouldn't break our top-ins. Is this the kind of criteria we all agree on?
> Does anyone already thought about the criteria that would allow a project to > contribute an "update" to all Eclipse IDE users out of the Simultaneous Release? > > Many people here are interested by providing ability for project to release > updates between SimRel builds, but we need to first agree on the criteria to > allow that... > For example, our vision for JBoss Tools is that this Eclipse "rolling update" > site should contain only maintenance fix, and no API change; to guarantee it > wouldn't break our top-ins. Is this the kind of criteria we all agree on? The update stream rules should be similar to the rules for the SR releases. That is no API breaking changes, but new features are allowed alongside bug fixes. PS: I was going to express the above in terms of major/minor/service versions, but decided against that since some folks only loosely follow the version convention.
(In reply to Konstantin Komissarchik from comment #18) > > Does anyone already thought about the criteria that would allow a project to > > contribute an "update" to all Eclipse IDE users out of the Simultaneous Release? > > > > Many people here are interested by providing ability for project to release > > updates between SimRel builds, but we need to first agree on the criteria to > > allow that... > > For example, our vision for JBoss Tools is that this Eclipse "rolling update" > > site should contain only maintenance fix, and no API change; to guarantee it > > wouldn't break our top-ins. Is this the kind of criteria we all agree on? > > The update stream rules should be similar to the rules for the SR releases. > That is no API breaking changes, but new features are allowed alongside bug > fixes. > > PS: I was going to express the above in terms of major/minor/service > versions, but decided against that since some folks only loosely follow the > version convention. Yes, that's already the rule for the simrel. No API changes between yearly releases.
(In reply to Konstantin Komissarchik from comment #18) > The update stream rules should be similar to the rules for the SR releases. > That is no API breaking changes, but new features are allowed alongside bug > fixes. Ok, and a project in this composite "updates/" site moving from a version to another wouldn't declare anything or require any change in the composite? The change wouldn't be traced/traceable, it's all based on trust in the contributed URL? (note that it's a pure question, not a "concern" at the moment)
> Ok, and a project in this composite "updates/" site moving from a version to > another wouldn't declare anything or require any change in the composite? The > change wouldn't be traced/traceable, it's all based on trust in the contributed > URL? > (note that it's a pure question, not a "concern" at the moment) Correct. The idea is that light-weight aggregation would span the gaps between more rigorous, but more labor-intensive current aggregation process. If a problem is detected, the fallback is to remove conflicting updates from the updates composite or (in the extreme case) to fallback to no updates (the last simultaneous release).
I think it is best to mark this as "won't fix". We have discussed at a few Planning Council meetings, and there is little consensus on if or how to make it a "policy". I think everyone agrees it should typically be avoided, but there were (or are) some cases where we felt it was the project itself that "knows best" -- that is, what their users wanted and expected. Hence, we are stuck that gray area of making things better for adopters at the expense of users or making things better for users at the expense of adopters. There has been some improvement in the general area of doing updates to EPP packages, where most but not all features are "root features" (but, see bug 505378) so that should make the "need for inserted reference repositories" a little less, but not zero. That is, we can, in theory, create a new Sim. Release repo to pick up fixes for major bugs without respinning all EPP packages, but, again, no clear policy on when that would be appropriate or desirable. We do think that particular cases are observed of features inserting reference repositories that are of no benefit then those would be valid bugs against the project that is doing it, but otherwise, we, the Planning Council, did not think we were in a position to dictate that projects could never insert a reference repository. I hope I have captured the discussions adequately, and people are still free to discuss the issue here but did not want to remain silent and leave this as an open issue forever. Thanks for opening, and thanks to everyone's participation.
(In reply to David Williams from comment #22) > There has been some improvement in the general area of doing updates to EPP > packages, where most but not all features are "root features" (but, see bug > 505378) so that should make the "need for inserted reference repositories" a > little less, but not zero. That is, we can, in theory, create a new Sim. > Release repo to pick up fixes for major bugs without respinning all EPP > packages, but, again, no clear policy on when that would be appropriate or > desirable. After I wrote this, I thought I should be a little bit clearer. I should have mentioned that while there is no clear-cut policy on when or if to "respin" the Sim. Release repo, that means, in practice, that it is handled on an exception basis. Any project is free to ask for one, ideally through their PMC and Planning Council representative, but there was just no obvious consensus that would specify when it would be appropriate. Some thought "any time for any reason", some thought it should depend on the magnitude of the bug, whether a leaf project or not, how big or risky the code change was, etc. And, it was my impression no one wanted to be on a "review board" reviewing these things on a frequent basis (even if they were technically able to). So ... that is all that leads me to describe the situation as "no consensus".
