Bugzilla – Full Text Bug Listing
| Summary: | Provide p2 repository containing all launcher fragments | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | [Eclipse Project] Equinox | Reporter: | Kim Moir <kim.moir> | ||||||
| Component: | Launcher | Assignee: | Bogdan Gheorghe <gheorghe> | ||||||
| Status: | RESOLVED FIXED | QA Contact: | |||||||
| Severity: | normal | ||||||||
| Priority: | P3 | CC: | aniefer, david_williams, Erik.Vanherck, jeffmcaffer, milesparker, pwebster, remy.suen, stepper, thomas, tjwatson, wayne.beaton | ||||||
| Version: | 3.7 | ||||||||
| Target Milestone: | Kepler M4 | ||||||||
| Hardware: | PC | ||||||||
| OS: | Windows XP | ||||||||
| Whiteboard: | |||||||||
| Bug Depends on: | |||||||||
| Bug Blocks: | 337357, 337358, 344048, 357140 | ||||||||
| Attachments: |
|
||||||||
|
Description
Kim Moir
The launcher binaries are compiled for all of these platforms and the fragments are available in CVS. The Eclipse SDK just isn't built for these platforms. The executable feature is intended as a built time feature for including the launchers in a product, removing fragments from this feature greatly increases the difficulty of building any eclipse based product/RCP app for these platforms. For a product to build these platforms, all that is required is to add the appropriate map file entry for fetching, or get the fragments from CVS in some other way if not using pde.build's fetch. Generally speaking, I am against removing fragments for which we are still compiling binaries. *** Bug 319345 has been marked as a duplicate of this bug. *** From bug 319345: win32.win32.ia64 wpf.win32.x86 I think this is unfair actually. We've been building using this feature for a long time. Now it breaks because the feature is no longer aligned with what is provided in the platform. A more sensible approach IMHO, would be to keep whatever is delivered in the platform repository coherent, build features or not really doesn't matter. If someone wants to add obscure binaries and fetch them from CVS using specific build technologies, then let them create a feature of their own that includes the launcher feature and then adds the extra fragments. That way, nobody gets any grief from dependencies that cannot be resolved. To us, this is a regression and I really wish that the bug gets fixed a.s.a.p. My opinion rimes well with John's conclusion in bug 213437 comment 7 which preceded (and I guess triggered) the fix that was made the last time this happened. (In reply to comment #5) > My opinion rimes well with John's conclusion in bug 213437 comment 7 which > preceded (and I guess triggered) the fix that was made the last time this > happened. org.eclipse.equinox.executable has a wider audience than even rcp. The executable feature was not modified for bug 213437. The only fragment that has ever been removed from the executable feature is motif.hpux.PA_RISC (bug 284865) because we haven't compiled the launcher for that platform since before the launcher was split into fragments. It has been so long that building a product for PA_RISC could potentially require source level changes. The dependencies on the fragments are all protected by appropriate platform os/arch/ws settings. The semantics of those settings have always been to ignore those entries when not targeting those platforms. In many ways the executable feature really is a PDE artifact. If alternative build systems consume the feature that is great, but this "missing fragment" issue is really a problem with those build systems. (In reply to comment #6) > ... but this "missing fragment" issue is really > a problem with those build systems. I strongly disagree. Unless you want to claim that the org.eclipse.equinox.executable should be thought of as an internal IBM build artifact, you have a responsibility against the community to: a) Not introduce regressions that break their builds. That is effectively what has happened now. b) See to that whatever you publish is coherent. At present, it isn't. Right now, you publish a feature that contains fragments that only very few will ever make use of. When they do, they need to write ant-script and use special 'fetch' tasks to get to it. You advocate that this should be the default because of it would be inconvenient for your internal build to remove them. Effectively that means that all others (the general public that the repository is intended for) must write their own features or add exceptions to repair builds that are now broken How hard can it be for you to fix this and maintain your own version of this feature where you add the obscure fragments? (In reply to comment #6) > org.eclipse.equinox.executable has a wider audience than even rcp. The > executable feature was not modified for bug 213437. > I didn't say it was. There are two ways to fix this. Either publish the missing fragments or remove them from the executable feature. I don't care how it's fixed as long as the resulting repository is coherent. Another related question. I use the PDE preferences to set up a new Target Platform. I add a "Software Site", uncheck the "Include required software" and check the "Include all environments". I then select the 3.7-I-builds and choose the launcher feature. To my surprise, the install succeeds although it failed to install all included bundles. There's no error message to indicate that not all bundles could be resolved. My question is, is this the general case? Will the PDE Target Platform provisioner silently install inconsistent IU's although it fail to resolve its dependencies? Or is there some special hack involved when the org.eclipse.equinox.executable feature is the culprit? (In reply to comment #7) > (In reply to comment #6) > > ... but this "missing fragment" issue is really > > a problem with those build systems. > > I strongly disagree. The simple fact is your build fails when an IU that is not needed by the thing you are building is missing. > Unless you want to claim that the > org.eclipse.equinox.executable should be thought of as an internal IBM build > artifact, This has nothing to do with IBM, we have told internal IBM teams the same thing: fix your tooling to respect the filter on the requirement. > you have a responsibility against the community to: > a) Not introduce regressions that break their builds. That is effectively what > has happened now. What has happened is that the Eclipse SDK dropping some platforms has revealed a general problem in your tooling. > b) See to that whatever you publish is coherent. At present, it isn't. The repository is perfectly coherent for everyone who does not actually target the missing platforms. There has never been any requirement that a p2 repository be entirely self contained. This is an artificial restriction introduced by your tooling. Every single Eclipse install is itself a p2 repository that is not self contained. > Right now, you publish a feature that contains fragments that only very few > will ever make use of. When they do, they need to write ant-script and use > special 'fetch' tasks to get to it. They do not need to write any ant-scripts, nothing special must be done to get these fragments. All build systems must obviously provide some way of getting source that will be built. For people who want to target these platforms, it is a very small change to add the fragments to this list of source to get. > How hard can it be for you to fix this and maintain your own version of this > feature where you add the obscure fragments? It isn't hard, but us making a change is at best a workaround to a specific instance of a problem that still exists for your users. You can ignore everything else, the fundamental point is this: Removing these fragments do not help you in general. You remain broken on missing IUs that are not actually required. There is a world of content with many different reasons for having filtered requirements that are not available. You can't go out to every content provider and ask them to remove dependencies because you don't happen to need them and your builder doesn't respect the filter. I have to say I'm at a bit of a loss as to why we have a feature that lists things we don't ship. IMHO we have the fragments, we should be putting them in the repo. Re comment 9. Yes, if you turn off "Include required software" the target platform may be inconsistent. This is on purpose and desired in many usecases we see. The target itself is not intended to be "runnable" rather ti represents a set of content that can be used to construct runnable/consistent things. For example, you would not be able to do cross platform development with out this. (In reply to comment #10) > The simple fact is your build fails when an IU that is not needed by the thing > you are building is missing. > The things we're building are p2 repositories for our headless products. We want those products to run on all conceivable platforms. Rather than hard-coding this list of platforms into the build system, we rely on the list in the org.eclipse.equinox.executable feature. This has worked well for us in the past. > The repository is perfectly coherent for everyone who does not actually target > the missing platforms. There has never been any requirement that a p2 > repository be entirely self contained. This is an artificial restriction > introduced by your tooling. Every single Eclipse install is itself a p2 > repository that is not self contained. > I hear (or read) everything that you say but I still see no argument for including IU's with requirements that can be resolved by p2 in a p2 repository. Why would you ever create such an IU if there's no intent to ever provide the missing pieces in a p2 repository? > They do not need to write any ant-scripts, nothing special must be done to get > these fragments. > That's simply not true. At present there's no way for p2 to resolve them. > > How hard can it be for you to fix this and maintain your own version of this > > feature where you add the obscure fragments? > > It isn't hard, but us making a change is at best a workaround to a specific > instance of a problem that still exists for your users. > I'd argue that including requirements that cannot be resolved, only for the sake of your internal build, is the workaround here. It has bad consequences for people who are relying on the consistency of what you produce. It has nothing whatsoever to do with what build system we are using. > Removing these fragments do not help you in general. Well, so don't remove them then. Provide the fragments instead, I don't care. But don't break the consistency. (In reply to comment #11) > I have to say I'm at a bit of a loss as to why we have a feature that lists > things we don't ship. IMHO we have the fragments, we should be putting them in > the repo. > Needless to say, I wholeheartedly agree. > Re comment 9. Yes, if you turn off "Include required software" the target > platform may be inconsistent. This is on purpose and desired in many usecases > we see. The target itself is not intended to be "runnable" rather ti > represents a set of content that can be used to construct runnable/consistent > things. For example, you would not be able to do cross platform development > with out this. OK, so then we have food for another bugzilla I guess. If I specify one of the missing environments, let's say linux.gtk.ppc under the Environment tab in the Target Platform wizard, and then check "Include required software" and load the Launcher feature, I get red markers with messages informing me about missing fragments. Since those fragmens are nowhere to be found in the p2 world, there is no way for me to resolve this. This is not a problem with "my build system". This is proof that special hacks needs to be inserted in order to produce a build with these fragments (if indeed it is at all possible using the IDE). As things stands now, they are not useful for general consumption in *any* build system (except perhaps some internal one used by Andrew). So please either remove them from the feature or add the fragments to the repository. Just a note that this breaks my bucky build as well. There is a workaround, but not knowing the intricacies of the platform build issues I think it is an inconsistency that should be addressed. (In reply to comment #10) > The repository is perfectly coherent for everyone who does not actually target > the missing platforms. There has never been any requirement that a p2 > repository be entirely self contained. This is an artificial restriction > introduced by your tooling. Every single Eclipse install is itself a p2 > repository that is not self contained. I may be misunderstanding this, but doesn't Indigo have a requirement that the repository be self-contained? In any case there are enormous advantages to not having to worry about special casing a repository's inconsistencies in order to get a simple build dependency working. Created attachment 192230 [details] Example project Here is a sample project demonstrating how trivially easy it is to build a product that targets these other platforms using PDE in the 3.7 SDK: 1) Import the equinox.motif project. 2) Open crossPlatform.target, click "Set as Target Platform" 3) Right click projectSet.psf and "Import Project Set" (user name: anonymous) 4) Open equinox.product and click "Use the Eclipse Product export wizard" 5) Enter a directory and select "Export for multiple platforms" 6) Next -> Select aix (motif/ppc) and hpux(motif/ia64_32) 7) Click finish. (Step 2 is adding the org.eclipse.equinox.executable feature to your target) Export from UI is as basic a setup for PDE as it gets. I found comment #12 > This is not a problem with "my build system". This is proof that special hacks > needs to be inserted in order to produce a build with these fragments (if > indeed it is at all possible using the IDE). As things stands now, they are not > useful for general consumption in *any* build system (except perhaps some > internal one used by Andrew). So please either remove them from the feature or > add the fragments to the repository. to be insulting. The only way it remotely resembles reality is if you call PDE "some internal [build system] used by Andrew". Removing the fragments from the executable changes targeting those platforms from trivial to next to impossible. Having the fragments in a repository isn't necessary for PDE. There may be valid reasons for adding them to the repo, but it is work for the releng team and the tone on this bug hasn't done anything to bring it to the top of the list of priorities. Regarding the Indigo, the executable feature is not part of the contribution there. It is a separate feature for use in a target platform and is not meant to be installed by the user. (In reply to comment #14) > Created attachment 192230 [details] > Example project > > Here is a sample project demonstrating how trivially easy it is to build a > product that targets these other platforms using PDE in the 3.7 SDK: > > 1) Import the equinox.motif project. > 2) Open crossPlatform.target, click "Set as Target Platform" > 3) Right click projectSet.psf and "Import Project Set" (user name: anonymous) > 4) Open equinox.product and click "Use the Eclipse Product export wizard" > 5) Enter a directory and select "Export for multiple platforms" > 6) Next -> Select aix (motif/ppc) and hpux(motif/ia64_32) > 7) Click finish. > > (Step 2 is adding the org.eclipse.equinox.executable feature to your target) > Anything is possible, but look at it from the consumers end: a) We don't need any of these 7 steps for the bundles that are included in the repository. b) It will require a lot more work to get this into a headless build. It would be so much easier for everyone if the repository was consistent. (In reply to comment #14) > ... The only way it remotely resembles reality is if you call PDE > "some internal [build system] used by Andrew". > I'm sorry if you found my comment insulting. The way I perceived it, you closed this bug in the middle of a discussion. You had your own conclusion, sure, but we were nowhere close to an agreement and that's what caused my strong reaction. Again, I'm sorry. I didn't mean to be offensive. What I wanted to highlight was this: Somewhere, some project apparently build RCP apps for platforms that are not normally supported by the Eclipse IDE. You defend exposing launchers for those platforms in the form of p2 requirements and then omit the same launchers from the repository. That seems like a subjective decision to me and I find it hard to digest. The two reasonable approaches from a repository consumers point of view would be to either remove the requirements or to provide the missing bundles. I tried the proposed seven steps. The build went smoothly and I produced a build that contained the expected launchers. There are however still several issues that needs a resolution. 1. The repository that I produce must contain the new launchers. This means that although I'm not the maintainer of the source for them, I now must take on a responsibility for tracking bugs and publish fixes. 2. The launchers get a version with a timestamp qualifier denoting the time when the bundle was built. There's no correlation with a CVS tag and hence no exact correlation with the source that was used when building the launcher host bundle. Solvable, I'm sure, but it requires more effort. 3. The result is not signed by Eclipse so I need to add that to my build to get these fragments on par with other launcher fragments. Luckily, I'm a committer with access to the Eclipse signer so for me it would be possible to actually do that but for most people that doesn't apply. (In reply to comment #16) > The way I perceived it, you closed this bug in the middle of a discussion. You > had your own conclusion, sure, but we were nowhere close to an agreement and > that's what caused my strong reaction. Again, I'm sorry. I didn't mean to be > offensive. The bug was closed because the request was to remove platforms. If it wasn't clear in comment #1, the feature is used in PDE's support for cross platform builds. These builds are not hard and removing the fragment greatly increases the difficulty of doing them. The fact that it is not hard was repeated again in comment #10. Also in comment #10 was this: > You can ignore everything else, the fundamental point is this: > > Removing these fragments do not help you in general. You remain broken on > missing IUs that are not actually required. There is a world of content with > many different reasons for having filtered requirements that are not available. > You can't go out to every content provider and ask them to remove dependencies > because you don't happen to need them and your builder doesn't respect the > filter. Comment #12 was offensive because it very clearly directly contradicts and dismisses my comments and made it clear that you were disregarding everything I said. Going forward this is what I see: 1) To be nice and helpful Equinox should provide a p2 repository containing all the fragments. 2) Buckminster has a general problem with filtered requirements that are not present. This should really be addressed there, any change from Equinox is really just a work around to a specific instance of this problem. 3) I am against removing functionality to work around (2). Equinox should not remove the fragments because they support cross platform development in PDE and I am against restructuring the feature in ways that would require enhancements from PDE in order to maintain the current level of functionality. As a meta comment, the focus here has been on the wrong thing from the beginning. Instead of arguing for removal, this would have gone better as a new bug requesting the addition of the fragments to a p2 repo. The launcher is a huge context switch, there is not a lot of time available for it. Conflating this bug with other things that belong to PDE, p2 or others does not help identify the path forward. As another meta-comment, I can understand that folks have frustrations around this. We all do. Because these are such longstanding issues and they cross so many areas of responsibility it is understandable that discussions get a bit heated, but we should get beyond that. As a consumer of Eclipse build systems, what i can say is that it is really difficult for mere mortals to deal with. The simple issue for me is that I want to build an RCP application in an automated way and I have to jump through a constantly evolving set of landmines to do it. With respect, I don't find the 7 step PDE process the least bit "trivial". ;) Frankly, I barely understand a lot of this thread, all I know is that I have an automated build that was working perfectly well and it broke. And again, speaking purely as a consumer, it doesn't matter to me whether the "blame" lies with Equinox, Platform, Buckminster, PDE, p2, build, Webmaster, or whatever. As I think Andrew's comment supports, a downside of the project structure is that with the best of intentions these kind of issues that implicate various systems get stove-piped especially when people are dealing with local concerns. I don't know what to do to get beyond that, but it would be worthwhile giving it some thought. What I do know is that Thomas and the Buckminster team along with the Maven/Tycho folks are the only two groups that are trying to do anything concrete about to insulate us from PDE, etc. so my $.02 is that wherever this particular issues heads we try to keep that in mind. (In reply to comment #18) > The bug was closed because the request was to remove platforms. I never requested removal in this bug. As I wrote in comment #8, there are two ways of fixing this. My concern is about consistency. The releng bug 337026 (referenced in Kim's first comment) was about removal. > The fact that it is not hard was repeated again in comment #10. Also > in comment #10 was this: > > You can ignore everything else, the fundamental point is this: > > > > Removing these fragments do not help you in general. Again, I didn't request removal. Adding the fragments will do just as well. > Going forward this is what I see: > 1) To be nice and helpful Equinox should provide a p2 repository > containing all the fragments. That will certainly resolve this issue. > 2) Buckminster has a general problem with filtered requirements that are not > present. This should really be addressed there, any change from Equinox is > really just a work around to a specific instance of this problem. I agree about the concerns and responsibilities. The fundamental problem is that today there's no way to get a canonical list of supported launchers. A smaller problem, yet annoying, is that we (and everybody else) must change the list of filters to keep up with inconsistencies. This is a general problem caused by inconsistencies and not a Buckminster problem. It will be solved in a good way by (1). Buckminster users encounter this problem head-on because we tend to add as little redundancy as possible to the build system. Ideally, we'd like to just build based on what's there without adding any artifacts whatsoever. That's why this canonical list is especially important to us and also why we get hit by the inconsistencies. As Miles pointed out, we do have a workaround (said filters) and we would be really happy to get rid of it. I'm very pleased if we can consider the launcher feature the artifact that provides this list and the platform repository a trusted source where they can be found. > 3) I am against removing functionality to work around (2). Agree. If this is the best solution for you, then it's the best for me. Whether you decide to remove the dependencies or add the missing fragments doesn't matter to me. That decision is more about what the Equinox team want included in their deliverable. > As a meta comment, the focus here has been on the wrong thing from the > beginning. Instead of arguing for removal, this would have gone better as a > new bug requesting the addition of the fragments to a p2 repo. I agree. And I'm sorry if I the focus was unclear. I tried to make a point that repository consistency was the important thing, not removal. > The launcher is a huge context switch, there is not a lot of time > available for it. Conflating this bug with other things that belong to > PDE, p2 or others does not help identify the path forward. I agree. There is one benefit outside of PDE that could be exploited once this bug is marked as fixed. It will deserve a bugzilla of its own and that is whether or not the launcher list in the org.eclipse.rcp feature should be removed in favor of an inclusion of the executable feature. This would of course elevate the executable feature from its current status as a "build only" feature, but would that cause any harm elsewhere? The benefits would be that there would be only one canonical list of launchers and nothing special about the executable feature. We have workarounds so this is not blocking. > Regarding the Indigo, the executable feature is not part of the contribution > there. It is a separate feature for use in a target platform and is not meant > to be installed by the user. It does seem to have slipped in to being contributed there in Indigo. Indirectly, by being included in org.eclipse.equinox.sdk? Is that a change in M7? (This hasn't shown up before, for M6, M5, for example). This Indigo repo specific problem is covered in bug 344048. In the b3 aggregator we do list the exact configurations we aggregate ... I'm not sure why that's not sufficient to "filter out" these requirements. A limitation or bug in aggregator? That just hasn't show up before, for some reason? (In reply to comment #22) > In the b3 aggregator we do list the exact configurations we aggregate ... I'm > not sure why that's not sufficient to "filter out" these requirements. A > limitation or bug in aggregator? That just hasn't show up before, for some > reason? Did this show up as a result of using a new version of the b3 aggregator? (In reply to comment #23) > (In reply to comment #22) > > In the b3 aggregator we do list the exact configurations we aggregate ... I'm > > not sure why that's not sufficient to "filter out" these requirements. A > > limitation or bug in aggregator? That just hasn't show up before, for some > > reason? > > Did this show up as a result of using a new version of the b3 aggregator? No. Based on Kim's comments in bug 344048, it may show up now since current aggregation file uses specific version of M7 repo for rcp product: <repositories location="http://download.eclipse.org/eclipse/updates/3.7-I-builds/I20110425-1800" description="Eclipse and Equinox 3.7 release"> <products name="org.eclipse.rcp.sdk.id" versionRange="3.7.0.I20110425-1800"/> </repositories> whereas previous, used a "general", "high level" composite which would have included "old" versions of the fragment ... if I'm reading it right. <repositories location="http://download.eclipse.org/eclipse/updates/3.7milestones" description="Eclipse and Equinox 3.7 release"> <products name="org.eclipse.rcp.sdk.id" versionRange="3.7.0.I20110310-1119"/> </repositories> Note the different location URLs: http://download.eclipse.org/eclipse/updates/3.7-I-builds/I20110425-1800 http://download.eclipse.org/eclipse/updates/3.7milestones So ... is that the "fix" ... or a work around? *** Bug 346056 has been marked as a duplicate of this bug. *** My issue (Bug 346056) was just marked as a duplicate of this, but most of the discussion in here revolves about the no longer being provided motif builds. However these were replaced with the newer GTK builds for which the SDK is being actively build. While I certainly sympathize with people trying to run builds, I would like to point out to those not reading every duplicate, that the current staging and Indigo repositories (so RC1 and M7 respectively) are broken with regards to _ALL_ AIX and HP-UX plugins. So will not get any launchers, no SWT, nothing at all and p2 will bark at you in the update, install new version, target platform and p2.mirror tools. As far as I see right now it is not possible to build or distribute an RCP application for AIX or HP-UX without manually getting the delta pack or platform SDK zip and adding them into 'insert random company product' shipping repository or online repository. I suspect this will also present problems for people with an SDK on one of these platforms wanting to do an update. I've seen that many builds are using the Eclipse milestone repository. This repository is a composite of all milestones. Because of that, those builds will succeed in finding the removed fragments (they are present in M3 and backwards). My concern now is that when 3.7 is released, all milestones will be cut off. When that happens several builds will break. (In reply to comment #18) > Going forward this is what I see: > 1) To be nice and helpful Equinox should provide a p2 repository containing all > the fragments. Any idea when this repository would be ready for consumption? I'd also need these fragments in my cross platform build. Has the new bugzilla for this already been created as recommended? Does this bug need to be restricted to committers only? No, fixed that. I would like to revisit this and remove old fragments for obsolete platforms (motif, 32-bit ppc, wpf) from the launcher feature. We haven't been building Eclipse platform or SWT on these architecture/os/ws permutations for at least a couple of years. We have no way to test these platforms and we have no confidence that they even still work. We have no people interested in helping maintain these fragments, and no evident demand for them. This has also caused many problems for downstream build systems, and although we can argue about who has the right semantics, it clearly creates work for others having these obsolete fragments in there. If it turns out there is indeed demand for these fragments and interest in helping maintain them, we could add them back to the feature, and add the corresponding fragments to our repository at the same time to maintain consistency. Created attachment 223536 [details]
Patch
Bogdan can you review this patch? Patch looks good. Checked it, compiled it and committed it. Closing. |