Bugzilla – Full Text Bug Listing
| Summary: | [repo] [ui] Need better way to model the various UM associate sites | ||
|---|---|---|---|
| Product: | [Eclipse Project] Equinox | Reporter: | Susan McCourt <susan> |
| Component: | p2 | Assignee: | John Arthorne <john.arthorne> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | P3 | CC: | david_williams, eclipse, greg, john.arthorne, mik.kersten, nboldt, pascal, samuelwu, simon_kaegi |
| Version: | 3.4 | ||
| Target Milestone: | 3.5 M5 | ||
| Hardware: | PC | ||
| OS: | Windows XP | ||
| Whiteboard: | |||
| Bug Depends on: | |||
| Bug Blocks: | 236025, 236485, 242396, 252443, 257773 | ||
|
Description
Susan McCourt
If we don't have time to fix this for 3.4.1, at the least we should consider including a link on the "No updates were found" dialog that leads the user to the manage sites dialog or even a more specific flavor of that dialog which shows the disabled repos only and allows the user to enable them and check again. *** Bug 236531 has been marked as a duplicate of this bug. *** Specific use case from the duplicate bug:
1. Download and unzipped the attached update site project
2. Install the feature Parent feature
3. Parent feature contains the following URL
<url>
<update label="my update site"
url="http://tpftoolkit.torolab.ibm.com/tpftoolkit/updates34p2/"/>
</url>
4. Child feature contains the following URL
<url>
<update label="my update site"
url="http://tpftoolkit.torolab.ibm.com/tpftoolkit/updates34/"/>
<discovery label="updateSiteName"
url="http://download.eclipse.org/eclipse/updates/3.4"/>
<discovery label="secondaryUpdateSiteName"
url="http://download.eclipse.org/releases/ganymede"/>
</url>
5. Expected behavior: the update site URL in parent feature shows in the
available software site and the ones of the child feature don't
6. Result: none of them shows
Both of them are found in the manage sites and you have to manually enable it
to add it to the available sites
On the feature properties pages of these two features, it's defined to let the
parent feature handle the update. It doesn't work in this case.
More information:
This request comes from the fact that a product may depend on specific version
of the common component. The product may be broken if the common components are
allowed to update themselves. The update to any of the common component has to
go through the main product feature.
There is no way to achieve this target in the current P2. I can't image how can
a user go through the huge list of all the update site URLs and pick up the
right one.
I've been thinking about this issue a bit, and would like to propose a solution to this problem that completely separates the generic disabled/enabled repo model from the issue of the content->site relationship. In previous discussions we've said that one advantage of p2 core is that it doesn't define IU->site relationships so that products can strictly control how site content is updated rather than letting producers of components do so. Yet to remain backward compatibility to UM, we read these URL references and hide them in the disabled sites list. The confusion placed upon the user is just one problem. In the spectrum of "unmanaged" vs. "highly managed" installs, we are not serving either community. "unmanaged/producer view" - users want to be able to do a one-click upgrade to the latest and greatest of everything, *as defined by the initial producer of the software*. Upgrading from other sites is a more advanced feature and should be made explicit by the user. "controlled/managed view" - product producers or installation admins want the ability to ignore the producer's view of the world and substitute new update sites (or none at all). This includes determining whether the user can be involved in updates at all. I think we need to allow for the following: - site relationships defined in the IU need to be maintained in the IU's properties (or at least need to be retrievable through API, regardless of where stored). These sites should *not* be added to the repository manager (not disabled, not enabled). - the existing disabled/enabled repo mechanism should be used only for truly disabling a site (for example, it is off line and the user doesn't want to keep retrying it). This would truly be a user mechanism for turning repos "on" and "off" and in fact it's an advanced concept from a user point of view. - site references defined in another site should be discovered but not automatically added by the repo manager. There should be API to get a list of discovered sites (from the repo manager point of view). Bug #218534 discusses the presentation of these discovered sites to the user. - we need policy/preference objects that allow the product to control how this site information is handled. (Note I do not mean user preference). Things that need to be controlled: ** are an IU's references to update sites considered when updating the IU? ** are a site's references to discovered sites shown to the user (if not then the discovery UI would not even be available)? ** what initial sites are in the available software list ** can users add sites at all to the site list (if not then the available software list should not even be shown) With this information, the UI can build the appropriate list of repos (using existing ProvisioningContext) to be consulted for each operation. "unmanaged/producer view" - installing something involves resolving against the list of enabled sites (as today) - checking for updates involves building a list of all the sites referred to by the various IU's. If no updates are found, user could be prompted to select other sites to check (or this could be controlled by a visible preference) "controlled/managed view" - all operations are resolved against the list of enabled sites - user may or not have the ability to control, or even see, this list I'm less concerned with modeling UM behaviour than with modeling real user requirements. I.e., I wouldn't want to revert back to UM behaviour here just for sake of compatibility. On the other hand, if we have made some scenarios that were possible in UM difficult or impossible in p2, we need to make improvements. The example in comment #3 is a case where someone redistributing an IU wants to define where updates to that IU should come from. I think that can almost be done today in p2 by creating your own p2 repository that does not refer back to the original producer's repository. The missing piece here is that there is no good way at build time to seed an install with a set of initial repositories. Ideally the director app would allow someone to add a repository to a profile at build or install time. Another use case that isn't well satisfied today is the aggregate repository scenario. UM associate sites allowed someone to coalesce a group of repositories into a single repository from the user's point of view. When the site was removed, all associate sites were removed as well. p2 repository references don't allow for this, because the association is not tracked across the repository lifecycle. When a repository is loaded, all of its referenced repositories are added to the repository manager. However, when that repository is removed all of its referenced repositories remain. The use case I'm most worried about, because I think we don't handle it well, is the simple "unmanaged" case. An Eclipse SDK end user installs an IU (let's call it IU "A") and when they check for updates for A, they want to check whatever update site the producer of A designates. Today, they have to do that by adding the site manually after they install the IU (or if they are already familiar with the disabled sites metaphor, they could disable the site in the site list.) The updates to A won't be found during an update check without the user having to take some explicit action, whereas before, the updates were "just found" without the user doing anything. We can call this a compatibility problem, since the site ref is currently defined in a feature, but it is a legitimate use case and is a usability regression from UM. How should a producer in the p2-world specify the preferred update repo for a downloaded IU? Yes, we need to give products a way to override this, but we also need to allow the producer-driven use case to function easily for the user. This is an install-time issue, not just a build-time issue. In this scenario, how was A installed in the first place? There are two possibilities: - A was installed in the profile from the beginning. In this case, whoever crafted the profile containing A should be able to set the list of available repositories. I.e., the person or build process that setup the profile has the ability to specify the "A provider" for that install. This is not actually easy to do today, which is the current limitation I was referring to above. - The user installed A. In this case the repository they obtained A from is their official "A provider", and will control the flow of updates to A. I.e., new versions of A would be published into that same repository where the user obtained A from in the first place. >new versions of A would be published into that same repository where the user
>obtained A from in the first place.
Is this a reasonable assumption? I was thinking of sites where the initial install and the updates should come from different places. For example, a stable release, nightly build, and milestone build are all offered on the same site and each IU is configured to get its updates from the correct site.
If this is a fringe use case then I guess it is primarily a build-time issue.
I think this is a good example of why the old feature->site association didn't work well. In the UM model a feature could only specify one site URL where all updates must come from. Say you have a release site, a milestone site, and an integration site. Which site you want to get updates from depends on how close the bleeding edge you want to live. A casual user might only want to consume releases, where committers want to consume integration builds. Thus different communities want to have a different "update provider". We can't put those associations directly on the IU because the same IU is promoted from integration->milestone or integration->release when the time comes. >I'm less concerned with modeling UM behaviour than with modeling real user >requirements. I.e., I wouldn't want to revert back to UM behaviour here just >for sake of compatibility. On the other hand, if we have made some scenarios >that were possible in UM difficult or impossible in p2, we need to make >improvements. Here's an example scenario where associate sites helped the user and we don't have a corresponding p2 answer. Suppose the user wants to install A. They add the site for A, find A, and choose install. A requires B, but user has never added site B. In UM world, A had a site reference to B's site, so B could be found. In the current p2 implementation, B won't be found because its site was added as a disabled site reference. Right now the best answer we have is to report the missing B and potentially guide the user to "manage sites." (see bug 232107). But that list may be quite large and it may not be obvious to the user which site to choose. The other problem I see in the current implementation is the performance issue. I understand John's comment #7 and comment #9. But without IU/site relationships, we have to check every single repo when we are looking for updates for any one IU. If we had a preferred repo for updates for an IU, we could check there, and always expand the search if it was not found there. >We can't put those >associations directly on the IU because the same IU is promoted from >integration->milestone or integration->release when the time comes. I assume that in this case we expect the user to add the new update site (milestone site) and delete or disable the old one (integration site), correct? All I'm suggesting is that instead we maintain an IU/site relationship on behalf of the user. The user would still have to make the changes, but it would be at the level of the IU, not a global pool of repos. Couldn't this information be kept in an IU profile property? That seems to me the right place to keep the information, because it depends on the user, not the IU from the producer point of view. Since we have so many bugs about network problems, broken sites, etc. etc., it seems worthwhile to reduce unnecessary contact of sites. If the site for one of my add-ons is broken and all I wanted to do was update releng tools, there's no need to suffer the performance problem and missing repo error. Yes, the user can disable the site, but again, this is more management we are putting on the users shoulders that UM did not force them to do. > All I'm suggesting is that instead we maintain an IU/site relationship on > behalf of the user. The user would still have to make the changes, but it > would be at the level of the IU, not a global pool of repos. Couldn't this > information be kept in an IU profile property? That seems to me the right > place to keep the information, because it depends on the user, not the IU from > the producer point of view. If we maintained a relationship between an IU and where to look for updates, such that anything visible on that site might be considered an update, it might solve some of the issues described in bug #245299 and bug #244872. I have started collecting some notes on this problem on the wiki: http://wiki.eclipse.org/Equinox/p2/Repository_Association Feedback most welcome. Susan, I noticed while re-reading this thread that there was some confusion about how p2 1.0 handled Update Manager's site association concepts, so I added some detail about this on the wiki. In particular, associate sites and feature update sites are currently added as *enabled* repositories, so no user action is needed to make them be searched. Only "discovery" sites are added as disabled sites. > Susan, I noticed while re-reading this thread that there was some confusion
> about how p2 1.0 handled Update Manager's site association concepts, so I added
> some detail about this on the wiki. In particular, associate sites and feature
> update sites are currently added as *enabled* repositories, so no user action
> is needed to make them be searched. Only "discovery" sites are added as
> disabled sites.
Thanks. This is explained well in the wiki. I didn't realize this until it came up recently in a different bug. For the update case, I see the main issue as one of performance, as you point out in the wiki.
Also, from a user perspective, the performance problem is often stated not as a performance problem, but one of confusion. "Why are you contacting the network when I know my feature is updated locally?" This issue comes up frequently for developers who are exporting sites and testing upgrades, etc.
(In reply to comment #10) > Suppose the user wants to install A. > They add the site for A, find A, and choose install. > A requires B, but user has never added site B. > In the current p2 implementation, B won't be found because its site was added > as a disabled site reference. I have this exact usecase, twice. a) The Ganymede SR1 site is missing one of BIRT's features (the BIRT-WTP integration feature) so to install something that depends on that feature, the user must add both the product's update site and BIRT's 2.3 update site (in addition to the Ganymede site). I've tried this: <site pack200="true" associateSitesURL="http://download.jboss.org/jbosstools/updates/nightly/trunk/associateSites.xml"> and this: <associateSites> <associateSite url="http://download.eclipse.org/releases/ganymede/" label="Ganymede Update Site"/> <associateSite url="http://download.eclipse.org/birt/update-site/2.3/" label="BIRT 2.3 Update Site"/> </associateSites> But the best I can get is that the site is prepopulated in the list of available sites, but NOT enabled. b) The same scenario presents for installing PDT [1], because it depends on Ganymede + the DLTK 1.0 site. Adding an associatedSitesURL would help pre-pop the list with the DLTK site, but still gets us no closer to "add one site and have it grab all its sister sites" in one step. [1]http://wiki.eclipse.org/PDT/Installation#Eclipse_3.4_.2F_Ganymede_.2F_PDT_2.0 --- Could the mechanism that allows me to import a bookmarks.xml file w/ sites -- and mark them enabled -- be used similarly when loading associated sites? Here's my bookmarks.xml for PDT install: <?xml version="1.0" encoding="UTF-8"?> <bookmarks> <site url="http://download.eclipse.org/releases/ganymede" selected="true" name="Ganymede Update Site"/> ... <site url="http://download.eclipse.org/tools/pdt/updates/2.0/interim/" selected="true" name=""/> <site url="http://download.eclipse.org/technology/dltk/updates-dev/1.0/" selected="true" name=""/> <site url="http://download.eclipse.org/webtools/updates/" selected="false" name=""/> ... </bookmarks> You could do something similar to control which ones are enabled/disabled by adding an attribute into <associateSite selected="true"> (if property omitted, default to false). If not via the associateSitesURL, could this functionality be done at the feature level [2]? For example, using <discovery selected="true"> ? [2]http://help.eclipse.org/ganymede/index.jsp?topic=/org.eclipse.platform.doc.isv/reference/misc/feature_manifest.html --- +1 for fixing this for Ganymede SR2, if possible. Nick, are you generating a p2 repository using that site.xml, or loading a "legacy" update site with only site.xml in the UI? Can you give a link to a site with associate sites that isn't working? Associate sites should be appearing as enabled, otherwise it's either a bug or you're not using it correctly. (In reply to comment #16) > Nick, are you generating a p2 repository using that site.xml, or loading a > "legacy" update site with only site.xml in the UI? Can you give a link to a > site with associate sites that isn't working? Associate sites should be > appearing as enabled, otherwise it's either a bug or you're not using it > correctly. Dunno if this is public or not... if not I'll try this for PDT and share that link: http://download.jboss.org/jbosstools/updates/nightly/trunk/ The site has: site.xml associateSites.xml artifacts.jar content.jar plugins/*.jar + *.jar.pack.gz features/*.jar + *.jar.pack.gz and even a very old-style index.html :) To install The whole JBoss Tools suite, start with Eclipse 3.4.1 "Classic" or JEE, then add the above site (it finds the p2 metadata and downloads the packed jars) and try to install the whole site's contents. It will fail on the BIRT install because it's missing one plugin. The solution is to enable the BIRT 2.3 update site, then try again. But because the assoc'd sites XML doesn't work, this is still a manual step. If I've done something wrong, please correct me, and I'll fix the XML. (Oh, and BTW, bug 257446.) Thanks! I've updated the site.xml per the example that Tasktop uses to associate mylyn and tasktop updates w/ a user's unique subscription update site. I've discovered two things: a) adding http://download.jboss.org/jbosstools/updates/nightly/trunk/ site does NOT add the associated BIRT site. b) adding http://download.jboss.org/jbosstools/updates/nightly/trunk/site.xml DOES add the associated BIRT site. This, furthermore, says two things: i) associated sites are only added if the URL is different from an existing one in the list. If the associated site already exists in the user's Manage Sites panel but is disabled, it will not be activated when re-adding it via an associated site reference. ii) associated sites are only read if the originating URL is NOT a p2-enabled site. The Tasktop site works; the jboss site.xml works (case b above). The jboss p2 repo, generated FROM the site.xml (case a above), DOES NOT. In fact, checking in content.xml from http://download.jboss.org/jbosstools/updates/nightly/trunk/content.jar shows that the associated sites are NOT included in that file. Would you like me to open two new bugs for these issues? i) is bug 241307, fixed in 3.5. Please enter a bug report for ii) and I will investigate. Some incremental progress has been made on this in M4: - New touchpoint instructions for adding/removing repositories. This allows any IU to associate itself with a repository location. The repository location will be added when that IU is installed - Ability to add touchpoint instructions via p2.inf on both features and products. - Composite repositories, which have some similar characteristics to associate sites in UM. Tying all this together, you can now add repositories at the product level, where they most belong. By removing this data from features and bundle it makes the features/bundles completely reusable for another publisher. Another publisher can take features from the producer, add them to their own custom "product" with references to their own custom repositories, and publish all of it to their own repository. No repositories from the original producer are leaked into the new publisher. I believe this properly enables the enterprise use cases that were served by the site policy files in UM. (In reply to comment #20) > Some incremental progress has been made on this in M4: > > - New touchpoint instructions for adding/removing repositories. This allows > any IU to associate itself with a repository location. The repository location > will be added when that IU is installed I assume the IU can remove the repo when it is uninstalled... > - Ability to add touchpoint instructions via p2.inf on both features and > products. > - Composite repositories, which have some similar characteristics to associate > sites in UM. > > Tying all this together, you can now add repositories at the product level, > where they most belong. By removing this data from features and bundle it makes > the features/bundles completely reusable for another publisher. Another > publisher can take features from the producer, add them to their own custom > "product" with references to their own custom repositories, and publish all of > it to their own repository. No repositories from the original producer are > leaked into the new publisher. I believe this properly enables the enterprise > use cases that were served by the site policy files in UM. > That sounds cool. So what we are missing, then, is the ability to be smart about where to check for updates for a particular IU? Or can the UI somehow use the touchpoint info on the IU to check for updates for an IU for one particular site? > I assume the IU can remove the repo when it is uninstalled... Yes, there is a "remove repository" action that can be run on uninstall. >That sounds cool. So what we are missing, then, is the ability to be smart >about where to check for updates for a particular IU? Or can the UI somehow >use the touchpoint info on the IU to check for updates for an IU for one >particular site? Yes, that is indeed the big remaining item that is missing. What I'm not sure of is how important this is, or even if we can get it right. Some of the work done to the UI workflows in 3.5 helps out with the performance. The deeper issue is that we're moving away from a model where a single repository contains the transitive closure of an IU's dependencies. So, an update on IU 'A' causes it to need a new version of 'B', we can't assume that the repo for 'A' will also contain 'B'. For example imagine each Galileo release train paricipant had their own repository containing only their bits. There is a composite repo that amalgamates the repos from each project into a single master repo from the user's perspective. None of the individual projects or their IUs have any knowledge of the master release train repository. To perform an update, you need to access not just the repo for individual IUs, but perhaps for IUs in other projects as well. So the update needs to have access to more repositories than a single IU knows about. Is this scoping of repositories accessible during updates "only" a performance issue, or are there other reasons why someone would want to do that scoping? > Is this scoping of repositories accessible during updates "only" a performance
> issue, or are there other reasons why someone would want to do that scoping?
>
The use case I keep hearing about is when at least one of the repos is local and user knows they are working with IUs from that repo, and the network is contacted. Particularly when they are disconnected. And the general surprise that the network would be contacted on uninstall.
I know that some of these are more related to how we build the list of repos to contact, but if the UI can't get any info about how IU's may have been related to repos then it can't be much smarter than "try everything" unless we want to change to a generic strategy like "try local" and then "try network"
Moving completion of this bug into M5. In the install wizard, there is now a way for the user to scope what they are looking at by repo. (This has been added so that user's can quickly type/paste/drag a repo, see only that repo's content, and install). Pascal and I were remarking that if a repo's references were remembered by the repo, instead of getting added to a global pool, that would be one way to scope an install. Of course scoping a search for updates would still require some way to look up the update site (and possibly that site's references) given an IU. *** Bug 229417 has been marked as a duplicate of this bug. *** I'd like to mark this resolved based on the progress described in comment #20. Also, all the blocking bugs have been fixed. The remaining issue of scoping the install is covered in other bugs such as bug 235923. I don't think we'll be addressing the scoping problem in 3.5. I agree this can be closed. Marking fixed. |