Community
Participate
Working Groups
Target definitions have been enhanced in 3.5 to support software site locations. A software site location allows software from an update site to be added to a target definition (which effectively retrieves all associated bundles in a runnable form for building, running and debugging). The current implementation uses the p2 planner to determine what bundles are downloaded and installed into the target. The planner is restrictive in that bundles are downloaded for a specific environment (OS, window system, architecture, locale), and only if all dependencies can be resolved. PDE would like to add two options to software site locations to ease these restrictions: (1) Allow the user to add software to target definitions even when all dependencies cannot be resolved. BENEFIT: Allow users to craft target definitions from incomplete repositories. Zipped repositories are produced by several projects and they will become more common. These repositories are generally incomplete, containing only the bundles that the producer is responsible for. For example, the Equinox SDK repository download contains only Equinox bundles but has dependencies on SWT which are not included. Enforcing all dependencies be met means developers cannot use the repository by itself to do server (non-UI) development. (2) Retrieve content for all available environments (i.e. all platform specific fragments) BENEFIT: Enhances the ability to build products for multiple platforms. The delta-pack provided with the SDK does not contain all platform specific bundles (only a subset deemed relevant by the SDK/RCP). This provides developers with a simple way to retrieve all platform specific content. EXTENT OF CHANGES: (1) Two new check boxes will be added to the software selection dialog used when adding a software site to a target definition. The check boxes will be labeled “Include all required software” (checked by default) and a sub-option “Include all available platforms” (unchecked and disabled by default – only becomes enabled when “Include all required software” is unchecked). (2) When the user disables dependency resolution (unchecks “Include all required software”), the p2 slicer is used in place of the p2 planner when retrieving content. When the user specifies all platforms, the slicer ignores platform filters.
We also need to "spoof a JRE IU" in the local bundle to pool since JRE's are not included in zipped repostiories. We don't intend developers to turn off dependency resolution when the only thing missing is a JRE. See bug 272750 comment#13 that provides a patch to spoof a JRE.
Created attachment 134486 [details] Screenshot of possible UI options Demonstrates how the provisioner UI could look with the extra options at the bottom of the page. By default the check box would be on, if the user unchecks it, the option to include multiple platforms would be enabled. If the user changes the value of this checkbox from the current value (when editing) or the default value (for the first time) a warning would be shown as the option applies to the entire target.
Do we want the option on or off by default? I would think off by default as most people that have been "using" the new functionality expect the platform specific stuff included. Anyone have an opinion on this?
(In reply to comment #3) > Do we want the option on or off by default? > > I would think off by default as most people that have been "using" the new > functionality expect the platform specific stuff included. > > Anyone have an opinion on this? > "include required software" should be enabled by default. When enabled the "include all environements" option should be disabled and should not be able to be enabled (grayed out). This is outlined in comment 0 by Darin :-)
Created attachment 134520 [details] Rough Fix This patch adds both the planner and slicer functionality to the IU bundle container. There are options on the container to set which one will be run (along with whether to grab multi-platform IUs). The UI looks like the screenshot attached previous and is hooked in. The patch has not been tested extensively and the unit tests will need to be updated and expanded. I haven't tested the multi-platform option at all. Playing with the option I am finding the slicer option more attractive than I did initially. While there are times where knowing what dependencies I am missing are useful, being able to just point to a repo and have the bundles added with no errors is quite useful. Feedback on the patch would be great, including some comments on the UI. Keeping in mind of course that this needs to be well contained for RC1.
Created attachment 134521 [details] mylyn/context/zip
One thing I noticed when using the slicer - when I don't specify "All available environments", no platform specific fragments get included. For example, org.eclispe.swt appears, but no fragments. Perhaps the p2 team can shed some more light on this? Is there a way we can use the slicer to retrive fragments appropriate for a specific environment? (I realize this is new/incomplete patch yet) These are the things I noticed that we need to do: * Add persistence of new attributes to the software site location (currently, if I close/re-open a target definition, it forgets the new settings) * We should still spoof a JRE IU in the local bundle pool
why do you need the JRE IU if the slicer is being used? It was only being added to satisfy dependencies on javax etc. but with the slicer these dependencies should not matter. related: the JRE IU does not have any artifacts so it should not be related to the bundle pool.
(In reply to comment #8) > why do you need the JRE IU if the slicer is being used? It was only being > added to satisfy dependencies on javax etc. but with the slicer these > dependencies should not matter. > related: the JRE IU does not have any artifacts so it should not be related to > the bundle pool. It's true - when using the slicer we don't need a JRE IU. This is a simpler story.
Created attachment 134669 [details] Fix v2.0 Adds persistence, some unit tests, adds specific target platform properties to slicer. 1) I am not 100% confident of the wording on the dialog (see first screenshot or try the patch). For those who plan to use this feature, pelase provide feedback. 2) I am getting some odd artifact missing errors when using the slicer. will attach screenshot next. 3) No JRE IU is being generated still.
Created attachment 134674 [details] Screenshot of artifact problem Happens when using the slicer. If 'include all platforms' is turned on, I get an error like this for every platform. If the option is off, I just get one error for my current platform.
On the Mac the options for "Included Software" get clipped off a the bottom such that the option for Include all platforms gets cut off. If I resize the dialog then it becomes visible.
In test this do you recommend we apply the latest patch from this bug and the one mentioned in bug 272750 comment#13 for the JRE IU spoofing?
Just this patch - the intent is to *not* spoof a JRE - instead view it as a missing requirement. All software that can be installed from an update should have all dependencies met (else it would not be installable?). Software in a zipped repo is generally incomplete, and thus the "Include required software" checkbox should be "off".
Created attachment 134701 [details] Updated Patch 3 additional changes: 1) Improved wording in the UI 2) Fixed bug where new containers always get default settings 3) Made the persistence more flexible for the future (storing mode string instead of boolean)
Created attachment 134708 [details] EditContentDialog (In reply to comment #15) > Created an attachment (id=134701) [details] > Updated Patch > > 3 additional changes: > > 1) Improved wording in the UI > 2) Fixed bug where new containers always get default settings > 3) Made the persistence more flexible for the future (storing mode string > instead of boolean) > This last patch is improving. A couple of observations: 1) if I leave a target file open and restart my workspace then I get the following in my log !ENTRY org.eclipse.equinox.p2.repository 4 0 2009-05-06 16:49:08.144 !MESSAGE ProvisioningEventBus could not be obtained. Metadata caches may not be cleaned up properly. And I get an error in the locations view for the target saying "Profile registry service not found". If I close the editor and reopen the target definition then the error goes away. 2) If I return to the location and edit it and select different features and deselect others the icons for the unselected features changes to not have any colors. They remain this way even if I restart. Something is getting persisted about these features even when they are not selected. Note that the bundles from the features are not actually included in the content tab of the target definition.
(In reply to comment #16) > 1) if I leave a target file open and restart my workspace then I get the > following in my log > !ENTRY org.eclipse.equinox.p2.repository 4 0 2009-05-06 16:49:08.144 > !MESSAGE ProvisioningEventBus could not be obtained. Metadata caches may not be > cleaned up properly. > And I get an error in the locations view for the target saying "Profile > registry service not found". If I close the editor and reopen the target > definition then the error goes away. This problem does not appear to be related to the new patch/code. I can reproduce the problem using the existing (unpatched) code. I will open a new bug for this.
Should we be allowed to add two different software site locations with different options set for "Include required software". I was trying to mix and match the Equinox SDK from the Galileo staging area at: http://download.eclipse.org/releases/staging with the RCP SDK repo from the latest I-Build. First I configured http://download.eclipse.org/releases/staging staging area which includes the Equinox SDK repository from M7 under the EquinoxRT target platform components category. I then installed the Equinox SDK into my target with the "Include required software" enabled. This seemed to work, except it downloaded the swt host without the fragment. Next I downloaded the RCP source p2repo from the latest I-Build and added it as a local archived repo. I then installed both the RCP features from this repo (including source). This time I disabled "Include required software". Should I be able to do this. I am not sure if we allow mixing the two types of repo options into the same target definition. This took a very long time to complete but it seem to finish eventually. But I had duplicates in my content tab for any bundles included in both the Equinox SDK repo from the Galileo staging area and my downloaded RCP repo. I also noticed that I could not save the target definition. It would never get rid of the dirty flag. If I closed the target definition file it will not ask me to save the file and if I reopen it then the last repo location I added will be missing.
(In reply to comment #16) > 2) If I return to the location and edit it and select different features and > deselect others the icons for the unselected features changes to not have any > colors. They remain this way even if I restart. Something is getting > persisted about these features even when they are not selected. Note that the > bundles from the features are not actually included in the content tab of the > target definition. > I have the feeling this is by design and is supposed to indicate that the bundles from these features are available in the bundle pool PDE is managing. Is this true?
(In reply to comment #18) > Should we be allowed to add two different software site locations with > different options set for "Include required software". You cannot mix modes for a single target. When you do, a warning appears at the top of the wizard saying that it will effect all software sites in your target definition. > I also noticed that I could not save > the target definition. It would never get rid of the dirty flag. This is another existing bug to be fixed in RC1 - see bug 274694.
(In reply to comment #19) > (In reply to comment #16) > > 2) If I return to the location and edit it and select different features and > > deselect others the icons for the unselected features changes to not have any > > colors. They remain this way even if I restart. Something is getting > > persisted about these features even when they are not selected. Note that the > > bundles from the features are not actually included in the content tab of the > > target definition. > > > I have the feeling this is by design and is supposed to indicate that the > bundles from these features are available in the bundle pool PDE is managing. > Is this true? Yes, this is the p2 UI showing what is already installed locally.
Created attachment 134802 [details] Proposed Fix Requesting PMC approval to apply this patch. The fix, while it requires changes to both the model and the UI is reasonable well contained. It only affects the new p2 provisioner wizard. To access this wizard, you must create a new target definition (File > New > PDE > Target Definition), in the editor on the Location page, hit the Add button. On the first page of the opened wizard select "Software Site". The options are available on the bottom of the second page. Changes: 1) UI gets a new group with two options (EditIUContainerPage). 2) Model (IUBundleContainer) gets two new settings for the new options. 3) Persistence (TargetDefinitionPersistenceHelper) updated to save the new options. 4) Persistence tests updated, new test added to check setting of new options on model.
Can the PMC please review the Proposed Fix for RC1 inclusion.
Note that this patch does not apply cleanly to HEAD. It has fix from bug 274852 in it which was applied to head already. This makes the patch hard to apply to head without getting the previous version of EditIUContainerPage
(In reply to comment #20) > (In reply to comment #18) > > Should we be allowed to add two different software site locations with > > different options set for "Include required software". > > You cannot mix modes for a single target. When you do, a warning appears at the > top of the wizard saying that it will effect all software sites in your target > definition. Just to clarify, going from include required to don't include required changes all sites in the target but does not have any ill-effects. Going the other way however changes all sites and may cause things to be removed from your target. > a local archived repo. I then installed both the RCP features from this repo > (including source). This time I disabled "Include required software". Should > I be able to do this. I am not sure if we allow mixing the two types of repo > options into the same target definition. This took a very long time to > complete but it seem to finish eventually. But I had duplicates in my content > tab for any bundles included in both the Equinox SDK repo from the Galileo > staging area and my downloaded RCP repo. Has the duplicates issue Tom mentions here been addressed in the latest patch?
(In reply to comment #25) > Has the duplicates issue Tom mentions here been addressed in the latest patch? The duplicates problem is not solved. The target will need to delete/recreate its profile when the resolution mode changes from plan to slice or visa versa, since the contents of the profile will change (since the resolution algorithm has changed). As well, all software site containers in the target need to be re-resolved when this happends.
Created attachment 135013 [details] Safer fix This fix eliminates the problems of having multiple sites. It disables the option to switch between the planner/slicer if there is already another site in the target.
(In reply to comment #27) > Created an attachment (id=135013) [details] > Safer fix > > This fix eliminates the problems of having multiple sites. It disables the > option to switch between the planner/slicer if there is already another site in > the target. > I was meaning to make a suggestion for this type of fix. Darin and I had discussed this possibility this morning. I will test this latest patch out. I think this will make the solution much more safe for the 3.5 release, otherwise the user could get into some very strange situations that made it hard to know what is going on. More advanced improvements can come in 3.6.
Tom, if you also use the patch in bug 275401, you will get an even better experience :-) It ensures the target updates when an existing software site location is removed from the target.
My recommendation is to release this patch for RC1. It is still possible to get duplicate bundles in your target when using the slicer against multiple repositories that contain different versions of the bundle. But this is similar to the case where you unzip a bunch of different versions into a directory and set the directory to be your target. In my view using the slicer is equivalent to asking PDE to just import all things available in the repository into my target regardless of dependencies and versions. If that is the case then this is working as designed.
(In reply to comment #30) > It is still possible to get duplicate bundles in your target when using the > slicer against multiple repositories that contain different versions of the > bundle. But this is similar to the case where you unzip a bunch of different > versions into a directory and set the directory to be your target. Yes, this is what I would expect - if different versions of the same plug-in are present, you will get both (a planner might be able to determine that they are compatible and only include one, but a slicer doesn't have that complexity).
I did some light testing and it looks pretty good. All my testing was done with Include Required *OFF*. Some comments: - when you deselect "Include Required software" there is a warning at the top "Changing this option will affect all sites in your target definition". However, if you don't happen to notice that warning's appearance in connection with the deselection it is out of context and you don't know what option is the problem. - Once the Include Required option was deselected it was not possible to turn it back on or change the multiplatform option. I understand that switching back and forth is problematic but this setup is a bit confusing for users. They were able to tweak the boxes previously but now cannot. Perhaps some dialog workflow that explains the perils of switching modes and why there is no going back... - It is also not clear that switching on and off multi platform mode need be target-global or permanent. It could/should just affect the one operation at hand. It is pretty easy to see scenarios where you want all platforms for one thing but then you go to add TPTP which has large wads of stuff for different platforms and doing all platforms will take days to download. - Adding the RCP SDK from the http://download.eclipse.org/eclipse/updates/3.5-I-builds/ repo understandably took some time. During this time progress was stuck at 85% saying "Resolving Target Definition" but my download bandwidth use was ~1MB/s. Clearly downloading all the stuff. Not sure if this is related to this patch and the change in workflow affecting progress.
my notes in comment 32 were more feedback than some sort of blocker for this fix. The target support overall and these changes are very high value. In looking at the patch I see it as a number of UI/workflow tweaks and the code that calls drives the slicer function. The UI tweaks are reasonably safe/sane. There could be better flows in a few places but that can come in the next release. The slicer function is basically just calling into p2 at a different point so is not inherently more or less safe/dangerous. it would be good to have the p2 team take a look over that part as part of the approval process. Barring issues that come up there I am happy to approve this for RC1. Irrespective I thank the PDE team and others invovled in putting the target provisioning support together and doing these last minute changes. As I said at the beginning, this is high value stuff.
Created attachment 135169 [details] Improved fix By fixing bug 275401 and bug 275458 we can do better in the UI. In this patch the controls are no longer disabled. Changing the settings will put up a warning if there are other site containers that will be affected. Because of the changes to those two bugs, pressing finish with the changed settings will cause the target definition to re-resolve all of it's sites properly. This patch also fixes one issue which is that the "include all platforms" setting must be the same for all sites in the target definition in the same way that the planner/slicer setting must be.
two clarifictions: - the new patch is much smaller than the previous one. Is it a replacement or supplement? (the previous one is not marked as obsolete) - When you say that it also "fixes one issue" are you saying that now the "include all platforms" option now must be teh same for all sites or no longer needs to be the same. In comment 32 I pointed out that requiring it to be target global (the same for all sites) seemed like overkill. So, a "fix" would be to allow it to differ between sites.
Created attachment 135215 [details] Correct improved fix My apologies, I missed part of Darin's work in the previous patch, which was causing hard to follow resolution errors when using multiple sites. This patch fixes the problems and merges all the patches I had floating around. The one patch was excessively large because CVS decided that every line had changed in a file (probably due to line endings). We tried to get the "include all environments" setting working on a per container/site basis, but it is difficult. We would have to store a list of per-site settings on the profile and persist them so the profile knows that something has changed and needs to be thrown away. After some discussion Darin and I think it is best that we just make the setting global (per target) for 3.5.
Pascal, if you are going to review the slicer code, the place to start is IUBundleContainer#resolveWithSlicer().
Great. Thanks Curtis. So this patch "Correct improved fix" is all that is needed? Re the global vs site setting for include all platforms. Understood. These things are never as easy as they appear. For 3.5 it is at least consistent to say that these settings (include *) are global to the target.
(In reply to comment #38) > Great. Thanks Curtis. So this patch "Correct improved fix" is all that is > needed? Yes - this patch is all that is needed for the "slice & dice" target provisioning. I just verified the patch applies cleanly against HEAD.
The patch is still big, but looks reasonably clean. I don't know the algorithm well enough to know whether the "// TODO: missing bundle" I found is something that needs to be fixed or not.
The "TODO" is for the rare case when we've provisioned bundles to a local repository, and then when we go to resolve the actual bundle locations (URI's) from the local repository, a bundle is missing. We can return a "resolved bundle" with a status code to indicate that it is missing in this case. In practice, I've never seen this happen, but it can be handled.
I believe this enhancement is sufficiently important that we should proceed.
Applied!
Tests pass, feature works. Marking as Verified. If you have the opportunity, please test this feature as part of your RC1 test pass.
