Bug 305642

Comment 1 Martin Oberhuber 2010-03-12 03:48:43 EST

Acceptance Criterion

7. The solution must be officially supported and documented.

Comment 2 Alex Blewitt 2010-03-12 04:21:10 EST

I'd just like to add to the list:

8. Ability to have bundles installed from more than 2 locations. 

Currently, the Eclipse dropins only permits 'up to two' by defining an 'alternate' site (the installed location's dropins is still consulted. For example, on Mac OS X, a typical application could consult:

/Applications/Foo.app/Contents/Bundles (per-app specific install)
~/Library/Application Support/Foo/Bundles
/Library/Application Support/Foo/Bundles
/Network/Application Support/Foo/Bundles

This kind of setup is impossible with P2 at the moment, though you can emulate the 'personal folder' use case.

This also permits a setup whereby an Eclipse user may have many bundles in a shared update location that don't share a root. For example, if I had an Eclipse install and I wanted to selectively add features, I could do so by defining other paths that were picked up at runtime e.g.

.../CDT/
.../JDT/
.../PDE/

This works with the 'dropins' case on the priviso that one of their parent directories is called 'dropins' (or they're all understand the same root). But you can't do, for example:

D:\CDT
E:\JDT
F:\PDE

Having multiple 'dropins' folders would permit this kind of install, which whilst impractical in this example, tends to be the case that there is a naming/folder structure in place which does not permit products to have a shared parent directory.

Comment 3 James Blackburn 2010-03-12 04:39:57 EST

Created attachment 161852 [details]
example feature management script

Thanks Martin! comment 0 is exactly what I do and I too want an officially sanctioned means of doing this.  (As it happens I'm supporting a multi-site installation, but NFS at all sites have the same mount-points.)

We have scripts here to achieve the comment 0 functionality. In particular:
  - Tools can be decided per team
  - Users get default version without any intervention
  - Users track tools-team default without any intervention
  - Power-Users can override the version of a particular feature, install other features and remove features

I'll attach the script, as it's may give a concrete example of how to achieve this functionality currently.

Examples:

eclipse-install: a script for managing centrally installed features, can be called directly by users to manage their installed feature. Also called by a top-level eclipse wrapper to keep extension features up to date.

- First launch: user types eclipse and, echo'd to the console:
Installing / Upgrading cvs to 3.5
Installing / Upgrading epic-perl to 0.6.35
Installing / Upgrading pydev to 1.5.4
Installing / Upgrading subclipse to 1.6.8

If the tool's default changes, a similar line is echo'd on subsequent start.

For most users this is all they ever see or need to know.


For power users, they can run eclipse-install directly to:

- Install a 'fixed' version of a tool (override default)
> eclipse-install -i ccrc D090713
Installing fixed version D090713 of tool ccrc

Install /track default version:
> eclipse-install -i ccrc
Installing default version D091111 of tool ccrc, tracking tools team default

- Show installed features:
> eclipse-install -s
ccrc => D090713
cvs => 3.5 (default)
epic-perl => 0.6.35 (default)
linuxtools => 0.4.1 (default)
mylyn => 3.3.1 (default)
...

- List features to install
> eclipse-install -l
ccrc (default: D091111)
cvs (default: 3.5)
egit (default: 0.6.0_20091013)
epic-perl (default: 0.6.35)
linuxtools (default: 0.4.1)
mylyn (default: 3.3.1)
...

- Remove a tool
> eclipse-install -d ccrc
> eclipse-install -s
eclipse-install -s
ccrc => removed
cvs => 3.5 (default)
...

NB: P2 doesn't notice a change in a .link file. So the script carefully creates new and removes old .link files.

Comment 4 James Blackburn 2010-03-12 04:51:54 EST

Also implicit in comment 0, but not stated:
 9) Features installed into a user's configuration from the central repository shouldn't be copied into user's configuration.  It turns out space on filers is expensive, and part of the reason we're doing this is so we don't use disk space ~ num_users x sizeof(features)

Comment 5 James Blackburn 2010-03-12 05:13:03 EST

(In reply to comment #2)
> I'd just like to add to the list:
> 
> 8. Ability to have bundles installed from more than 2 locations. 
>  ...
> This works with the 'dropins' case on the priviso that one of their parent
> directories is called 'dropins' (or they're all understand the same root). But
> you can't do, for example:
> 
> D:\CDT
> E:\JDT
> F:\PDE
> 

I'm not sure I understand this.  In the shared configuration case, we've got the read-only configuration on one filesystem, and the user's configuration in ~/.eclipse/config on another filesystem.  The .link files specify the absolute path to the extension feature.
Is there something specific about Windows which means you can't have:
cdt.link contains D:\CDT
jdt.link contains E:\JDT
etc.

Comment 6 Martin Oberhuber 2010-03-12 05:39:13 EST

(In reply to comment #3)
Thanks for the script! - I'll carefully inspect it since this is exactly what we need :) I only have one central file system today, but I need to go multi-site so your script might save my day. One extra complexity of my setup is that I also have Windows users (but they all use the same "mount point", we force the install into C:\WindRiver today; proper UNC path support might help here).

> NB: P2 doesn't notice a change in a .link file. So the script carefully 
> creates new and removes old .link files.

Incidentally I looked at the code in o.e.e.p2.reconciler.dropins recently,
and I now know why that's the case: the code which checks for up-to-date status of the reconciler cache just inspects the timestamp of the directories. So when you change contents of a .link file, you'll just need to touch the parent folder and p2 will re-load it.

Changing the code to keep a timestamp of all .link files would not be too hard I think, and also the performance degradation of doing so should be acceptable since the number of disk accesses doesn't really change (the directory contents needs to be read anyways). I didn't check whether a bug already exists asking for this. The code is in Activator#isUpToDate() of that bundle.

I should also add another criterion:

10. Keep user config-scope Preferences when upgrading central Platform.
    I filed bug 301531 for this independent of this discussion.

Today, we start with a fresh -configuration folder when the base Eclipse Platform changes to another milestone or I-build (yes, we are rolling out Eclipse Platform milestones to our users :). We do so in order to make sure everything's fresh and fine. The drawback of this is that configuration scope preferences (such as the recently used workspace list) get lost during a Platform update. James, how are you dealing with this?

Comment 7 Alex Blewitt 2010-03-12 06:05:10 EST

(In reply to comment #5)
> (In reply to comment #2)
> > I'd just like to add to the list:
> > 
> > 8. Ability to have bundles installed from more than 2 locations. 
> 
> I'm not sure I understand this.  In the shared configuration case, we've got
> the read-only configuration on one filesystem, and the user's configuration in
> ~/.eclipse/config on another filesystem.  The .link files specify the absolute
> path to the extension feature.
> Is there something specific about Windows which means you can't have:
> cdt.link contains D:\CDT
> jdt.link contains E:\JDT

Yes. These need/expect/mandate there to be an eclipse\plugins and eclipse\features hierarchy under the folders pointed to by the .link files. There is no reason such a restriction needs to exist; further, it prevents the linked resource being a more complex hierarchy below. On the other hand, the dropins directory needs no such folder structure in order to work.

Comment 8 James Blackburn 2010-03-12 06:13:41 EST

(In reply to comment #6)
> Today, we start with a fresh -configuration folder when the base Eclipse
> Platform changes to another milestone or I-build (yes, we are rolling out
> Eclipse Platform milestones to our users :). We do so in order to make sure
> everything's fresh and fine. The drawback of this is that configuration scope
> preferences (such as the recently used workspace list) get lost during a
> Platform update. James, how are you dealing with this?

I have the _same_ problem & this is a major issue.  You may have noticed $ECLIPSE_VERSION. My platform is basically Platform + CDT. I build CDT from source + a platform feature patch with o.e.c.resource & o.e.c.ide (getting fixes & features into the platform takes a while: bug 86973 :s). 

I only change the configuration at major platform changes and users get annoyed that Eclipse 'forgets' the location of their workspaces.  My versions look like 3.5-n where n is a minor-ish respin that doesn't require a configuration change (and by 'doesn't require' I mean fingers-crossed...). 

This is a poor compromise: it seems like a reconcile that takes place while the user has multiple eclipse instances open, sharing a configuration, breaks the existing instances (random exceptions happen in the future which requires a restart)!  Certainly eclipse -clean (which prunes cached .so's, images, and who knows what else) also breaks running instances.

So the answer is: not very well.  Separating user settings from plugin configuration would really ease this pain!

> (In reply to comment #3)
> Incidentally I looked at the code in o.e.e.p2.reconciler.dropins recently,

I was wondering whether p2 used the metadata in the 'extension location', or does the reconciler generate it all afresh anyway? It doesn't seem to much matter if the extension loc has metadata or not (ccrc, for example, which can't be installed with p2, can be used by pointing at the UM installed extension location -- even though there is no metadata there).  That would certainly explain the long reconcile time.

> I didn't check whether a bug already exists asking
> for this. The code is in Activator#isUpToDate() of that bundle.

Bug 288755

So far the dependencies of the features I'm providing have all worked out. I dread to think what will happen when a user requests 2 features which depend on incompatible versions of a another feature...

Comment 9 James Blackburn 2010-03-12 06:20:09 EST

(In reply to comment #6)
> James, how are you dealing with this?

Also you get phantom perspectives: bug 288758 in workspaces which are used after a configuration change. Note this seems to be workspace specific, creating a new one fixes the problem.

The duplication can actually be quite funny:
https://bugs.eclipse.org/bugs/attachment.cgi?id=146610

I asked for pointers on where the bug might so I could investigate and provide a patch, sadly no pointers :(

Comment 10 Martin Oberhuber 2010-03-12 06:24:20 EST

(In reply to comment #8)
> I dread to think what will happen when a user requests 2 features which
> depend on incompatible versions of a another feature...

Wouldn't that be cought by the admin? - In my case the admin has a
"godzilla-i-got-it-all" master install with all the features to verify
compatibility and protect from such issues. Without this, the whole "I check it
first before I roll out, just take my versions and you'll be good" wouldn't be
of much benefit.

Of course we cannot protect users from shooting themselves by downloading an
unapproved add-on themselves. But we ask them to report any need for an addon
back to the admin for integration and testing.

(In reply to comment #9)
> Also you get phantom perspectives: bug 288758 in workspaces which are used

Ah, that's where these come from. Looks like we're safer re-initializing the config and going for shared config-level prefs as per bug 301531 then, at least for now. The "should-have", of course, is that any version-to-version upgrade should be able to work with the old config, so I'm going to track your bugs.

Comment 11 James Blackburn 2010-03-12 06:32:26 EST

(In reply to comment #10)
> Wouldn't that be cought by the admin? - In my case the admin has a
> "godzilla-i-got-it-all" master install with all the features to verify
> compatibility and protect from such issues. Without this, the whole "I check it
> first before I roll out, just take my versions and you'll be good" wouldn't be
> of much benefit.

Oh sure: I'll make sure all the defaults are self consistent. And a user sticking to these should be OK. But given we don't intend to delete old version (for fear of breaking some unknown user, somewhere). In the future when we have a new platform and new features, a user might elect to install incompatible feature-sets. 

This can no doubt be fixed-up by making the script & the admin cleverer.  Make the script aware of feature dependencies so the user isn't allowed to go wrong. Make the admin more God-like and remember which versions of features all her users are using.  At this point it feels we've gone full-circle and arrived back at UM!  Isn't all this supposed to be p2's job :)?

Comment 12 Pascal Rapicault 2010-03-12 17:00:59 EST

Thanks for coming up with a clear list of requirements.

> 1. No need for end user interaction. End users ever just launch the central
>    Eclipse and get updates automatically. There should not even be a single
>    dialog when a minor update has been applied to the central install.
  The prestartup update example shows how to do this if you things to happen before the app completely starts (org.eclipse.equinox/p2/examples/org.eclipse.equinox.p2.examples.rcp.prestartupdate), or otherwise I'm pretty sure we have a way to disable the prompting for updates and just prompt the user for restart once everything has happened.

> 2. Easy central update. An administrator can update any individual Eclipse
>    extension easily and roll that update out to the end users.
  Thinking of extensions in the old way is the issue. In the scenario you are talking about, an extension becomes a repository (maybe processed to be in a ready to run format). The issue for the admin is now how to keep the repo in sync with the source it got it from. For this problem p2 does not provide any particular tools aside from the mirroring tools, you may need some wrappering logic that would go and check for updates on the source site.

> 3. Users can configure their install by pick-and-choose extensions as they 
>    need. In case of an update of any extension, their configuration remains
>    the same.
  If all individual repos are presented as one repo (e.g. composite) and the content filtered for consumption for your users then he just has to use the standard p2 UI or the p2.ui.discovery.

> 4. Protected shared install. End users can only influence their own 
>    installation, but not destroy any config of somebody else. Only admins 
>    can update packages.
   This is already supported today (since 3.4), but some fixes could be done (see bugs with [shared] prefix).

> 5. Add-your-own. End users can add their own extensions such that they are
>    not restricted to what has been centrally provided.
   This is already supported today (since 3.4), but some fixes could be done (see bugs with [shared] prefix)

> 6. Team-level configs. Groups of users can define their own config and share
>    that config in a team.
    Here again, this could be seen as a repo management issue, where each team could have its own repo, or just be an issue of publishing an IU that refers to all the necessary tools for the team.

> 7. The solution must be officially supported and documented.
    This will only happen with code and extensive test suite from the interested parties.

>8. Ability to have bundles installed from more than 2 locations.
   Installing by just adding a folder was just plain wrong and resulted in many problems in UM at the time because ppl would break themselves and then complain. I'm against reintroducing something like that and so are the other p2 committers, which is why I closed the original bug as wontfix. p2 provides bundle pooling which provide a much better way to do this.

> 9. Features installed into a user's configuration from the central repository shouldn't be copied into user's configuration
   This is the only thing that we are not able to do at this point without mucking around with some internal properties. Though even with this fixed (provide an API for changing the property), this would require some changes in the UI to accommodate this.

Comment 13 James Blackburn 2010-03-12 17:57:11 EST

Thanks for the reply Pascal!

It feels like there's a long way to go before users or admins, can provision with p2 commands as easily as they can by tweaking dropins.  Note the usage I gave in the shell script:
  - users can track
  - users can specify a specific version
  - users can upgrade and downgrade and uninstall.

(In reply to comment #12)
> > 1. No need for end user interaction. End users ever just launch the central
> ...
> and just prompt the user for restart once everything has happened.

With dropins + the reconciler, no restart required.

> > 4. Protected shared install. End users can only influence their own 
> >    installation, but not destroy any config of somebody else. Only admins 
> >    can update packages.
>    This is already supported today (since 3.4), but some fixes could be done
> (see bugs with [shared] prefix).

How does this work in practise? Does anyone use it? If I have 100 users with running Eclipse instances, and I change the central p2 shared install, do all the running Eclipse instances fall over?

Versioning any part of the toolchain (for other tools) happens in phases:
   - New version released to early-birds
   - Most users migrate to new version
   - Old versions deprecated
We still keep the old version around for building finished projects (e.g. tagged from CVS), or for any stragglers we've missed. It's an insurance policy.

Having a directory on a filesystem I can chmod -R -w *  sets a release in stone for me.  And this is what we're missing.  

With p2 how can I, the admin, be sure I don't mess things up, and when I do mess things up, how do I press undo?  (Note those 100 users are still running their now-broken Eclipse installs.).

This is why I'm scared of opaque data stores which we touch in magical ways using a clever tool I also don't understand. If I release something I want it fixed in stone.  Is there anyway to marry p2's immense flexibility with the requirements and assurances that shared installs demand?  Can you say that it's safe to install features into a shared pool while Eclipse continue to run in the background?

Comment 14 Pascal Rapicault 2010-03-12 19:23:29 EST

> It feels like there's a long way to go before users or admins, can provision
> with p2 commands as easily as they can by tweaking dropins.  Note the usage I
> gave in the shell script:
>   - users can track
>   - users can specify a specific version
>   - users can upgrade and downgrade and uninstall.
    p2 install reigstry and director app provide all this and more.

> > > 1. No need for end user interaction. End users ever just launch the central
> > ...
> > and just prompt the user for restart once everything has happened.
> 
> With dropins + the reconciler, no restart required.
   You are missing one big thing here. Dropins is a best effort mechanism. If the user ends up installing something else through the director or the UI, it may run the risk of disabling some of the plug-ins you installed through dropins. And no this is not a bug.
   
> > > 4. Protected shared install. End users can only influence their own 
> > >    installation, but not destroy any config of somebody else. Only admins 
> > >    can update packages.
> >    This is already supported today (since 3.4), but some fixes could be done
> > (see bugs with [shared] prefix).
> 
> How does this work in practise? Does anyone use it? If I have 100 users with
> running Eclipse instances, and I change the central p2 shared install, do all
> the running Eclipse instances fall over?
   Some ppl use this, not sure in which degree, but they do. When the instance is updated on the server,  old plug-ins have to be kept around so that running instances just don't fail because the old plug-ins disappear. As for notification goes, if the shared instance was exposed as a repository to the client instance, then the client could be notified of an update.


> Versioning any part of the toolchain (for other tools) happens in phases:
>    - New version released to early-birds
>    - Most users migrate to new version
>    - Old versions deprecated
> We still keep the old version around for building finished projects (e.g.
> tagged from CVS), or for any stragglers we've missed. It's an insurance policy.
> 
> Having a directory on a filesystem I can chmod -R -w *  sets a release in stone
> for me.  And this is what we're missing.  
    I don't follow why this is missing. You can create an install, chmod and away you go. If you want even more control you can disable the p2 UI and the dropins for your end users...

> With p2 how can I, the admin, be sure I don't mess things up, and when I do
> mess things up, how do I press undo?  (Note those 100 users are still running
> their now-broken Eclipse installs.).
   I'm not sure how this is different than what you have to do when you upgrade your linux install. p2 offers a rollback functionality (now on the director app) that will take you back to a previous state.

>  Can you say that it's safe to install features into a shared pool while Eclipse continue to run in
> the background?
    Absolutely. But if it was not then we would fix it.

Comment 15 Alex Blewitt 2010-03-17 17:50:27 EDT

(In reply to comment #12)
> >8. Ability to have bundles installed from more than 2 locations.
> Installing by just adding a folder was just plain wrong and resulted in many
> problems in UM at the time because ppl would break themselves and then
> complain. I'm against reintroducing something like that and so are the other p2
> committers, which is why I closed the original bug as wontfix. p2 provides
> bundle pooling which provide a much better way to do this.

What's up with the Orwellian "two dirs good, four dirs bad" statement? The fact is that you already have the ability to add plugins into one, or two, directories at the moment (by using the eclipse/dropins folder or the one defined by argument variable. 

Furthermore, as already noted elsewhere, you can *almost* get to a situation where multiple folders are possible by adding many .link files. The problem with that approach is that a .link file is (arbitrarily) constrained to a folder which contains an eclipse/plugins folder, which places additional and unnecessary constraints on the structure of the folders.

So by flat-out writing off this request, you haven't changed anything. People can still shoot themselves in the foot if they want to; but what it does do is prevent a perfectly legitimate (and useful) way of managing Eclipse installations. Indeed, it's possible to do this out-of-the-box with Eclipse as it currently stands, and if Eclipse won't do it for you, then installing utilities like FileInstall will.

Lastly, your point about 'many problems in UM' really go away since we're not talking UM here - we're talking P2. It has nothing to do with 'reintroducing' and everything to do with generalising the current solution.

Comment 16 Pascal Rapicault 2015-03-26 06:20:13 EDT

An overall solution to this problem has been released in Luna.