Community
Participate
Working Groups
At today's architecture council meeting, I brought up the topic of implementing a third party library policy (based on some of Andrew Overholt's gripes). The main motivator of this is the pain we feel at Fedora as we tend to ship the latest version of a library and have had to patch Eclipse to use the latest library. eclipse.org projects, especially the SDK has historically not been good about moving to the latest version of libraries (e.g., Lucene). There are many benefits in moving to the latest version of a library, from security fixes to new features. On the call, we discussed some potential downsides, from the time it takes to move to a new library via the Eclipse IP process to the testing of the actual new library. Another topic that came up was that Eclipse sometimes ends up in a situation where projects are using two different versions of a library and we are in a case where three projects are using three different versions of say log4j, instead of just the latest version. There was some discussion by Wayne Beaton on how to implement a tool using iplog data that can notify projects if a newer version of a library is available in Orbit (or elsewhere) and they aren't using it. Another strategy could we make this part of the release review process, when you hand in your IP log, the IP team could do a quick check to see if there are newer versions of any of the libraries used... projects would have to justify their usage of the older library. Another strategy could be to make a strong requirement for the simultaneous release that you're on the latest version of a library available. Those are my two cents. So what are people's thoughts here?
I am new to AC so maybe I bring up silly questions. But where do we communicate what the library of choice is for the release train. I in Riena for instance always find myself looking at the finished packaged SDK to detect what is obviously the current choice. How gets to decide what the "right" third-party library version is for a release train and how is it communicated to the projects. That would be my 1 1/2 cents christian
Ideally, projects would just spec a well defined dependency range when importing 3rdParty packages (eg. [1.0.x.whatever,2.0.0)). They would then build and assemble using the latest available in Orbit (assuming that Orbit becomes the only authorized place for 3rdparty libraries). I think a first step is to provide better visualization of what is available and approved today. As Christian points out, it's hard to discover that. Next we could develop tooling to generate usage reports for projects. Another step would be to make the upgrading process easier. Very often a diff-scan is sufficient which makes upgrades a fly. In those cases it seems that the IP submission process is a bit too bureaucratic (see bug 300717). I also think that PDE support for OSGi-fying libraries can be improved (eg., bug 330256). This would make upgrading easier. However, I think the most critical issue is that every external library has a different (sometimes none) understanding of API and compatibility. At some point you start testing and then you have to commit to a specific version, especially when you intend to provide long support for it. ----- On a side node, I don't yet understand why Fedora needs to patch Eclipse. Eclipse dependencies are very self contained. One of the key features of OSGi is to run multiple versions in parallel. Thus, technically it shouldn't really matter what version of a library an Eclipse project depends on. I have the feeling that this could turn into a support nightmare. For example, some Fedory user opens a bug in Eclipse Bugzilla which just isn't reproducible in any of the vanilla Eclipse.org Linux packages.
(In reply to comment #2) > On a side node, I don't yet understand why Fedora needs to patch Eclipse. We have to ship the latest version of a library available generally (everything has to be built from source). And those libraries are maintained separately from Eclipse (e.g., yum install commons-net). And those libraries are the ones that need to be used by Eclipse.
(In reply to comment #1) > I am new to AC so maybe I bring up silly questions. > > But where do we communicate what the library of choice is for the release > train. I in Riena for instance always find myself looking at the finished > packaged SDK to detect what is obviously the current choice. We don't really do this anywhere now explicitly... generally this happens when you do the release review for the train... this year it's June 1-8, 2011 for Indigo... you have to submit your iplogs as part of the review. My idea involves the IP team informing you that there's a later version of a library available used by project XYZ or in Orbit. > How gets to decide what the "right" third-party library version is for a > release train and how is it communicated to the projects. Well, it should generally be the latest version of the library available in most cases. Sure, there are some cases like junit3/junit4 but that's different.
(In reply to comment #4) > (In reply to comment #1) > > I am new to AC so maybe I bring up silly questions. > > > > But where do we communicate what the library of choice is for the release > > train. I in Riena for instance always find myself looking at the finished > > packaged SDK to detect what is obviously the current choice. > > We don't really do this anywhere now explicitly... generally this happens when > you do the release review for the train... this year it's June 1-8, 2011 for > Indigo... you have to submit your iplogs as part of the review. My idea > involves the IP team informing you that there's a later version of a library > available used by project XYZ or in Orbit. > > > How gets to decide what the "right" third-party library version is for a > > release train and how is it communicated to the projects. > > Well, it should generally be the latest version of the library available in > most cases. Sure, there are some cases like junit3/junit4 but that's different. ok… Looking at the ressources in the IP team, anything that does NOT involve them is a better idea I believe. Strictly its not a license issue (so we dont necessarily need lawyers for this). Its more an EMO topic or a release train policy topic right and then a script controlling it. Another idea could be that we built something into IPZilla that that you could mark a CQ as being a later version of the same library of a previous CQ. Then once the new CQ is approved it could automatically send emails to all the Piggyback CQs of the previous CQ. :-)
(In reply to comment #3) > And those libraries are the ones > that need to be used by Eclipse. What if those libraries don't contain any OSGi headers? Do you patch them as well? I also think that this policy has room for multiple versions. Very often I see packages tight to a *much* older version (eg. gtk1, kde2, apache1). Not sure if they don't count because of the version being part of the package name. ;) (In reply to comment #4) > Indigo... you have to submit your iplogs as part of the review. My idea > involves the IP team informing you that there's a later version of a library > available used by project XYZ or in Orbit. I'm afraid that this will generally be too late in the game for projects to change. Especially larger projects tend to freeze major updates at M6/M7 time. I also don't think that external dependencies can be changed during the RCs. Remember, I'm always pushing for staying on the latest and greatest. I just don't think that we can *force* projects to upgrade to a specific version. However, maybe we can change the general thinking by pushing for package level imports only in combination with version ranges. This would allow projects to build and test with what's approved in Orbit and what they are able to adopt. Other distributors would simply take the project bundles and re-assemble using a higher version. If we could only get rid of Require-Bundle for non "org.eclipse" dependencies... (In reply to comment #5) > Another idea could be that we built something into IPZilla that that you could > mark a CQ as being a later version of the same library of a previous CQ. When approving CQs (as PMC member) I generally try to comment whether a newer version is available. Very often it's just a matter of search/visibility.
<snip> > > When approving CQs (as PMC member) I generally try to comment whether a newer > version is available. Very often it's just a matter of search/visibility. But when you approve a CQ for new third party library version you dont notify the user of old version of the same lib which they manifest through their existing Piggybacks, dont you ?
(In reply to comment #7) > But when you approve a CQ for new third party library version you dont notify > the user of old version of the same lib which they manifest through their > existing Piggybacks, dont you ? I'm not sure I understand your questions. When a user submits a CQ I generally check whether they are asking for an old or the most recent library. Not sure that this case should be complicated with different CQs of other libraries. I think a tool for pulling an IP log and comparing that with what's in available in Orbit would be very valuable. It would provide more insights into how far off my dependencies are. But I (as a committer) must be able to run this tool myself for my project. Another valuable report could be generated as part of the release-train build.
(In reply to comment #8) > (In reply to comment #7) > > But when you approve a CQ for new third party library version you dont notify > > the user of old version of the same lib which they manifest through their > > existing Piggybacks, dont you ? > > I'm not sure I understand your questions. When a user submits a CQ I generally > check whether they are asking for an old or the most recent library. Not sure > that this case should be complicated with different CQs of other libraries. > > I think a tool for pulling an IP log and comparing that with what's in > available in Orbit would be very valuable. It would provide more insights into > how far off my dependencies are. But I (as a committer) must be able to run > this tool myself for my project. > > Another valuable report could be generated as part of the release-train build. Aparently I am not good in explaining this. I make an example. Say there exists an approved CQ for coollib-1.0 and there are another 5 PB CQs referencing this one. Once a CQ for coollib-2.0 is approved, wouldnt it be cool if we notify the CQ report of coollib-1.0 that there is a new version approved and he/she should upgrade. Chance are its the same person, so the supercool effect that we could also notify all the PB CQ owners of that CQ. "Look there is a new version and its already approved and all you have to do is create a PB CQ". This could be automatic, no user intervertion and maybe solve 80 % of the usecases. The only information missing I believe that there is no indication in the CQ of coollib-2.0 that its an UPGRADE of a previous version. (maybe there is and I am not aware of it) Once coollib-3.0 comes around we could notify everyone down the chain whether CQ or PB CQ who is still active because I believe a project is expected to withdraw a CQ if it is no longer using it. Does that make sense ? And it is way earlier than when you submit your IP Log but right when a lib version enters the Eclipse eco system.
(In reply to comment #6) > (In reply to comment #3) > > And those libraries are the ones > > that need to be used by Eclipse. > > What if those libraries don't contain any OSGi headers? Do you patch them as > well? I also think that this policy has room for multiple versions. Very often > I see packages tight to a *much* older version (eg. gtk1, kde2, apache1). Not > sure if they don't count because of the version being part of the package name. > ;) > I'll try to explain the Fedora usage a bit: 1. Yes we have patched a number of projects to properly build OSGi bundles and even pushed the changes upstream whenever upstream accepted them. 2. Yes there are several versions of certain packages but this doubles the amount of work for us - older versions fail to build from source with newer maven or ant, security issues have to be tracked in 2 packages, some projects change api in x.y.z+1 release and so on and so on All these issues forced us to work on a one single version of a bundle.
And there is one more thing to be considered: When upstream project ships an OSGi bundle maybe we(Eclipse.org) should ship the bundle as is or at least with unmodified MANIFEST.MF. We can use for example Apache Commons projects - most of them are ready to use as they are. Shipping without modification will prevent further differences and if there is a problem we will help others that are trying to be part of the OSGi ecosystem. Should something like this be part of such policy?
I suspect the reason that most don't go through updating, is because it means filing a CQ. I personally would love to always be on the lastest updated version, but if it means going through the CQ process with a smaller staff, I won't update unless there is a Security reason to do so. Streamline the third party CQ process and projects may update more often.
(In reply to comment #12) > Streamline the third party CQ process and projects may update more often. How can it be better? I'd love to hear ideas. If there's a library already approved, you simply piggy back on it via the portal. It's pretty painless.
(In reply to comment #13) > (In reply to comment #12) > > Streamline the third party CQ process and projects may update more often. > > How can it be better? I'd love to hear ideas. > > If there's a library already approved, you simply piggy back on it via the > portal. It's pretty painless. Right but here is the issue. Getting approved is not the issue, somebody then still has to get it checked into orbit. So say a new release or a security fix comes out for a third party library like Apache HTTP Commons or something. A new CQ has to be created, the code has to be reviewed, if there are questions and concerns about pedigree, those have to be answered, etc. Then a new CQ has to be created to get it created for Orbit, and then it has to be checked in, built, etc. The fact of the matter is that the IP process is the slowest portion of this. It would be nice if say for those items that are already converted to OSGI bundles if an equivalent version is hosted at say Spring Sources Open Source Repository, Maven Central, or other place, that we could just use it. I've had it take over 6 months because of backlogs to get things approved for distribution and use in the past. And that was before the staff reductions over the last year. Developers typically aren't going to want to go through the hassle and time it takes to get things approved. If you are using an Apache project it isn't as bad, but if you want to use something else, where you have to track down all parties concerned that worked on it...it can take way longer than they have time to devote to it.
(In reply to comment #14) > So say a new release or a security fix comes out for a third party library like > Apache HTTP Commons or something. A new CQ has to be created, the code has to > be reviewed, if there are questions and concerns about pedigree, those have to > be answered, etc. Then a new CQ has to be created to get it created for > Orbit, and then it has to be checked in, built, etc. A minor update to an already approved library is usually a low hanging fruit for the IP team if a diff-scan applies. A while back I've opened bug 300717 in order to simplify entering "update CQs" through the portal.
(In reply to comment #15) > (In reply to comment #14) > > So say a new release or a security fix comes out for a third party library like > > Apache HTTP Commons or something. A new CQ has to be created, the code has to > > be reviewed, if there are questions and concerns about pedigree, those have to > > be answered, etc. Then a new CQ has to be created to get it created for > > Orbit, and then it has to be checked in, built, etc. > > A minor update to an already approved library is usually a low hanging fruit > for the IP team if a diff-scan applies. A while back I've opened bug 300717 in > order to simplify entering "update CQs" through the portal. This can depend greatly on the library and where it is coming from. Typically Apache hosted projects not as difficult as say projects that don't have a Foundation backing them.
This bug hasn't had any activity in quite some time. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. If you have further information on the current state of the bug, please add it. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. -- The automated Eclipse Genie.
This issue has been migrated to https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/140.
