Community
Participate
Working Groups
Created attachment 96264 [details] rough listing of 135 features jars that are not signed. One of our "must do" conditions for Ganymede was to sign jars: "Projects must use signed plugins using the Eclipse certificate. Exceptions authorized by the planning council for technical reasons." But, running jarsigner -verify on the Ganymede update site makes it appear only about 3/4ths are. I've only looked at features, so far, in detail, thinking that would be indicative of projects that have not done it yet. I'll attach full list here, that all projects should look at, but the ones that jumped out at me were from modeling and gmf projects, Mylin, DLTK, stp, ecf. So ... what's up? It's unfortunate these are not all signed now, since many are starting on their "final testing", checking performance data, etc. But ... I think if people do it by M7, we can still let you stay in Ganymede :) If anyone has trouble doing this, there's plenty of people to help. If anyone wants an exception for technical reasons, it's time to seek approval!
Odd that those modeling project plug-ins are listed, as those using the common builder should have been signed properly. The org.eclipse.m2m.qvto.* plug-ins will be signed upon switching to this common build as well, and is planned to be done for M7.
(In reply to comment #1) > Odd that those modeling project plug-ins are listed, as those using the common > builder should have been signed properly. Yes, the common modeling build WILL do signing, but only if the build itself (buildAll.xml, customTargets.xml) are set up correctly. http://wiki.eclipse.org/Modeling_Project_Releng/Building/Signing_And_Packing To date, I've only converted over 5 of my 8 builds. The other three are waiting until I can enhance the way the system works by using PDE Packager so that there's less custom repackaging code, but to date I've had nothing but problems getting that to work. See bug 226777.
Mylyn hasn't done the signing yet, but we will be doing it in the near future (created bug 227671).
Created attachment 99643 [details] M7 list of features (only) that are not signed
updated for M7
Created attachment 101908 [details] RC1 list of features (only) that are unsigned The list of unsigned feature jars is now down to 21, from the M7 number of 49. It appears these 21 some from three projects, at least from namespaces, of org.eclipse.dd... org.eclipse.m2m.alt... org.eclipse.stp... The list of unsigned plugin jars seems to map pretty directly to these three projects. There are 77 of these, not attached here.
See bug 233875 for cross reference.
Re: ATL ATL is running signing, but may not be contributing those signed jars correctly to Ganymede. Checking the atl.releng/promoteToEclipse.atl.properties file, I see that they're using the following zips as contribution to the update site, which ought to be simplifed to just include the Master zip, with its signed & packed jars. filePrefixesToUnzipArray=( "$projectName-$subprojectName-Master-" "$projectName-$subprojectName-SDK-" "$projectName-$subprojectName-automated-tests-" "$projectName-$subprojectName-examples-" ); -changes-to-> filePrefixesToUnzipArray=( "$projectName-$subprojectName-Master-" ); William: can you update your .properties file to include ONLY the master zip, which ought to already include SDK + examples? Tests aren't contributed to Ganymede so need not be included.
(In reply to comment #8) > filePrefixesToUnzipArray=( "$projectName-$subprojectName-Master-" ); BTW, I've patched this file on emft.eclipse.org and re-run the latest M2M ATL RC1 update site contrib. Ganymede is green, so if David can re-run his bad-jar-counter, we'll know if this fixes the missing signed jar contribution problem.
Just in case you're wondering, STP is not ignoring this bug report, however I am still working on understanding how to get the signing done using Buckminster. Please pardon my learning curve!
(In reply to comment #10) > Just in case you're wondering, STP is not ignoring this bug report, however I > am still working on understanding how to get the signing done using > Buckminster. > > Please pardon my learning curve! > I was wondering! :) Thanks for letting us know. So, sounds like you still anticipate having all signed for this release. I might point out, in the worst case, you could sort of do it "manually". While it's better to do automatically every build, that is not technically required, and should be pretty easy to do manually once and a while (e.g. with every milestone, release candidate, and final bits). To do "manually", I mean, basically, you'd sit down with some SSH terminal, copy a zip of jars to the magic signing spot, and check on it every hour or so until it was done, and then copy back and unzip and distribute how you needed to. While this approach would have it's own complications and challenges, it would provide a fall back plan, and might be a good way to get started, and help you take smaller steps on the learning curve. Thanks again,
If you are interested, I have some shell scripts that pack + sign an existing update site. No integration with your build is required... only drawback is that it signs the update site only, but not your downloadable ZIPs.
David - thanks for the instructions for manual signing, that should get me out of the woods while I get the automation sorted! (In reply to comment #12) > If you are interested, I have some shell scripts that pack + sign an existing > update site. No integration with your build is required... only drawback is > that it signs the update site only, but not your downloadable ZIPs. Martin - I am interested...very interested...anything that can help me produce those little signed bags of classes asap would be most appreciated. If you have a link or a pointer it would be great!
(In reply to comment #13) > Martin - I am interested...very interested...anything that can help me produce > those little signed bags of classes asap would be most appreciated. If you have > a link or a pointer it would be great! > Oisin, how about the eclipse-signing.ant ant-scripts that I mentioned to you? Were they of any help? Are there any particular issues you have when adding signing as part of the Buckminster workflow? As you know, I'm more then happy to help you get passed this hurdle.
(In reply to comment #14) > Oisin, how about the eclipse-signing.ant ant-scripts that I mentioned to you? > Were they of any help? Are there any particular issues you have when adding > signing as part of the Buckminster workflow? As you know, I'm more then happy > to help you get passed this hurdle. Thomas, your commitment to help is beyond doubt, and you have been vital in getting us up to speed with Buckminster (beers are on me). I've plagiarized chunks of the Buckminster build system that are part of the normalize/signing procedure and I'm mutating them to apply to the STP system. Because of the Ganymede juggernaut approaching, I want to hedge my bets just in case I don't grok things in time, or in case something blows up. However, I will be asking more questions on the dev newsgroup!
Created attachment 102640 [details] current list of usinged jars on near-RC2 site Things are looking pretty good, only stp project (known) but there is also a lot of plugins from org.eclipse.dd namespace that are not signed. Who owns those?
Guess that would be Ted or Pawel.
(In reply to comment #13) > Martin - I am interested...very interested...anything that can help me produce build.eclipse.org:/home/data/httpd/download.eclipse.org/dsdp/tm/signedUpdates/bin/mkTestUpdates.sh Much DSDP specific stuff there, but should be easy to adapt. Copies plugins from ../testUpdates/ and signs them. The packing is done in ../testUpdates already. One speciality of the script is that it signs each plugin separately , and does a jarsigner -verify step .
Created attachment 104943 [details] current list of unsigned jars as of 6/13 near-final staging area STP now is signing jars, but still have the DD jars that are unsigned? By DD jars I mean those in the name space of org.eclipse.dd.* Also, there are these three ... org.eclipse.emf.cdo.server.hibernate.libraries org.eclipse.net4j.db.hsqldb org.eclipse.net4j.db.mysql Anyone care to own up to these? Or track them down?
(In reply to comment #19) > org.eclipse.emf.cdo.server.hibernate.libraries > org.eclipse.net4j.db.hsqldb > org.eclipse.net4j.db.mysql These should be exempt from signing. We tried signing them then removing the build-time/test-time non-EPL 3rd party libs (HSQLDB, MySQL, and Hibernate) prior to packaging, but on installation p2 reported the plugins were corrupt due to missing files (as it should). With the libs removed, the unsigned plugins install fine and can later have the missing libs installed by the end user in order to activate these plugins. Eike Stepper is the component owner CDO and Net4j, which have both recently graduated from the EMFT incubator to the EMF project proper.
(In reply to comment #20) > (In reply to comment #19) > > org.eclipse.emf.cdo.server.hibernate.libraries > > org.eclipse.net4j.db.hsqldb > > org.eclipse.net4j.db.mysql > > These should be exempt from signing. > > We tried signing them then removing the build-time/test-time non-EPL 3rd party > libs (HSQLDB, MySQL, and Hibernate) prior to packaging, but on installation p2 > reported the plugins were corrupt due to missing files (as it should). With the > libs removed, the unsigned plugins install fine and can later have the missing > libs installed by the end user in order to activate these plugins. > > Eike Stepper is the component owner CDO and Net4j, which have both recently > graduated from the EMFT incubator to the EMF project proper. > Thanks Nick, it's good to get these issues documented here. It seems this indicates a problem, somewhere, but not sure how to overcome it. It is a side issue (not related to signing, per se) ... but it sounds like this project (via these plugins) has dependencies on third party code, that the project does not have permission to ship. Am I reading that right? I know that happens, and is sometimes ok, but wanted to read more about it in EMF's IP Log but couldn't find any mention of it there. But, I am not sure I know exactly where to look. Eike, can you provide a link to that "dependencies on third party code" section of your IP Log, that would mention these three? Thanks,
> > the non-EPL 3rd party > > libs (HSQLDB, MySQL, and Hibernate) ... can later have the missing > > libs installed by the end user in order to activate these plugins. > it sounds like this > project (via these plugins) has dependencies on third party code, that the > project does not have permission to ship. ... > Eike, can you provide a link to that "dependencies on third > party code" section of your IP Log, that would mention these three? Eike, yes, please do explain this situation as an incorrect IP log is a stop ship situation and I'd really hate to have that happen at this late a date.
Hi, And sorry for my late answer. I was away over the weekend... Yes, there are three plugins that are wrapper bundles for EPL-incompatible 3rd party libraries: hsqldb.jar, mysql-connector.jar and hibernate.jar (along with some hibernate support libs). I don't ship these jars! Nick and I have applied considerable efforts to have these jars included during build and test and remove them from all published stuff afterwards. This is also the reason why these plugins are not signed: The user is expected to add the jars if the plugins should be used. These plugins are purely optional since for each of them there is an EPL-compatible soultion available and published. We did not know that there must be mention of these in the IP Log. Can you please confirm that we were wrong with this assumption and that there's work to be done on our side?
> > > Eike, can you provide a link to that "dependencies on third > > > party code" section of your IP Log, that would mention these three? > > Eike, yes, please do explain this situation as an incorrect IP log is a stop > > ship situation and I'd really hate to have that happen at this late a date. > We did not know that there must be mention of these in the IP Log. Can you > please confirm that we were wrong with this assumption and that there's work to > be done on our side? We were specifically asked by Sharon to remove things that are NOT shipped from our IP logs. This includes JPOX & JDO2, which are not shipped with Teneo. I would assume the same logic applies here -- if not shipped, there's no IP log required. Are you now saying that code that's used but not shipped needs a CQ, too?
(In reply to comment #23) > Hi, And sorry for my late answer. I was away over the weekend... > What? You mean there's some people that don't work every weekend? :) > Yes, there are three plugins that are wrapper bundles for EPL-incompatible 3rd > party libraries: hsqldb.jar, mysql-connector.jar and hibernate.jar (along with > some hibernate support libs). > Not that I would know, but "EPL-incompatible" sounds "off limits" as far as anything we (Eclipse) would encourage by providing wrapper jars for! (I guess this is in fact the reason for the new documentation and review procedures ... it is either ok, and should be documented as to why, how, etc., or ... the Eclipse Legal staff may decide it is not ok, just like they do other IP items that are shipped). > > I don't ship these jars! Nick and I have applied considerable efforts to have > these jars included during build and test and remove them from all published > stuff afterwards. This is also the reason why these plugins are not signed: The > user is expected to add the jars if the plugins should be used. > Sounds like there are some technical issues to be solved here, in the future. 1) we should not ship bundles that have to be modified by users, 2) There are (sometimes) code patterns where our code can make use of "third party code" that is on the users file system (similar to how users can specify a new/different ant library, but they do it via preferences, not modifying a plugin. 3) believe it or not, there are sometimes license issues related to "copying jars" (not just using them) ... or, so I've heard as we went through our IP reviews :) > > We did not know that there must be mention of these in the IP Log. Can you > please confirm that we were wrong with this assumption and that there's work to > be done on our side? > Yep, required. (I'm sure your PMC knows all about it :) See, the Board inspired policy document at http://www.eclipse.org/org/documents/Eclipse_Policy_and_Procedure_for_3rd_Party_Dependencies_Final.pdf We in WTP have the following document in our IP Log, to provide this required documentation, if it helps serve as an example. http://www.eclipse.org/webtools/iprelated/2008dependancies.php This might just be an "IP documentation issue" ... but does sound like requires some review and involvement by Janet, et. al. (I'll let Bjorn guide you on next steps, if any ... just thought I'd document here what I know, and what we did in in WTP).
(In reply to comment #25) > http://www.eclipse.org/webtools/iprelated/2008dependancies.php BTW, it's "dependencies" not "dependancies". :) http://www.google.com/search?q=define%3Adependancies (from http://www.eclipse.org/org/documents/Eclipse_Policy_and_Procedure_for_3rd_Party_Dependencies_Final.pdf) > Works-with Dependencies (ii): > The Eclipse software is designed to work with multiple third party > software choices that provide similar functionality - the choice of > which to use is up to the user. At least one of those must be a > pre-req (see below) or approved by the EMO for distribution by the > project. Example: If a project requires a persistence mechanism, > it can allow the user to select from several different implementations. ... > "works-with" dependencies as determined by the PMC are > approved for use by the projects without further EMO review. > These discussions and decisions must occur transparently either > via email on the public PMC mailing list, or in the minutes of > meetings distributed to the public PMC mailing list. > These decisions must be documented by the project in their IP log to > ensure their visibility during release and other review checkpoints. > All PMCs are expected to seek the advice of the EMO if they are > uncertain with respect to a determination on whether a dependency > is “works with” or “pre-req”. Given that we're talking about alternate persistence mechanisms, I'd expect that the only thing we didn't do here is mention all this on a public mailing list. It's also now clear that we need IP log details for things we don't ship, despite previous instructions to the contrary.
Please note the context by which JPOX & JDO2 were discussed with respect to Teneo's IP Log for Ganymede. These two packages were approved for inclusion in Eclipse and simply were not shipping with Teneo for Ganymede. As such, the project could choose to notate them out or remove them for IP review purposes. There was no discussion concerning pre-req or depedency packages during the review. Hope this helps, Sharon
(In reply to comment #27) > Please note the context by which JPOX & JDO2 were discussed with respect to > Teneo's IP Log for Ganymede. These two packages were approved for inclusion in > Eclipse and simply were not shipping with Teneo for Ganymede. As such, the > project could choose to notate them out or remove them for IP review purposes. Apologies, you're right.
(In reply to comment #24) > Are you now saying that code that's used but not shipped needs a CQ, too? Yes. As per the "Third-Party Dependencies" policy. The Board instituted this policy because a number of projects were using dependencies rather than inclusions to get around the IP Policy. So now all third-party IP that is involved in a project, be it included in the zip or required/pre-req'ed by the code, must have a CQ and be in the IP log. (In reply to comment #25) > 3) believe it or not, there are sometimes license issues related to "copying > jars" (not just using them) ... or, so I've heard as we went through our IP > reviews :) That is correct. > > We did not know that there must be mention of these in the IP Log. Can you > > please confirm that we were wrong with this assumption and that > > there's work to be done on our side? I can confirm that, yes. Please have your PMC approve these pre-reqs as per the url in comment #25. > This might just be an "IP documentation issue" ... but does sound > like requires some review and involvement by Janet, et. al. Yes. Each of pre-reqs must have a CQ even if it is eventually approved as an exempt pre-req by the PMC. (In reply to comment #26) > It's also now clear that we need IP log details for things we don't ship, > despite previous instructions to the contrary. (In reply to comment #27) > There was no discussion concerning pre-req or depedency packages during the > review. My understanding of the conversations that occurred about the IP log was that the conversations involved "if we ship the bits, we need to include the CQ" and did not include "if we pre-req the bits, we need to include the CQ". My understanding is that the misunderstanding was the difference between "if we do not ship the bits..." and "if we do not ship the bits *and* do not pre-req the bits..." However, now that we are at a happy place (mis-communications cleared up), please make sure there are CQs for everything and have the Modeling PMC approve the pre-reqs as per the policy. Thanks.
The issue about CDO and Teneo optional hibernate integration is not the topic of this bugzilla. The involved parties will have a discussion with Janet to find out how best to conform to the policies for these types of issues that seem to fall between the cracks in the policies. We'll work hard together to ensure that whatever meaningful steps need to be taken are taken quickly.
(In reply to comment #30) > The issue about CDO and Teneo optional hibernate integration is not the topic > of this bugzilla. ... > True enough, and on that topic, I have realized we should raise this to planning council, since the statement of the requirement is similar to "... exceptions based on technical reasons can be approved by the Planning Council ... " So, I'll send note, but for the official record, I think even if approved this release, that this should not be taken as precedent that it will always ok. I think there are other technical solutions possible and it is bad practice to ship "partial jars" that require a user touching them before they work. And, the topic of this bugzilla, it does "leave a hole" in security efforts.
(In reply to comment #19) The final build of DD is now up on the Ganymede update site and we've verified that these jars are signed. The signing is a manual step in our build and we haven't had a check in place to verify with each milestone/rc that the jars are were actually signed.
Re Comment #19 , Installing org.eclipse.dd.* from site http://download.eclipse.org/releases/ganymede/staging, Help -> About Eclipse SDK -> Plug-in Details shows a non-broken signed icon with Signing Info "L=Ottawa CN=Eclipse Foundation ST=Ontario OU=Digital ID Class 3 - Java Object Signing O=Eclipse Foundation C=CA" Is there something more we must do?
Yes, I see now the org.eclipse.dd* plugins are signed. Thanks for updating them. BTW, on mailing list someone asked "how I see them unsigned" so just thought I'd mention here it is from running jarsigner -verify on each jar file in the staging area. I just re-check the dd files right now, I'll re-check everything again soon and on final release, etc. Thanks again,
I have just rechecked all of the staging area, and there are only these three "incomplete jars" that are not signed. org.eclipse.emf.cdo.server.hibernate.libraries_1.0.0.v200806180411.jar org.eclipse.net4j.db.mysql_1.0.0.v200806180305.jar org.eclipse.net4j.db.hsqldb_1.0.0.v200806180305.jar Since no one (besides me :) has complained about these three, I guess they are are approved by silence. (for this one release only! :) For the recored, I thinking jar signing is important for security reasons, I think Eclipse has been embarrassingly way behind on security efforts such as this (compared to the industry), and I think part of the key in getting signed jars to actually improve security is getting all users accustomed to the fact (or for them to have the expectation) that any jar they install should be signed, so they at least know "where it comes from". Currently, I think Eclipse users just blindly install unsigned jars out of habit, since few are signed, and it is "hit or miss" on who signs and who doesn't. So, this Ganymede release is a significant step forward in changing users expectations and setting a good example for the community of plugin providers. I know it's hard to see the benefit of these "long term" advantages of changing expectations, setting good examples, etc., but it _is_ worth it in the long run for the betterment of Eclipse as a viable way to do open source development. I know first hand it's not easy, so much thanks to everyones efforts to get all this signing stuff to work.
