Community
Participate
Working Groups
As the old modeling-common based build system is no longer maintainable, we should set-up a new build infrastructure. Hudson/Tycho seem to be the technologies that are commonly accepted for this.
Before investigating to migrate GEF proper to Tycho/Hudson, I have - as stated on bug #347636 - started to set up a Tycho/Hudson based infrastructure for the GEF4 components. The build is set-up on hudson.eclipse.org (gef4-nighly-tycho). Things I want to clarify/implement by means of this build before migrating GEF proper are: 1) Checking out sources from cvs git mirror instead of cvs (so build does not have to be migrated afterwards, except changing the url to the git repo) 2) generation of javadoc/wikitext (already implemented and tested locally for org.eclipse.gef4.geometry.doc bundle, testing on server still has to be performed (currently blocked because 1) is not working yet) 3) source bundle generation 4) signing
Generation of source bundles and signing are now working for GEF4 based Tycho build. Last thing will be promotion, then we can transfer the results and setup a new build for GEF proper as well.
Requested Hudson build job for GEF proper nightly builds by means of bug #365005, so adding a dependency on it.
(In reply to comment #2) > Generation of source bundles and signing are now working for GEF4 based Tycho > build. Last thing will be promotion, then we can transfer the results and setup > a new build for GEF proper as well. Promotion for GEF4 is working well now. The Tycho-based build produces a signed update site, which can be published to download.eclipse.org via a shell script (publish.sh within org.eclipse.gef4.repository), which allows to replace the remote update site content as well as to merge it with the existing content. As such, I am now ready to transfer everything to GEF proper. The hudson job is already in place, so there is no blocker left. As I will have to do changes to the sources, I think creating a branch for this will be the best option, so the old build is still usable until the new one is online.
Created branch TYCHO_BUILD_MIGRATION.
Created two new CVS sub-modules, which take the new projects needed for Tycho-based builds 1) dev: org.eclipse.gef.baseline org.eclipse.gef.releng org.eclipse.gef.repository org.eclipse.gef.target 2) source-features org.eclipse.draw2d.source-feature org.eclipse.gef.examples.source-feature org.eclipse.gef.source-feature org.eclipse.gef.test.source-feature org.eclipse.zest.source-feature
OK, first part of the mission accomplished. We have a running Tycho-based build for the development stream in place (which up to now builds against the TYCHO_BUILD_MIGRATION branch). The build has support for: - source bundle generation - test execution - javadoc generation - signing It produces an update site, which can be published to download.eclipse.org using a shell script I have added to org.eclipse.gef.repository. The publishing script does not provide sdk drop files yet. It would be easy to set that up (using repo2runnable), but I doubt whether this is still needed (as people may easily produce them based on the update site as well).
In case drop files are still needed, it would be the next step to update the publish script to respective sdk drop files are produced. If we can live without, we could skip this step and merge the changed back into HEAD (and switch the nightly build to HEAD).
Anthony, what do you think w.r.t. to the drop files? I would like to not further provide them, as users can easily extract them themselves using repo2runnable.
BTW, the build job is located here: https://hudson.eclipse.org/hudson/job/GEF-tycho-nightly/, the update site (which does not get promoted to download.eclipse.org yet; I will change that as soon as we have merged the changes and build against HEAD) can be found here: https://hudson.eclipse.org/hudson/job/GEF-tycho-nightly/lastSuccessfulBuild/artifact/update-site/
OK, I have started to work on extending the publish script to produce drop files as well (I think we should provide a bundled update site and at least the ALL-zip in the download area). I will further extend the script to generate some static .html page to list the contents within the created drop directory. As soon as this is working, I will have to migrate the download-webpages to include these generated .html files properly. Anthony, as a preparation for migrating to this new infrastructure, could you please archive all old 3.5.x and 3.6.x versions? I don't know how this is currently performed...
OK, publish script now supports generation of SK drop files as well, with the exception of the GEF-Automated-Tests, which are now available within the Hudson build job only. Accordingly, the publish script does not longer publish any buildNotes, testResults, etc. but only a build.cfg, which contains a link to the Hudson job it was published from. I published a 3.8.0 integration build using the script for test purposes. Concerning the update site, I used http://download.eclipse.org/tools/gef/update-sites/integration instead of http://download.eclipse.org/tools/gef/updates/interim/ to avoid any collision (the existing update site seems to be a mix of p2-repo and old-style-site, maybe we should start with a fresh one?). The webpages still have to be adopted to do not show links to missing test results, etc. but to point to the Hudson build job for build details (the common-modeling infrastructure does not handle such a case properly; I think it would be best to make our php pages self-encapsulated anyway, so this is a topic on its own).
I uploaded a modified version of the downloads index page (http://www.eclipse.org/gef/downloads/index2.php) for testing purposes. It does not expect any test results and does also skip the RSS feed linking, but it shows a link to the Hudson build job in the build details.
OK. I think last thing that has to be decided is whether we can start with fresh update site urls (what I would recommend) or whether we reuse the existing ones. After that decision has been made, I can merge my changes into HEAD and put everything live (including the modified download pages on our homepage). Anthony, are we good to go or are you missing something?
An addition concerning the update site. I think it will be best to replace the current contents of the interim and milestones pages, but to preserve the current contents of the releases update sites (so they are all kept in place). I will thus try to convert the releases update site to the new form (so it can be merged without problems with updates sites published by this build).
Created attachment 209058 [details] Re-created GEF releases update site. Having investigated the contents of our current releases update site, I recognized that it contains the 3.2.2 and 3.3.2, but only the 3.4.1, 3.4.2, 3.5.0, 3.5.2, 3.6.2, and 3.7.1 releases, so that 3.4.0, 3.5.1, 3.6.0, 3.6.1, and 3.7.0 are missing. Furthermore, the releases are grouped under their respective own categories (e.g. GEF SDK 3.4.1), which is (in my opinion) not a good idea, because the category will have to be changed for each release (in case somebody forgets it, the site gets inconsistent). As such, I used the update-site zip drop files from our downloads page and re-created the releases update site to contain all releases from 3.4.1 onwards. I also re-catorized them all to be in the GEF category. I used an ant script with p2.mirror and the p2 category publisher application to achieve this (and had to manually process all update zips with jarProcessor, because they - beginning from 3.5.0 only contain packed bundles). Having put our Tycho/Hudson build structure live, I propose to clean the milestones and interim update sites (and publish fresh 3.7.2 and 3.8.0 milestones and integrations there) and to replace the releases site with the one I have uploaded here. We may backup the current contents however (as it was done when migrating to the modeling build infrastructure) so it is still accessible.
OK, I think everything is now pretty much prepared, and the new build system could be put live for the 3.8.0 stream. In order to prevent that we have to merge it also into the 3.7 maintenance branch, I would propose that we release this first (using the old build mechanism) and afterwards add its contents to the re-created update site I have created for all builds up to 3.7.1 before replacing the releases site with its contents. The only thing that in my eyes could favor migrate the new build system into the 3.7 maintenance branch also, is that we would get our javadoc and help pages back; bug #327261, which I have fixed in the new build system). Assuming that we promote the 3.7.2 build using the old build system in advance, I would propose the following migration strategy (responsibilities in brackets): 1) Move all pre-3.7.0 builds from downloads.eclipse.org to archive.eclipse.org (ahunter) 2) Promote the 3.7.2 release using the old build system (ahunter). I would propose to do this as soon as possible, and not wait to the official release dates, because there are no outstanding issues that would have to be addressed for 3.7.2 (except bug#327261, which we may only solve by applying the new build system also in the 3.7 maintenance branch). 3) Add the 3.7.2 release to the re-created releases update site, and clean all existing sites (i.e. move the old contents to other locations, replace the releases site with the re-created one) (anyssen) 4) Merge the changes of the TYCHO-BUILD-MIGRATION branch to HEAD (3.8.0) (anyssen) 5) Put the new downloads page life (index2.php, adding a link to the old one) (anyssen) 6) Disable the old nightly build system on modeling.eclipse.org 7) Create cronjob for publishing weekly integration builds based on the new build infrastructure to downloads.eclipse.org; a publish script to do so is there, nightly builds can directly be consumed from hudson.eclipse.org, so we do not need to promote them (nyssen)
(In reply to comment #17) > OK, I think everything is now pretty much prepared, and the new build system > could be put live for the 3.8.0 stream. In order to prevent that we have to > merge it also into the 3.7 maintenance branch, I would propose that we release > this first (using the old build mechanism) and afterwards add its contents to > the re-created update site I have created for all builds up to 3.7.1 before > replacing the releases site with its contents. The only thing that in my eyes > could favor migrate the new build system into the 3.7 maintenance branch also, > is that we would get our javadoc and help pages back; bug #327261, which I have > fixed in the new build system). > > Assuming that we promote the 3.7.2 build using the old build system in advance, > I would propose the following migration strategy (responsibilities in > brackets): > > 1) Move all pre-3.7.0 builds from downloads.eclipse.org to archive.eclipse.org > (ahunter) > > 2) Promote the 3.7.2 release using the old build system (ahunter). I would > propose to do this as soon as possible, and not wait to the official release > dates, because there are no outstanding issues that would have to be addressed > for 3.7.2 (except bug#327261, which we may only solve by applying the new build > system also in the 3.7 maintenance branch). > > 3) Add the 3.7.2 release to the re-created releases update site, and clean all > existing sites (i.e. move the old contents to other locations, replace the > releases site with the re-created one) (anyssen) > > 4) Merge the changes of the TYCHO-BUILD-MIGRATION branch to HEAD (3.8.0) > (anyssen) > > 5) Put the new downloads page life (index2.php, adding a link to the old one) > (anyssen) > > 6) Disable the old nightly build system on modeling.eclipse.org > > 7) Create cronjob for publishing weekly integration builds based on the new > build infrastructure to downloads.eclipse.org; a publish script to do so is > there, nightly builds can directly be consumed from hudson.eclipse.org, so we > do not need to promote them (nyssen) 8) Change hudson job to build against HEAD instead of TYCHO-BUILD-MIGRATION (anyssen) 9) Add tagging support to build so we can keep track of which sources were used in each build (anyssen). And a last note on future release policy: The Hudson job should be renamed from gef-nightly-tycho to gef-HEAD or gef-master (anticipating the migration to Git), because the builds produced there will also be promoted as milestone or release builds. After 3.8.0 is released, we will have to create an additional job (gef-maintenance) which produces builds accordingly against the 3_8_maintenance branch.
Hi Alexander, sounds good, I will try to get these things done this week. Will update this work item as I progress.
(In reply to comment #17) > 2) Promote the 3.7.2 release using the old build system (ahunter). I would > propose to do this as soon as possible, and not wait to the official release > dates, because there are no outstanding issues that would have to be addressed > for 3.7.2 (except bug#327261, which we may only solve by applying the new build > system also in the 3.7 maintenance branch). I have ran a 3.7.2 release build and promoted it to the downloads site.
(In reply to comment #17) > 6) Disable the old nightly build system on modeling.eclipse.org Done
> > 3) Add the 3.7.2 release to the re-created releases update site, and clean all > > existing sites (i.e. move the old contents to other locations, replace the > > releases site with the re-created one) (anyssen) Done. Merged 3.7.2 release into update-site. Moved old update sites (updates -> updates-pre-3_8). Re-created updates/releases with merged content.
> > 4) Merge the changes of the TYCHO-BUILD-MIGRATION branch to HEAD (3.8.0) > > (anyssen) Done. Created tag PRE_MERGING_TYCHO_BUILD_MIGRATION_BRANCH, then merged all changes into current HEAD. Switched gef-nighly-tycho job to build against HEAD. Unfortunately the build currently fails with the following message, so I was not yet able to produce and publish a first HEAD build: a cvs [checkout aborted]: connect to dev.eclipse.org(172.25.25.51):2401 failed: No route to host
> > 5) Put the new downloads page life (index2.php, adding a link to the old one) > > (anyssen) Done. replaced downloads/index.php with index.2.php. Provided a link to the old version in case it is still needed. > 8) Change hudson job to build against HEAD instead of TYCHO-BUILD-MIGRATION > (anyssen) Done, as already stated earlier. Will promote the first successful build as soon as we have one...
10) Clean-up the CVS repository and move all old releng projects (there are quite a few ones) and old pom.xml within root of repository to archive.
Published first integration build 3.8.0 (2012/01/17) to interim update site and drop ins.
(In reply to comment #26) > Published first integration build 3.8.0 (2012/01/17) to interim update site and > drop ins. As stated on the cross-platform mailing list, also published a first milestone build (so the site is not empty). However, I think we should replace this with an official M5 build.
11) Update the contribution guide to mention that we now have a target and a baseline project, and how to locally perform a Tycho based build.
> 7) Create cronjob for publishing weekly integration builds based on the new > build infrastructure to downloads.eclipse.org; a publish script to do so is > there, nightly builds can directly be consumed from hudson.eclipse.org, so we > do not need to promote them (nyssen) Done with the following settings: 10 23 * * 5 Build will be merged into the interim site, drop files are created as well.
(In reply to comment #25) > 10) Clean-up the CVS repository and move all old releng projects (there are > quite a few ones) and old pom.xml within root of repository to archive. Done. - Moved releng folder to archive - Moved org.eclipse.gef.releng to archive - Moved genpom.scala, parent-pom.xml, and README.build.with.Tycho.txt files to archive
OK, I think we are done now despite the following issues: 1) Move all pre-3.7.0 builds from downloads.eclipse.org to archive.eclipse.org (ahunter) 9) Add tagging support to build so we can keep track of which sources were used in each build (anyssen). 11) Update the contribution guide to mention that we now have a target and a baseline project, and how to locally perform a Tycho based build. I will try to investigate 11) this week. As hudson.eclipse.org does not deliver a tagging plug-in, and as cvs checkout is performed via date, tagging support is not crucial and may be postponed until we have migrated to Git. As such, 1) and 11) remain to be done.
(In reply to comment #30) > (In reply to comment #25) > > 10) Clean-up the CVS repository and move all old releng projects (there are > > quite a few ones) and old pom.xml within root of repository to archive. > > Done. > > - Moved releng folder to archive > - Moved org.eclipse.gef.releng to archive > - Moved genpom.scala, parent-pom.xml, and README.build.with.Tycho.txt files to > archive Additionally moved test/org.elipse.test and test/org.eclipse.ant.optional.junit and test/.cvsignore to archive.
(In reply to comment #31) > 11) Update the contribution guide to mention that we now have a target and a > baseline project, and how to locally perform a Tycho based build. Done.
> 9) Add tagging support to build so we can keep track of which sources were used > in each build (anyssen). Added a comment to bug #351232 that tagging has to be enabled for the build, as soon as the transition to Git has been completed, so this no longer is an issue here.
> Additionally moved test/org.elipse.test and test/org.eclipse.ant.optional.junit > and test/.cvsignore to archive. Also ensured, they are no longer bundled by the test-feature.
> 1) Move all pre-3.7.0 builds from downloads.eclipse.org to archive.eclipse.org > (ahunter) I moved 3.5.2 into archive. I think it is save to leave the 3.6 builds in place until we have released 3.8.0. As such all points seem to have been resolved. Mission accomplished! GEF has a new build infrastructure.
As commented in bug #383103, the build job to build the origin/master branch of our repo has been renamed from gef-nightly-tycho to gef-master. Accordingly, we now also have a gef-maintenance build job which I will use for the respectively latest maintenance branch (as soon as we have created it after the Juno release).
The updatesite http://download.eclipse.org/tools/gef/updates/releases/ contains partial IUs and is not usable with tycho > 0.17.0! Can you please tell me which site to use with tycho 0.18.1?
