Community
Participate
Working Groups
Eclipse needs to setup and publish a CBI-built build from build.eclipse.org. In general, we need to get the source, run maven using BREE libs, signing, and the comparator, and extract the build artifacts so they can be published. I'll open bugs as a starting place for the steps we need. PW
A set of steps to start from: - Bug 393923 - [CBI] clone the aggregator to run the build - Bug 393925 - [CBI] replace auto-tagging - Bug 393926 - [CBI] execute the maven build - Bug 393927 - [CBI] Generate Deployable build directory PW
I've re-ordered the build steps a little: 1) clone or clean the repos 2) update the build input 3) tag the build inputs 4) install the signing plugin, parent, cbi test support 5) run the pom version updater (no checkin) 6) run the build PW
David, I've made some progress on this, the starting point is http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/tree/scripts/master-build.sh?h=R4_2_maintenance I'm trying to find where the files like linPlatform.php come from, and where the tests are run and the results packaged up. What's doing the work? I found eclipse.platform.releng.eclipsebuilder/eclipse/helpernew.xml ... is that what drives packaging everything up into its respective M20121121-1200 dir? Right now I drive it with a simple set of copy scripts (with a little bit of massaging), see http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/tree/scripts/gather-parts.sh?h=R4_2_maintenance and http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/tree/scripts/build-functions.sh?h=R4_2_maintenance#n353 PW
(In reply to comment #3) > David, I've made some progress on this, the starting point is > http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/ > tree/scripts/master-build.sh?h=R4_2_maintenance > > I'm trying to find where the files like linPlatform.php come from, and where > the tests are run and the results packaged up. What's doing the work? > > I found eclipse.platform.releng.eclipsebuilder/eclipse/helpernew.xml ... is > that what drives packaging everything up into its respective M20121121-1200 > dir? > You can see linPlatform.php is created in either helper.xml or helpernew.xml. This was something that was "half done" when I took over from Kim. She was, I think, starting to refactor "helper.xml", instead using "helpernew.xml" to be for "build.eclipse.org" specific changes. I once tried to get rid of one of them, and found it wasn't so simple ... both are still being used, by different parts of the main scripts ... and that's about as far as I got. See bug 392520. I believe currently the build uses "publish" from helper.xml and the tests use "publish" from helpernew.xml ... but ... it'd take some study and experiments to sort that all out. I'm not sure if they do much different, or not. Welcome to my world! :)
(In reply to comment #4) > You can see linPlatform.php is created in either helper.xml or > helpernew.xml. > I believe currently the build uses "publish" from helper.xml and the tests > use "publish" from helpernew.xml ... but ... it'd take some study and > experiments to sort that all out. I'm not sure if they do much different, or > not. Thanks for the update. Which parts of the scripts actually take the automated tests zip and kick off a test run on hudson? That's somewhere else than helpernew.xml right? And then helpernew is responsible for waiting for the test results to come available, I guess. PW
David, something more general to think about: Right now I've based my work on scripts, similar the my tag/release scripts and build scripts. They'll run on linux, and maybe to a lesser degree in git bash shell on windows. The main build contains a lot of ant scripts as well. Should our build be controlled by ant scripts instead of shell scripts? That sounds like more work (although it's an opportunity to learn ant :-), but if that's the direction you were taking the build scripts I don't want to set us back. PW
(In reply to comment #5) > (In reply to comment #4) > Which parts of the scripts actually take the automated tests zip and kick > off a test run on hudson? That's somewhere else than helpernew.xml right? > And then helpernew is responsible for waiting for the test results to come > available, I guess. > No, not exactly. The big picture is as the build finishes, it creates a "promote script" in a special directory, I have a cronjob that checks there every now and then for a build to promote (since that must be done by a committer id). At the end of the promotion, the promote script invokes startTests.sh which is mostly a signal to Hudson to start _its_ tests jobs, which are defined in hudson, of course. Those hudson jobs have (basically) three steps (and can be invoked "manually" if you provide the buildId and build stream). [A good way to get started with CBI builds, and then automate the cron jobs, etc.]. a) gets eclipse builder using wget with cgit URL b) uses a script from eclipse builder to get basebuilder (which is mostly to runAntrunner, with our custom tasks) c) invokes the tests with "runTests2.xml" (in eclipsebuilder). Collecting/publishing the results is a whole other matter which I assume you are not asking about right now, but it too is essentially started from another of my own cronjobs, checking for certain "data files" (which describe where to get the test results from, as well as signaling the results are ready). When you are ready, I can "duplicate" our existing hudson jobs, and you can tweak how you'd like. Some of the locations in some of the scripts, such as "runtests2.xml" are semi-hard coded, so that for example, from the simple input of buildId and buildStream, we know the downloads should some from http://download/.../drops/${buildId} or .../drops4/${buildId}. I'm assuming your CBI builds would be in a different directory, if not still on build machine?
(In reply to comment #6) > Right now I've based my work on scripts, similar the my tag/release scripts > and build scripts. They'll run on linux, and maybe to a lesser degree in > git bash shell on windows. > > The main build contains a lot of ant scripts as well. Should our build be > controlled by ant scripts instead of shell scripts? That sounds like more > work (although it's an opportunity to learn ant :-), but if that's the > direction you were taking the build scripts I don't want to set us back. > No, for builds, we have the right amount of shell scripts (e.g. masterBuild.sh) which invokes ant and build.xml at a good point). I've not thought of changing that ... though obviously is a good long term idea. The tests script do differ a little, naturally, but for example, webmasters installed a "unix-like" wget on the windows machine for us ... but had to resort to xcopy there instead of rsync ... a good way to learn Windows :)
I've managed to run a build to completion and collect artifacts from it: http://download.eclipse.org/eclipse/staging/cbi/drops4/M20121128-1918/ Some of the zipped repos don't look right (like org.eclipse.rcp) but the SDK products and swt*zips are correct. PW
The current state of the scripts. You can run our builds as e4Build on build.eclipse.org. The root build directory is /shared/eclipse/builds. I've only been testing the 4.2.2 build, 3.8.2 and 4.3 would probably need their properties file updated, and possibly the helper scripts in aggregator/eclipse.platform.releng.tychoeclipsebuilder. To run a build from /shared/eclipse/builds: bash$ /bin/bash scripts_R4_2_maintenance/scripts/master-build.sh scripts_R4_2_maintenance/scripts/build_eclipse_org.env build_eclipse_org.env is the starting build properties file, and should be configured per branch. The build script should create a <branch> directory and everything else is contained in there. * R4_2_maintenance/dirs - the build output directories ** all of the output is captured in a buildlogs/buildOutput.txt file, which can grow quite large. * R4_2_maintenance/gitCache - the git repo directories. It also currently has the maven local repo for that build * R4_2_maintenance/org.eclipse.releng.basebuilder_R38M6PlusRC3D - the basebuilder used by that build I've currently been deploying the 4.2.2 builds to a staging directory, after changing the download.php file: bash$ sed 's!downloads/drops!staging/cbi/drops!g' download.php >/tmp/t1 bash$ mv /tmp/t1 download.php bash$ cd .. bash$ scp -r M20121231-0716/ pwebster@build.eclipse.org:/home/data/httpd/download.eclipse.org/eclipse/staging/cbi/drops4 bash$ wget -O t.txt http://download.eclipse.org/eclipse/staging/cbi/createIndex4x.php bash$ scp t.txt pwebster@build.eclipse.org:/home/data/httpd/download.eclipse.org/eclipse/staging/cbi/index.html PW
It looks like https://hudson.eclipse.org/platform/job/eclipse-aggregator-master is a valid solution to this ticket. Could this report be closed then?
(In reply to Mickael Istria from comment #11) > It looks like > https://hudson.eclipse.org/platform/job/eclipse-aggregator-master is a valid > solution to this ticket. Could this report be closed then? The setup of the build was done in Bug 420101, not sure if the Hudson build I created is sufficient for this ticket. The result is currently not used AFAIK, Thanh and I use it only to monitor the stability of the build.
*** This bug has been marked as a duplicate of bug 372792 ***
