Community
Participate
Working Groups
A couple of weeks ago we decided to try a new model where builds are automatically tagged from a designated build branch. I am opening this bug to cover logistics and builder changes to convert to this approach. A brief summary of the approach: - Nightly builds run from "master" branch. No tagging is done. I think the builder stay unchanged here. It will continue to hammer the map files to point to master rather than the tags in the maps. - Integration builds run from "integration" branch. The builder tags all repositories, and updates the map files for any bundles that have changed. I believe the builder itself doesn't need changes here, but we need an extra step in the bootstrap script to perform the tagging. On logistics, the question is how it fits into the current priority list: 1) Git migration 2) Conversion of Eclipse 4 to full builder 3) Moving build to eclipse.org. I currently see this as part of 2). There's not a lot of value in switching 3.x development to a two-branch model because soon we won't be doing 3.x nightly builds anymore. Without a nightly build there is no longer much value in having two branches.
Kim, Paul and I talked about this yesterday afternoon. Doing conversion of builder to Eclipse 4 will probably require forking our map file project. Once we do this, it will be painful for a human to perform the tagging. Therefore we think the best approach is to switch to auto-tagging immediately for 3.8 builds. Then when the 4.x full build is introduced committers don't have to deal with forked map files.
Assigning to myself to start working on script changes for this.
First step is a list of all repositories used in our build. This looks complete: johna@build:~/test/eclipse.platform.releng.maps/org.eclipse.releng/maps> cat *.map |cut -d, -f3 | grep ^repo= | cut -c6- | sort | uniq git://git.eclipse.org/gitroot/equinox/rt.equinox.bundles.git git://git.eclipse.org/gitroot/equinox/rt.equinox.framework.git git://git.eclipse.org/gitroot/equinox/rt.equinox.incubator.git git://git.eclipse.org/gitroot/equinox/rt.equinox.p2.git git://git.eclipse.org/gitroot/jdt/eclipse.jdt.core.binaries.git git://git.eclipse.org/gitroot/jdt/eclipse.jdt.core.git git://git.eclipse.org/gitroot/jdt/eclipse.jdt.debug.git git://git.eclipse.org/gitroot/jdt/eclipse.jdt.git git://git.eclipse.org/gitroot/jdt/eclipse.jdt.ui.git git://git.eclipse.org/gitroot/pde/eclipse.pde git://git.eclipse.org/gitroot/pde/eclipse.pde.build.git git://git.eclipse.org/gitroot/pde/eclipse.pde.ui.git git://git.eclipse.org/gitroot/platform/eclipse.platform.common.git git://git.eclipse.org/gitroot/platform/eclipse.platform.debug.git git://git.eclipse.org/gitroot/platform/eclipse.platform.git git://git.eclipse.org/gitroot/platform/eclipse.platform.resources.git git://git.eclipse.org/gitroot/platform/eclipse.platform.runtime.git git://git.eclipse.org/gitroot/platform/eclipse.platform.swt.binaries.git git://git.eclipse.org/gitroot/platform/eclipse.platform.swt.git git://git.eclipse.org/gitroot/platform/eclipse.platform.team.git git://git.eclipse.org/gitroot/platform/eclipse.platform.text.git git://git.eclipse.org/gitroot/platform/eclipse.platform.ua.git git://git.eclipse.org/gitroot/platform/eclipse.platform.ui.git
Created attachment 206864 [details] Shell script Here is an initial script. It looks for a file org.eclipse.releng/tagging/repositories.txt, which has a list in the following format: <repo> <branch> <repo> <branch> .. etc .. For each repository it does a pull and checkout of correct branch (or clone if it doesn't exist). It then runs the e4 tagging scripts, and finally pushes the map file changes to git. Note I have commented out the final two "git push" operations so it can be used for testing without modifying the releng repository. One potential issue is this requires creating a separate clone of all our repositories. It's possible this could run directly on /gitroot but I would be scared to try that for worry of damaging the canonical repository. Otherwise, if there is some other part of our build that does a full clone perhaps we could reuse that here. Note after the first clone, subsequent runs are relatively fast because it just does a git pull. So if we have a space where disk footprint is not an issue maybe it is ok.
Thanks John. Will have to try it out. Yes, cloning all our repos is very expensive in terms of the time it consumes. Today, the build clones all our repos to the build machine but it's a sequential process as the fetch scripts are generated for the features and called, the appropriate repos are cloned. As a side note, I should test reusing repositories across builds using fetchCacheLocation to avoid cloning the repos from scratch each build. I've opened bug 363603 for this issue.
Created attachment 207053 [details] Updated script A few minor updates. Note you'll need to change to user=kmoir and writableBuildRoot to some reasonable writable location for the clones.
Moving to Kim to integrate with the build bootstrap script. Note for future reference or for anyone else trying this: To fully automate tagging you need to add your own public ssh key to your ~/.ssh/authorized_keys file. This enables you to ssh as yourself from build.eclipse.org to git.eclipse.org. Actually even if you switch your git URLs to use build.eclipse.org you need to authorize yourself to ssh as yourself. Also to save people from the same n00b mistake I made, don't forget to 'chmod 600 authorized_keys' or it won't work.
I've been working on a version of this for our current 4.2 short build/e4 0.12 build. There are some fixes over the default attached script. http://git.eclipse.org/c/e4/org.eclipse.e4.releng.git/tree/org.eclipse.e4.builder/scripts PW
The fixes I included in the script: 1) in pull() use -d instead of -f to see if the repo has been cloned: if [ ! -d $directory ]; then 2) if the pull() function has to clone, it should set user.email and user.name (e4Build doesn't have a global default, and we probably don't want to set one) 3) in pull(), do git checkout and then git pull (so you merge into the correct branch) 4) always copy the git-map.sh and git-submission.sh scripts so you can get the latest available. 5) wget the git-map.sh and git-submission.sh scripts from http://git.eclipse.org/c/e4/org.eclipse.e4.releng.git now 6) make sure clones.txt has the repos in the git://git.eclipse.org/... form since that's what is in the maps. PW
Thanks Paul. I think these all make sense. Kim do you want me to attach an updated script, or are you already hacking on it?
I have hacked it already thanks.
I have it working on the build machine. The scripts didn't pick up any changes that were new to the integration branch compared to the last integration build. Paul is going to look and see if he can push some stuff to the integration stream so we can verify this. One question: The tag is currently in the format $buildTypeYYYYMMDDHHMM but previously the tag was in the format $buildTypeYYYYMMDD-HHMM which corresponds to the $buildId which reflects the directory name and the name included in the zips. Should we change this?
(In reply to comment #12) > One question: The tag is currently in the format $buildTypeYYYYMMDDHHMM but > previously the tag was in the format $buildTypeYYYYMMDD-HHMM which corresponds > to the $buildId which reflects the directory name and the name included in the > zips. Should we change this? I'd like to see it consistent with the build id, to. I've pushed new changes to R3_integration so there should be a diff. PW
Created attachment 207250 [details] updated script for 3.8 stream builds
(In reply to comment #13) > (In reply to comment #12) > > One question: The tag is currently in the format $buildTypeYYYYMMDDHHMM but > > previously the tag was in the format $buildTypeYYYYMMDD-HHMM which corresponds > > to the $buildId which reflects the directory name and the name included in the > > zips. Should we change this? > > I'd like to see it consistent with the build id, to. I tend to agree, but there is one problem with this tag pattern: currently most projects use a pattern that starts with a 'v'. If we decide to go with the new scheme (which I think is good), then we must ensure that the tag still increases compared to the previous build. To achieve this most projects will have to increase their service number.
(In reply to comment #15) > I tend to agree, but there is one problem with this tag pattern: currently most > projects use a pattern that starts with a 'v'. If we decide to go with the new > scheme (which I think is good), then we must ensure that the tag still > increases compared to the previous build. To achieve this most projects will > have to increase their service number. Kim can correct me if I'm wrong, but we're just talking about the tag on org.eclipse.releng repo (I20111122-0800 for example). Our git-release.sh script was just generating I201111220800. The auto-tagging scripts will use v<YYYYmmDD-HHMM> in the map files and on the source repos. PW
> The auto-tagging scripts will use v<YYYYmmDD-HHMM> in the map files and on the > source repos. > > PW Beauty then!
regarding comment #16, yes that's what I meant
(In reply to comment #12) > One question: The tag is currently in the format $buildTypeYYYYMMDDHHMM but > previously the tag was in the format $buildTypeYYYYMMDD-HHMM which corresponds > to the $buildId which reflects the directory name and the name included in the > zips. Should we change this? +1
I think this can be closed. There is bug 364676 to implement these changes in the maintenance branches.
