Community
Participate
Working Groups
DJ and I are testing running the Eclipse SDK and Equinox build from a git repos. We have branched the builder (org.eclipse.releng.eclipse.builder and org.eclipse.releng.basebuilder to git_test from v20110426. The org.eclipse.releng project has also been branched to git_test. A Hudson job has been created here https://hudson.eclipse.org/hudson/view/Eclipse%20and%20Equinox/job/eclipse-equinox-git-test-N/? as part of bug 344133. Andrew's blog describes how to build from git using pde build http://aniefer.blogspot.com/2011/01/building-from-git.html The git project's pde build fetch factory is available here https://hudson.eclipse.org/hudson/job/egit-pde/lastSuccessfulBuild/artifact/org.eclipse.egit.pde-updatesite/target/site/
For a point of reference, I branched org.eclipse.releng from version I20110428-0848.
Created attachment 194285 [details] builder changes Patch contains: - addition of the Git fetch factory bundle to the plugins/ folder of the basebuilder - set the mapVersionTag property in the build.properties to be the stream name (git_test)
Kim, please commit the patch to the git_test branch. Thanks.
Created attachment 194290 [details] patch
The egit.fetchfactory bundle was corrupt. Released a new jar and have started a new test build.
Created attachment 194396 [details] patch to update maps The fetch factory is now working but the git command to mirror the sources was not. I ran the following command to mirror the repo on build.eclipse.org kmoir@build:~/git> git clone http://dev.eclipse.org/git/org.eclipse.equinox.p2/org.eclipse.equinox.p2.git . Cloning into .... remote: Counting objects: 155562, done. remote: Compressing objects: 100% (32318/32318), done. remote: Total 155562 (delta 68320), reused 155274 (delta 68157) Receiving objects: 100% (155562/155562), 64.13 MiB | 20.31 MiB/s, done. Resolving deltas: 100% (68320/68320), done. kmoir@build:~/git> ls bundles examples features org.eclipse.equinox.p2.releng kmoir@build:~/git> The patch adds http:// to the maps for the git repo. The previous generated git clone command was git clone dev.eclipse.org/org.eclipse.equinox.p2/org.eclipse.equinox.p2.git which didn't work.
Created attachment 194399 [details] another patch for maps
Ran a test build yesterday http://eclbld1.ottawa.ibm.com/downloads/bogus/downloads/drops/I20110509-1555/index.php It's failing checking out a tag from the local repo. Investigating.
DJ, the problem is that the v20110419-0147 tag associated with many of the p2 bundles in the git_test branch of org.eclipse.releng doesn't exist in the git repository. git clone http://dev.eclipse.org/git/org.eclipse.equinox.p2/org.eclipse.equinox.p2.git cd org.eclipse.equinox.p2.git git tag (many tags) -bash-3.2$ git tag | grep v20110419-0147 -bash-3.2$ Can you fix this and I can run another test build?
How do I fix this? I thought that repo was a clone of the CVS repo with all the tags copied. I have verified that this version does exist in CVS.
(In reply to comment #10) > How do I fix this? I thought that repo was a clone of the CVS repo with all the > tags copied. I have verified that this version does exist in CVS. For the read-only git repos, because they track the CVS repo, they're using git-cvsimport. It has a big warning: "...Tags and branches may be missing entirely or contain missing / corrupted files..." For an actual conversion of a CVS module to a git repo, they use cvs2git. But that creates an entire repo out of its input, it doesn't do incremental. From: http://wiki.eclipse.org/Git/Migrating_to_Git PW
Cool, thanks. So basically we can't build from tags from these repos. Ok I'll look at the Git fetch factory in PDE/Build tomorrow and see if there is a "HEAD"-like keyword that means just build the latest from the repo. Although given the above warnings of corrupt files, I wonder if we are wasting our time trying to build from these repos.
(In reply to comment #12) > Cool, thanks. So basically we can't build from tags from these repos. Ok I'll > look at the Git fetch factory in PDE/Build tomorrow and see if there is a > "HEAD"-like keyword that means just build the latest from the repo. Although > given the above warnings of corrupt files, I wonder if we are wasting our time > trying to build from these repos. The equivalent to CVS HEAD is simply master PW
Yep, just need to make sure that PDE/Build handles it. :-)
The fetch factory looks like it just passes the tag as-is to git so I changed the map file entries and kicked off another build.
The build completed successfully https://hudson.eclipse.org/hudson/view/Eclipse%20and%20Equinox/job/eclipse-equinox-git-test-N/10/artifact/builds/transfer/files/master/downloads/drops/I20110510-2043/ I downloaded it and tried it out. It seems fine. One thing I noticed is that the source references on the bundle manifests are not being generated. Will investigate further. Thanks DJ! Just as an aside, this build is run as in integration build. Which works because it will fetch the master tag for bundles from the git mirror or the repo. I think in order to have a nightly build we would have to change all the bundles to point to the git repo because you can only specify one value to replace the tags. Not HEAD(CVS) and master(git).
Yeah, the source references is on my list of things that would be effected by the move to git and something that needs to be looked at. Is the nightly/integration build difference controlled by fetchTag? If so, it looks like we could specify different tags for different repository types. # The tags to use when fetching elements to build. # By default the builder will use whatever is in the maps. # This value takes the form of a comma separated list of repository identifier (like used in the map files) and the # overriding value # For example fetchTag=CVS=HEAD, SVN=v20050101 # fetchTag=HEAD
Yes, it's controlled by fetchTag. I didn't know it could be replaced by different values for different SCMs. Learn something new every day :-)
I think this whole "move to git" thing is a learning experience for everyone. :-)
As an aside I decided to check out the size of our CVS repos. On the CDT bug, it mentioned their repo was around 100M. kmoir@build:/home/data/cvs> du -sh eclipse 15G eclipse kmoir@build:/home/data/cvs/rt/org.eclipse.equinox> du -sh * 12M compendium 5.0M components 188M framework 6.8G incubator 269M p2 2.0M security 1.4M server-side 1.2M weaving
DJ, I need a mapping of committer id and email address to do the conversion. See https://bugs.eclipse.org/bugs/show_bug.cgi?id=316208#c6 Is http://www.eclipse.org/equinox/p2/ a complete list of p2 committers?
I assume the list on Dash is up to date. http://dash.eclipse.org/dash/commits/web-api/commit-committers-by-project.php aniefer (aniefer@ca.ibm.com) bdealwis (briandealwis@gmail.com) dj (dj_houghton@ca.ibm.com) dleberre (leberre@cril.univ-artois.fr) helindbe (henrik.lindberg@cloudsmith.com) irbull (irbull@eclipsesource.com) iyamasaki (yamasaki.ikuo@lab.ntt.co.jp) jeff (jeff@eclipsesource.com) jglander (jens.glander@sap.com) johna (john_arthorne@ca.ibm.com) kmoir (kmoir@ca.ibm.com) mpiggott (mpiggott@sonatype.com) mzhu (kane.zhu@windriver.com) prapicau (pascal@sonatype.com) sfranklin (susan_franklin@us.ibm.com) skaegi (simon_kaegi@ca.ibm.com) sliebig (Stefan.Liebig@compeople.de) spingel (steffen.pingel@tasktop.com) thallgren (thomas@tada.se) toberlies (tobias.oberlies@sap.com) One user (dstevenso) is no longer working at IBM and is inactive. I don't know what his email would be. One user (ibull) I am not sure is correct... Ian uses irbull.
Created attachment 198337 [details] UI map to test repo released to git_test
The test build https://hudson.eclipse.org/hudson/view/Eclipse%20and%20Equinox/job/eclipse-equinox-git-test-N/16/ died. /opt/users/hudsonbuild/workspace/eclipse-equinox-git-test-N/builds/I201106210947/src/fetch_org.eclipse.sdk.tests.xml:1030: /opt/users/hudsonbuild/workspace/eclipse-equinox-git-test-N/builds/I201106210947/org.eclipse.releng.basebuilder/plugins/org.eclipse.pde.build_3.7.0.v20110512-1320/scripts/scmCache/file____gitroot_e4_eclipse_platform_ui_git/${path} does not exist. I had tests=tests/... instead of path=tests/... on one line. I've committed a fix. PW
I've opened bug 349996 for the generated source references. PW
I think this bug can be closed as we are working through the git migration.
