Bugzilla – Full Text Bug Listing
| Summary: | [releng] migrate Mylyn projects to Git | ||||||
|---|---|---|---|---|---|---|---|
| Product: | z_Archived | Reporter: | Steffen Pingel <steffen.pingel> | ||||
| Component: | Mylyn | Assignee: | Steffen Pingel <steffen.pingel> | ||||
| Status: | RESOLVED FIXED | QA Contact: | |||||
| Severity: | enhancement | ||||||
| Priority: | P3 | CC: | b.muskalla, caniszczyk, d_a_carver, eclipse, elh.mailgate, florian, greensopinion, holger.staudacher, jtk499, Kenn.Hussey, manuel.doninger, matthias.sohn, mik.kersten, pwebster, robert.munteanu, steffen.pingel, torkildr | ||||
| Version: | unspecified | Keywords: | plan | ||||
| Target Milestone: | 3.7 | ||||||
| Hardware: | PC | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Bug Depends on: | 351225, 351253, 351844, 354497 | ||||||
| Bug Blocks: | |||||||
| Attachments: |
|
||||||
|
Description
Steffen Pingel
I am very interested in seeing this happen. For me a couple of things need to be in place before considering a migration to git: * Git integration for Eclipse (EGit) that is ready for day-to-day use. * A Mylyn integration for EGit capable of generating commit comments and managing change sets. * A build system for Mylyn that fully supports git. (In reply to comment #1) > I am very interested in seeing this happen. For me a couple of things need to > be in place before considering a migration to git: > > * Git integration for Eclipse (EGit) that is ready for day-to-day use. this is pretty close for the most common tasks. Some of the edge cases are still not handled, but Vex devs have been using it regularly. It's one of the reasons they migrated to help test out EGit. > * A Mylyn integration for EGit capable of generating commit comments and > managing change sets. That is in progress. Chris is working on this. > * A build system for Mylyn that fully supports git. Already exists. Maven 3/Tycho work fine with git. (In reply to comment #2) > > * Git integration for Eclipse (EGit) that is ready for day-to-day use. > > this is pretty close for the most common tasks. Some of the edge cases are > still not handled, but Vex devs have been using it regularly. It's one of the > reasons they migrated to help test out EGit. From my experience it's still a bit rough around the edges and some parts that I frequently use such as Synchronize view are still buggy. > > * A Mylyn integration for EGit capable of generating commit comments and > > managing change sets. > > That is in progress. Chris is working on this. Right, the code is at https://github.com/caniszczyk/mylyn-git but I was under the impression that it still needs work before it supports the required feature set. > > * A build system for Mylyn that fully supports git. > > Already exists. Maven 3/Tycho work fine with git. Yes, but Mylyn is still built using PDE build. Some related discussion about the build is on bug 306191 and bug 319652. (In reply to comment #3) > Yes, but Mylyn is still built using PDE build. Some related discussion about > the build is on bug 306191 and bug 319652. Regarding bug 319652, Buckminster also supports git (and has for some time). (In reply to comment #4) > (In reply to comment #3) > > Yes, but Mylyn is still built using PDE build. Some related discussion about > > the build is on bug 306191 and bug 319652. > > Regarding bug 319652, Buckminster also supports git (and has for some time). Yep, and there is also a GIT FetchFactory for PDE Build that the e4 folks are using for those that MUST continue to use the MAP files (personally I don't see the point when git makes creating a a Release Branch so simple and easy to merge items into). Just an FYI, ECF has moved to git so might be able to learn from them as well. (In reply to comment #3) > (In reply to comment #2) > > > * Git integration for Eclipse (EGit) that is ready for day-to-day use. > > > > this is pretty close for the most common tasks. Some of the edge cases are > > still not handled, but Vex devs have been using it regularly. It's one of the > > reasons they migrated to help test out EGit. > > From my experience it's still a bit rough around the edges and some parts that > I frequently use such as Synchronize view are still buggy. Actually, I don't find the need for the synchronization view in my work flow with Git. I create a topic branch and do my development there, commit my changes, or ammend as needed, and then merge back into the main branch. My development cycle is fairly short so I know what is what, and what has changed. > > > > * A Mylyn integration for EGit capable of generating commit comments and > > > managing change sets. > > > > That is in progress. Chris is working on this. > > Right, the code is at https://github.com/caniszczyk/mylyn-git but I was under > the impression that it still needs work before it supports the required feature > set. Great opportunity to help him get it working then. :) > > > > * A build system for Mylyn that fully supports git. > > > > Already exists. Maven 3/Tycho work fine with git. > > Yes, but Mylyn is still built using PDE build. Some related discussion about > the build is on bug 306191 and bug 319652. Git FetchFactory exists, at the egit project for PDE Build. http://aniszczyk.org/2010/10/26/git-fetch-factory-for-pde-build/ (In reply to comment #3) > (In reply to comment #2) > > > * Git integration for Eclipse (EGit) that is ready for day-to-day use. > > > > this is pretty close for the most common tasks. Some of the edge cases are > > still not handled, but Vex devs have been using it regularly. It's one of the > > reasons they migrated to help test out EGit. > > From my experience it's still a bit rough around the edges and some parts that > I frequently use such as Synchronize view are still buggy. Could you open EGit bugs for the rough edges you see ? Yup, Synchronize view isn't ready for prime time, it's much too slow and lacks quite some operations. > > > * A Mylyn integration for EGit capable of generating commit comments and > > > managing change sets. > > > > That is in progress. Chris is working on this. > > Right, the code is at https://github.com/caniszczyk/mylyn-git but I was under > the impression that it still needs work before it supports the required feature > set. Mik and lately also Benny promised to work on that hence we rather concentrate on other things :-) Are there any news on this topic? EGit is to be released in version 1.0, and the feature of generating commit comments is included since one of the last versions. Using CVS if you one is used to use Git is painful :( It's on top of my list past Indigo. Trust me, I know all about CVS pains ;). If you need some help, give a ping. I migrated our company repositories from CVS to Git earlier this year and have a little experience with CVS2Git. (In reply to comment #10) > If you need some help, give a ping. I migrated our company repositories from CVS > to Git earlier this year and have a little experience with CVS2Git. Thanks, that's good to know. We migrated the org.eclipse.mylyn.docs repository as part of the restructuring and ran into some weird problems with change set aggregation (which resulted in bug 333620) and it sucked up a lot of time to get that sorted. although i'm not a "standard committer" i have heard of git here and there, and to me it seems "yet-another-version-control", perhaps i myself am deprecated by saying why not use SVN? but even so , just for the record, the choices we make today should be recorded for future re-visions of ourselfs.. (ok, maybe i got a little carried away.. but still.. ;-) ) Well, Git (and other distributed VCS) has one key advantage over SVN (and other central VCS): A contributor doesn't need commit rights on the central repository to make commits for himself, because he has cloned the whole repository and works completely local. I myself am currently working on a new feature for Mylyn (context, thus CVS repository), and i can't record my work with commits, because i don't have commit rights to the CVS repository (and even if i would have, i wouldn't want my commits to go to a central repository, because they are just for my history). With Git and a local repository i don't have this problem. (In reply to comment #13) > Well, Git (and other distributed VCS) has one key advantage over SVN (and other > central VCS): A contributor doesn't need commit rights on the central > repository to make commits for himself, because he has cloned the whole > repository and works completely local. > I myself am currently working on a new feature for Mylyn (context, thus CVS > repository), and i can't record my work with commits, because i don't have > commit rights to the CVS repository (and even if i would have, i wouldn't want > my commits to go to a central repository, because they are just for my > history). With Git and a local repository i don't have this problem. so what do you do when you DO wanna "commit"? i mean eventually something has to go central somewhen? In the Mylyn project (with CVS): I create a patch and attach this patch to the Bugzilla bug. In the EGit project (with Git): I squash my local commits into a single one per feature, and push those commits to the Gerrit code review server (egit.eclipse.org). I don't know, what the workflow with Git will be with projects, which don't use Gerrit. It would be an easy way, if pull requests on Github would work (since all Eclipse Git repositories are mirrored to Github). But i don't know, if this complies with the Eclipse IP rules. Steffen: I checked our migrated repositories another time, and there this bug with the backdated commit didn't occur, so i don't know, what causes this problem. To reflect the dependency structure outlined in http://wiki.eclipse.org/Mylyn/Architecture I would like to move some bundles as part of the EGit migration: h3. org.eclipse.mylyn.incubator bc. org.eclipse.mylyn.tests.performance (org.eclipse.mylyn) org.eclipse.mylyn.tests.performance-feature (org.eclipse.mylyn) org.eclipse.mylyn.tests.ui (org.eclipse.mylyn) Moving UI tests to the incubator is due to the SWTbot dependency. SWTbot is still in incubation and has not provided stable releases which poses a risk to the reproducibility of the build that is acceptable in the Inubator project but not in the Tasks project. h3. org.eclipse.mylyn.tasks bc. org.eclipse.mylyn.help.sdk (org.eclipse.mylyn) org.eclipse.mylyn.help.ui (org.eclipse.mylyn) org.eclipse.mylyn.sdk-feature (org.eclipse.mylyn) org.eclipse.mylyn.test-feature (org.eclipse.mylyn) h3. org.eclipse.mylyn We would end up with the following bundles in org.eclipse.mylyn to support our releng processes. The test bundles would not be shipped in the download distribution any longer but still be used for integration testing: bc. org.eclipse.mylyn.discovery-directory org.eclipse.mylyn.releng org.eclipse.mylyn.tests org.eclipse.mylyn.tests.standalone In addition I would like to purge all .refactorings folders from the history. These add substantially to the size of the repository and no one complained after we removed these from CVS and disabled recording of refactoring history a while ago (bug 287019). Proposed plan: 1) Do a test migration and create Git repositories for: org.eclipse.mylyn org.eclipse.mylyn.commons org.eclipse.mylyn.context org.eclipse.mylyn.incubator org.eclipse.mylyn.tasks org.eclipse.mylyn.versions These repositories will be available for a few days to verify that branches, tags, etc. were migrated properly and change sets were aggregated correctly. 2) If everyone is happy with the Git repositories I will ask the webmasters to put CVS in read-only mode and re-run the migration. 3) From that point on the new repositories will be open for commits and Hudson builds etc. will get updated accordingly. 4) Remove CVS under /cvsroot/mylyn. CVS clean-up and restructuring commands: pre.. rm -rf ~/tmp/cvs cp -a /cvsroot/mylyn ~/tmp/cvs cd ~/tmp/cvs find -name .refactorings -type d | xargs rm -rf find -name "*.jar,v" -path "*Attic*" | xargs rm find org.eclipse.mylyn.commons/org.eclipse.mylyn/ -name Attic -not -path "*src/*" -not -path org.eclipse.mylyn.commons/org.eclipse.mylyn/Attic -not -path org.eclipse.mylyn.commons/org.eclipse.mylyn/.settings/Attic | xargs rm -rf rm -rf org.eclipse.mylyn/org.eclipse.mylyn.releng/scripts/tools/ rm -f org.eclipse.mylyn/org.eclipse.mylyn.tests.performance/Attic/resourceExclusionTestPaths.txt,v rm -rf org.eclipse.mylyn/Attic mv org.eclipse.mylyn/org.eclipse.mylyn.tests.performance* org.eclipse.mylyn/org.eclipse.mylyn.tests.ui org.eclipse.mylyn.incubator/ mv org.eclipse.mylyn/org.eclipse.mylyn.help.* org.eclipse.mylyn/org.eclipse.mylyn.sdk-feature/ org.eclipse.mylyn/org.eclipse.mylyn.test-feature/ org.eclipse.mylyn.tasks/ mv org.eclipse.mylyn.contexts org.eclipse.mylyn.context Created attachment 198781 [details]
migration script
We started looking at doing this in e4 [1] and platform.ui [2] and we're following the same basic steps you are. Some of the things we found: 1) if you only tag projects that change for each integration build, cvs2git can create a lot of tag commits that are just off master. I use org.eclipse.migration/scripts/fix_tags.sh to migrate the tags back to the main branch. 2) if (like platform ui) you tag all projects at a release but the only branch maintenance projects that change cvs2git creates a branch history that looks horrible. We're trying a "pre-conditioning" step on our important maintenance releases in CVS so that all of our projects are tagged and branched it (R3_6, R3_6_maintenance, etc). PW [1] http://git.eclipse.org/c/e4/org.eclipse.migration.git/tree/scripts/e4_ui_prep.txt [2] http://git.eclipse.org/c/e4/org.eclipse.migration.git/tree/scripts/cvs2git_prep.txt Thanks a lot for the helpful comment and links to the scripts! I noticed a bunch of artificial commits after the conversion and will run fix_tags.sh for Mylyn as will. Additionally, I think I'll take an even more radical approach and delete all v20... and I20... tags since these provide little value going forward. We can't reproduce these old builds easily anyways and would probably end up "backporting" the Tycho build if there was a unexpected need to rebuild older version that Mylyn 3.5. I have committed Git migration scripts to org.eclipse.mylyn/org.eclipse.mylyn.releng/git-migration/ . Testing repositories are available here: https://github.com/spingel/org.eclipse.mylyn https://github.com/spingel/org.eclipse.mylyn.commons https://github.com/spingel/org.eclipse.mylyn.context https://github.com/spingel/org.eclipse.mylyn.incubator https://github.com/spingel/org.eclipse.mylyn.tasks https://github.com/spingel/org.eclipse.mylyn.versions The current plan is to verify these repositories and if everything checks out to re-run scripts on the latest CVS on Tuesday next week to switch over to Git. Steffen, I found some problem while I try to setup an new bootstrap workspace. 1) org.eclipse.mylyn.commons.http is in the new git repository but we dit not have this in the mylyn-extssh.psf => this generate some errors so I remove this project 2) org.eclipse.test.performance is now the only cvs project the we need from dev.eclipse.org:/cvsroot/eclipse =>if this is not present it generate some errors 3) there are several new projects connector-tutorial org.eclipse.mylyn.cvs-feature org.eclipse.mylyn.emf.tests org.eclipse.mylyn.emf.ui org.eclipse.mylyn.git-feature org.eclipse.mylyn.incubator-site org.eclipse.mylyn.sandbox.search-feature org.eclipse.mylyn.sandbox.search.ui org.eclipse.mylyn.tests.standalone org.eclipse.mylyn.versions-feature org.eclipse.mylyn.versions.sdk-feature Thanks for testing the Git repositories! > 1) org.eclipse.mylyn.commons.http is in the new git repository but we dit not > have this in the mylyn-extssh.psf => this generate some errors so I remove this > project Yes, that bundle is still work in progress but it should compile. > 2) org.eclipse.test.performance is now the only cvs project the we need from > dev.eclipse.org:/cvsroot/eclipse =>if this is not present it generate some > errors That's why I moved the bundle to the Incubator. AFAIK there is no repository for the performance testing framework and hence we can't easily include it in the target. > 3) there are several new projects > connector-tutorial > org.eclipse.mylyn.cvs-feature > org.eclipse.mylyn.emf.tests > org.eclipse.mylyn.emf.ui > org.eclipse.mylyn.git-feature > org.eclipse.mylyn.incubator-site > org.eclipse.mylyn.sandbox.search-feature > org.eclipse.mylyn.sandbox.search.ui > org.eclipse.mylyn.tests.standalone > org.eclipse.mylyn.versions-feature > org.eclipse.mylyn.versions.sdk-feature These projects are also in CVS but haven't been added to the PSF files. After the migration we may end up removing the PSF files since EGit does not support that, yet. The migration is now in progress: (~/releng/git-migration/copy-cvs.sh && ~/releng/git-migration/migrate.sh) 1> ~/tmp/migration.log 2> ~/tmp/migration.err The Mylyn CVS has now been migrated to Git! The new repository locations are listed here: http://git.eclipse.org/c/mylyn/ |