Community
Participate
Working Groups
We are recognizing more and more issues with Subversive and we no longer can tolerate that. I've herad that EGit has matured a lot as of now, so we'd like to give it a try. Is it possible that you copy our SVN repo from http://dev.eclipse.org/svnroot/modeling/org.eclipse.emf.cdo to a new Git repo? This is for EGit evaluation only at the moment, i.e. *both* repos (SVN and Git should be writable! If, after an evaluation period, we decide to permanently switch to Git we'd like to start with a new SVN copy (Git import). The description of the Git repo shall be "CDO Model Repository" and the owner "Eike Stepper" (userid=estepper).
I've created /gitroot/cdo, and have sent you the 'setup' guide for git. -M.
I've spent a couple of hours attempting to import from SVN. Without the least bit of success. I could init a bare repo /gitroot/cdo/org.eclipse.emf.cdo.git In my home dir on build.eclipse.org the following happens: estepper@build:~> git clone /gitroot/cdo/org.eclipse.emf.cdo.git Cloning into org.eclipse.emf.cdo... done. warning: You appear to have cloned an empty repository. estepper@build:~> cd org.eclipse.emf.cdo/ estepper@build:~/org.eclipse.emf.cdo> svn2git -v file:///svnroot/modeling/org.eclipse.emf.cdo Running command: git svn init --prefix=svn/ --no-metadata --trunk=trunk --tags=tags --branches=branches file:///svnroot/modeling/org.eclipse.emf.cdo Can't locate SVN/Core.pm in @INC (@INC contains: /usr/local/lib/perl5/site_perl/5.10.0/x86_64-linux-thread-multi /usr/local/lib/perl5/site_perl/5.10.0 /usr/lib/perl5/5.10.0/x86_64-linux-thread-multi /usr/lib/perl5/5.10.0 /usr/lib/perl5/site_perl/5.10.0/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.10.0 /usr/lib/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.10.0 /usr/lib/perl5/vendor_perl .) at /usr/local/libexec/git-core/git-svn line 42. Running command: git svn fetch [svn-remote "svn"] unknown Running command: git branch -l --no-color Running command: git branch -r --no-color Running command: git checkout -f master error: pathspec 'master' did not match any file(s) known to git. Running command: git gc Nothing new to pack. I have no clue what that means and need help ;-(
(In reply to comment #2) > I've spent a couple of hours attempting to import from SVN. Without the least > bit of success. I could init a bare repo /gitroot/cdo/org.eclipse.emf.cdo.git I don't understand why you're posting this in this bug, AND you sent us an email. I know this is important to you, but it doesn't make us work any faster. Anyway, for completeness, the email you have received with instructions on setting up your Git repos specifically suggests you SSH to 'git.eclipse.org' and you're trying to run these commands on build.eclipse.org. Again, for completeness, because Git and SVN are not part of our core OS packages, we must compile and maintain them manually -- so we tend to simply not install all the packages on all the servers to reduce the maintenance burden.
(In reply to comment #3) > I don't understand why you're posting this in this bug, AND you sent us an > email. I know this is important to you, but it doesn't make us work any > faster. That was absolutely not my intention. If you look at what I've exactly written you'll notice that all 4 issues/questions I've reported are different. Anyway, I know you're working hard and I've never had reason to complain ;-) > Anyway, for completeness, the email you have received with instructions on > setting up your Git repos specifically suggests you SSH to 'git.eclipse.org' > and you're trying to run these commands on build.eclipse.org. Silly me! Okay, on git.e.o I could run the initrepo command. The I've followed http://wiki.eclipse.org/Git/Migrating_to_Git#Using_svn2git_on_build.eclipse.org and got a new error: ---- estepper@build:~> cd /gitroot/cdo estepper@build:/gitroot/cdo> export GIT_DIR=org.eclipse.emf.cdo.git estepper@build:/gitroot/cdo> svn2git svn://dev.eclipse.org/modeling/org.eclipse.emf.cdo Can't locate SVN/Core.pm in @INC (@INC contains: /usr/local/lib/perl5/site_perl/5.10.0/x86_64-linux-thread-multi /usr/local/lib/perl5/site_perl/5.10.0 /usr/lib/perl5/5.10.0/x86_64-linux-thread-multi /usr/lib/perl5/5.10.0 /usr/lib/perl5/site_perl/5.10.0/x86_64-linux-thread-multi /usr/lib/perl5/site_perl/5.10.0 /usr/lib/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi /usr/lib/perl5/vendor_perl/5.10.0 /usr/lib/perl5/vendor_perl .) at /usr/local/libexec/git-core/git-svn line 42. [svn-remote "svn"] unknown fatal: This operation must be run in a work tree ---- The "can't locate" error can be fixed by not using svn:// but just the files system path. The "must be run in a work tree" error can be fixed by creating a clone in my home folder. A new error then (so I'm not sure if my fixes were ok): ---- estepper@build:~> git clone /gitroot/cdo/org.eclipse.emf.cdo.git Cloning into org.eclipse.emf.cdo... done. warning: You appear to have cloned an empty repository. estepper@build:~> cd org.eclipse.emf.cdo/ estepper@build:~/org.eclipse.emf.cdo> svn2git /svnroot/modeling/org.eclipse.emf.cdo E: 'trunk' is not a complete URL and a separate URL is not specified Migrating from a git-svn v1 layout... Data from a previous version of git-svn exists, but org.eclipse.emf.cdo.git/svn (required for this version (1.7.4.1) of git-svn) does not exist. Done migrating from a git-svn v1 layout [svn-remote "svn"] unknown fatal: This operation must be run in a work tree ---- I'm lost again. Did I made a simple typo somewhere again?
> Can't locate SVN/Core.pm in @INC (@INC contains: Matt recently upgraded our SVN packages, which always causes us grief -- Perhaps he can take a look?
Hmmm, looks like there is some disconnect between perl and SVN. I've tried a few things but so far nothing has made any difference. I'll see if I can think of some more options. -M.
Ok, I think we're finally there. I've managed to get the perl libraries built and I've run a test import now (on build.eclipse.org) that's actually finished. -M.
Great! svn2git does properly start the fetch step now. I discover some problems, though: r3597 = 3c83886cf31032185af9f7b51eeedf13598c0df6 (refs/remotes/svn/trunk) M plugins/org.eclipse.emf.cdo.edit/src/org/eclipse/emf/cdo/edit/CDOItemProviderAdapter.java r3598 = 04a2db48dd54b79b2a5a5765742d59c731ebc193 (refs/remotes/svn/trunk) Found possible branch point: svn://dev.eclipse.org/modeling/org.eclipse.emf.cdo/trunk => svn://dev.eclipse.org/modeling/org.eclipse.emf.cdo/ branches/emf-on-server, 3594 Use of uninitialized value $u in substitution (s///) at /usr/local/libexec/git-core/git-svn line 1735. Use of uninitialized value $u in concatenation (.) or string at /usr/local/libexec/git-core/git-svn line 1735. refs/remotes/svn/HibernateStore: 'svn://dev.eclipse.org/modeling/org.eclipse.emf.cdo' not found in '' Running command: git branch -l --no-color svn/HibernateStore svn/R0_7_maintenance svn/R1_0_maintenance [...] svn2git does continue after this problem but in the end I'm missing many important branches in the resulting git repo. The svn HEAD revision is at ~r8714, not anywhere near the last imported r3597 ;-( Digging in the dark I found out that r3598 (last logged revision + 1) has "(no author)" in the commit message. If that can be the cause? BTW. is there a central committerid->email mapping maintained by you or the portal? I've used this command to initialize an authors file: mkdir -p ~/.svn2git && echo "(no author) = estepper stepper@esc-net.de" > ~/.svn2git/authors && for name in `svn log file:///svnroot/modeling/org.eclipse.emf.cdo | grep -E "r[0-9]+" | grep "|" | grep -v "(no author)" | awk '{print $3}' | sort | uniq`; do echo "$name = $name <$name@XXXXXXX.com>" >> ~/.svn2git/authors; done Then I manually looked up all bugzilla email addresses of my committers and ended with this ~/.svn2git/authors file: (no author) = estepper <stepper@esc-net.de> adietisheim = adietisheim <andre.dietisheim@gmail.com> cdegroot = cdegroot <caspar_d@nomagicasia.com> estepper = estepper <stepper@esc-net.de> evaisnora = evaisnora <egiva@nomagic.com> isallam = isallam <ibrahim.sallam@objectivity.com> mfluegge = mfluegge <martin.fluegge@gmx.de> mruzicka = mruzicka <michal.ruzicka@cloudsmith.com> mtaal = mtaal <mtaal@elver.org> nickb = nickb <nickboldt@gmail.com> plehmann = plehmann <pascal.lehmann@ruag.com> smcduff = smcduff <smcduff@hotmail.com> swinkler = swinkler <sw2312+bugs@googlemail.com> vroldanbet = vroldanbet <vroldan@opencanarias.com> I'm currently running a new import. Will report here...
Bucky RMAP for Git: https://bugs.eclipse.org/bugs/attachment.cgi?id=186912
The new import attempt was unsuccessful, too. This time it even stopped at/after r1826: r1825 = 6951cfceef18e2a647a46e1c025138bd4d9bec53 (refs/remotes/svn/trunk) M deprecated/org.eclipse.emf.cdo.protocol/src/org/eclipse/emf/cdo/internal/protocol/model/CDOFeatureImpl.java M deprecated/org.eclipse.emf.cdo.protocol/src/org/eclipse/emf/cdo/internal/protocol/model/CDOPackageImpl.java M deprecated/org.eclipse.emf.cdo.protocol/src/org/eclipse/emf/cdo/internal/protocol/model/CDOClassImpl.java r1826 = cf71a451990fc61eb373affe0d7a4431de2aeaec (refs/remotes/svn/trunk) Found possible branch point: file:///svnroot/modeling/org.eclipse.emf.cdo/trunk => file:///svnroot/modeling/org.eclipse.emf.cdo/tags/Root_HibernateStore, 1826 Use of uninitialized value $u in substitution (s///) at /usr/local/libexec/git-core/git-svn line 1735. Use of uninitialized value $u in concatenation (.) or string at /usr/local/libexec/git-core/git-svn line 1735. refs/remotes/svn/R0_7_maintenance: 'file:///svnroot/modeling/org.eclipse.emf.cdo' not found in '' Running command: git branch -l --no-color Running command: git branch -r --no-color svn/R0_7_maintenance svn/lazy_contents [...] Even less branches and objects were found, ~50% of the first run. I guess I need help again...
Well, I used the following to init a repo at org.eclipse.emf.cdo.git: git svn clone --no-checkout --shared 1 file:////svnroot/modeling/org.eclipse.emf.cdo org.eclipse.cdo1.git And it didn't generate any errors that I saw, but it takes a long long time to run. Can you take a look at the repo and tell me if it's closer to what you expect? -M.
Created attachment 200021 [details] Screenshot of Progress view I used EGit to clone the repo you've created but that quickly looks stuck at 100%, see the attached screenshot. On my local file system it seems that EGit creates an exact copy of the SVN layout, see http://dev.eclipse.org/svnroot/modeling/org.eclipse.emf.cdo , and puts everything (all branches, all tags, etc) as a huge object tree under the Git master branch. When I started the clone operation I was only offered the master branch as a checkout source. That doesn't look right. Could it be a problem that I have SVN folders at the root of my SVN repo in addition to trunk/, branches/ and tags/ ?
Created attachment 200022 [details] Screenshot of local Git clone Correction! It just did end, after a long long time, as you said. But, as I expected, the resulting repo layout seems wrong to me. Please see the attached screenshot.
(In reply to comment #12) > Could it be a problem that I have SVN folders at the root of my SVN repo in > addition to trunk/, branches/ and tags/ ? Possible. The import tool does it's best to convert the SVN repo directly into Git, so I don't think this is a problem per se. You may want to take a look at bug 350188, where the Jetty folks worked some magic(comment 17) on their conversion. They may be able to give you a breakdown of what they did. My only other idea is to clone the full import, re-arrange it as you like, and then push it as the initial commit to a new repo. -M.
The process we took was to clone the repo into a migration repository locally, clean up any tags or branches that we didn't want to migrate and then copy the repository wholesale over to the eclipse git server. First I had to extract the authors and build a mapping file: #!/usr/bin/env bash authors=$(svn log -q | grep -e '^r' | awk 'BEGIN { FS = "|" } ; { print $2 }' | sort | uniq) for author in ${authors}; do echo "${author} = NAME <USER@DOMAIN>"; done I created a text file with the results of that and fixed them all. Then I used the latest svn2git tool I could find and used a command like: > svn2git svn+ssh://jmcconnell@dev.eclipse.org/svnroot/rt/org.eclipse.jetty/jetty --no-minimize-url --authors ~/eclipse-authors.txt --metadata I edited the .git/config file so it was aligned with the ones in the other gitroot/ projects and then scp'd the .git repo over to the eclipse machines and renamed it to org.eclipse.jetty.foo.git to align with that convention. Then the webmasters went in and added in the correct hooks to honor IP policies. The config file when you use svn2git ends up with a lot of trash in it, branches in svn, etc which are all there to maintain the ability to go back to the svn repo...so I just smoked those references and I was really only interested in git moving forward with it. You can also fairly easily add a remote for a bare repository that the admins could setup but scp worked fine for me.
if you have shell access to dev then you can look in the /gitroot/jetty/org.eclipse.jetty.project.git/config file for all you need to have in it.. Also add in a description file as that is what shows up in the git.eclipse.org/c listing any issues let me know on here and I'll try and help
Thank you, Jesse! I already knew the authors thing but something was better with your other svn2git options. Now it ran for ~2 hours until my Putty shell got network problems. This always happens here after being connected to eclipse.org for a while. Webmasters, can you try to run this job for me? svn2git file:///svnroot/modeling/org.eclipse.emf.cdo --no-minimize-url --authors ~estepper/.svn2git/authors --metadata BTW. there have been quite some console messages like this: Couldn't find revmap for file:///svnroot/modeling/org.eclipse.emf.cdo/trunk/plugins/org.eclipse.emf.cdo/doc Couldn't find revmap for file:///svnroot/modeling/org.eclipse.emf.cdo/trunk/plugins/org.eclipse.emf.cdo.tests/logic Couldn't find revmap for file:///svnroot/modeling/org.eclipse.emf.cdo/trunk/plugins/org.eclipse.emf.cdo.server.objectivity/.settings But the script always continued.
Trying again with: nohup svn2git file:///svnroot/modeling/org.eclipse.emf.cdo --no-minimize-url --authors ~estepper/.svn2git/authors --metadata & Will report in a couple of hours...
Okay, after ~5 hours the import finished. The only error messages are a bunch of these: fatal: Not a valid object name: 'remotes/svn/tags/2.0-end-of-maintenance'. fatal: Not a valid object name: 'remotes/svn/tags/2.0.2'. fatal: Not a valid object name: 'remotes/svn/tags/3.0'. fatal: Not a valid object name: 'remotes/svn/tags/3.0-SR1'. fatal: Not a valid object name: 'remotes/svn/tags/3.0-SR2-RC4'. [...] Nevertheless I moved the .git folder of my import clone to /gitroot/cdo, renamed it to org.eclipse.emf.cdo.git and ran git config --bool core.bare true. Then I cloned it locally with EGit, which worked without errors and fast enough. There seems to be 2 problems with branches and tags, though: 1) "nested" tags and branches, e.g., /tags/drops/I20110701-1201 have not been imported. I suspect I have to flatten the SVN tags/ and branches/ folders prior to the next import attempt. 2) On the other hand a lot of branches and tags have been imported that I did not expect to see in Git. In some cases, for example, I had created a branch in SVN, later realized that it had a typo in its name and renamed it. Now I do see the branch with the typo. This pretty much pollutes the Git branch and tag lists. Is this expectable/acceptable?
No reason you can't just delete those, imo.
Directly in the file system below /gitroot/cdo or via git in my clone? In any case, how? I'm an absolute Git beginner.
with git, best not muck directly with anything other then the git config bits directly i found using a client like Tower on the mac was a handy learning step, see what it let you do and then figure out how to do it via the cli
> Now it ran for ~2 hours until my Putty shell > got network problems. This always happens here after being connected to > eclipse.org for a while. FWIW, if you have stuff to run on *nix terminals which will take minutes/hours, I usually spawn a screen session so it doesn't get interrupted by network closures etc. http://www.gnu.org/software/screen/manual/screen.html
(In reply to comment #22) > with git, best not muck directly with anything other then the git config bits > directly Ok, will follow that. > i found using a client like Tower on the mac was a handy learning step, see > what it let you do and then figure out how to do it via the cli Also a good idea. I've just flattened 70 nested barnches and tags. Will start another import now...
(In reply to comment #23) > FWIW, if you have stuff to run on *nix terminals which will take minutes/hours, > I usually spawn a screen session so it doesn't get interrupted by network > closures etc. http://www.gnu.org/software/screen/manual/screen.html I'm doing all this on build.o.e because I have no other linux box at hand. Isn't "nohup xyz &" good enough?
> I'm doing all this on build.o.e because I have no other linux box at hand. > Isn't "nohup xyz &" good enough? Sure, but I think Denis suggested screen since it will let you 'resume' control of the import process even if your ssh connection gets hosed. -M.
*** This bug has been marked as a duplicate of bug 360970 ***
