Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 345474

Summary: test migration of org.eclipse.equinox.p2 to git
Product: Community Reporter: Kim Moir <kim.moir>
Component: GitAssignee: Eclipse Webmaster <webmaster>
Status: RESOLVED INVALID QA Contact:
Severity: normal    
Priority: P3 CC: dj.houghton, d_a_carver, jamesblackburn+eclipse, katya.stoycheva, matthew, pwebster, tjwatson
Version: unspecified   
Target Milestone: ---   
Hardware: PC   
OS: Windows XP   
Whiteboard:
Bug Depends on:    
Bug Blocks: 345479    
Attachments:
Description Flags
output from the git import
none
log from running verify-cvs2svn.py none

Description Kim Moir CLA 2011-05-11 13:43:51 EDT
This is a request for a test migration of the p2 bundles and features from CVS to git at eclipse.org.  We've been able to run a successful test build using the git mirror of the p2 CVS content (org.eclipse.equinox.p2) (see bug 344152).  However, the current git mirrors are missing many tags so it's not a complete test. (See https://bugs.eclipse.org/bugs/show_bug.cgi?id=344152#c11).  We'd like to do a compare of the resulting p2 source bundles from a build created with CVS versus git.  The repo could be made read only, we don't want anybody to commit to it, it's just being used for testing.  This repo can be deleted and recreated when the actual migration takes place.
Comment 1 Eclipse Webmaster CLA 2011-05-12 13:10:46 EDT
Ok, I've created /gitroot/p2 to hold this repo, and given you ownership.  

Do you want to do the import or would you prefer we do it?

-M.
Comment 2 Paul Webster CLA 2011-05-13 10:36:11 EDT
I wasn't able to see into that directory:

drwxr-x---  2 kmoir       root                       48 2011-05-12 13:08 p2

Also, the wiki suggests using cvs2git (part of svn2git) on build.eclipse.org, but those executables don't appear to exist: http://wiki.eclipse.org/Git/Migrating_to_Git#Using_cvs2git  Was svn2git installed at one time on build.eclipse.org and lost in an upgrade?

I did see the CDT guys wrote down a lot of their experiences using cvs2git in bug 316208

PW
Comment 3 Eclipse Webmaster CLA 2011-05-16 10:43:43 EDT
I've updated the group permissions, and re-installed svn2git (must have been lost with the build crash last year).

-M.
Comment 4 Denis Roy CLA 2011-06-09 10:25:02 EDT
Have we made progress here?
Comment 5 Kim Moir CLA 2011-06-09 10:53:43 EDT
I ran a test migration of the p2 bundles yesterday following the instructions that the cdt project used in bug 316208. It looks like the export worked, but the import to git did not. I'm working on it :-)
Comment 6 Kim Moir CLA 2011-06-09 13:58:05 EDT
Created attachment 197716 [details]
output from the git import
Comment 7 Kim Moir CLA 2011-06-09 15:38:14 EDT
Actually the migration seemed to work after all. Trying a build from this copy now.
Comment 8 James Blackburn CLA 2011-06-09 17:50:05 EDT
(In reply to comment #6)
Those warnings are weird - I've never seen that before...    I would guess those tags are broken?  Are you using head of cvs2git?  
Also worth running cvs2svn-verify agains the git and CVS repo to check that they're identical.  The cvs2svn mailing list is quite responsive if you're encountering an issue.
Comment 9 Kim Moir CLA 2011-06-09 18:05:56 EDT
Thanks James, I have to run the verify step.  I'm using your recipe from bug 316208.  Yes, I downloaded head of cvs2git yesterday. Thanks for the suggestions!
Comment 10 Kim Moir CLA 2011-06-14 14:22:42 EDT
Test build is running here with p2 bundles from git repository that I migrated as a test yesterday.  

https://hudson.eclipse.org/hudson/view/Eclipse%20and%20Equinox/job/eclipse-equinox-git-test-N/15/

Not sure if the p2 projects are organized in git as they should be in /gitroot/p2, perhaps an p2 committer could comment.
Comment 11 Kim Moir CLA 2011-06-15 09:33:55 EDT
Created attachment 198018 [details]
log from running verify-cvs2svn.py

104 failures from comparing repos.  Not really sure how to fix this.

Here is the command I used to compare the repos.
http://wiki.eclipse.org/Platform-releng/Juno_Git_Migration_Recipe#Verify_repo
Comment 12 James Blackburn CLA 2011-06-15 11:34:11 EDT
(In reply to comment #11)
> Created attachment 198018 [details]
> log from running verify-cvs2svn.py
> 
> 104 failures from comparing repos.  Not really sure how to fix this.

Interesting.

Nearly all the issues are incorrect expansion of $Id.  The file content is otherwise correct.  Note that in git the $Id tags will never change again so in some sense you don't need these in the future: after all what is revision 1.1?  
That said, cvs2git can get this right. I think it depends on which RCSRevisionReader you choose and how you configure keyword expansion:
http://cvs2svn.tigris.org/cvs2svn.html
In CDT we had a two instances of incorrect $Ids, which I've ignored...


The other issue appears to be that the line-endings are broken on the Version.java, as reachable by some tags in the repo.  The line-ending seems to be just '\r'.  The import will likely have used the unix standard line-ending (which matches HEAD).   If you really wanted you can make the import match the original CVS:  http://cvs2svn.tigris.org/faq.html#eol-fixup

However in this case, I think that would be wrong.  For text files you really want all the text files in the repo to have the same line-ending.  When the local working copy is checked out, line-endings are updated to match the user's preference (and the reverse on commit).  So in this case it looks to me like the import is _better_ than the original :)

It should be relatively straightforward to tweak the diff in cvs2git to such that the '\r' ending is ignored if you decide you that '\n' is correct for text files in your git repo.
Comment 13 Katya Stoycheva CLA 2011-07-11 09:25:01 EDT
I tried cloning the test repo and the operation seems to finish successfully but afterwards most of the projects (when imported in Eclipse) don’t compile properly.
For example p2.repository doesn’t see some of the exported packages by p2.core though p2.core has correct manifest/project structure and no compilation errors.

Is that a known issue or something on my side went wrong?
Comment 14 Kim Moir CLA 2011-07-11 11:16:05 EDT
I don't know if this was the state of the p2 repository when I converted it.  Do any other p2 committers see these problems?  Do you have all your dependencies loaded in your workspace, for instance equinox.common which resides outside the p2 repository?
Comment 15 Matthew Piggott CLA 2011-07-11 11:31:09 EDT
(In reply to comment #13)
> I tried cloning the test repo and the operation seems to finish successfully
> but afterwards most of the projects (when imported in Eclipse) don’t compile
> properly.
> For example p2.repository doesn’t see some of the exported packages by p2.core
> though p2.core has correct manifest/project structure and no compilation
> errors.
> 
> Is that a known issue or something on my side went wrong?

I cloned and imported, what you may have run into is two projects have the name: org.eclipse.equinox.p2.repository

One of the projects, org.eclipse.equinox.p2.repository.impl isn't part of the p2 psf, and appears to be an unused project referring to old packages that no longer exist (eg org.eclipse.equinox.internal.provisional.p2.core).  When we update the psf people shouldn't run into it (though maybe we should just remove it if it has no use)
Comment 16 Katya Stoycheva CLA 2011-07-11 15:37:20 EDT
(In reply to comment #15)
> One of the projects, org.eclipse.equinox.p2.repository.impl isn't part of the
> p2 psf, and appears to be an unused project referring to old packages that no
> longer exist (eg org.eclipse.equinox.internal.provisional.p2.core).  When we
> update the psf people shouldn't run into it (though maybe we should just remove
> it if it has no use)
What you described is probably the reason for my compilation issues - I imported just a couple of projects instead of all and everything compiles now. Thank you for the hint.
Comment 17 John Arthorne CLA 2011-08-08 15:11:05 EDT
p2 has now been fully migrated to git.