Bugzilla – Full Text Bug Listing
| Summary: | Eclipse e4 repositories should be mirrored by Git | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Community | Reporter: | Lars Vogel <Lars.Vogel> | ||||||||||
| Component: | Servers | Assignee: | Eclipse Webmaster <webmaster> | ||||||||||
| Status: | RESOLVED WORKSFORME | QA Contact: | |||||||||||
| Severity: | normal | ||||||||||||
| Priority: | P3 | CC: | bernhard.merkle, bokowski, bokowski, bsd, contact, eduard.bartsch, Lars.Vogel, pwebster, simon_kaegi, Szymon.Brandys | ||||||||||
| Version: | unspecified | Keywords: | helpwanted | ||||||||||
| Target Milestone: | --- | ||||||||||||
| Hardware: | PC | ||||||||||||
| OS: | Windows XP | ||||||||||||
| Whiteboard: | |||||||||||||
| Bug Depends on: | 345479 | ||||||||||||
| Bug Blocks: | |||||||||||||
| Attachments: |
|
||||||||||||
|
Description
Lars Vogel
Moving to Community/Server. I started the import hours ago, and the index should be generated soon. I'm mirroring all the org.eclipse.* directories in this CVS location: http://dev.eclipse.org/viewcvs/index.cgi/e4/ e4 is now listed on the git page: http://dev.eclipse.org/git/ Closing fixed. Reopen if I have missed something. Hi Denis, If it try to clone the repository I receive the message that the directory is empty: Here my command and the error message: vogella@ubuntu:~/eclipsesource$ git clone git://dev.eclipse.org/org.eclipse.e4/org.eclipse.e4.ui.git Initialized empty Git repository in /home/vogella/eclipsesource/org.eclipse.e4.ui/.git/ warning: You appear to have cloned an empty repository. I tried several repos but the same for all of them. Can you please re-check? Best regards, Lars /usr/local/libexec/git-core/git-cvsimport -i -p -x -d :local:/home/data/cvs/eclipse e4/org.eclipse.e4.resources -C /home/data/git/org.eclipse.e4 Initialized empty Git repository in /home/data/git/org.eclipse.e4/org.eclipse.e4.resources.git/ revision 1.4 of file bundles/org.eclipse.ui.ide/icons/full/etool16/undo_edit.gif is tagged but not present revision 1.4 of file bundles/org.eclipse.ui.ide/icons/full/etool16/saveas_edit.gif is tagged but not present [snip] git-cvsimport: fatal: cvsps reported error This is happening on two e4 repos. I'll see if I can make sense of that amazingly cryptic error. (In reply to comment #4) > revision 1.4 of file > bundles/org.eclipse.ui.ide/icons/full/etool16/saveas_edit.gif is tagged but not > present Have a look at this file: /cvsroot/eclipse/e4/org.eclipse.e4.resources/bundles/org.eclipse.ui.ide/icons/full/etool16/Attic/saveas_edit.gif,v It contains a line: v20040316:1.4; Which is bogus because the revision history says that there were revisions 1.1 and 1.2, and then a "deletion record" with revision 1.3. I would suggest deleting the offending line in the files that cause these problems. The same error should happen when converting /cvsroot/eclipse/org.eclipse.ui.ide to Git - that's where the "org.eclipse.ui.ide" was copied from when it was "hard forked" for e4. Boris, that type of error happens in a lot of our CVS repos, but it doesn't prevent the import from functioning. I'll have to dig deeper. *** Bug 299135 has been marked as a duplicate of this bug. *** Hi Denis, Any progress on this? Since there are EGit builds available now, I wanted to try it out on e4 code I am working on but then hit this problem. (In reply to comment #5) > The same error should happen when converting > /cvsroot/eclipse/org.eclipse.ui.ide to Git - that's where the > "org.eclipse.ui.ide" was copied from when it was "hard forked" for e4. Just as a data point - trying to import git://dev.eclipse.org/org.eclipse.ui/org.eclipse.ui.ide.git does not work. (In reply to comment #5) > I would suggest deleting the offending line in the files that cause these > problems. Sorry for the third bug spam... Since the offending lines listed in comment 5 don't make sense at all, not even when just using CVS, wouldn't it be worth trying to fix the ",v" files, followed by a new git import attempt? I would volunteer to change the two ",v" files, including taking full responsibility if it breaks anything on the CVS side. :-) > Since the offending lines listed in comment 5 don't make sense at all, not even
> when just using CVS, wouldn't it be worth trying to fix the ",v" files,
> followed by a new git import attempt?
I'm not convinced the lines are 'offending' at all. cvsps is segfaulting on a specific set of e4 repositories. I'll try some experimentation offline so that I don't bamboozle the live repo.
Any news on the investigation? Sorry to ask again but it would be really nice to have the possibility to clone the e4 repository. I'm mostly caught behind a firewall with no access to the cvs repository and in the evening my internet connection is slow and unstable. It would be nice to sync during the day via http git and then be able to work on / look at the code in the evening. same for me. company firewall blocks CVS and workarounds. Denis, no updates from you since January. Is there anything we can do to help? For example, what are the command line options (or script) you are using to convert from CVS to Git? If you made a CVS tarball of /cvsroot/eclipse available, the community could try to track down the cause of the problem. The wiki page at [1] points to a mostly empty web page. [1] http://wiki.eclipse.org/index.php/CVS_Howto#CVS_tarball_snapshots Boris, if you tar/gzip me the original CVS e4 repo (means all e4 ,v files) I can give it a try. I have done several CVS2SVN and CVS2Git conversions so maybe i can help. thanks, Berni. (In reply to comment #14) > Denis, no updates from you since January. Is there anything we can do to help? > For example, what are the command line options (or script) you are using to > convert from CVS to Git? I'm using git-cvsimport: git-cvsimport -i -p -x -d :local:/home/data2/cvs/eclipse e4/ -C ./e4/ Not that although I am using :local: anyone can use :pserver: and run this from anywhere. You don't need to download tarballs. > If you made a CVS tarball of /cvsroot/eclipse > available, the community could try to track down the cause of the problem. The > wiki page at [1] points to a mostly empty web page. > > [1] http://wiki.eclipse.org/index.php/CVS_Howto#CVS_tarball_snapshots I fixed the blank page but ... don't download a 5G tarball just for this. Just use pserver. > Denis, no updates from you since January.
Sorry, I just can't do everything.
(In reply to comment #16) >> You don't need to download tarballs. > > > If you made a CVS tarball of /cvsroot/eclipse > > available, the community could try to track down the cause of the problem. The > > wiki page at [1] points to a mostly empty web page. > > > > [1] http://wiki.eclipse.org/index.php/CVS_Howto#CVS_tarball_snapshots > > I fixed the blank page but ... don't download a 5G tarball just for this. Just > use pserver. the problem with pserver is that it will take quite some time from germany to do this. I have not tested this but I could imagine. also, as I understand the problem there is a issue with the current CVS ,v file in one or two cases which causes problems for the conversion, so I need really the ,v file to fix this there. (pserver does not work in this case) regarding the tar bar. I do not need the whole 5GB (I do not want this actually). Just tar me up the e4 directory and that should be really small i think. I only need the e4 part for this IIRC... thanks, Berni. (In reply to comment #17) > > Denis, no updates from you since January. > > Sorry, I just can't do everything. Thats why I offer help. tar me up the e4 directory and put is somewhere i can download. :-) I imagine Bernhard will also need the stuff under /cvsroot/eclipse/CVSROOT for the CVS metadata. Denis, could you put that in another tarball for Bernhard to download? Thanks! question: - will we make a direct switch (after some testing of course ;-) an then proceed with git and retire cvs for the e4 stuff or - stay for some time with cvs and only mirror into git ? for testing the result i need a few relevant labels which i then checkout and compare the cvs and git stuff. Can you tell me the interesting candidates :-) ? Not sure if you mean this but plugin "org.eclipse.e4.ui.model.workbench" is pretty important. (In reply to comment #21) > I imagine Bernhard will also need the stuff under /cvsroot/eclipse/CVSROOT for > the CVS metadata. Denis, could you put that in another tarball for Bernhard to > download? Thanks! Thanks but I do not need CVSROOT. The converter I use does not need that information and actually all information needed is in the ,v files. But what I can use is a mapping from the cvs user names to e-mail adresses if you plan to use the notation "J. Random <jrandom@example.com>" in git. The default used at the moment is "cvsauthor <cvsauthor>" I suggest that you send me this information via private e-mail :-) if you want to use the e-mail adresses. Boris, Denis the conversion for the complete 4 repository is done. :-) I tested HEAD and 3 tags and the output of CVS and git is identical. Please tell me where I can upload the converted stuff. I will also write up a summary how to do the conversion, so you can setup it at your site. Please also see #comment25 for the login/email stuff i could need. (In reply to comment #26) > Boris, Denis the conversion for the complete 4 repository is done. :-) I meant "complete e4 repository" of course, sorry Thanks for your efforts. Just to be clear, I would have appreciated finding a solution to the current issue, where the git-cvsimport process used to mirror, on a regular basis, is segfaulting with the cvsps call. Unless I misunderstand, a one-off conversion is not what is needed now, unless e4 is ready to move off CVS entirely. Even then, such a conversion would have to be performed by either myself or a committer using data and tools they know and trust. I hope you understand -- I don't know what is inside your "converted stuff" :-) If we can figure out what is preventing the repo from being mirrored correctly, that would be awesome. (In reply to comment #28) > Thanks for your efforts. > > Just to be clear, I would have appreciated finding a solution to the current > issue, where the git-cvsimport process used to mirror, on a regular basis, is > segfaulting with the cvsps call. > > Unless I misunderstand, a one-off conversion is not what is needed now, unless > e4 is ready to move off CVS entirely. Even then, such a conversion would have > to be performed by either myself or a committer using data and tools they know > and trust. I hope you understand -- I don't know what is inside your > "converted stuff" :-) > > If we can figure out what is preventing the repo from being mirrored correctly, > that would be awesome. there are no secrets :-) I use cvs2svn / better cvs2git to perform the conversion. This is _much_ more reliable than git-cvsimport which uses cvsps, which is essentially dead as i understand. also cvs2git gives you the reasons why cvsps segfaults and cvs2git is much faster than git-cvsimport. The workflow is a one-time cvs2git conversion which i now succeeded + a following mirroring/sync of modifications in cvs to the git repo (e.g. via git-cvsimport) I will write up detailed summary how _you_ can do this also so you do not have to trust my converted git repo. However it would be great it some of the e4 commiters can do just a test with my converted stuff, just as a test. Do you see any problems with that ? Is there a dir where i can upload the converted stuff ? thanks, Berni. (In reply to comment #29) > Do you see any problems with that ? Is there a dir where i can upload the > converted stuff ? GitHub? Apologies if we haven't been clear enough - we cannot afford the churn that would be caused by moving e4 to Git right now. But I am all for having a Git mirror that is continually synced with CVS. My idea was that we could help Denis figure out how to improve the current CVS->Git mirroring that he set up. @Denis: is using cvs2git an option for you @Bernhard: Did you get an error message regarding the segfault which you can share with Denis so that he can fix this issue? Quote: ------------- cvs2git also gives you the reasons why cvsps segfaults and cvs2git is much faster than git-cvsimport. --------- (In reply to comment #30) > My idea was that we could help Denis figure out how to improve the current > CVS->Git mirroring that he set up. yes thats a good idea. I will post the instructions how to convert and mirror the current CVS e4 repo with cvs2git. I tested the converted stuff and it is ok. This is much more stable than using git cvs-import :-) (In reply to comment #31) > @Denis: is using cvs2git an option for you > > @Bernhard: Did you get an error message regarding the segfault which you can > share with Denis so that he can fix this issue? the problem is that some of the cvs files are broken and cvsps segfaults on them. cvs2git is more fault tolerant and outputs the reason. We can fix this in the cvs ,v files, however cvs2git is so much faster and faulttolerant that it IS the way to go. > Quote: > ------------- > cvs2git also gives you the reasons why cvsps segfaults and cvs2git is much > faster than git-cvsimport. > --------- yes, see comment #29 :-) (In reply to comment #33) > the problem is that some of the cvs files are broken and cvsps segfaults on > them. How many files are broken? > cvs2git is more fault tolerant and outputs the reason. We can fix this in the > cvs ,v files, however cvs2git is so much faster and faulttolerant that it IS > the way to go. You are probably right, but the current set up (using cvsps) seems to work for most other projects. It might be easiest (for now) to fix the CVS ,v files and look at changing the way the import works later. ok, i used http://cvs2svn.tigris.org/ it has not only a svn but also git/hg/bzr output. I used the current trunk version in addition to cvs2svn 2.3.0. - install cvs2svn 2.3.0 (just to get a clean cvs2svn environment) sudo apt-get install cvs2svn - get the current trunk version of cvs2svn (has better git support) http://cvs2svn.tigris.org/svn/cvs2svn/trunk/ run the tests .e.g in cvs2svn-trunk ./run-tests.py - in cvs2svntrunk ./cvs2git --options=e4-cvs2git.options now conversion runs about 40min or so... in the first stage it also gives infos about the broken cvs ,v files -import into git via fast import # git init cat cvs2svn-tmp/git-blob.dat cvs2svn-tmp/git-dump.dat | git fast-import - then perform tests, like checkout head from cvs and master from git. compare the two working copies with a recursive diff gave no differences for me. please see also http://cvs2svn.tigris.org/cvs2git.html e4-cvs2git.options file is attached. I changed the dirs and encoding. Created attachment 165945 [details]
options for e4 cvs repository for cvs2git
(In reply to comment #34) > (In reply to comment #33) > > the problem is that some of the cvs files are broken and cvsps segfaults on > > them. > > How many files are broken? 38 > > > cvs2git is more fault tolerant and outputs the reason. We can fix this in the > > cvs ,v files, however cvs2git is so much faster and faulttolerant that it IS > > the way to go. > > You are probably right, but the current set up (using cvsps) seems to work for > most other projects. It might be easiest (for now) to fix the CVS ,v files and > look at changing the way the import works later. okay attached you can find the list of broken ,v files, found by cvs2git. If you fix them (usually delete that invalid tag/label) cvsps will work (i hope) I would (recommend to) switch to cvs2svn/git in the longer term :-) At work I converted a large amount of our repos from CVS/PVCS/<insert other VCS> with cvs2svn. Created attachment 165948 [details]
broken_komma_v_cvs_files_found_by_cvs2git
(In reply to comment #37) > okay attached you can find the list of broken ,v files, found by cvs2git. > If you fix them (usually delete that invalid tag/label) cvsps will work (i > hope) Done. Denis, can you try to run the Git import for e4 to confirm that it no longer segfaults? (Or will the script run anyway overnight?) The script runs every day at 1pm ET. I'm not sure if it has processed e4 yet (doubt it) but it will run automatically. I tried to test but dev.eclipse.org seems to be down since Sat evening...
Ping dev.eclipse.org [206.191.52.50] mit 32 Bytes Daten:
Zeitüberschreitung der Anforderung.
Ping-Statistik für 206.191.52.50:
Pakete: Gesendet = 1, Empfangen = 0, Verloren = 1 (100% Verlust),
You'll never get a ping reply from dev.eclipse.org. (In reply to comment #40) > The script runs every day at 1pm ET. I'm not sure if it has processed e4 yet > (doubt it) but it will run automatically. testing with the "ui" and "resources" project still fails... did the script run ? C:\_test>C:\Programme\Git\bin\git clone git://dev.eclipse.org/org.eclipse.e4/org.eclipse.e4.ui.git Initialized empty Git repository in C:/_test/org.eclipse.e4.ui/.git/ fatal: The remote end hung up unexpectedly C:\_test>C:\Programme\Git\bin\git clone git://dev.eclipse.org/org.eclipse.e4/org.eclipse.e4.resources.git Initialized empty Git repository in C:/_test/org.eclipse.e4.resources/.git/ fatal: The remote end hung up unexpectedly This morning I completely deleted all the e4 Git mirrors and re-started the import. Some had succeeded, but I'm guessing others have not. I'll provide some more details tomorrow since I'm not at the computer that did the import. (In reply to comment #43) > testing with the "ui" and "resources" project still fails... It works for me now with "ui" but not with "resources". It looks like we have at least some progress. Denis, do you have a log file from the last import? Or could you create an updated tar for Bernhard to check for any files that I might have missed? (In reply to comment #45) > (In reply to comment #43) > > testing with the "ui" and "resources" project still fails... > > It works for me now with "ui" but not with "resources". It looks like we have > at least some progress. Denis, do you have a log file from the last import? Or > could you create an updated tar for Bernhard to check for any files that I > might have missed? we can do this again. Please tar me up again the e4 rep and I will have a look at this. I will additionally test with cvsps this time and locate the crash. However in the longer term, please consider #comment 30 and switching to cvs2svn resp. cvs2git. http://cvs2svn.tigris.org/cvs2git.html It is actively maintained, faster and more reliable than cvsps. You said that is is working with cvsps, and I believe that, but i am unsure if the conversion really took over all tags/branches etc. in the right way. Have you tested this ? (compare the converted output). ? There are numerous bug reports for cvsps that it that problems with various scenarios in branches and tags... Also #comment 35 includes a complete description how to do this with cvs2svn/git. maybe the following links are also worth to consider [1] http://www.gelato.unsw.edu.au/archives/git/0506/5389.html [2] http://www.oak.homeunix.org/~marcel/blog/2009/06/03/tracking-cvs-with-git-using-cvs2git > Denis, do you have a log file from the last import?
These are the final lines from the 'resources' import:
cvs_direct: rlog: read E cvs rlog: Logging e4/org.eclipse.e4.resources/bundles/org.eclipse.ui.ide/icons/full/elcl16
cvs rlog: Logging e4/org.eclipse.e4.resources/bundles/org.eclipse.ui.ide/icons/full/elcl16
state: 0 read line:=============================================================================
cvs_direct: rlog: read M
state: 0 read line:
cvs_direct: rlog: read M RCS file: /home/data2/cvs/eclipse/e4/org.eclipse.e4.resources/bundles/org.eclipse.ui.ide/icons/full/elcl16/Attic/addbkmrk_tsk.gif,v
state: 0 read line:RCS file: /home/data2/cvs/eclipse/e4/org.eclipse.e4.resources/bundles/org.eclipse.ui.ide/icons/full/elcl16/Attic/addbkmrk_tsk.gif,v
stripped filename bundles/org.eclipse.ui.ide/icons/full/elcl16/addbkmrk_tsk.gif
new file: bundles/org.eclipse.ui.ide/icons/full/elcl16/addbkmrk_tsk.gif
cvs_direct: rlog: read M head: 1.3
state: 2 read line:head: 1.3
cvs_direct: rlog: read M branch:
state: 2 read line:branch:
cvs_direct: rlog: read M locks:
state: 2 read line:locks:
cvs_direct: rlog: read M access list:
state: 2 read line:access list:
cvs_direct: rlog: read E Terminated with fatal signal 11
Terminated with fatal signal 11
state: 2 read line:access list:
cvs_direct: rlog: read error
state: 2 read line:access list:
(In reply to comment #46) > However in the longer term, please consider #comment 30 and switching to > cvs2svn resp. Consider it considered. (In reply to comment #45) > (In reply to comment #43) > > testing with the "ui" and "resources" project still fails... > > It works for me now with "ui" but not with "resources". It looks like we have > at least some progress. Denis, do you have a log file from the last import? Or > could you create an updated tar for Bernhard to check for any files that I > might have missed? Boris could it be that not all ,v have been patched correctly ? I did this now also in my patched e4 CVS repo and got no error with the approach "git cvs-import" which you are using. There are some variation with that invalid labels (e.g. rev number) so I you have patched this with sed maybe not all patterns were replaced ? anyway, appended i a tar.gz of the relevant e4.resources part. this should fix the crash. I did not produce a patch file but just look for ~ files in the tar.gz and you will see what I have changed. the "git cvs-import" rans fine now (but it is still running and takes ages)... Created attachment 166585 [details]
patched ,v files for e4.ressources project
patched ,v files for e4.ressources project
I'm looking into cvs2svn today... @Bernhard: As far as I can tell you did the conversion from the tar file. How would I run this conversion directly from cvs? Lets say I checkout on plugin from cvs, e.g. cvs -d ':pserver:anonymous@dev.eclipse.org/cvsroot/eclipse' co e4/org.eclipse.e4.ui/bundles/org.eclipse.e4.ui.workbench3 What would be the next step to convert the repo to git? Perhaps I can help cleaning up the errors. So I believe this has been fixed — at least I am now able to pull from the repository, and it appears to be up-to-date with recent changes. (In reply to comment #52) > @Bernhard: As far as I can tell you did the conversion from the tar file. How > would I run this conversion directly from cvs? > > Lets say I checkout on plugin from cvs, e.g. > > cvs -d ':pserver:anonymous@dev.eclipse.org/cvsroot/eclipse' co > e4/org.eclipse.e4.ui/bundles/org.eclipse.e4.ui.workbench3 > > What would be the next step to convert the repo to git? Perhaps I can help > cleaning up the errors. @Lars, please see comment#50. there were some broken ,v CVS files and i fixed them. I think Dennis has applied them to the CVS repo and now the cvs2git mirror process runs, so it seems to be fixed now. (In reply to comment #53) > So I believe this has been fixed — at least I am now able to pull from the > repository, and it appears to be up-to-date with recent changes. I never got a chance to look into cvs2svn, but I did upgrade Git to the latest version... I think that has solved numerous problems. Closing as FIXED since everything seems to be working ok. Please reopen if I've missed something. Seems to be working fine. Thanks to all. Special thanks Denis and Bernhard for spending time on tracking this down. I am reopening the bug because the repository org.eclipse.e4.resources.git is still empty. org.eclipse.e4.ui.git and some others seem to work well. Is the issue supposed to be solved for all components in e4? The import process for e4.resources always fails with: cvs rlog: Logging e4/org.eclipse.e4.resources/bundles/org.eclipse.ui.ide/icons/full/elcl16 Terminated with fatal signal 11 I'll strace further to hopefully determine why. (In reply to comment #58) > The import process for e4.resources always fails with: > > cvs rlog: Logging > e4/org.eclipse.e4.resources/bundles/org.eclipse.ui.ide/icons/full/elcl16 > Terminated with fatal signal 11 > > I'll strace further to hopefully determine why. When doing a CVS tagging operation on e4/org.eclipse.e4.resources recently, I ran into problems in that directory. It was complaining about waiting for an anonymous lock so it could proceed. I don't know if that's the problem, or if the git import is leaving the read-lock there. PW (In reply to comment #59) > > The import process for e4.resources always fails with: > > > > cvs rlog: Logging > > e4/org.eclipse.e4.resources/bundles/org.eclipse.ui.ide/icons/full/elcl16 > > Terminated with fatal signal 11 Looks like it's the same problem as the one we had before, see comment 38 and earlier. Denis, did you apply Bernhard's patch? See comments 49 and 50. > When doing a CVS tagging operation on e4/org.eclipse.e4.resources recently, I > ran into problems in that directory. It was complaining about waiting for an > anonymous lock so it could proceed. In the 'Just so you know' dept., the anonymous lock forcibly comes from build.eclipse.org, since it is the only machine which can access pserver using the 'live' CVS files. Anonymous pserver uses a shadow copy. > I don't know if that's the problem, or if the git import is leaving the > read-lock there. Like anonymous pserver, the git import also uses the shadow copy of CVS, so it cannot interfere with committer writes. > Denis, did you apply Bernhard's patch? See comments 49 and 50. I have, but it's not really a 'patch' -- it is complete replacement files for 443 files in CVS. I realize most of these files are Attic files, I'm not comfortable replacing CVS with files in such a brute-force fashion without diffing them first. Another alternative is to just share the command line/script you used to remove the offending lines (I'm guessing it's an awk/sed one-liner) ? > it is complete replacement files for 443 files in CVS.
Sorry, I misread... There are 38 files in there with the ~. I'll just diff them and replace as necessary. Sorry for the noise.
Created attachment 175606 [details]
Diff log
I've applied the 'patch'. Attached is a diff log.
This will soon no longer be an issue after e4 migrates fully to git. > This will soon no longer be an issue after e4 migrates fully to git.
This is no longer an issue since e4 has migrated to git.
|