Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.
Bug 332365 - CDT git mirror missing some files
Summary: CDT git mirror missing some files
Status: CLOSED WORKSFORME
Alias: None
Product: Community
Classification: Eclipse Foundation
Component: Git (show other bugs)
Version: unspecified   Edit
Hardware: All All
: P3 major (vote)
Target Milestone: ---   Edit
Assignee: Eclipse Webmaster CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-12-11 13:07 EST by Marc-André Laperle CLA
Modified: 2011-01-23 02:16 EST (History)
3 users (show)

See Also:


Attachments
screenshot (153.02 KB, image/png)
2011-01-19 14:38 EST, Andrew Gvozdev CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Marc-André Laperle CLA 2010-12-11 13:07:59 EST
Some missing files:
org.eclipse.cdt.codan.core.internal.checkers.ReturnStyleCheckerTest
org.eclipse.cdt.codan.core.internal.checkers.SuspiciousSemicolonCheckerTest

org.eclipse.cdt.dsf.mi.service.IMIExpressions
org.eclipse.cdt.dsf.mi.service.command.commands.CLIMaintenance
and probably others in dsf.gdb

Also, it looks like the HEAD of CVS is in the origin branch, see bug 315076.
Comment 1 Eclipse Webmaster CLA 2011-01-06 09:42:42 EST
Is this still an issue?

-M.
Comment 2 Marc-André Laperle CLA 2011-01-06 12:38:33 EST
(In reply to comment #1)
> Is this still an issue?

Yes, I just did a fresh clone and the files are still missing.
Comment 3 Marc Khouzam CLA 2011-01-12 15:35:08 EST
(In reply to comment #2)
> (In reply to comment #1)
> > Is this still an issue?
> 
> Yes, I just did a fresh clone and the files are still missing.

I can also confirm that the problem was still there as of Friday the 7th of Jan.
Comment 4 Andrew Gvozdev CLA 2011-01-14 15:39:38 EST
Now it seems that some branches are missing which were there before. Those are missing:
  remotes/origin/cdt_6_0
  remotes/origin/cdt_6_0_2_special
  remotes/origin/cdt_7_0
  remotes/origin/cdt_7_0_1

Also, I am getting after a fresh clone message "warning: remote HEAD refers to nonexistent ref, unable to checkout."
"git log origin/origin" shows last commit as being done in March. Is the CDT repository corrupted?
Comment 5 Eclipse Webmaster CLA 2011-01-14 16:08:38 EST
I've forced a rebuild of the git mirror, so you may have caught the mirror as it's updating.

It should be finished in a couple of hours, so let me know if it's still misbehaving.

-M.
Comment 6 Andrew Gvozdev CLA 2011-01-14 16:15:56 EST
How is it being rebuilt, are you doing every time from scratch or incrementally?
Comment 7 Eclipse Webmaster CLA 2011-01-14 16:35:46 EST
In this case I simply deleted the git copy and started the mirror script which re-creates the git mirror from cvs with a full import.

-M.
Comment 8 Andrew Gvozdev CLA 2011-01-14 23:15:53 EST
(In reply to comment #5)
> I've forced a rebuild of the git mirror, so you may have caught the mirror as
> it's updating.
> It should be finished in a couple of hours, so let me know if it's still
> misbehaving.
It does not look like it advanced anywhere, still March 2009 in git log on another fresh clone. Same as comment 4.
Comment 9 Marc Khouzam CLA 2011-01-15 14:40:16 EST
(In reply to comment #8)
> (In reply to comment #5)
> > I've forced a rebuild of the git mirror, so you may have caught the mirror as
> > it's updating.
> > It should be finished in a couple of hours, so let me know if it's still
> > misbehaving.
> It does not look like it advanced anywhere, still March 2009 in git log on
> another fresh clone. Same as comment 4.

For me too.  I actually ran a diff -r between by old clone of the rep and the new one and did not see the missing files having appeared.
Comment 10 Andrew Gvozdev CLA 2011-01-17 11:36:32 EST
After most recent pull the recent CDT changes are there but I am getting message:
# On branch origin
# Your branch and 'cdt/origin' have diverged,
# and have 9709 and 14014 different commit(s) each, respectively.

That is pretty scary. I did the previous pull just a few days earlier and kept my "origin" branch up to date. I guess that is the risk of recreating the git repo from cvs with full import from scratch. Need to figure out how to rebase my nontrivial changes to the new different origin.
Comment 11 Eclipse Webmaster CLA 2011-01-17 15:43:22 EST
I'm 're-importing' CVS into a new location beside the current repository as the import on friday failed(around 10pm) with 'unkown error'.  I'm using the CVS mirror in case the problem was caused by activity in the repository.

-M.
Comment 12 Eclipse Webmaster CLA 2011-01-18 09:09:00 EST
Well the new import has finished(without issue).

Could I impose on someone to take a look at gitmirror/org.eclipse.cdt/org.eclipse.cdt.1.git and tell me if that is 'closer' to what the repository should look like?

-M.
Comment 13 Marc Khouzam CLA 2011-01-18 09:13:13 EST
(In reply to comment #12)
> Well the new import has finished(without issue).
> 
> Could I impose on someone to take a look at
> gitmirror/org.eclipse.cdt/org.eclipse.cdt.1.git and tell me if that is 'closer'
> to what the repository should look like?
> 
> -M.

I've just started a
git clone git://dev.eclipse.org/org.eclipse.cdt/org.eclipse.cdt.1.git
Comment 14 James Blackburn CLA 2011-01-18 09:21:28 EST
When CDT is migrated once and for all to git. The migration should use cvs2git not git cvsimport as the latter is prone to corruption, missing tags and branches.
Comment 15 Marc Khouzam CLA 2011-01-18 09:36:03 EST
(In reply to comment #0)
> Some missing files:
> org.eclipse.cdt.codan.core.internal.checkers.ReturnStyleCheckerTest
> org.eclipse.cdt.codan.core.internal.checkers.SuspiciousSemicolonCheckerTest
> 
> org.eclipse.cdt.dsf.mi.service.IMIExpressions
> org.eclipse.cdt.dsf.mi.service.command.commands.CLIMaintenance
> and probably others in dsf.gdb

The missing files below are now there.

(In reply to comment #4)
> Now it seems that some branches are missing which were there before. Those are
> missing:
>   remotes/origin/cdt_6_0
>   remotes/origin/cdt_6_0_2_special
>   remotes/origin/cdt_7_0
>   remotes/origin/cdt_7_0_1
> 
> Also, I am getting after a fresh clone message "warning: remote HEAD refers to
> nonexistent ref, unable to checkout."
> "git log origin/origin" shows last commit as being done in March. Is the CDT
> repository corrupted?

I got no errors or warning when doing my clone.
"git log origin/origin" shows the last commit to be yesterday.

I haven't check the above remote branches yet.  I'm just getting familiar with git and I'm not sure how to do that yet.

But, things seem good up to now.

Thanks!
Comment 16 Marc Khouzam CLA 2011-01-18 09:36:42 EST
(In reply to comment #15)
> (In reply to comment #0)
> > Some missing files:
> > org.eclipse.cdt.codan.core.internal.checkers.ReturnStyleCheckerTest
> > org.eclipse.cdt.codan.core.internal.checkers.SuspiciousSemicolonCheckerTest
> > 
> > org.eclipse.cdt.dsf.mi.service.IMIExpressions
> > org.eclipse.cdt.dsf.mi.service.command.commands.CLIMaintenance
> > and probably others in dsf.gdb
> 
> The missing files below are now there.

I meant the missing file _above_ are now there.
Comment 17 Marc Khouzam CLA 2011-01-18 09:38:35 EST
When will the mirror get refreshed to the code from HEAD?
Comment 18 Andrew Gvozdev CLA 2011-01-18 18:35:28 EST
(In reply to comment #12)
> Well the new import has finished(without issue).
> Could I impose on someone to take a look at
> gitmirror/org.eclipse.cdt/org.eclipse.cdt.1.git and tell me if that is 'closer'
> to what the repository should look like?
It seems that one difference is that the earlier imports would combine CVS commits with the identical comments into one git commit and the newer imports (both org.eclipse.cdt.git and org.eclipse.cdt.1.git) created separate commits for each. I'd rather prefer combined one.
Comment 19 Eclipse Webmaster CLA 2011-01-19 11:27:12 EST
Ok, I've moved org.eclipse.cdt.1.git into place as org.eclipse.cdt.git.

The update scripts should start bringing the repo in line with the cvs HEAD as the data is mirrored.

I don't see any flags that indicate they would 'condense' cvs comments.

-M.
Comment 20 James Blackburn CLA 2011-01-19 11:32:56 EST
(In reply to comment #19)
> I don't see any flags that indicate they would 'condense' cvs comments.

cvsps should do this by default, combining any   See the -z switch on git cvsimport / cvsps. This defaults to 5 mins.  Any files changed not within the fuzz window are treated as a separate commit.
Comment 21 Eclipse Webmaster CLA 2011-01-19 13:59:46 EST
I can re-run the import and increase the fuzz time.  The only question is how large should the fuzz time be?

Frankly if the git mirror is now up to date(or only a couple of hours out), I'm inclined to leave it alone and let everyone get back to work.  Unless of course the commit grouping is really blocking someone.

-M.
Comment 22 James Blackburn CLA 2011-01-19 14:14:16 EST
The defaults should be fine -- most commits are within seconds of each other.  
It's odd that Andrew's seeing different behaviour; Andrew what files are you seeing this on, and what are the timestamps on the commits?
Comment 23 Andrew Gvozdev CLA 2011-01-19 14:38:21 EST
Created attachment 187140 [details]
screenshot

I just fetched again and I see the origin I was working with diverged from the new one in the repo currently on commit from 2006. The reason is this commit, see screenshot. The last file BaseUITestCase.java got to a separate commit and it seems that there is 5 min difference. Maybe we could try 10 min?
Comment 24 Andrew Gvozdev CLA 2011-01-19 17:33:03 EST
(In reply to comment #23)
> Created attachment 187140 [details]
> screenshot
> I just fetched again and I see the origin I was working with diverged from the
> new one in the repo currently on commit from 2006. The reason is this commit,
> see screenshot. The last file BaseUITestCase.java got to a separate commit and
> it seems that there is 5 min difference. Maybe we could try 10 min?
Wait, that was not 5 min but 5 sec., oops :). 10:18:08 and 10:18:13. But they went to different commits even the comment is the same.
Comment 25 Andrew Gvozdev CLA 2011-01-19 22:31:15 EST
(In reply to comment #21)
> Frankly if the git mirror is now up to date(or only a couple of hours out), I'm
> inclined to leave it alone and let everyone get back to work.  Unless of course
> the commit grouping is really blocking someone.
The only thing slightly blocking me is switching repositories. :). I am kind of waiting for the repository to stabilize on something then I'll rebase against it.
Comment 26 Eclipse Webmaster CLA 2011-01-20 10:34:30 EST
Ok, well unless there are more missing files I'm not going to 'rebuild' the repo again.  So the only updates going forward should be the automated ones to keep up with CVS head.

-M.
Comment 27 Andrew Gvozdev CLA 2011-01-23 02:16:38 EST
(In reply to comment #25)
> I am kind of waiting for the repository to stabilize on something then I'll rebase against it.
I spent a couple of evenings and successfully rebased my stuff. But it was a major PITA. I learned a bunch of things on git in the process, things I really don't want to know. Sure, this rebuild was justified (and thanks for that!), but I hope we won't do it again before CDT moves to git.