Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.
Bug 330271 - [git mirror] git mirror does not correspond to state in CVS HEAD
Summary: [git mirror] git mirror does not correspond to state in CVS HEAD
Status: RESOLVED FIXED
Alias: None
Product: Community
Classification: Eclipse Foundation
Component: Git (show other bugs)
Version: unspecified   Edit
Hardware: PC Windows 7
: P3 blocker (vote)
Target Milestone: ---   Edit
Assignee: Eclipse Webmaster CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-11-15 13:03 EST by Istvan Ballok CLA
Modified: 2012-03-23 11:05 EDT (History)
6 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Istvan Ballok CLA 2010-11-15 13:03:41 EST
Problem
=======
The git mirror does not correspond to state in CVS HEAD.

Example
=======
This file was added on Fri Oct 22 17:41:22 2010 UTC
http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.rap/runtime.rwt/org.eclipse.rap.rwt.q07/js/org/eclipse/swt/widgets/Scrollable.js?logsort=date&view=markup&revision=1.1&root=RT_Project

This is the corresponding commit in 
http://dev.eclipse.org/git/org.eclipse.rap/org.eclipse.rap.git
a00ad92fd59b86fb5f5874159dc7fca9061a75b9

The file Scrollable.js is missing from the changeset in git!! 
(and of course from all subsequent commits)

If this bug is not appropriate for the RAP product (I think so), please forward it
to the appropriate "eclipse infrastructure" product, or post the correct
product name as a comment here, and I would create a new bug for them. 
(Unfortunately, I don't know for what product should I report this bug)

It is fairly important because if the git mirror is not a 1:1 mirror,
it is (almost) useless.
Comment 1 Rüdiger Herrmann CLA 2010-11-15 13:46:40 EST
I think this is the right slot.
Comment 2 Istvan Ballok CLA 2010-11-17 04:54:00 EST
FYI

in the commit: c04153eac45f2dae8a024896c5c1b03b83b86149,
the Scrollable.js was added to the rap git mirror repository.

This git commit corresponds to the CVS commit that introduced the
1.2 reversion of this file:
http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.rap/runtime.rwt/org.eclipse.rap.rwt.q07/js/org/eclipse/swt/widgets/Scrollable.js?logsort=date&view=markup&revision=1.2&root=RT_Project

i.e., the Scrollable.js is no longer missing in these git commits: c04153eac..HEAD

However, of course, it does not solve the problem, that 
in the 74 commits that cover a 4 week long interval:
a00ad92fd59b86f..c04153eac45f2d, the Scrollable.js in its reversion 1.1 is missing.
Comment 3 Austin Riddle CLA 2011-01-04 17:52:20 EST
Hello, 

I hope no one minds me escalating the importance of this bug.  I am doing work on a GEF/Draw2D port in RAP and the Git mirrors are incredibly important for allowing me to stay in sync between the projects.  As it is now, I am at a standstill because I can't get anymore updates from the RAP Git mirror.  I can see that certain remedies have been applied in the following bugs:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=322148
https://bugs.eclipse.org/bugs/show_bug.cgi?id=322885
https://bugs.eclipse.org/bugs/show_bug.cgi?id=332365
https://bugs.eclipse.org/bugs/show_bug.cgi?id=298528

Is there any relief in sight for these kind of problems? Is that something short term that can be done in this case?  Thanks.
Comment 4 Eclipse Webmaster CLA 2011-01-05 13:56:14 EST
I can rebuild the RAP mirror, if that will help.

Your best bet for more 'permanent' fix is to either: use CVS like RAP does or to convince RAP to migrate to Git.  

-M.
Comment 5 Austin Riddle CLA 2011-01-05 14:20:48 EST
(In reply to comment #4)
> I can rebuild the RAP mirror, if that will help.
> 
> Your best bet for more 'permanent' fix is to either: use CVS like RAP does or
> to convince RAP to migrate to Git.  
> 
> -M.

A rebuild of the mirror might at least resolve the current blockage.  We are planning on migrating RAP to Git with our 2.0 release which is still a while off.
Comment 6 Austin Riddle CLA 2011-01-06 14:12:13 EST
(In reply to comment #4)

How long will the rebuild take?
Comment 7 Eclipse Webmaster CLA 2011-01-06 14:45:30 EST
It's done now.

-M.
Comment 8 Austin Riddle CLA 2011-01-06 15:35:43 EST
(In reply to comment #7)
> It's done now.
> 
> -M.

Thanks....I tried to fetch and now I receive errors about not being able to load the git URL:  git://dev.eclipse.org/org.eclipse.rap/org.eclipse.rap.git
Comment 9 Eclipse Webmaster CLA 2011-01-07 13:39:27 EST
There was a delay between the mirror rebuild and it's sync to the front end.

-M.
Comment 10 Austin Riddle CLA 2011-05-03 13:51:55 EDT
For the last 2 days now, I cannot fetch from the Git mirror at all.

The error I receive from eGit is: 

Cannot get repository refs.
Connection reset.

Is the mirror down?
Comment 11 Austin Riddle CLA 2011-05-03 13:54:32 EDT
(In reply to comment #10)

I also forgot to mention that previous to the aforementioned error, new changes could not be obtained by fetching.  So it seems that the out of sync problem still exists.
Comment 12 Florian Waibel CLA 2011-05-11 09:09:34 EDT
(In reply to comment #11)
> (In reply to comment #10)
> 
> I also forgot to mention that previous to the aforementioned error, new changes
> could not be obtained by fetching.  So it seems that the out of sync problem
> still exists.

Hello,

I can second that. Milestone 7 is *not* present in the git mirror.

florian
Comment 13 Austin Riddle CLA 2011-05-11 11:00:11 EDT
It is unfortunate that it has come to this. Since the Git mirror is seemingly not functional and at best is unreliable, I have moved to the more drastic solution of doing a CVS/Git merge on my machine. Following an approach found online, this entailed checking out from CVS, copying my .git folder into that checkout adding a .cvsignore for the .git directory and a .gitignore for the CVS subfolders. This has enabled me to synchronize CVS with my Git working directory. It is distasteful, but at least I can continue working reliably.
Comment 14 Denis Roy CLA 2011-05-11 11:03:00 EDT
(In reply to comment #13)
> It is unfortunate that it has come to this.

If you have any suggestions on what we can do to make the Git mirrors of CVS reach a level approaching 'reliable' without rebuilding them from scratch every night, then by all means, suggest away.  git-cvsimport is functional at best.
Comment 15 Denis Roy CLA 2011-05-11 11:06:27 EDT
I just tried this:


$ git clone git://dev.eclipse.org/org.eclipse.rap/org.eclipse.rap.git
Cloning into org.eclipse.rap...
remote: Counting objects: 128729, done.
remote: Compressing objects: 100% (29948/29948), done.
remote: Total 128729 (delta 78068), reused 128729 (delta 78068)
Receiving objects: 100% (128729/128729), 62.05 MiB | 1.77 MiB/s, done.
Resolving deltas: 100% (78068/78068), done.
warning: remote HEAD refers to nonexistent ref, unable to checkout.

$ cd org.eclipse.rap/
$ git checkout master
Checking out files: 100% (12447/12447), done.
Branch master set up to track remote branch master from origin.
Already on 'master'
$ ls -l
total 36
drwxrwxr-x  8  4096 May 11  2011 incubator
drwxrwxr-x 14  4096 May 11  2011 obsolete
drwxrwxr-x 14  4096 May 11  2011 releng
drwxrwxr-x  5  4096 May 11  2011 runtime.rwt
drwxrwxr-x  9  4096 May 11  2011 runtime.rwt.test
drwxrwxr-x 18  4096 May 11  2011 runtime.ui
drwxrwxr-x  7  4096 May 11  2011 runtime.ui.test
drwxrwxr-x 14  4096 May 11  2011 sandbox
drwxrwxr-x 14  4096 May 11  2011 tooling

I can obviously clone, and checkout.... What do I need to do?
Comment 16 Austin Riddle CLA 2011-05-11 11:13:19 EDT
(In reply to comment #15)

In my case the problem is with fetching.  I consistently get no new changes. In your case after cloning, does master point to a changeset that is recent, like earlier this week?
Comment 17 Florian Waibel CLA 2011-05-11 11:23:27 EDT
(In reply to comment #16)
> (In reply to comment #15)
> 
> In my case the problem is with fetching.  I consistently get no new changes. In
> your case after cloning, does master point to a changeset that is recent, like
> earlier this week?

$ git log -1
commit 94f8e8de3693c90fa5d84815eafd6ec928c81782
Author: rsternber <rsternber>
Date:   Wed Mar 30 19:37:40 2011 +0000

    formatted copyright

Mirroring the other way around:

Does any eclipse project walk the migration path with http://www.kernel.org/pub/software/scm/git/docs/git-cvsserver.html as Austin proposed here https://bugs.eclipse.org/bugs/show_bug.cgi?id=331780 ?
Comment 18 Austin Riddle CLA 2011-05-11 11:32:43 EDT
(In reply to comment #14)
> (In reply to comment #13)
> > It is unfortunate that it has come to this.
> 

Sorry, didn't mean to seem so dramatic.

> If you have any suggestions on what we can do to make the Git mirrors of CVS
> reach a level approaching 'reliable' without rebuilding them from scratch every
> night, then by all means, suggest away.  git-cvsimport is functional at best.

I definitely think that rebuilding the mirrors every night is not a solution. I believe rebuilding the mirror causes downstream consumers to have to rebase or even reclone their local repos to get in sync.

The behavior I have been experiencing seems to be sporadic.  For instance, I went for a long time without receiving any changes, and then when RAP 1.4M6 was tagged, it came down from the mirror.  But 1.4M7 has yet to do so. I am not familiar with how the mirrors are synced. Denis, could you fill me in?
Comment 19 Denis Roy CLA 2011-05-11 13:26:41 EDT
No worries, Austin.  I understand your frustration.

I had discontinued importing the RT repository because of problems I could not resolve with the import.  Alas, I must have been distracted since I didn't delete the repos.

I'm running it now, and I'll report back the problem.  In the meanwhile, this is the actual script I use:

#!/bin/bash

umask 0002

# exit;  # droy -- this thing explodes after a while for whatever reason

BASEDIR=/(snip)
TARGETDIR=/(snip)

for i in $(ls -1 $BASEDIR | egrep "^org.eclipse." | awk -F. {'print $1 "." $2 "." $3'} | sed -e 's/-.*//' | sort | uniq); do
        echo "Processing $i"
        if [ ! -d $TARGETDIR/$i ]; then
                mkdir $TARGETDIR/$i
        fi

        # Iterate all directories in /cvsroot/rt that start with $i

        for j in $(ls -1 $BASEDIR | egrep "^$i"); do
                echo "Importing $j into $i";
                export GIT_DIR=$j.git
                git-cvsimport -v -i -p -x -o master -d :local:/cvsroot/rt $j -C $TARGETDIR/$i
        done

done
Comment 20 Denis Roy CLA 2011-05-12 13:34:11 EDT
The import for RT seems to have run successfully.  A git clone / git show reveals this:
$ git show
commit f8c22210852ab8576807074d9e10356efed2eb88
Author: rsternber <rsternber>
Date:   Wed May 11 20:48:34 2011 +0000


It looks like it's in good shape -- for now  :/
Comment 21 Florian Waibel CLA 2011-05-12 14:00:44 EDT
(In reply to comment #20)
> The import for RT seems to have run successfully.  A git clone / git show
> reveals this:
> $ git show
> commit f8c22210852ab8576807074d9e10356efed2eb88
> Author: rsternber <rsternber>
> Date:   Wed May 11 20:48:34 2011 +0000
> 
> 
> It looks like it's in good shape -- for now  :/

Thanks a lot.

The git mirror now contains the 1.4 M7 tag(s).
Comment 22 Denis Roy CLA 2011-05-12 14:05:43 EDT
Ok, let's dare to close this as fixed.  Please reopen if you see it get skewed again.

The Git sync runs daily at 13:00 Eastern time and takes several hours to complete all the CVS projects.
Comment 23 Austin Riddle CLA 2011-05-12 14:47:37 EDT
OK, thanks Denis.
Comment 24 Denis Roy CLA 2012-03-23 11:05:20 EDT
> Ok, let's dare to close this as fixed.  Please reopen if you see it get skewed
> again.

Actually setting this to "fixed".  Regardless, CVS is deprecated at Eclipse, so these cloned repos will be going away soon.