Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 362076

Summary: Better policy to guard against deleting all branches and tags from our public repos
Product: Community Reporter: Paul Webster <pwebster>
Component: GitAssignee: Eclipse Webmaster <webmaster>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: P3 CC: bugs.eclipse.org, caniszczyk, contact, daniel_megert, david_williams, deepakazad, denis.roy, dj.houghton, gunnar, hargrave, info, irbull, john.arthorne, kim.moir, markus.kell.r, matthias.sohn, michael.wenz, Mike_Wilson, mober.at+eclipse, ob1.eclipse, overholt, remy.suen, sbouchet, sptaszkiewicz, steffen.pingel, stephan.herrmann, stepper, tjwatson, tomasz.zarna, wayne.beaton
Version: unspecified   
Target Milestone: ---   
Hardware: PC   
OS: Linux   
Whiteboard:
Bug Depends on: 343150, 362358, 362361, 362363    
Bug Blocks:    
Attachments:
Description Flags
gc-all configuration script none

Description Paul Webster CLA 2011-10-26 10:08:10 EDT
ISSUE:

A committer (through use of a tool) was able to delete most of the tags and branches from our public eclipse.platform.ui git repo.  See bug 361707

FALLOUT:

With the refs gone, it looks like a prune or gc was done on the public repo, because the missing branch's commits were deleted from the repo as well.  We're still not sure what caused the gc/prune to run, as it would have to run on build.eclipse.org, right?  Can the git client request a prune or GC on the remote?

With the tags gone, the Eclipse SDK builds failed because the map files were pointing to nothing.

FIXES:

The quickest fix is: Find a repo that still has everything and have the Foundation disable the commit hooks.  Then push all the tags and refs (which will push the commits).  Con: Finding that repo can be a challenge.  How do you verify how correct the public repo will be?

Another fix:  Restore from backup and have the Foundation disable the commit hooks.  Then add back any missing commits since the backup was done.   Con:  You still have to recreate any missing commits from somewhere, but the plus is potentially less churn to the public repo. In this specific case, though, Denis requested a restore but still hadn't received it 10 hours later.

POLICY:

What kind of policy should we put in place to prevent or limit the damage from these kind of operations?  Let's discuss!


I'll capture the first set of suggestions in comments.

PW
Comment 1 Paul Webster CLA 2011-10-26 10:17:13 EDT
via Chris A and Kim M:

We can set properties on the public repos:
receive.denyNonFastForwards=true
  - prevents monkeying around with the remote branch pointers.
receive.denyDeletes=true
  - denies branch and tag deletion over a push

There's general agreement that receive.denyNonFastForwards=true is probably a good policy.

The good side to receive.denyDeletes=true is that it would protect us against run-away scripts and slightly-misconfigured tools removing the branches and tags (and allowing a GC to remove the commits).  If you push it to the public repo, it's there (although it can be cleaned out manually but only on the server directly).

The downside is that committers would not be able to clean up pushed topic or feature branches, which is a common git workflow.  These would continue to accumulate.

PW
Comment 2 Paul Webster CLA 2011-10-26 10:18:59 EDT
BJ commented on the pain of receive.denyDeletes=true to the common git workflow:

receive.denyNonFastForwards=true is a good idea

However receive.denyDeletes=true is not. It prevents people from sharing "feature" branches in the repo and then deleting them when they are done.

At OSGi, we have a pre-receive hook which allows committers to create and delete branches matching the pattern */* (e.g hargrave/myidea) but only admins can create or delete branches w/o a /. This protects the main branches (master, next) from inadvertent deletes (unless the admin screws up :-) while allowing all the committers to share feature branches in the repo.
Comment 3 BJ Hargrave CLA 2011-10-26 10:58:58 EDT
(In reply to comment #2)
> At OSGi, we have a pre-receive hook which allows committers to create and
> delete branches matching the pattern */* (e.g hargrave/myidea) but only admins
> can create or delete branches w/o a /. This protects the main branches (master,
> next) from inadvertent deletes (unless the admin screws up :-) while allowing
> all the committers to share feature branches in the repo.

You will also want the pre-receive hook to prevent committers from creating or modifying tags in the repo. This should only be allowed for special parties like build bot and admins. OSGi also does not allow un-annotated tags to be pushed to the central repo.
Comment 4 BJ Hargrave CLA 2011-10-26 11:05:39 EDT
(In reply to comment #0)
> We're
> still not sure what caused the gc/prune to run, as it would have to run on
> build.eclipse.org, right?  Can the git client request a prune or GC on the
> remote?

Sometimes, gc will be run automatically:

"Some git commands run git gc --auto after performing operations that could create many loose objects."

We could use a pre-auto-gc hook to disable auto gc on the foundation repos.
Comment 5 David Williams CLA 2011-10-26 11:25:38 EDT
In my view, the default behavior on all repos at the Eclipse Foundation should
be to "delete nothing" ... that's the only way to guarantee long term
availability of source some adopter or downstream project may be depending on. 

[I am surprised this was not already the case, and well understood, after all this time ... easy for me to say as a Git newbie :) ] 

Granted, there could be mechanisms to allow some certain patterns of branches
(or tags) to be created and later removed by a committer ... but, I think that
should be the exception, rather than the rule. 

(Thanks for all the tips, BJ! Very helpful to have your expertise and "real world" experience.)
Comment 6 Tomasz Zarna CLA 2011-10-26 11:43:26 EDT
(In reply to comment #3)
> [...] prevent committers from creating or modifying tags in the repo. 
> This should only be allowed for special parties like build bot and admins.

Does it mean releasing (tagging repos and updating the map files) for an I-build would be limited to a narrow group of committers?
Comment 7 Denis Roy CLA 2011-10-26 11:48:18 EDT
(In reply to comment #0)
> FIXES:
> 
> The quickest fix is: Find a repo that still has everything and have the
> Foundation disable the commit hooks.  Then push all the tags and refs (which
> will push the commits).  Con: Finding that repo can be a challenge.

There is a delayed copy of the repos in /home/data2/git, which is available to all the committers with a shell.  In this specific case, by the time we saw the bug the shadow copy was already "up to date".
  
> Another fix:  Restore from backup and have the Foundation [snip]
> In this specific case, though, Denis
> requested a restore but still hadn't received it 10 hours later.

We are investigating what's taking so long. Restoring from tape is no fast task, but it shouldn't take days.
Comment 8 Ian Bull CLA 2011-10-26 11:50:52 EDT
(In reply to comment #6)
> (In reply to comment #3)
> > [...] prevent committers from creating or modifying tags in the repo. 
> > This should only be allowed for special parties like build bot and admins.
> 
> Does it mean releasing (tagging repos and updating the map files) for an
> I-build would be limited to a narrow group of committers?

For now I don't think we can do this with the Eclipse project since we do need access to tag. However, we will (hopefully) be moving towards a automated tagging system, so this shouldn't be an issue.

As discussed on the Eclipse call today, I think we should start by disabling ALL deletes.  (receive.denyDeletes=true). I understand that we won't be able to delete feature branches, but that's a small price to pay -- compared to loosing our entire repository.

I think we should investigate BJ's suggestion (receive hooks that don't allow the master branch to be deleted), but let's start by protecting our repositories first.
Comment 9 BJ Hargrave CLA 2011-10-26 11:53:40 EDT
(In reply to comment #5)
> In my view, the default behavior on all repos at the Eclipse Foundation should
> be to "delete nothing" ... that's the only way to guarantee long term
> availability of source some adopter or downstream project may be depending on. 

A delete nothing policy is really only necessary for build tags. These will record the important history and pin the referenced objects in the repo. No one should be depending upon anything in branches other than master which has not been tagged for a build.

Everything else is fair game for changing as dictated by the workflow of the committers on the repo.

A too rigid policy about disallowing deleting branches will likely drive people away from using the foundation repo as a place to share "feature" branches with others. For example, I would clone to github and do my feature work there. Only when I am ready to promote to master would my work enter the foundation repo.

That workflow is a valid choice but should be decided upon after careful consideration. I would much rather keep all the work in the foundation repos.
Comment 10 Chris Aniszczyk CLA 2011-10-26 12:01:02 EDT
(In reply to comment #5)
> Granted, there could be mechanisms to allow some certain patterns of branches
> (or tags) to be created and later removed by a committer ... but, I think that
> should be the exception, rather than the rule. 

+1

There's nothing wrong with people doing the feature branch work elsewhere... say on github... git makes this really easy to do and our IP process handles it now. On top of that, github (and other services like it) make it easy to see who has forked your code and what they are up to.

I would rather the canonical git repos always be monotonic and nothing be deleted.
Comment 11 BJ Hargrave CLA 2011-10-26 12:09:10 EDT
(In reply to comment #8)
> > Does it mean releasing (tagging repos and updating the map files) for an
> > I-build would be limited to a narrow group of committers?
> 

I think this all depends on how it is decided what goes into the build. For example, it could be that the tip of master is always what goes into the build. So the project puts commits in the build by putting them on the master branch. The next (or develop) branch is used for commits that may then be promoted to master.

Then the build bot will tag master when it builds from it. 

> As discussed on the Eclipse call today, I think we should start by disabling
> ALL deletes.  (receive.denyDeletes=true). I understand that we won't be able to
> delete feature branches, but that's a small price to pay -- compared to loosing
> our entire repository.

Isn't this overreacting to one mistake? And the repository was not lost :-) 

If you end up doing this, let people know first so they can delete their feature branches from the foundation repos and relocate them to github (or somewhere else). I don't really want my experiments permanently recorded in history :-)
Comment 12 BJ Hargrave CLA 2011-10-26 12:14:03 EDT
(In reply to comment #10)
> (In reply to comment #5)
> There's nothing wrong with people doing the feature branch work elsewhere...
> say on github... git makes this really easy to do and our IP process handles it
> now. On top of that, github (and other services like it) make it easy to see
> who has forked your code and what they are up to.

This is certainly a valid way to operate. I just think this decision should be carefully considered.

> 
> I would rather the canonical git repos always be monotonic and nothing be
> deleted.

If this is the case, then we only need a single branch in each repo (master) which is what is built and tagged. Everything else should be done in other repos (e.g. on github) and only pushed to master in the foundation repo when ready to enter the build.
Comment 13 Matthias Sohn CLA 2011-10-26 12:23:34 EDT
At SAP we use Gerrit as Git server and use the following policy to prevent such problems:
- reflog is auto-configured to never expire so that no changes get gc'ed except corrupt data created by interrupted pushes, we learnt this from Shawn
- this way the centrally stored reflog can be used to restore destroyed branches in case somebody nuked them
- we schedule the attached script to run weekly in order to ensure this configuration and gc all repositories (stolen from the Android project and slightly modified)
- we strictly disallow tag deletion for all repositories through Gerrit permission settings, we are considering to relax this to only prevent deletion of tags in a certain namespace used to tag releases. This would allow deletion of tags used for less permanent purposes. But still I would consider it bad practice to delete published tags.
- if you want temporary tags, e.g. in order to tag each CI build you may consider to let Hudson push these tags to a read-only replica (except for Hudson pushing the tags) of the public repository maintained only for this purpose (we did that earlier for JGit and EGit [1] but stopped doing that since nobody had a need for these tags)

[1] http://egit.eclipse.org/w/?p=build-egit.git;a=summary
Comment 14 Matthias Sohn CLA 2011-10-26 12:24:29 EDT
Created attachment 206001 [details]
gc-all configuration script
Comment 15 Dani Megert CLA 2011-10-27 09:00:58 EDT
(In reply to comment #1)
> There's general agreement that receive.denyNonFastForwards=true is probably a
> good policy.
+1.

(In reply to comment #5)
> In my view, the default behavior on all repos at the Eclipse Foundation should
> be to "delete nothing" ... that's the only way to guarantee long term
> availability of source some adopter or downstream project may be depending on. 
+1. Each project could then decide to relax this, but out of the box a new install should be safe/protected.

> (In reply to comment #6)
> As discussed on the Eclipse call today, I think we should start by disabling
> ALL deletes.  (receive.denyDeletes=true). I understand that we won't be able to
> delete feature branches, but that's a small price to pay -- compared to loosing
> our entire repository.
+1. We can relax it later when we've decided what we want to allow.


One approach we discussed in the PMC call would be to allow committers the creation of branches with a certain name pattern (e.g. committerID + "_" + branchName). This would be enforced by a hook. Such branches could afterwards (only) be deleted by the corresponding committer. For tags we could take an almost similar approach except that a committer can create tags with arbitrary names but only delete those that start with the committerID.
Comment 16 BJ Hargrave CLA 2011-10-27 09:14:03 EDT
(In reply to comment #15)
> One approach we discussed in the PMC call would be to allow committers the
> creation of branches with a certain name pattern (e.g. committerID + "_" +
> branchName). This would be enforced by a hook. Such branches could afterwards
> (only) be deleted by the corresponding committer. For tags we could take an
> almost similar approach except that a committer can create tags with arbitrary
> names but only delete those that start with the committerID.

The correct pattern to use is committer + "/" + branch name. "/" fits into the branch and tag hierarchies of git: refs/heads/committer/branchname.
Comment 17 Deepak Azad CLA 2011-10-27 09:33:48 EDT
(In reply to comment #15)
> One approach we discussed in the PMC call would be to allow committers the
> creation of branches with a certain name pattern (e.g. committerID + "_" +
> branchName). 
+1

> This would be enforced by a hook. Such branches could afterwards
> (only) be deleted by the corresponding committer. For tags we could take an
> almost similar approach except that a committer can create tags with arbitrary
> names but only delete those that start with the committerID.
That sounds a bit too strict. What happens if a committer stops working on a project and these branches and tags are left behind? Do the delete restrictions also apply to rename, I mean can another committer rename a branch and then delete? just asking... :)
Comment 18 Dani Megert CLA 2011-10-27 09:38:27 EDT
>> This would be enforced by a hook. Such branches could afterwards
>> (only) be deleted by the corresponding committer. For tags we could take an
>> almost similar approach except that a committer can create tags with arbitrary
>> names but only delete those that start with the committerID.
> That sounds a bit too strict. What happens if a committer stops working on a
> project and these branches and tags are left behind?
In such a case there's always the possibility to ask the webmaster to do this. It shouldn't be up to you to delete my branches ;-).
Comment 19 BJ Hargrave CLA 2011-10-27 09:43:57 EDT
(In reply to comment #17)
> That sounds a bit too strict. What happens if a committer stops working on a
> project and these branches and tags are left behind? 

I think it is easier to allow any committer to create and delete any branch that matched the pattern .*/.*

> Do the delete restrictions
> also apply to rename, I mean can another committer rename a branch and then
> delete? just asking... :)

There is no such thing as branch renaming in git. You create a new branch and delete the old one.
Comment 20 BJ Hargrave CLA 2011-10-27 09:45:29 EDT
(In reply to comment #18)
> In such a case there's always the possibility to ask the webmaster to do this.
> It shouldn't be up to you to delete my branches ;-).

So you trust committers to change all of the code in the repo, but not to manage committer branches (i.e branches matching the pattern .*/.*)? Seems a bit paranoid.
Comment 21 Deepak Azad CLA 2011-10-27 09:51:36 EDT
(In reply to comment #19)
> There is no such thing as branch renaming in git. You create a new branch and
> delete the old one.
oh ok. EGit has a 'Rename' option, I guess it creates a new branch and deletes the old one?

(In reply to comment #18)
> It shouldn't be up to you to delete my branches ;-).
I just don't expect a committer to delete branches willy nilly and hence think the restriction is a bit unnecessary and overkill. 

I prefer BJ's idea 
> I think it is easier to allow any committer to create and delete any branch
> that matched the pattern .*/.*
Comment 22 Dani Megert CLA 2011-10-27 09:56:40 EDT
> (In reply to comment #20)
> So you trust committers to change all of the code in the repo, but not to
> manage committer branches (i.e branches matching the pattern .*/.*)? Seems a
> bit paranoid.
Mmh, no. The big difference is that I can track all changes and easily revert them while a deletion of a tag or branch is permanent and can even result in lost commits in case of a GC. Hence it is way more dangerous as the current incident showed.


> (In reply to comment #21)
> > It shouldn't be up to you to delete my branches ;-).
> I just don't expect a committer to delete branches willy nilly
See bug 361707.

> I prefer BJ's idea 
> > I think it is easier to allow any committer to create and delete any branch
> > that matched the pattern .*/.*
I guess that could work too for us.
Comment 23 Paul Webster CLA 2011-10-27 09:58:11 EDT
I think the */* pattern for developer branches is a good one.

(In reply to comment #19)
> 
> I think it is easier to allow any committer to create and delete any branch
> that matched the pattern .*/.*


This would just move the problem we experienced in platform.ui from all of the branches to all of the developer day-to-day-work branches.  I don't see a problem with restricting developers to deleting their own branches ... if the project lead is willing to clean up any stragglers in their repos at release time.

PW
Comment 24 Markus Keller CLA 2011-10-27 10:01:40 EDT
Committers should still be able to create maintenance and perf baseline
branches, so please add these patterns to allowed branches (only create, not delete):

R*_maintenance*
perf_*

In most projects, we could also clean up the existing branches and move all
non-master branches that don't match these patterns to zzz_archive/<branchName>.
Comment 25 Dani Megert CLA 2011-10-27 10:14:58 EDT
(In reply to comment #24)
> Committers should still be able to create maintenance and perf baseline
> branches, so please add these patterns to allowed branches (only create, not
> delete):
> 
> R*_maintenance*
> perf_*
> 
Good catch! We could add this to the hook or only allow project leads to do this. This would be similar to adding new versions and target milestones to bugzilla, which is also not open to all.
Comment 26 BJ Hargrave CLA 2011-10-27 11:56:16 EDT
(In reply to comment #23)
> This would just move the problem we experienced in platform.ui from all of the
> branches to all of the developer day-to-day-work branches.  I don't see a
> problem with restricting developers to deleting their own branches ... 

So what if some other committer deletes my branch? I can just recreate it. I have that branch on my local repo along with all the objects behind it.
Comment 27 BJ Hargrave CLA 2011-10-27 12:00:41 EDT
(In reply to comment #24)
> In most projects, we could also clean up the existing branches and move all
> non-master branches that don't match these patterns to
> zzz_archive/<branchName>.

The migration from CVS created a lot of crap branches. Only the master and maint branches should really be branches. The others should just be tags. I went through the rt.equinox.framework repo and "converted" all the crap branches to tags of the form CVS/oldbranchname to clearly mark them as CVS conversion residue.

My point is that these crap branches should be "archived" as tags not branches. If someone ever really needs to put new commits after them, a new branch can be created from the tag.
Comment 28 John Arthorne CLA 2011-10-27 12:03:28 EDT
(In reply to comment #26)
> So what if some other committer deletes my branch? I can just recreate it. I
> have that branch on my local repo along with all the objects behind it.

The current eclipse commit hook may prevent you pushing back some of those objects. In the Platform UI case a committer who did have all the branches in their local clone was not able to push those back. Also, one of the main reasons for pushing those topic branches to the remote in the first place is to have a backup, so the answer that we can fall back to restoring the remote based on our local clone isn't very satisfying.
Comment 29 Paul Webster CLA 2011-10-27 12:21:36 EDT
(In reply to comment #26)
> 
> So what if some other committer deletes my branch? I can just recreate it. I
> have that branch on my local repo along with all the objects behind it.

- If you had taken a pull before the problem was well known your commits might have been pruned (unless you disabled that auto-prune thing).  
- Having all the developers try re-assemble their branches and commits is time-consuming (and hence error prone).
- While it's true that if you *can* track down a repo and disable the commit hooks, someone can push the repo back ... "we might be able to restore it" is the same as having no policy at all.  The public repos have to be more dependable than that.

I'm not willing to have my topic branches deleted by other developers.  I don't need more hunting around to put things back that should not be removed in the first place.  If this is restricted to our project and other projects would like less restrictive options, I'm fine with that as well.

PW
Comment 30 BJ Hargrave CLA 2011-10-27 12:29:32 EDT
(In reply to comment #28)
> The current eclipse commit hook may prevent you pushing back some of those
> objects. 

I suspect you mean branches. Perhaps the push was not a non-fastfoward.

> In the Platform UI case a committer who did have all the branches in
> their local clone was not able to push those back. 

Are the full details of that problem written down somewhere for me to read? All I have seen so far was that there was a problem and this are now better. But not a proper post-mortem. It is possible there were other ways to recover.

Again, that issue was the result of not having proper hooks in place. We should not let this color how we will operate once we have proper hooks in place.

> Also, one of the main
> reasons for pushing those topic branches to the remote in the first place is to
> have a backup, 

That is a bad reason. The reason is to share them with others. One should be responsible for backing up their own systems.

> so the answer that we can fall back to restoring the remote
> based on our local clone isn't very satisfying.

Using git means thinking differently. The git repos at the foundation are copies like everyone elses. They are the common sharing point and are what the builds use. But they are not the only repos like SVN and CVS repos were. Having multiple copies is very satisfying (and is the perfect backup strategy). We need to make sure all foundation repos are mirrored to github on a regular basis as an additional copy.

When k.org was taken down due to compromise, git development chugged along because the git repos had copies everywhere (including github and google). As k.org began to come back up and it's git repo went back online, all the git work done since k.org went down was pushed to k.org.
Comment 31 BJ Hargrave CLA 2011-10-27 12:34:59 EDT
(In reply to comment #29)
> - If you had taken a pull before the problem was well known your commits might
> have been pruned (unless you disabled that auto-prune thing).  

No one can delete anything in your repo. If someone deletes a branch on the repo you pull from, that does not delete the local branch you have.

> - Having all the developers try re-assemble their branches and commits is
> time-consuming (and hence error prone).

You don't need to reassemble anything. The branch is still in your local repo. Just push it back to the central repo.

> - While it's true that if you *can* track down a repo and disable the commit
> hooks, someone can push the repo back ... "we might be able to restore it" is
> the same as having no policy at all.  The public repos have to be more
> dependable than that.
> 

I don't understand this probably because I don't have a full accounting of the recent problem and the attempts to correct it.

> I'm not willing to have my topic branches deleted by other developers.  I don't
> need more hunting around to put things back that should not be removed in the
> first place.  If this is restricted to our project and other projects would
> like less restrictive options, I'm fine with that as well.

No one can delete anything in your repo. If someone deletes a branch on the repo you pull from, that does not delete the local branch you have.
Comment 32 Paul Webster CLA 2011-10-27 12:35:37 EDT
(In reply to comment #30)
> Are the full details of that problem written down somewhere for me to read? All
> I have seen so far was that there was a problem and this are now better. But
> not a proper post-mortem. It is possible there were other ways to recover.

bug 361707 is the original problem, comment #0 is the summary.

PW
Comment 33 Paul Webster CLA 2011-10-27 12:39:15 EDT
(In reply to comment #31)
> 
> No one can delete anything in your repo. If someone deletes a branch on the
> repo you pull from, that does not delete the local branch you have.

OK, that makes sense.  In this case the problem  is that there were branches (like R3_5_maintenance and lower, never mind the CVS to archive branches) that no one had as local branches, so when the remote branches disappear they were all wiped out.

PW
Comment 34 BJ Hargrave CLA 2011-10-27 13:23:32 EDT
(In reply to comment #32)
> bug 361707 is the original problem, comment #0 is the summary.

OK, looking at bug 361707 comment 1, it seems the hook was validating incoming commit object to make sure they are from current known committers. So the hook rejected the commit because the commit was created from the CVS migration using an email that is not a known current committer.

So yes, the hook would have to be set aside to push back those commits. However, this problem was allowed because we did not have a proper hook in place. A hook that would prevent deleting important branches (master, maint). Once we have that hook in place, this particular issue will no longer reoccur.

(In reply to comment #33)
> In this case the problem  is that there were branches
> (like R3_5_maintenance and lower, never mind the CVS to archive branches) that
> no one had as local branches, so when the remote branches disappear they were
> all wiped out.

This doesn't make sense. I have a clone of rt.equinox.framework on my system:

$ git branch -a
  R3_7_maintenance
* hargrave/osgi44
  master
  remotes/origin/BETA_JAVA7
  remotes/origin/HEAD -> origin/master
  remotes/origin/R3_0_maintenance
  remotes/origin/R3_1_2_bug191652
  remotes/origin/R3_1_maintenance
  remotes/origin/R3_2_1_criticalFixes
...

All the remote branches exist in my local repo. Unless I do git fetch --prune, those remotes/origin branches are not deleted. So I still have them and all the objects behind them.

[Of course I manage my repo with git command line. I went through the motions of cloning the repo with egit and it gives me a dialog to select which branches to fetch in the clone. So people using egit may not be getting the full repo (sigh) which reduces their use as full copies and is another reason to mirror the full repo at github. People should really clone the full repo.]
Comment 35 Matthias Sohn CLA 2011-10-27 18:26:08 EDT
(In reply to comment #19)
> (In reply to comment #17)
> ...
> There is no such thing as branch renaming in git. You create a new branch and
> delete the old one.

look at: git branch -m
Comment 36 BJ Hargrave CLA 2011-10-27 18:46:47 EDT
(In reply to comment #35)
> look at: git branch -m

Yes, but under the covers it is a delete and a create. Certainly from the perspective of the hooks. Hooks will see a branch being deleted and a branch being created.
Comment 37 David Williams CLA 2011-10-28 02:34:56 EDT
For what its worth, according to "git help init", if you use --shared, you automatically get receive.denyNonFastForwards, and surely all repos at Eclipse Foundation are "shared" ... so, I think this bug mostly becomes about "delete". Anyone else find it odd the default is to guard against changing history ... but not, by default, guard on deleting the whole thing?  (Sounds like a work around to me :) 

I did check and the simple act of "cloning" the --bare --shared denyNonFastForwards repository does not clone/copy all the properties ... the local repo does not prevent NonFastForwards, which is, I think, what allows rebase to work locally, but not remotely. 

Now, about that delete ... I think its obvious we need to provide a stable, rock solid, high integrity, immutable complete repository of code to be used by Eclipse Members for vast periods of time ... and that deletes from there can not be be decided simply by the committers or projects (with the exception of maybe those "userid/featurework" branches and perhaps we could trust PMCs if they really had reason to delete certain things ... but it is basically a service, or protection offered by the Eclipse Foundation ... I like to call that the central repository, but I now those with just love Git just start snickering when they hear the word "central". So, what ever you want to call it, we need one. And it needs to be guarded by Eclipse IT staff and mechanisms, not left up to individual committers and projects. I guess we could call it the canonical monotonic repos, as Chris did. 

Adding Wayne to CC as I think this comes down to "Eclipse Foundation Policy" ... whether thy intend to continue to provide a canonical monotonic repo data base as they have been ... or, if they just want to be another git-hub and let people do what ever they want. If the later ... better get the word out to members quickly, as they will each have to snag their own copies to have canonical monotonic versions that they need for their business.
Comment 38 Wayne Beaton CLA 2011-10-28 06:59:23 EDT
(In reply to comment #10)

> I would rather the canonical git repos always be monotonic and nothing be
> deleted.

+1

I actually assumed that this was the case.

(In reply to comment #12)
 
> If this is the case, then we only need a single branch in each repo (master)
> which is what is built and tagged. Everything else should be done in other
> repos (e.g. on github) and only pushed to master in the foundation repo when
> ready to enter the build.

I disagree. The standard development practice, AFAIK, is to create a new branch in the local repository, do the work, merge back into the master branch and then push. So, relatively few branches will make it into the canonical repository only. It'd be only shared development branches that would need to be in the repository, and I would think that keeping them around might be useful (at least for historical context). 

Or is this naive?

(In reply to comment #37)

> Adding Wayne to CC as I think this comes down to "Eclipse Foundation Policy"
> ... whether thy intend to continue to provide a canonical monotonic repo data
> base as they have been ... or, if they just want to be another git-hub and let
> people do what ever they want. If the later ... better get the word out to
> members quickly, as they will each have to snag their own copies to have
> canonical monotonic versions that they need for their business.

You have a way with words... IMHO, the short version is that we need to follow the principle of least surprise. Things can't just start disappearing from the repository. If we end up in a situation where adopters feel a need to retain their own canonical monotonic version of any of our repositories, then we've failed miserably.

There are cases where we have to rewrite history, but they should be rare and isolated (e.g. rejections from parallel IP)
Comment 39 BJ Hargrave CLA 2011-10-28 08:24:54 EDT
(In reply to comment #37)
> Now, about that delete ... I think its obvious we need to provide a stable,
> rock solid, high integrity, immutable complete repository of code to be used by
> Eclipse Members for vast periods of time ... 

I don't think anyone will dispute this. The question is what is the important part of the repo that should be preserved for ever. Not every wild brained experiment needs to be kept forever. What needs to be preserved is what is built. This is what tags are for. Tags created by the build process must not be allowed to be deleted.

> 
> (In reply to comment #12)
> 
> > If this is the case, then we only need a single branch in each repo (master)
> > which is what is built and tagged. Everything else should be done in other
> > repos (e.g. on github) and only pushed to master in the foundation repo when
> > ready to enter the build.
> 
> I disagree. The standard development practice, AFAIK, is to create a new branch
> in the local repository, do the work, merge back into the master branch and
> then push. So, relatively few branches will make it into the canonical
> repository only. 

This is not entirely true. I often want to share my experiments with other committers for them to comment on or amend. These experiments, if successful, will eventually be merged onto master and then become part of the record. Otherwise the branch is discarded. 

> It'd be only shared development branches that would need to be
> in the repository, and I would think that keeping them around might be useful
> (at least for historical context). 

Assuming people do need to share experimental branches, if we retain every branch ever shared in the foundation's repos, we will eventually be consumed by branches. We will not be able to see the forest for all of the dead branches.

> You have a way with words... IMHO, the short version is that we need to follow
> the principle of least surprise. Things can't just start disappearing from the
> repository. If we end up in a situation where adopters feel a need to retain
> their own canonical monotonic version of any of our repositories, then we've
> failed miserably.

No one is suggesting the built code is ever deleted. But git makes branching easy and people should and will branch. And they will need to share their branches with other committers and the community for feedback. We can either encourage them to do that sharing in the foundation's repos or in external repos (e.g. github). But dead experiments do not need to be kept forever.

> There are cases where we have to rewrite history, but they should be rare and
> isolated (e.g. rejections from parallel IP)

I don't think this discussion is about rewriting (non-fastforward) history of the master branch.
Comment 40 David Williams CLA 2011-10-28 16:00:04 EDT
I think best to divide this bug into 3 parts, representing 3 pieces of work. 

bug 362358 (denyNonFastforwards) is pretty independent of the rest and seems non-controversial. 

For the rest, I propose that we give focus to preventing deletes (bug 362361) and then handle separately the issue of when and how to relax that rule (bug 362363). 

I should add, I think we all know there are "exceptions to the rules" where some project accidentally created some odd or some confusingly mis-named branches and they really do want to delete them ... but ... those could be handled as exceptions ... open a bug, work with webmasters, documents what's done, etc. 

QED?
Comment 41 Denis Roy CLA 2014-05-07 10:11:34 EDT
I think we are done here.  All new repos are configured with FF only, no forces and through the magic of Gerrit, we don't allow writing to the master branch.

Of course, none of this applied to the repos we host at Github, but there's not much we can do about it.

What am I missing?
Comment 42 Matthias Sohn CLA 2014-05-07 11:37:11 EDT
(In reply to Denis Roy from comment #41)
> ... and through the magic of Gerrit, we don't allow writing to the master
> branch.

I don't understand what you say here
Comment 43 Denis Roy CLA 2014-05-07 14:35:36 EDT
I guess I'm saying this has been a board policy for over a year now, and we're doing everything we can.  Closing as fixed.