Community
Participate
Working Groups
As discussed in bug 358191, the UI team currently has to apply most of their code changes to multiple branches: R4_development and R3_development. This is error prone and there is a risk that these branches get out of sync in the bundles where they should be the same. We need to consider a different way to organize development that is easier to manage and less error prone.
Eric/Kim/myself had a meeting today to discuss. We like the following direction: - All development occurs in master branch - The small set of bundles that have truly forked are placed in a separate directory: bundles/ org.eclipse.ui.workbench <all other bundles> tests/ bundles3x/ org.eclipse.ui.workbench org.eclipse.ui With this setup, Platform UI developers can switch from R4 to R3 development by: - Removing org.eclipse.ui.workbench and org.eclipse.ui from their workspace - Importing the different copy of the bundles from the "bundles3x" folder. Open question: Can we reorganize into this shape while maintaining version history for split bundles? I think it should be possible, with it appearing in the history as a merge of the R4 and R3 streams into master. From that point onwards the 3x and 4x development history would be mixed together (a bit of a bummer but not nearly as bad as the current problem).
I'm sorry, but -1 So far, the bundle that's the most out of sync is ui.workbench, as we have 3.6 and 3.7 to scan for commits that we need to propagate into 4.x. That's the only bundle we *did* have in a separate location (with no tool support to compare them). Moving to separate locations for forked bundles (and ui.ide and jface are the next candidates, and ui.navigator if we try and add that deferred stuff) seems to me to be more likely to cause these kind of out of sync failures (as the last 2 releases prove :-) I'm loathe to abandon branching and what the tools can support (be they command line or EGit) and add a process that any developer working on anything forked must remember that it's forked *and* create a patch and apply it to a different file location, with nothing in Team that can tell that these are really just 2 different branches of the same project. What parts of this are "not working"? We have 2 streams to manage, and "it's a pain in the arse" isn't the same as not working. Did anybody activate Andrew's scripts, which was an automated way of solving 85% of the cases we care about? PW
A big -1 for the suggested setup. This makes quickly switching branches a pain and goes against the Git workflow. Though I agree with the "pain" part that Paul mentions I'm fine with the current setup. Having said that, there's another possible alternative: we split the repo into 'org.eclipse.platform.ui.common.git' with 'master' for 3.x and 4.x 'org.eclipse.platform.ui.git' with 'master' for 4.x and 'R3_development' for 3.x That way - we would not have to copy the fixes - we can use the standard Git workflows - we can easily switch from 3.x to 4.x and back inside the same workspace - everyone would know what's common
This conversation was started as a result of bug 358191 where it came as a surprise to me that 'ide' was indeed forked. How could I tell ? Dani's last point is the one that is the main issue in the current structure, "everyone would know what's common". I'm trying to prevent humans being expected to cherry picking all their fixes between two branches by trying to come up with some way to get the common code into one place. ..we're bad that way...;-). This is made worse by having no easy way to know whether the project you've changed is indeed 'common'. If there's only one copy of common code there are no synchronization issues and 'everybody knows what's common'. From the 3.x point of view it's even more important to know what's *not* common to avoid the danger of accidently cherry picking over unwanted changes.
Dani, you're right on the mark as to the solution I'm looking for, you raise all the correct points. We talked about something like this at the meeting but couldn't get around the fact that two repositories means two histories and there seems to be no way to tie changes in one to a commit in the other. That's how we ended up at the original suggestion. It has most of the attributes we want except for the clunky open/close operation when switching between a 3.x / 4.x focus. Perhaps something like PSF's could help here ? We could also tool this up properly by having a 'dev' feature that adds a command to toggle between the two project sets. It would have to be maintained whenever a project is forked of course but at the moment we have to maintain the build script whenever that happens (and any committer can change the command but not the script). The most important thing is that there's only one copy of the common bundles but it's worth noting that it's also able to track changes in common, R3 *and* R4 in a single commit, something that will help keep both sides in synch.
> We talked about something like this at the meeting but couldn't get around the > fact that two repositories means two histories and there seems to be no way to > tie changes in one to a commit in the other. Do you mean two histories for the same bundle? I don't think this would be the case. Either it would be in the normal or the common repo. If we'd decide to fork a bundle we could move it over to the other repo along with all the history.
(In reply to comment #4) > This conversation was started as a result of bug 358191 where it came as a > surprise to me that 'ide' was indeed forked. Just to be absolutely clear, org.eclipse.ui.ide has _not_ been forked. org.eclipse.ui has though. (In reply to comment #5) > The most important thing is that there's only one copy of the common bundles > but it's worth noting that it's also able to track changes in common, R3 *and* > R4 in a single commit, something that will help keep both sides in synch. Not sure I understand what's being said here. Please rephrase.
Here's the scenario I was thinking of: There's a new SWT flag (think "SWT.SHEET") where we have to make changes in both the 3.x and 4.x streams (and perhaps JFace as well) to use the new flag. Under John's scenario *all* of this work could be done under a single commit which would allow us to be sure that the changes had actually taken place in both 'branches'...
(In reply to comment #8) > Here's the scenario I was thinking of: > > There's a new SWT flag (think "SWT.SHEET") where we have to make changes in > both the 3.x and 4.x streams (and perhaps JFace as well) to use the new flag. > > Under John's scenario *all* of this work could be done under a single commit > which would allow us to be sure that the changes had actually taken place in > both 'branches'... Can you explain how that would work for branched bundles like ui.workbench?
Dani, John's proposal doesn't have the 3.x and 4.x versions in different branches...they're just in different folders on the same branch.
(In reply to comment #10) > Dani, John's proposal doesn't have the 3.x and 4.x versions in different > branches...they're just in different folders on the same branch. Yep, but you said you can do all under a single commit. But you would still have to make the change in the two different folders and before doing that make sure both changes compile against the corresponding Eclipse version.
I've toyed around with a couple of scenarios for managing the 3.8 bundles, none of them scream simple :-) The current setup means cherry-picking anything that's not ui.workbench related, and as with any manual step it's easy to get out of sync. I haven't tried writing a auto-sync script yet, I'm not sure how scriptable cherrypicking is. With the switch to a 4.2 full build in M5, I was thinking again of maybe having everything in master and only the forked bundles (ui.workbench for example) in R3_development. Working on R3-only changes in the workbench is simple, but working on an R3-only change in ui.ide, ui.workbench, jface change would be more difficult as you would need 2 repos, the R3_development one for ui.workbench and the other one for everything else. Of course then any R3 changes would need to be checked against master ui.workbench to avoid killing all involved. But... if we still require a full 3.8 build (instead of a short 3.8 build) that would limit the auto-tagging scripts to only ever updating ui.workbench for 3.8 I builds. So then would we take 4.2 changes to jface, and manually merge them into 3.8 map branch? PW
(In reply to comment #12) > The current setup means cherry-picking anything > that's not ui.workbench related, I think that's not 100% true: if there's a change in ui.workbench which is not split, then this also needs to be cherry-picked, right? > With the switch to a 4.2 full build in M5, I was thinking again of maybe having > everything in master and only the forked bundles (ui.workbench for example) in > R3_development. That's what we should strive for. > but > working on an R3-only change in ui.ide, ui.workbench, jface change would be > more difficult as you would need 2 repos, the R3_development one for > ui.workbench and the other one for everything else. It should work to have the same repo checked out twice (to a different local directory) and then check out different branches (for different projects). I prefer a more complex initial setup if it allows us to avoid duplicating the fixes for non-split bundles.
>> With the switch to a 4.2 full build in M5, I was thinking again of maybe >> having everything in master and only the forked bundles (ui.workbench for >> example) in R3_development. >That's what we should strive for. If that's not possible then we should invest into a commit hook which first compares the commit hash of the file in both streams. If it's the same the change gets committed/cherry-picked into both branches. If not, the commit is rejected. Actually, I like that solution even better than the one with the two (split) repos.
(In reply to comment #12) > With the switch to a 4.2 full build in M5, I was thinking again of maybe having > everything in master and only the forked bundles (ui.workbench for example) in > R3_development. After working with the split repo in eclipse.platform and looking at the Eclipse Foundation Long Term Support build work, I'm thinking again that all bundles in both branches is the way to go. More comments still welcome. PW
My understanding of problems with only having the split bundle in one branch: This is not the established convention for branches. To work in 3.8, you need to clone a given repo twice. Once to checkout master for normal plugins and once to checkout R3_development for split plugins. This also impacts our auto-tagging since there is not one commit we can use for build input, but 2, and we cannot tag 2 commits with the same buildId tag. It also effects the LongTermSupport work that the Eclipse Foundation is looking at. Since the build cannot run from one commit in a repo with split plugins, it must clone that repo twice (and must know which of the bundles to ignore in submodule 1 to pick them up in submodule 2). PW
I start to agree that the approach where we have all bundles in both branches (no splitting) ***and*** a commit hook that automatically cherry-picks if possible, would be the right one, because none of the splitting based solutions solves the problem where the common stuff and the split stuff is in the same bundle, like e.g. the platform doc. Of course we cannot reject the commit in case of conflicts like I previously suggested in comment 14, because then we would be doomed. The hook has to do the "normal" commit and inform the committer (via UI if possible or by sending an e-mail).
Created attachment 209073 [details] report missing commits in common projects Here's an example report script that can generate a list of commits in common projects that are in master but not in R3_development. Here's the current output for eclipse.platform.ui: bug 349224 [CommonNavigator] Navigator content provider "appearsBefore" creates hard reference to named id: bundles/org.eclipse.ui.navigator bug 349224 [CommonNavigator] Navigator content provider "appearsBefore" creates hard reference to named id: tests/org.eclipse.ui.tests.navigator Bug 351841 - Some of our examples now have errors because of deprecated elements Change to warning: examples/org.eclipse.ui.examples.contributions Bug 351841 - Some of our examples now have errors because of deprecated elements Change to warning: examples/org.eclipse.ui.examples.fieldassist Bug 351851 - [Compatibility] we need to fix up 122 errors in our org.eclipse.ui.tests bundle: tests/org.eclipse.ui.tests.harness Bug 356605 - Performance tests have compile errors in 4.x stream: tests/org.eclipse.ui.tests.performance Bug 96373: [ErrorHandling] ErrorDialog details area becomes huge with multi-line strings https://bugs.eclipse.org/bugs/show_bug.cgi?id=96373: bundles/org.eclipse.jface Fixed copyrfight date: bundles/org.eclipse.ui.ide Fix for Bug 339130 - Unable to save perspective: bundles/org.eclipse.ui.ide You can also provide an "exclude commit subject" file that helps reduce false positives. PW
A third option that is probably the most easily supported by git is to follow the suggestions in gitworkflow. 1) always put fixes into the "earliest" branch the apply to. So work touching jface or ui.ide would go into R3_development, work touching ui.workbench would go into master or R3_development as appropriate. 2) regularly merge from R3_development into master. There are simple steps like "git checkout master -- bundles/org.eclipse.ui.workbench" that can insure the merge won't touch split projects, and for some fixes like the recent Progress comparator fix we want to merge them as-is into master anyway (progress isn't different between the 2 streams). 3) if a commit goes into master and we determine it should be in R3_development, those would be cherry-picked. The pros include that it is a well understood git workflow, getting the log of outstanding R3_development changes to be merged is easy, etc. In testing my initial merge (the most work) from R3_development to master was relatively painless. I'll do it again and comment with more detailed results. The cons include that you have to know beforehand you're working on common plugins, that you have to switch workspaces to get the right repo/branch combination, etc. PW
I've pushed my test in eclipse.platform.ui to 2 branches: pwebster/bug359616_R3_development * pwebster/bug359616_master After the initial merge, it only considers changes from the merge point on. PW
(In reply to comment #19) > A third option that is probably the most easily supported by git is to follow > the suggestions in gitworkflow. > > 1) always put fixes into the "earliest" branch the apply to. So work touching > jface or ui.ide would go into R3_development, work touching ui.workbench would > go into master or R3_development as appropriate. > I don't care too much about the branch where the commit is made as long as we have a commit hook that automatically cherry-picks if possible and reminds the developer to merge conflicting changes.
(In reply to comment #21) > I don't care too much about the branch where the commit is made as long as we > have a commit hook that automatically cherry-picks if possible and reminds the > developer to merge conflicting changes. The point of option 3 is to get rid of the cherry-picking for each commit and replace it with a merge (as the developer goes or at checkpoints throughout the week). The advantage of the merge over reporting on cherry-picking is that the merge will catch changes to ui.workbench like the progress change I had to make, but the cherry-pick report will exclude ui.workbench. PW
I had a discussion with the team, and the prefer the consistency of dealing with 3.8 as a "past branch" of 4.2. 1) We will work in master (4.2). 2) Developers are responsible for cherry-picking fixes into R3_development (3.8) for the common plugins *or* for common areas of the ui.workbench (like progress). 3) we'll run a report every day and post it to platform-ui-dev@eclipse.org of fixes to common plugins in master that are not in R3_development ... pointy sticks and clown noses to be handed out appropriately. We'll review it 2 weeks into M6 to determine how it is working. PW
I've released my script to the e4 releng project: http://git.eclipse.org/c/e4/org.eclipse.e4.releng.git/tree/org.eclipse.e4.builder/scripts/git-stream-report.sh Running the script against eclipse.platform.ui.git (including the ui.workbench org.eclipse.ui.progress java package as well) gives the output below. The next step would be to either cherry-pick the commits back to R3_development or to add those commit comments to the REPO/git_report/git_stream_exclude_commits.txt file. Output: Bug 296599 - Add ability to run e4 based tests using PDE's JUnit Plug-in Test (initial commit of compatibility version of org.eclipse.ui.workbench): bundles/org.eclipse.ui.workbench/Eclipse UI/org/eclipse/ui/progress Bug 304940 [Progress] remove deprecated IProgressConstants.COMMAND_PROPERTY: bundles/org.eclipse.ui.workbench/Eclipse UI/org/eclipse/ui/progress bug 349224 [CommonNavigator] Navigator content provider "appearsBefore" creates hard reference to named id: bundles/org.eclipse.ui.navigator bug 349224 [CommonNavigator] Navigator content provider "appearsBefore" creates hard reference to named id: tests/org.eclipse.ui.tests.navigator Bug 351841 - Some of our examples now have errors because of deprecated elements Change to warning: examples/org.eclipse.ui.examples.contributions Bug 351841 - Some of our examples now have errors because of deprecated elements Change to warning: examples/org.eclipse.ui.examples.fieldassist Bug 351851 - [Compatibility] we need to fix up 122 errors in our org.eclipse.ui.tests bundle: tests/org.eclipse.ui.tests.harness Bug 356605 - Performance tests have compile errors in 4.x stream: tests/org.eclipse.ui.tests.performance Bug 96373: [ErrorHandling] ErrorDialog details area becomes huge with multi-line strings https://bugs.eclipse.org/bugs/show_bug.cgi?id=96373: bundles/org.eclipse.jface Fixed copyrfight date: bundles/org.eclipse.ui.ide Fix for Bug 305395 - Need IProgressConstants2 moved into the compatibility workbench: bundles/org.eclipse.ui.workbench/Eclipse UI/org/eclipse/ui/progress Fix for Bug 339130 - Unable to save perspective: bundles/org.eclipse.ui.ide
> bug 349224 [CommonNavigator] Navigator content provider "appearsBefore" creates hard reference to named id: bundles/org.eclipse.ui.navigator I never got around to putting this into the 3.8 stream, it's already in 3.7.1 and 4.2. Paul would you mind going this because I'm not set up right now?
> 2) Developers are responsible for cherry-picking fixes into R3_development > (3.8) for the common plugins *or* for common areas of the ui.workbench (like > progress). This is for platform.ui only at this point, right? > 3) we'll run a report every day and post it to platform-ui-dev@eclipse.org of > fixes to common plugins in master that are not in R3_development ... pointy > sticks and clown noses to be handed out appropriately. Is this part of the build or a manual task?
(In reply to comment #26) > > This is for platform.ui only at this point, right? Yes. I think we'll see if this type of process is enough for 4.2/3.8 or if we'll need something else. > > Is this part of the build or a manual task? This will be an automated report from build.eclipse.org, say every morning at 05:00 EST PW
> I never got around to putting this into the 3.8 stream, it's already in 3.7.1 > and 4.2. Paul would you mind going this because I'm not set up right now? Yup, it's no problem. PW
> > Is this part of the build or a manual task? > > This will be an automated report from build.eclipse.org, say every morning at > 05:00 EST > > PW Nice!
I've tried an enhancement, tomorrow's report will include 2 sections, master commits missing in R3_development and R3_development commits missing from master PW
(In reply to comment #30) > I've tried an enhancement, tomorrow's report will include 2 sections, master > commits missing in R3_development and R3_development commits missing from > master > > PW Thanks Paul!
I'm guessing we're done working on this bug :-) PW
