Bugzilla – Full Text Bug Listing
| Summary: | Avoid creating "Dirt" during the CBI build | ||
|---|---|---|---|
| Product: | [Eclipse Project] Platform | Reporter: | Lars Vogel <Lars.Vogel> |
| Component: | Releng | Assignee: | Platform-Releng-Inbox <platform-releng-inbox> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | P3 | CC: | akurtakov, daniel_megert, david_williams, fabian.pfaff, Lars.Vogel, loskutov, mat.booth, thanh.ha |
| Version: | 4.3 | ||
| Target Milestone: | --- | ||
| Hardware: | PC | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Bug Depends on: | 419503, 419506, 419508, 419510, 419522, 419524, 419527, 419528, 419531, 459323, 459584, 461002, 463150, 481710, 487723, 488935, 490410, 490412, 490413, 498988, 498992 | ||
| Bug Blocks: | 434596, 480588 | ||
| Attachments: | |||
|
Description
Lars Vogel
There's already specific bugs open for these against each component, search eclipse bugs for subject "Dirty working tree". https://bugs.eclipse.org/bugs/buglist.cgi?list_id=7249258&short_desc=Dirty%20working%20tree&classification=Eclipse&query_format=advanced&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED&short_desc_type=allwordssubstr The issue you are hitting now, based on your post to CBI mailing list is actually a case of "over cleaning" :) See bug 419811. Thanks David. Maybe a "Master Bug" which depends on all the component bugs for the dirty issue would be helpful? (In reply to Lars Vogel from comment #2) > Thanks David. Maybe a "Master Bug" which depends on all the component bugs > for the dirty issue would be helpful? You are welcome to re-open this one and use for that purpose, if you'd like. (Be sure to include the fixed ones :) (which were not in my query above). Reopening the bug to track the detailed dirty bugs. David, adding &bug_status=RESOLVED and and &bug_status=CLOSED to your query did not show me more bugs. Is the assumption correct that after all these "dirt" bugs are fixed, the reset of the submodule via Git (see below) is not required anymore? git submodule foreach git clean -f -d -x git submodule foreach git reset --hard HEAD git clean -f -d -x git reset --hard HEAD (In reply to Lars Vogel from comment #5) > Is the assumption correct that after all these "dirt" bugs are fixed, the > reset of the submodule via Git (see below) is not required anymore? > > git submodule foreach git clean -f -d -x > git submodule foreach git reset --hard HEAD > git clean -f -d -x > git reset --hard HEAD In theory yes. Created attachment 236823 [details] post build clean status after I20131023-2000 shows that all "about.mappings" are now no longer dirt (bug 419503). (I'll periodically attach "git status" listings here, after one of our production builds ... but I'm not expecting this to be fixed "real quick" ... but we'll make progress!) Created attachment 237072 [details]
post build clean status after I20131030-0800
While not quite final build for M3 ... close enough to say this is the state of things at M3.
Some progress made ... much still to go.
(In reply to David Williams from comment #8) > Created attachment 237072 [details] > post build clean status after I20131030-0800 > > While not quite final build for M3 ... close enough to say this is the state > of things at M3. > > Some progress made ... much still to go. Thought I'd check today, after I20131126-0800 build, and there has been no change since last attachment. Looks like the "dirt" is produced by the builder by placing temp files into the root of bundles. In that case the builder should actually remove those files. The source providers can't and don't want to know what a certain build system puts into their source tree. If the builder is not capable deleting that temp stuff or use a temp directory outside the source tree, then it remains to clients to call git clean. Having said that, most of the "dirt" could be avoided by changing the packaging of those bundles to get rid of nested JARs. Created attachment 239293 [details]
dirty repo after M5 build (I20140123-1600)
I think there as been progress.
Created attachment 243454 [details]
statis of build tree after RC2
This file shows things about the same as they were in previous check ...
maybe progress during Mars?
Created attachment 250689 [details] result of "git status" after I20150210-0800 build I have attached a check of dirt in working tree, taken after today's "production I-build". Seems some improvement (though, I've lost track) and did notice there is a new one .. I've opened bug 459584 to track it. (In reply to David Williams from comment #13) > Created attachment 250689 [details] > result of "git status" after I20150210-0800 build > > I have attached a check of dirt in working tree, taken after today's > "production I-build". > > Seems some improvement (though, I've lost track) and did notice there is a > new one .. I've opened bug 459584 to track it. Would you please share the state after latest I-build? I have my own builds and check against them but would rather have eclipse.org results for comparison too. Created attachment 251096 [details] dirt "report" after 20150224 I-build (In reply to Alexander Kurtakov from comment #14) > Would you please share the state after latest I-build? I have my own builds > and check against them but would rather have eclipse.org results for > comparison too. Sure. I've added attachment. Does it match what you see? I "get" these by running, in the "build area" (under /shared/eclipse/builds/4I/gitCache/eclipse.platform.releng.aggregator) the following sort of commands: git status > ~/temp/dirt<date>.txt git submodule foreach git status >> ~/temp/dirt<date>.txt If you agree that is "the right way", I will make a "dirt report" part of every build. Eventually, we'd want a "releng tests" to make sure we don't regress. (In reply to David Williams from comment #15) > Created attachment 251096 [details] > dirt "report" after 20150224 I-build > > (In reply to Alexander Kurtakov from comment #14) > > > Would you please share the state after latest I-build? I have my own builds > > and check against them but would rather have eclipse.org results for > > comparison too. > > Sure. I've added attachment. Does it match what you see? Yes, it's exactly the same with mine. > > I "get" these by running, in the "build area" (under > /shared/eclipse/builds/4I/gitCache/eclipse.platform.releng.aggregator) > the following sort of commands: > > git status > ~/temp/dirt<date>.txt > git submodule foreach git status >> ~/temp/dirt<date>.txt > > If you agree that is "the right way", I will make a "dirt report" part of > every build. Eventually, we'd want a "releng tests" to make sure we don't > regress. Having such a report would be nice. When you say releng test you mean only when we have all the dirt gone? Or one comparing against current status? (In reply to Alexander Kurtakov from comment #16) > Having such a report would be nice. When you say releng test you mean only > when we have all the dirt gone? Or one comparing against current status? Either. I've not really thought it through, but for such tests, we often simply look at "file size" of the report to be sure it has not "grown". And then as improvements are made, we reduce the hard coded "checked file size". Did I ever document the "dirty report". Created every build. For example, for M6, the report is at http://download.eclipse.org/eclipse/downloads/drops4/S-4.5M6-201503200800/buildlogs/dirtReport.txt To navigate there, from DL page, there is a link to logs for the current build and from that page, there is a link to Release engineering build logs and on that page, there is a link to dirtReport.txt There is also a "releng test" that fails with "regression" if the file get's larger ... and every now and then, I'll have to change the constant in the test on what the "maximum file size" should be. Currently set as 37179 bytes. I also wanted to point out one solution to "cleaning" mentioned in bug 461837 and in gerrit at https://git.eclipse.org/r/#/c/43539/1 In short, the idea being to do more "cleaning" from the pom ... less that ideal, I think ... but ... perhaps a tactical solution? (If anyone had time to work on it). All the "depends on" bugs show "fixed", but the "dirt report" shows we are not clean. Did we miss opening some initially? Or have things changed? Entering 'eclipse.jdt' Entering 'eclipse.jdt.core' ?? org.eclipse.jdt.core/jdtCompilerAdapter.jar Entering 'eclipse.jdt.core.binaries' Entering 'eclipse.jdt.debug' ?? org.eclipse.jdt.debug.tests/javadebugtests.jar ?? org.eclipse.jdt.debug/jdi.jar ?? org.eclipse.jdt.debug/jdimodel.jar Entering 'eclipse.jdt.ui' Entering 'eclipse.pde' Entering 'eclipse.pde.build' Entering 'eclipse.pde.ui' Entering 'eclipse.platform' Entering 'eclipse.platform.common' ?? bundles/org.eclipse.jdt.doc.isv/index/ ?? bundles/org.eclipse.jdt.doc.user/index/ ?? bundles/org.eclipse.pde.doc.user/index/ ?? bundles/org.eclipse.platform.doc.isv/index/ ?? bundles/org.eclipse.platform.doc.isv/samples/org.eclipse.compare.examples.xml/ ?? bundles/org.eclipse.platform.doc.isv/samples/org.eclipse.compare.examples/ ?? bundles/org.eclipse.platform.doc.isv/samples/org.eclipse.swt.examples.launcher/ ?? bundles/org.eclipse.platform.doc.isv/samples/org.eclipse.swt.examples.ole.win32/ ?? bundles/org.eclipse.platform.doc.isv/samples/org.eclipse.swt.examples.views/ ?? bundles/org.eclipse.platform.doc.isv/samples/org.eclipse.swt.examples/ ?? bundles/org.eclipse.platform.doc.isv/samples/org.eclipse.team.examples.filesystem/ ?? bundles/org.eclipse.platform.doc.isv/samples/org.eclipse.ui.examples.fieldassist/ ?? bundles/org.eclipse.platform.doc.isv/samples/org.eclipse.ui.examples.javaeditor/ ?? bundles/org.eclipse.platform.doc.isv/samples/org.eclipse.ui.examples.multipageeditor/ ?? bundles/org.eclipse.platform.doc.isv/samples/org.eclipse.ui.examples.propertysheet/ ?? bundles/org.eclipse.platform.doc.isv/samples/org.eclipse.ui.examples.readmetool/ ?? bundles/org.eclipse.platform.doc.isv/samples/org.eclipse.ui.examples.undo/ ?? bundles/org.eclipse.platform.doc.user/index/ Entering 'eclipse.platform.debug' Entering 'eclipse.platform.releng' Entering 'eclipse.platform.resources' Entering 'eclipse.platform.runtime' Entering 'eclipse.platform.swt' Entering 'eclipse.platform.swt.binaries' Entering 'eclipse.platform.team' Entering 'eclipse.platform.text' Entering 'eclipse.platform.ua' Entering 'eclipse.platform.ui' Entering 'eclipse.platform.ui.tools' Entering 'rt.equinox.binaries' Entering 'rt.equinox.bundles' Entering 'rt.equinox.framework' ?? features/org.eclipse.equinox.executable.feature/bin/cocoa/macosx/x86_64/Eclipse.app/Contents/MacOS/ ?? features/org.eclipse.equinox.executable.feature/bin/gtk/ ?? features/org.eclipse.equinox.executable.feature/bin/win32/ Entering 'rt.equinox.p2' (In reply to David Williams from comment #20) > All the "depends on" bugs show "fixed", but the "dirt report" shows we are > not clean. Did we miss opening some initially? Or have things changed? I'm not aware of any change. > (In reply to David Williams from comment #20) > All the "depends on" bugs show "fixed" I've opened 3 new bugs (and attached possible patches for .gitignore files) for the "dirt" I saw in my broken build: Bug 498986 - [releng] Dirty git status after build in eclipse.jdt.core Bug 498988 - [releng] Dirty git status after build in eclipse.jdt.debug Bug 498992 - [releng] Dirty git status after build in eclipse.platform.common *** Bug 498992 has been marked as a duplicate of this bug. *** Only Bug 490411 and Bug 498986 left, lets close this tracking bug and work on these directly. |