Bugzilla – Full Text Bug Listing
| Summary: | build server doesn't always pick up changes from git | ||
|---|---|---|---|
| Product: | Community | Reporter: | David Green <greensopinion> |
| Component: | CI-Jenkins | Assignee: | CI Admin Inbox <ci.admin-inbox> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | P3 | CC: | caniszczyk, d_a_carver, ian.skerrett, manfred, mcculls, milesparker, nathan, peter.kullmann, steffen.pingel, thomas, webmaster |
| Version: | unspecified | ||
| Target Milestone: | --- | ||
| Hardware: | All | ||
| OS: | All | ||
| Whiteboard: | |||
|
Description
David Green
(In reply to comment #0) > The MPC project uses Hudson's Git plug-in to poll the git repository for > changes and pull necessary changes to perform a new build. > > On several occasions the Hudson job has detected changes and kicked off a new > build. The git pull appears to complete successfully, however upon examination > of the completed build some changes were not pulled into the local git > repository on the build machine. Wiping the workspace and kicking off a new > build resolves the problem (temporarily). > > This problem is particularly troublesome because there's no indication of any > problem. > > The "MPC project job on > Hudson":https://build.eclipse.org/hudson/job/epp-mpc-nightly/ Hmm..may need to open a bug with the Hudson Git plugin, as that is what we use. An alternative is to add a Shell statement, that clears the workspace and reclones the git repository, using the git clone command. This sounds like an upstream issue with Hudson. (In reply to comment #2) > This sounds like an upstream issue with Hudson. I agree, however given the number of projects using Git at ecilpse.org (and the expectation that more will move to Git) we should continue to track the issue here, and if possible document/implement a workaround for all projects. (In reply to comment #3) > (In reply to comment #2) > > This sounds like an upstream issue with Hudson. > > I agree, however given the number of projects using Git at ecilpse.org (and the > expectation that more will move to Git) we should continue to track the issue > here, and if possible document/implement a workaround for all projects. Yep..also ideally, be a great way for Eclipse projects to contribute back to the Git plugin development for Hudson. Currently I think it requires git to be installed, when it probably should be using jGit's API. David quick question have you tried setting up your Git polling and builds using the recommended steps in Advanced section of the wiki page: http://wiki.hudson-ci.org/display/HUDSON/Git+Plugin You might also be able to do a git clean as part of the build process to make sure you are working with a clean workspace before building. I checked the clean workspace flag. It didn't seem to make any difference. It's been enabled for some time now. (In reply to comment #6) > I checked the clean workspace flag. It didn't seem to make any difference. It's > been enabled for some time now. Best bet is to open a bug with the Hudson git developers. The other alternative is we use jGit to start creating our own git plugin for Hudson. The current git plugin doesn't use jGit. For those that are interested, I've done a fork of the Git plugin for Hudson. http://github.com/kingargyle/Hudson-GIT-plugin One of the things I found after forking the current git plugin code, is that it was using jGit, but a very old version of jGit. If anybody is interested you can get the new code at: git://github.com/kingargyle/Hudson-GIT-plugin.git Branch: HUDSON-6408 I've contributed this small upgrade back to the maintainer of the plugin and we'll see how long it takes to get this change into the system. With the latest Hudson restart, I also updated the git plugin from version 0.8.2 to 0.8.3. Thanks for all of the advice/feedback on this issue. I'm still seeing the problem. Could it be that we've got a broken version of git on the build machine? It appears to be using /usr/local/bin/git (see https://build.eclipse.org/hudson/job/epp-mpc-nightly/56/consoleFull) we have git 1.6.5.1 installed. The latest version available from the git-scm download site is git 1.7.0.6 we were just hit by this problem again, see bug 314570 comment 5 for related discussion In case it helps, here's the console log showing Git commands: pre. Started by an SCM change ln -s 2010-05-28_19-38-11 /opt/users/hudsonbuild/.hudson/jobs/epp-mpc-nightly/builds/73 failed: -1 Building on master Checkout:workspace / /opt/users/hudsonbuild/.hudson/jobs/epp-mpc-nightly/workspace - hudson.remoting.LocalChannel@22a522a5 Using strategy: Default Last Built Revision: Revision a48a64ccfe95c233a14fae108e9c97fb07649a94 (origin/master) Checkout:workspace / /opt/users/hudsonbuild/.hudson/jobs/epp-mpc-nightly/workspace - hudson.remoting.LocalChannel@22a522a5 GitAPI created Fetching changes from the remote Git repository Fetching upstream changes from git://git.eclipse.org/gitroot/mpc/org.eclipse.epp.mpc [workspace] $ /usr/local/bin/git fetch -t git://git.eclipse.org/gitroot/mpc/org.eclipse.epp.mpc +refs/heads/*:refs/remotes/origin/* From git://git.eclipse.org/gitroot/mpc/org.eclipse.epp.mpc b016c20..d41cd14 master -> origin/master [workspace] $ /usr/local/bin/git ls-tree HEAD Seen branch in repository origin/master Commencing build of Revision d41cd143c83e90f21f2219c4581b906dafaf949e (origin/master) GitAPI created Checking out Revision d41cd143c83e90f21f2219c4581b906dafaf949e (origin/master) [workspace] $ /usr/local/bin/git checkout -f d41cd143c83e90f21f2219c4581b906dafaf949e [workspace] $ /usr/local/bin/git tag -a -f -m "Hudson Build #73" hudson-epp-mpc-nightly-73 Recording changes in branch origin/master [workspace] $ /usr/local/bin/git log --pretty=format:%H a48a64ccfe95c233a14fae108e9c97fb07649a94..d41cd143c83e90f21f2219c4581b906dafaf949e [workspace] $ /usr/local/bin/git log -M --summary --pretty=raw -n 1 d41cd143c83e90f21f2219c4581b906dafaf949e [workspace] $ /usr/local/bin/git diff-tree -M -r d41cd143c83e90f21f2219c4581b906dafaf949e [workspace] $ /usr/local/bin/git log -M --summary --pretty=raw -n 1 b016c2069dc7db76789695d1fcdaf7ae08d18121 [workspace] $ /usr/local/bin/git diff-tree -M -r b016c2069dc7db76789695d1fcdaf7ae08d18121 [workspace] $ /usr/local/bin/git log -M --summary --pretty=raw -n 1 c19c246102986cdee3c01ff66afd31ee78d759f5 [workspace] $ /usr/local/bin/git diff-tree -M -r c19c246102986cdee3c01ff66afd31ee78d759f5 Cleaning workspace [workspace] $ /usr/local/bin/git clean -fdx [org.eclipse.epp.mpc.releng] $ ant -file build.xml clean.all buckminster.perform If you are using the Git plugin, the best place to report this is to the Hudson issue tracking system. http://issues.hudson-ci.org/secure/Dashboard.jspa The alternative is to set SCM to none, and then add the appropriate Git commands to check out the code yourself. Note, that you loose the polling ability of Hudson by doing this. (In reply to comment #14) > In case it helps, here's the console log showing Git commands: > > pre. > Started by an SCM change > ln -s 2010-05-28_19-38-11 > /opt/users/hudsonbuild/.hudson/jobs/epp-mpc-nightly/builds/73 failed: -1 > Building on master > Checkout:workspace / > /opt/users/hudsonbuild/.hudson/jobs/epp-mpc-nightly/workspace - > hudson.remoting.LocalChannel@22a522a5 > Using strategy: Default > Last Built Revision: Revision a48a64ccfe95c233a14fae108e9c97fb07649a94 > (origin/master) > Checkout:workspace / > /opt/users/hudsonbuild/.hudson/jobs/epp-mpc-nightly/workspace - > hudson.remoting.LocalChannel@22a522a5 > GitAPI created > Fetching changes from the remote Git repository > Fetching upstream changes from > git://git.eclipse.org/gitroot/mpc/org.eclipse.epp.mpc > [workspace] $ /usr/local/bin/git fetch -t > git://git.eclipse.org/gitroot/mpc/org.eclipse.epp.mpc > +refs/heads/*:refs/remotes/origin/* > From git://git.eclipse.org/gitroot/mpc/org.eclipse.epp.mpc > b016c20..d41cd14 master -> origin/master > [workspace] $ /usr/local/bin/git ls-tree HEAD > Seen branch in repository origin/master > Commencing build of Revision d41cd143c83e90f21f2219c4581b906dafaf949e > (origin/master) > GitAPI created > Checking out Revision d41cd143c83e90f21f2219c4581b906dafaf949e (origin/master) > [workspace] $ /usr/local/bin/git checkout -f > d41cd143c83e90f21f2219c4581b906dafaf949e > [workspace] $ /usr/local/bin/git tag -a -f -m "Hudson Build #73" > hudson-epp-mpc-nightly-73 > Recording changes in branch origin/master > [workspace] $ /usr/local/bin/git log --pretty=format:%H > a48a64ccfe95c233a14fae108e9c97fb07649a94..d41cd143c83e90f21f2219c4581b906dafaf949e > [workspace] $ /usr/local/bin/git log -M --summary --pretty=raw -n 1 > d41cd143c83e90f21f2219c4581b906dafaf949e > [workspace] $ /usr/local/bin/git diff-tree -M -r > d41cd143c83e90f21f2219c4581b906dafaf949e > [workspace] $ /usr/local/bin/git log -M --summary --pretty=raw -n 1 > b016c2069dc7db76789695d1fcdaf7ae08d18121 > [workspace] $ /usr/local/bin/git diff-tree -M -r > b016c2069dc7db76789695d1fcdaf7ae08d18121 > [workspace] $ /usr/local/bin/git log -M --summary --pretty=raw -n 1 > c19c246102986cdee3c01ff66afd31ee78d759f5 > [workspace] $ /usr/local/bin/git diff-tree -M -r > c19c246102986cdee3c01ff66afd31ee78d759f5 > Cleaning workspace > [workspace] $ /usr/local/bin/git clean -fdx > [org.eclipse.epp.mpc.releng] $ ant -file build.xml clean.all > buckminster.perform I'm hitting this..at the worst possible time, naturally, i.e. the last day of the release train build. I see the changes, I check git web front end, everything looks good. Even the git workspace looks fine. But when the build itself occurs..yep, it's using the wrong artifacts. It typically seems to be trailing by one build iteration. David, were you using Buckminster when this came up? Miles, quite possibly we were using Buckminster. My workaround was to use the Hudson UI to delete the workspace and manually trigger a build. (In reply to comment #17) > Miles, quite possibly we were using Buckminster. My workaround was to use the > Hudson UI to delete the workspace and manually trigger a build. Yep, that's what I've ended up doing. It does sort of take the continuous out of continuous integration. ;) And I end up forgetting about it at the worst possible times.. I guess it isn't possible to make "wipe out workspace" an actual build step. For a better long-term solution I'm wondering what sort of artifacts might be hanging around that confuse Buckminster. IIRC I've inspected the contents of that repos at build time with the Buckminster import disabled and they've been correct, but I can't swear to it. (Is the Hudson bucky and/or git plugins hiding something somewhere else in the workspace?) The fact that this only occurs after a full build makes me think that there is something that Buckminster is doing to the repos that the SCM system doesn't like. And now I remember Thomas saying something at some point about Buckminster *not* doing an automated pull or fetch when changes occur. Could there be some aspect of that that is confounding Buckminster? Or perhaps Buckminster is setting itself back to the refid that it thinks it should be at from the original resolution.. What AMP is doing is cloning the entire repos and then pointing the rmap to those now local repos with the git defined repos as a fall-back. My thinking was that that would get around the issue of having to grab the whole repos just to check the change state and get the releng project, and then reusing the results of that. Are other people using a similar strategy, or are you materializing into a different git repos? If I wasn't deathly afraid to muck with anything right now, I'd try blowing away the automated repos after the SCM check or at least populating a different git repos location for the build itself. At least then I could see if there was a delta between the Hudson Git SCM provisionend repos and the Buckminster one. Miles, It was my understanding that the pull was done by Hudson and Buckminster just uses the master branch of the clone from it's rmap. Isn't that the case here? (In reply to comment #19) > Miles, It was my understanding that the pull was done by Hudson and Buckminster > just uses the master branch of the clone from it's rmap. Isn't that the case > here? That's what is specified, i.e. it's relative to [Hudson] workspace root. But the buckminster provider handles setting up the initial remote replication as well, right? That is, if I have a completely new local workspace and materialize form a cquery URL, Buckminster creates a new local repos and clones the contents. Presumably if that already exists, it is simply left as is..? (In reply to comment #19) > Miles, It was my understanding that the pull was done by Hudson and Buckminster > just uses the master branch of the clone from it's rmap. Isn't that the case > here? But in fact it is even simpler than that, because unless I have something messed up, buckminster shouldn't even be invoking a git provider as it will have first found the local provider at the repos location and used that. But now that I think about it, that might not be a smart or necessary thing to do if both providers point to the same file system location. When the Buckminster git provider senses that the clone is already in place (cloned by Hudson), then it will just use that clone. Using a local provider shouldn't be necessary. If you do however, then we can be certain that Buckminster isn't the culprit. The local provider will not perform clones or check-outs. There are currently 68 unresolved issues logged on the Hudson Git plug-in (4 critical and 51 major). Judging from http://issues.hudson-ci.org/browse/HUDSON-6408 the JGit in use is 0.7.1 and that particular bug is unassigned (again). This is a bad situation and it affects all Eclipse projects that has moved to Git. Is there anyone out there that can spend some time on this plug-in? (In reply to comment #22) > When the Buckminster git provider senses that the clone is already in place > (cloned by Hudson), then it will just use that clone. Using a local provider > shouldn't be necessary. If you do however, then we can be certain that > Buckminster isn't the culprit. The local provider will not perform clones or > check-outs. Yeah, I got rid of my local provider setup -- it was really a remnant from CVS/SVN days -- and so far so good. > There are currently 68 unresolved issues logged on the Hudson Git plug-in (4 > critical and 51 major). Judging from > http://issues.hudson-ci.org/browse/HUDSON-6408 the JGit in use is 0.7.1 and > that particular bug is unassigned (again). This is a bad situation and it > affects all Eclipse projects that has moved to Git. > > Is there anyone out there that can spend some time on this plug-in? All I can offer it to ask some other people. :) Is Jenkins side any better? I know that I posted a bug there on Buckminster plugin not supporting new version and KK responded to it. (In reply to comment #22) > When the Buckminster git provider senses that the clone is already in place > (cloned by Hudson), then it will just use that clone. Using a local provider > shouldn't be necessary. If you do however, then we can be certain that > Buckminster isn't the culprit. The local provider will not perform clones or > check-outs. > > There are currently 68 unresolved issues logged on the Hudson Git plug-in (4 > critical and 51 major). Judging from > http://issues.hudson-ci.org/browse/HUDSON-6408 the JGit in use is 0.7.1 and > that particular bug is unassigned (again). This is a bad situation and it > affects all Eclipse projects that has moved to Git. > > Is there anyone out there that can spend some time on this plug-in? FYI, Manfred went through and moved issues that had been assigned to people who weren't looking at the Hudson JIRA anymore back to unassigned. Committers are now picking up issues based on the number of votes, priority, etc. Nikita and Anton have been actively working on the git plugin recently (https://github.com/hudson-plugins/git-plugin/commits/master/) so if you need something fixed let them know on the dev list. PS. the current git plugin snapshot was recently updated to use JGit 1.0.0.201106090707-r: https://github.com/hudson-plugins/git-plugin/blob/master/pom.xml http://hudson-ci.org/hudson/job/git-plugin/ Thanks for finding out this problem. We are very much looking at anything that comes into our issue tracker as well as on the users and dev list. The git plugin is actively worked on and as Stuart said currently using a very new version of the jgit. Btw the Hudson community maintained plugins are here: https://github.com/hudson-plugins/ and we welcome any pull requests and fixes and bug reports. We just need to find out about it somehow. Please try the latest version of the plugin (2.0.1) and report back on our issue tracker or here. Does Webmaster need to do anything here? -M. From the lack of feedback I'll assume all is well here. Please reopen if there's something we need to fix. |