Community
Participate
Working Groups
The user ids that are using by Eclipse to check out and build code, do not allow one of the best practices from a Continuous Integration standpoint to occur. The HudsonBuilder id does not have write access rights to the repositories that it checks code out from. This makes it impossible for Hudson to Tag the builds either pre-build or post build. The same is true for when using Cruisecontrol, or other continuous integration servers. While I realise that there is a legacy build process in place, I think in general from a release engineering standpoint we need to revisit why we don't allow write access rights by the build machine user id. It is causing unnecessary hoop jumping to have to occur by the various build processes. +++ This bug was initially created as a clone of Bug #264713 +++ We should tag SVN each time we do a build so that it's reproducible. This will also alleviate our rather hairy release engineering procedures.
There is only one userid used to checkout and build code in Hudson: hudsonbuild@build.eclipse.org. This id would need to be authorized as a committer in all projects, so that both tagging and map file updates (using that new tag) can be done in an automated fashion. IMHO we should tag before a build, but ONLY if the sources have changed since the last build (https://bugs.eclipse.org/bugs/show_bug.cgi?id=264713#c2). That way you're not introducing clutter into the repository.
I sympathize, but you do know that build IDs can not literally be given write access to CVS, right? That'd hose up the strict IP tracking requirements that Eclipse has. That's not to say there's not some other technical solution, though. Perhaps to "notify" a user with write access who could then tag what was in a Hudson directory? Perhaps some "indirect" tagging ... where there'd be a way to track versions built with by some other means than CVS commits? I supposed maps could be tagged first (by a committer) and then that tagged version built? Just ideas.
This needs to be addressed very carefully. IMHO such a generic account would need really limited permissions. For example, the build should be allowed to create new tags but never modify existing tags or even code. I remember situations where someone committed code (configuration changes) from the build machine using the build account by accident. If you recognize this weeks later it will be almost impossible to trace it down.
I agree that tagging projects built by Hudson is problematic. However, I don't know that it's kosher from a security standpoint for the hudson user have rights to tag all the projects. Our project tags the maps and builder for each integration or maintenance build we run. This ensures that the build is reproducible. There are occasions where the maps don't change, but we do fix a bug in the builder, so we want to ensure that this build can be reproduced. There are hudson plugins to tag SVN and CVS http://wiki.hudson-ci.org/display/HUDSON/Subversion+Tagging+Plugin http://wiki.hudson-ci.org/display/HUDSON/CVS+Tagging+Plugin but I don't see that they use ssh keys or similar to tag as a different user.
(In reply to comment #3) > This needs to be addressed very carefully. IMHO such a generic account would > need really limited permissions. For example, the build should be allowed to > create new tags but never modify existing tags or even code. I remember > situations where someone committed code (configuration changes) from the build > machine using the build account by accident. If you recognize this weeks later > it will be almost impossible to trace it down. While I understand and appreciate the concern. I've been running builds this way for the last 5 years, with no particular problems. It does mean that you have to keep an eye on the builds to make sure they are still good. If you have good Continuous Integration practices on your team, they should notice these issues with Build Failures, or when somebody goes to bring in code. Configuration changes and items generated by the Build shouldn't be in directories that are under version control. That is what cvs ignore and svn ignore is for, to exclude items from version control so they aren't checked in. Implementing good Releng and Build practices is a key to this, not limiting the build id rights. That just masks the problem instead of addressing it. (In reply to comment #2) > I sympathize, but you do know that build IDs can not literally be given write > access to CVS, right? That'd hose up the strict IP tracking requirements that > Eclipse has. One could filter out the build ids from the tracking requirements. Do we have a link to the requirement where this is documented? > > That's not to say there's not some other technical solution, though. > > Perhaps to "notify" a user with write access who could then tag what was in a > Hudson directory? that is the current hoop that has to be jumped through, and it can lead to human error. Best to let the build process do the tagging. > > Perhaps some "indirect" tagging ... where there'd be a way to track versions > built with by some other means than CVS commits? > > I supposed maps could be tagged first (by a committer) and then that tagged > version built? What about the situation where the build occurs directly from head, with out Map files. Not every project uses Map files now, several are building directly from head. Following recommendations from Continuous Integration. In order to tag you need either to have the build tag the files that were checked out by the Hudson user, or better just let Hudson tag them. Easiest solution is to lift the restriction that is currently in place, but to do that we need to understand why ( I know IP tracking has been mentioned), but we are talking about Tagging files, not checking in new code. We also need to promote good releng practices (i.e. cvs ignore/svn ignore and not to check in code during a build to the repository.)
(In reply to comment #5) > Implementing good Releng and Build practices is a key to this, not limiting the > build id rights. That just masks the problem instead of addressing it. I'm not so sure about this statement. See below. > What about the situation where the build occurs directly from head, with out > Map files. Not every project uses Map files now, several are building directly > from head. Following recommendations from Continuous Integration. It might be a recommendation from CI. However, I don't think that building from head is a good practice for software release processes in general. One might produce regular builds from a branch. I also know that CI should kick in building whenever someone commits code. But do we really need to reproduce every CI build? I say no. Having said that, I do see a need for tagging in some situations (for example, as mentioned in comment #4). However, those situation do not require full write permissions to a repository. Really just "add tags" permission is totally sufficient. This can be implemented in SVN today. But as far as I remember, the SVN implementation at Eclipse.org does not support this and would need some modification.
Tagging SVN w/o the map being updated too is pretty pointless, unless you're going to tag everything and build from tht tag (sorta like building from a CVS branch) I still think the set up we have for GEF, GMF, and PDT today (tagging every week by script & running a build from those tags, building from HEAD if bleeding edge is desired). Here's how that works: https://bugs.eclipse.org/bugs/show_bug.cgi?id=264713#c9 --- > What about the situation where the build occurs directly from head, with out Map files. Not every project uses Map files now, several are building directly from head. Dave, who in their right mind does a PDE build w/o map files? Seriously, that's just bad mojo. Who does this? I can see temporarily overriding the tags in the map w/ a branch name, but building w/o a map? No thanks.
(In reply to comment #7) > > Dave, who in their right mind does a PDE build w/o map files? Seriously, that's > just bad mojo. Who does this? I can see temporarily overriding the tags in the > map w/ a branch name, but building w/o a map? No thanks. I'll refer to: http://martinfowler.com/articles/continuousIntegration.html That has the reasoning for building with out maps. In some ways we are getting off topic with the later, there are reasons, and I do these types of builds not with PDE build but with a Ant4Eclipse build script, and it works well. Map files are just Tagged builds, that is all they really are, just says which tags to get. (In reply to comment #6) > > What about the situation where the build occurs directly from head, with out > > Map files. Not every project uses Map files now, several are building directly > > from head. Following recommendations from Continuous Integration. There are different types of builds in Continuous Integration. So we will agree to disagree as I don't want to get this too far off topic. > Having said that, I do see a need for tagging in some situations (for example, > as mentioned in comment #4). However, those situation do not require full write > permissions to a repository. Really just "add tags" permission is totally > sufficient. This can be implemented in SVN today. But as far as I remember, the > SVN implementation at Eclipse.org does not support this and would need some > modification. Having SVN/CVS tag rights would fit the bill for this particular issue, and address the problem that is occurring. Mainly needing to jump through extra hoops to get a build tagged so that it can be rechecked out later if it has failed or we need to go back to the last good build's source.
(In reply to comment #8) > Map files are just Tagged builds, that is all they really are, > just says which tags to get. Well, a *simple* map file [0] (where all the tags are the same) is the same as a tagged build, sure. [0]http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.atf/components/releng/org.eclipse.atf.releng/maps/atf.map?root=Tools_Project&view=markup But without a map you can't have plugin A from tag X and feature B from tag Y. If you want tagging at the IU level, you need a map [1]. Then you can tag that map so that ITS iterations are reproduceable too. [1]http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.gef/releng/org.eclipse.gef.athena.releng/maps/gef.map?revision=1.1&root=Tools_Project&view=markup And while this seems increasingly off topic, I think it's germane to the conversation because I'm not convinced that something fine-grained like GEF's map (or indeed the Platform, or WTP) ought to be left to a server process which can't be easily turned off by the project lead or the committers should something unplanned happen. Couple that with the security issues of having a bot committing tags and changes into CVS/SVN, and I wonder... is this a non-starter?
Now we are talking about stagged builds and which Hudson already handles. Maps are irrelevant, it's all about dependencies, and maps are one way to handle that. As far as tags, having a separate job that tags is an unnecessary step, and doesn't that just go around the IP issue that David Williams raised? Personally, I think just letting Hudson/Cruise/Luntbuild just do the tagging is a good thing. It's a best practice, as it removes the human error that can and will occur from a manual process. That is probably one of the reasons the releng plugin was developed, to remove the human equation as much as possible from tagging the maps.
(In reply to comment #7) > Dave, who in their right mind does a PDE build w/o map files? Seriously, that's > just bad mojo. Who does this? I can see temporarily overriding the tags in the > map w/ a branch name, but building w/o a map? No thanks. Please don't offend people writing unit tests and doing CI :) Dave's recent cramp hits the nail on the head (http://intellectualcramps.blogspot.com/2009/07/build-types-cramps.html). As someone who practices CI at work, and writes commercial(ThoughtWorks Cruise) and open source(CruiseControl and CruiseControl.Rb) tools that do CI, and open source testing tools (http://eclipse.org/swtbot, http://openqa.org/frankenstein, http://seleniumhq.org/) heres my 2 cents: We do build from trunk all the time. And we build from branches -- all the time. 1. The reason for moving to a CI system, and please correct me if I'm wrong -- is to provide quick feedback about changes. Just doing a build on every checkin does not imply CI. Without unit/integration/functional testing would mean testing the (ahem) compiler. A lot of eclipse projects test the compiler. Such projects are better off running nightly or weekly builds for the testers to provide feedback. 2. That said it is still essential that it should be possible to re-run historical builds for a variety of reasons and a possibility to tag builds once a day (or whatever is preferred by projects). Seriously map files are just a way to work around CVS issue it does not support directory versioning, moving to a new generation VCS SVN and you'll see life without map files. I'd rather that we discuss this point elsewhere. This is essential for point (1) which Dave mentions in his blog, it is the map files that should *ideally* be version controlled and tags seems like the only way to do it given the legacy that is CVS, and therefore most projects at eclipse.
(In reply to comment #8) > Having SVN/CVS tag rights would fit the bill for this particular issue, and > address the problem that is occurring. That's good to hear. I think we've reached consensus here. I adapted the summary to reflect the consensus. Hope you don't mind. old summary: Build User IDs should have Commit Rights to Projects
(In reply to comment #10) > As far as tags, having a separate job that tags is an unnecessary step, and > doesn't that just go around the IP issue that David Williams raised? I should clarify that "running a cronjob as nickb to tag&release GEF once a week" is not the same as "running a Hudson process to tag as hudsonbuild". In the first case, it's user-controlled and happens based on a user's id, so those source tagging operations (and changes/commits/tags of map files) happen as an actual committer. There's accountability and ownership. One might argue that a hands-off process done as a committer is just as IP-risky as a hands-off process done as a generic userid, but I'll let the Councils and the IP lawyers figure that one out. Oh, and in my mind, if you're going to tag sources you also need to update the maps that reference those sources to record the new mix of tags (see the GEF example in comment 9). This change requires a commit operation as well as a tag operation, so the title of this bug is no longer correct, IMHO.
(In reply to comment #13 > Oh, and in my mind, if you're going to tag sources you also need to update the > maps that reference those sources to record the new mix of tags (see the GEF > example in comment 9). This change requires a commit operation as well as a tag > operation, so the title of this bug is no longer correct, IMHO. I'll have to disagree with the latter (i.e. you need to update the tags in the map as well). Depends on how you do it. 1. Maps tags are done, and point to particular versions to build. 2. The map is checked out, hudson tags the map file for the build number. 3. Build completes or fails. If somebody wants to get the source for the build that just occcurred and their project is relying on Map files, then all they need to do is identify the Map file tagged for that particular build. The map already contains tags to the code that was used itself. Now if no Map files are being used, and you build directly from head, then yes, all code involved in the build and used should be tagged so you can get it out at a later time. Depends what your build methodology when something gets tagged.
Resolving this as won't fix as this really comes down to a philosophical opinion on how builds should be done.
