Bugzilla – Full Text Bug Listing
| Summary: | Create cron'able script for tagging CVS or SVN | ||
|---|---|---|---|
| Product: | z_Archived | Reporter: | Andrew Overholt <overholt> |
| Component: | Dash Athena | Assignee: | Common Build Inbox <dash.commonbuilder-inbox> |
| Status: | RESOLVED WONTFIX | QA Contact: | |
| Severity: | normal | ||
| Priority: | P3 | CC: | bernhard.merkle, contact, dash.commonbuilder-inbox, d_a_carver, jacek.pospychala, linux.distros-inbox, nboldt, spektom, wayne.beaton |
| Version: | unspecified | ||
| Target Milestone: | --- | ||
| Hardware: | PC | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Bug Depends on: | 252041, 276662, 285074, 292157 | ||
| Bug Blocks: | |||
|
Description
Andrew Overholt
Moving to Athena. This is a difficult issue due to the userid who runs the build. We may not be able to solve it. Renaming. Due to limitations in Hudson, Hudson cannot change CVS or SVN (ie., by tagging). However, we can implement something for scheduled tagging (eg., weekly) so that maps are kept up to date. We already have the option to build from checked-out sources in the workspace or to override map tags w/ some other value (for CVS, anyway; untested w/ SVN). So all we need here is to, for example, spin nightlies every day from HEAD/trunk, and a weekly I build from the tags in the map. As long as the autotagger runs right before the I build is scheduled, that build will include the latest tagged sources. This is how GEF is currently built using the Common Modeling Build; we would need to port that system into Athena and extend it for use w/ SVN (and eventually Git). Here's the current implementation (a shell script, of course). http://dev.eclipse.org/viewcvs/index.cgi/releng-common/tools/org.eclipse.releng.tools.tagandrelease/?root=Modeling_Project We could look at rewriting this as Ant so it's more easily adopted, as it need not be run on build.eclipse.org (could be done on users' own systems). See also http://wiki.eclipse.org/Modeling_Project_Releng/Releasing/Scheduled_Releases https://bugs.eclipse.org/bugs/show_bug.cgi?id=252041#c2 (In reply to comment #3) > See also > > http://wiki.eclipse.org/Modeling_Project_Releng/Releasing/Scheduled_Releases > https://bugs.eclipse.org/bugs/show_bug.cgi?id=252041#c2 > The hudson bug for this is: https://hudson.dev.java.net/issues/show_bug.cgi?id=1930 Ideally the best solution is to pitch in and help the hudson developers track down the issue with SVN tagging. Or, you can add the ability to Tag the build yourself at the end of the job using the appropriate svn commands or ant tasks. (In reply to comment #4) > Ideally the best solution is to pitch in and help the hudson developers track > down the issue with SVN tagging. Or, you can add the ability to Tag the build > yourself at the end of the job using the appropriate svn commands or ant tasks. Well, the issue here is that the build runs as hudsonbuild, and that "person" isn't a committer. So, no, the build cannot pre- or post- tag sources. That has to be done *by a user*, by hand in Eclipse, using the releng tools in Eclipse, or via crontab on a schedule. (In reply to comment #5) > (In reply to comment #4) > > Ideally the best solution is to pitch in and help the hudson developers track > > down the issue with SVN tagging. Or, you can add the ability to Tag the build > > yourself at the end of the job using the appropriate svn commands or ant tasks. > > Well, the issue here is that the build runs as hudsonbuild, and that "person" > isn't a committer. So, no, the build cannot pre- or post- tag sources. That has > to be done *by a user*, by hand in Eclipse, using the releng tools in Eclipse, > or via crontab on a schedule. > > > I think we have a couple of issues here: 1. Map builds. How to Tag them. 2. Hudson checked out/workspace builds. (i.e. hudson checks everything out). 3. Ant only builds. (i.e Ant handles all check ins and checkouts). For option 3, I would just rely on a specific user id that had the necessary rights to check out and tag files (i.e. a committer/releng). Use either the svn commands directly, or use the svn ant task commands to tag the files after the build completes. Option 2, would be a modificaiton of option 3, except you let Hudson check everything out, and then you use the tagging step in option 3, to tag the files. Option 1 is the more painful option, and why I hate Map files. This depends on eclipse support for SVN tasks, which I haven't looked into so don't know what tagging support there is for SVN from eclipse ant tasks. None of these options requires a Cron job to do the work. Ideally the hudsonbuilder should have the same rights as a committer on the various projects. Personally this is a limitation imposed by eclipse admins due to the way Map files have been used in the past to control everything. > 1. Map builds. How to Tag them. See comment 2. Already have a headless automated implementation of this for CVS. For SVN I'll need something new, but simpler. Can just query Hudson for SVN revision numbers and if # has changed since last good build, tag repo accordingly. > 2. Hudson checked out/workspace builds. (i.e. hudson checks everything out). Still needs that hudsonbuild can tag svn or cvs (or, later, git). > 3. Ant only builds. (i.e Ant handles all check ins and checkouts). Out of scope. Dash Athena does PDE, not pure Ant. Non-starter. > None of these options requires a Cron job to do the work. Option 1 does because nickb and hudsonbuild are not the same user, so one has to "listen" to changes posted by the other, and the best way to do that is to have a crontab check at intervals for changes to the repo. Unless, of course, the "hudson can't tag because it's not a committer" limit is lifted. You're on the Council - raise this issue and get the rule changed. (Good luck with that.) -- However... the more I work with CVS amd SVN, the more I realize that tagging is ONLY useful when you want to build from a specific version of sources. For a lot of projects, tagging once a milestone seems sufficient to project that air of "reproduceability" - even if no one EVER asks for a build to be respun from a specific tag or map. (I've done it to prove Athena works as well as older Modeling system, but that's it.) So, if that's the case, then once-a-milestone use of the releng.tools plugin against your workspace is sufficient to get an updated map file. (In reply to comment #7) > Unless, of course, the "hudson can't tag because it's not a committer" limit is > lifted. You're on the Council - raise this issue and get the rule changed. > (Good luck with that.) > I've openend an issue with the Architecture Council on this. > -- > > However... the more I work with CVS amd SVN, the more I realize that tagging is > ONLY useful when you want to build from a specific version of sources. For a > lot of projects, tagging once a milestone seems sufficient to project that air > of "reproduceability" - even if no one EVER asks for a build to be respun from > a specific tag or map. (I've done it to prove Athena works as well as older > Modeling system, but that's it.) > > So, if that's the case, then once-a-milestone use of the releng.tools plugin > against your workspace is sufficient to get an updated map file. Builds should be tagged every time, not prior tagging done. The main reason is that it allows people to go back and get the sources that were in that particular build that failed. With the map tagging now, it's not really the same thing, as the map files can change, and that then means going back through a map file history to get the particular information. If you you tag before hand or after the build completes/fails, you have an easier way to get the version that was failing, and still allow others to continue development. Once the change is found it can be merged back in. Basic Continuous Integration practices outlined in Martin Fowler's article. http://martinfowler.com/articles/continuousIntegration.html IMHO there are two ways to run a build: a) from tagged sources (using map to fetch specific versions of plugins/features) b) from HEAD / trunk / branch / tag Usecases? (a) - weekly I build, stable milestones. (b) - nightly N builds pulled from the latest in CVS/SVN. I've had requests over the years for BOTH scenarios, so that you can get immediate CI feedback via an N build (case b), but also, once you're satisfied that the changes in the N don't break you, you can tag (manually, releng.tools, or scheduled script) and run a reproduceable build from tagged map(s) - case a. Best of both worlds. (In reply to comment #9) > IMHO there are two ways to run a build: > > a) from tagged sources (using map to fetch specific versions of > plugins/features) > > b) from HEAD / trunk / branch / tag > > Usecases? > > (a) - weekly I build, stable milestones. > (b) - nightly N builds pulled from the latest in CVS/SVN. > > I've had requests over the years for BOTH scenarios, so that you can get > immediate CI feedback via an N build (case b), but also, once you're satisfied > that the changes in the N don't break you, you can tag (manually, releng.tools, > or scheduled script) and run a reproduceable build from tagged map(s) - case a. > > Best of both worlds. Yep, and All of the above are different stages in a Continuous Integration build process. why do you need a tag in _svn_ ? the revision number is global per project and hence serves technically like a tag. (in svn is only a pointer to that specific revision number, nothing more). (In reply to comment #11) > why do you need a tag in _svn_ ? > the revision number is global per project and hence serves technically like a > tag. (in svn is only a pointer to that specific revision number, nothing more). A valid point. But can you reference a revision # in a map file? Here's a map using branches; using tags would be the same basic syntax. http://dev.eclipse.org/svnroot/technology/org.eclipse.linuxtools/releng/branches/0.4/org.eclipse.linuxtools.releng/maps/linuxtools.map How do you reference the revision of a file or folder via URL? If there's no way to do so, then we need tagging so that we can point at a specific revision rather than the latest from that branch. Seems that there is an available syntax for using revisions within a branch: http://wiki.eclipse.org/Subversive_PDE_Fetch So tagging is not needed (if this works); instead all that's needed is to put the revision number(s) directly into the map file(s) so as to lock down a "tagged" version of that plugin or feature. Thus, once a week, a script could run against all the plugin/feature folders in a given branch of an SVN tree, and collect all the revision numbers for those folders using svn info | grep Revision | sed 's/Revision: //g' Then the map file could be updated w/ those revision #s for a more stable map. Possible pitfalls: s) pde-svn-plugin may not support the ":revsision" syntax; may have to switch to Subversive SVN Fetch Factory (bug 276662) b) script should be Ant, not shell; is there a way to run `svn info` without doing an <exec> ? Grep and sed can be done using ant-contrib. c) need to verify that these revision numbers (or indeed the branch or tag) can be ignored using a commandline override (bug 292157) (In reply to comment #13) > Seems that there is an available syntax for using revisions within a branch: > > http://wiki.eclipse.org/Subversive_PDE_Fetch > > So tagging is not needed (if this works); instead all that's needed is to put > the revision number(s) directly into the map file(s) so as to lock down a > "tagged" version of that plugin or feature. > > Thus, once a week, a script could run against all the plugin/feature folders in > a given branch of an SVN tree, and collect all the revision numbers for those > folders using > > svn info | grep Revision | sed 's/Revision: //g' > > Then the map file could be updated w/ those revision #s for a more stable map. > > Possible pitfalls: > > s) pde-svn-plugin may not support the ":revsision" syntax; may have to switch > to Subversive SVN Fetch Factory (bug 276662) > b) script should be Ant, not shell; is there a way to run `svn info` without > doing an <exec> ? Grep and sed can be done using ant-contrib. Could also create a Java ANT task to do this and access the various SVN java apis that are available. > c) need to verify that these revision numbers (or indeed the branch or tag) can > be ignored using a commandline override (bug 292157) My personal preference is for a Tag still...it's more human readable. Revisions are really for machines. > How do you reference the revision of a file or folder via URL? If there's no > way to do so, then we need tagging so that we can point at a specific revision > rather than the latest from that branch. you can if you put the /!svn/bc/<revisionnumber>/ into the svn URL, e.g. http://svn-merge-fairy.googlecode.com/svn/!svn/bc/4/ get revision 4. really usefull (In reply to comment #14) > My personal preference is for a Tag still...it's more human readable. > Revisions are really for machines. I could imagine that you result in LOTs of tags over time. Here in information as we do it in our CI system: - we do a regular CI build via a svn time trigger (e.g. 10 minutes after previous checking). This only compiles the sources and does not check for other things. maybe it also runs unit tests. Essentially it shows if someone breaks the build and/or the unit tests. - we have a nightly build where we do more expensive things like: --running findbugs --running stylechecks --running API checks --running architectural checks --running extensive and expensive (long) unit tests --running automated GUI tests we never tag the regular ("10 minute build") nor the nightly build because we have the unique revision number (per repository in svn) and thats enough. (remember if you are using external in svn you have to have the revision number for the external repositories also, but thats a minor detail).. That revision number gets into the executable and is also in the buildserver logs. now when we go for release candidates (as the software gets more stable), we select a certain "good build" via the revision number and also give them a tag. (e.g. alpha_1). But tagging each build is a bit overkill IMO. We had good experience with this approach and i think also other people use this approach and it may be a best practice in the meantime. just me 2 euro cents ;-) Berni. > (e.g. alpha_1). But tagging each build is a bit overkill IMO. > just me 2 euro cents ;-) > Berni. +1. I don't think you ever need to tag EVERY build, especially if your sources are in SVN. But having an automated way to tag weekly or monthly, including updating the maps, would certainly be of value IMHO. See also these blogs on the subject of when/how/where to tag before/after builds: http://divby0.blogspot.com/2009/12/feeding-right-sources-to-your-builds.html http://intellectualcramps.blogspot.com/2009/12/tagbuild-buildtaghow-about-both.html Athena's design goal is to make simple things simple, and complex things possible. I want to give you enough rope to hang yourself (or save yourself), but never declare exactly how much rope to use (or when/where). Every build, every team, every situation is different, so there's no ONE answer that will work for everyone. (In reply to comment #17) > > (e.g. alpha_1). But tagging each build is a bit overkill IMO. > > just me 2 euro cents ;-) > > Berni. > > +1. I don't think you ever need to tag EVERY build, especially if your sources > are in SVN. But having an automated way to tag weekly or monthly, including > updating the maps, would certainly be of value IMHO. I agree, tagging every build is overkill. The way I use Hudson at work is to setup a scheduled build for Milestone and Integration builds. Only these scheduled builds are tagged after the build completes. I'll tag other builds as necessary manually. WONTFIX; Athena has been terminated. |