Bugzilla – Full Text Bug Listing
| Summary: | Publish Now plugin for Hudson backed by buildpublisher genie | ||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | z_Archived | Reporter: | Nick Boldt <nboldt> | ||||||||||||
| Component: | Dash Athena | Assignee: | Project Inbox <athena.publish-inbox> | ||||||||||||
| Status: | RESOLVED WONTFIX | QA Contact: | |||||||||||||
| Severity: | normal | ||||||||||||||
| Priority: | P3 | CC: | ahunter.eclipse, ashw.kumar, denis.roy, d_a_carver, ed, gunnar, kaloyan, konstantin, sbouchet, stepper, wayne.beaton, webmaster | ||||||||||||
| Version: | unspecified | Keywords: | helpwanted | ||||||||||||
| Target Milestone: | --- | ||||||||||||||
| Hardware: | PC | ||||||||||||||
| OS: | Linux | ||||||||||||||
| Whiteboard: | |||||||||||||||
| Attachments: |
|
||||||||||||||
Sounds like we need to hack one of the existing Hudson promotion plugins a bit. http://wiki.hudson-ci.org/display/HUDSON/Promoted+Builds+Plugin http://wiki.hudson-ci.org/display/HUDSON/Promoted+Builds+Simple+Plugin http://wiki.hudson-ci.org/display/HUDSON/Plugins#Plugins-Artifactuploaders Talked to Denis about this at lunch today. Some thoughts: a) Publish Now should also include stringent rules for cleaning up old builds; part of the process of publishing a new build must be to purge old builds before adding a new one to the pile b) explicit documentation must be in place to explain that clicking "Publish Now" will enable Hudson to publish on your behalf (then fix permissions and ownership). If you have a problem with a server doing this for you (ie., around auditing and backups) then you should not use this tool. c) An audit trail is required. Ideally, the plugin would push a status update into the hudson build log [1], or something similar. [1] https://build.eclipse.org/hudson/builds d) hudsonBuild should be able chgrp the files to the correct group for that project so that anyone in that group (project/component/subproject) can manipulate those files. The group cannot simply be "common" so that WTP committers cannot accidentally delete Modeling files - the current group permission scheme must be respected. Any progress update? Getting this implemented would be of tremendous benefit for lowering the entry bar for new projects, especially smaller ones. In terms of design, I'd like to add that this plugin should support running it as a post-build step. Some projects that practice true continuous integration have no use for a separate publish step in the development process. It should be possible to push a build without manual intervention if it has passed unit tests. One more vote from me for having this implemented. (In reply to comment #4) > One more vote from me for having this implemented. BTW, why the vote for this bug is unavailable ? Well somebody is going to have to write the plugin for this. Voting for it is nice, but if people really want it, somebody is going to have to write it. Unfortunately higher priorities are preventing me from taking a stab at this. But it isn't going to write itself. Well somebody is going to have to write the plugin for this. Voting for it is nice, but if people really want it, somebody is going to have to write it. Unfortunately higher priorities are preventing me from taking a stab at this. But it isn't going to write itself. I talked to a colleague who has experience with Hudson, including development of plugins. He is willing to help us for any Hudson development needed, if we are able to tell him what exactly we need. We actually sit down together and evaluated the plugins suggested in comment 1. We found out that the Promoted Builds Plugin [1] and the SCP Plugin [2] work very well together and can provide us with a kind of solution without any need of additional coding - just with configuration. [1] http://wiki.hudson-ci.org/display/HUDSON/Promoted+Builds+Plugin [2] http://wiki.hudson-ci.org/display/HUDSON/SCP+plugin The SCP Plugin contributes a SCP publisher to Hudson. The webmaster can configure "SCP repository hosts" in the Hudson configuration. This should be the SCP settings for the download.eclipse.org server. The Promoted Builds Plugin then enables Hudson users to configure their Jobs with "promotion logic". A build can be promoted automatically if meets specific criteria (like successful JUnits) or manually (by hitting a button). More than one promotion steps can be configured. Now, here is the integration with the SCP Plugin - a promotion step can execute publishing the build artifacts to a SCP site - the one configured by the webmaster in the Hudson configuration. Another nice thing is that there can be configured more than one Promotion types. So, for one Job we can have promotion for night build, promotion for milestone, etc. The plugin has also good security integration. It contributes new security role - "Promote". So, only users assigned to this security role can execute manual promotions. I think this is a very good start. However, I see one problem. The SCP plugin provides configuration only on "Hudson configuration" level. There is no "per Job" configuration. Therefore, we would need the webmaster to configure a superuser who has unrestricted access to download.eclipse.org. This way we need to trust users to correctly configure the destination folder of the SCP Publish promotion step in their Jobs. Otherwise, they can mess someone else update site. I will attach some screenshots that will give a visual explanation to the above. Created attachment 178444 [details]
1.SCP-Configuration.png
Here is the "SCP repository hosts" configuration available in Hudson Configuration. We need the webmaster to configure a superuser that can access download.eclipse.org.
Created attachment 178445 [details]
2.Configuration_of_promotion_processes.png
Here is how promotion logic can be configured in Jobs. One of the available Actions is publishing build artifacts to an SCP site.
More than one "Promotion processes" can be configured.
Created attachment 178446 [details]
3.AccessControl.png
Security integration. New security role is introduced - Promote.
Created attachment 178447 [details]
4.Promoted_NightlyAndMileStone.png
A view displaying promotions in a Project/Job.
Note the Promotion Status link on the left side.
Created attachment 178448 [details]
5.PromotedBuildsList.png
Promoted builds a decorated.
Hi, for my point of view, this is exactly what i need, that is a button i can clic manaully to promote the specified build to a remote server. +1 to that solution. > The SCP plugin > provides configuration only on "Hudson configuration" level. There is no "per > Job" configuration. Therefore, we would need the webmaster to configure a > superuser who has unrestricted access to download.eclipse.org. All the Hudson servers mount the download.eclipse.org networked file system, so publishing a build is simply a matter of copying files from the working directory. We don't need to use scp since everything is local. With that being said, comment 3 outlines the issues I have with this solution, or with a "superuser who has unrestricted access to download.eclipse.org". Here there are, in order of how importantly I rank them 1. Allowing the Hudson Build account, or any other super account, to have unrestricted access to download.eclipse.org (or even parts of it) is problematic in that there is no access control any more. One project could accidentally 'mess up' the downloads area of other projects in interesting ways, and tracking such actions would be nearly impossible. -> the proposed solution here is to: - enable an audit log, so that we can see what file operations were done by which job - we only allow the Hudson user to write to download directories of projects that have opted in 2. The promote mechanism needs to incorporate a maintenance mechanism, otherwise Gigabytes upon Gigabytes will be dumped into download.eclipse.org and left there forever. WONTFIX; Athena has been terminated. |
It's been suggested (by a number of people) that Athena (and everything running on build.eclipse.org's Hudson) can't be considered complete until there's web UI available to do a publish of a build. Today, the story for publishing requires a cron job or manual shell script invocation because Hudson (hudsonBuild) cannot put files into /home/data/httpd/download.eclipse.org/. By the same token, no one can sign jars except the jarsigner genie - and this works fine because we can all request that the genie do the work for us, then collect the results. Can we not use a similar message queueing mechanism here? I propose the following: a) a hudson plugin for "Promote Now" -> knows who is pushing the button -> asks what promote.properties file to use (eg., for Nightly or Integration, to add/replace, to update Helios or not) -> drops a lockfile into the buildpublisher genie's queue -> file contains: folder path to the build to publish userid to use to do the promote path to properties file b) new genie that: -> watches for files dropped into a folder on build.eclipse.org -> sudo's to the userid specified -> runs a generic promote script using the specified promote.properties -> when promote is done, lockfile gets deleted I can assist with writing the promote script (it's essentially already there, just may need some tweaks), but someone will need to look at -> writing the hudson plugin (me?, Dave C?) -> setting up the new genie user on build.eclipse (Denis R?) -> creating the genie's crontab script to watch & respond (I can write it but I can't test it - need Denis for that)