Bugzilla – Full Text Bug Listing
| Summary: | Define and implement the automated promotion and publication of update-sites | ||
|---|---|---|---|
| Product: | [Modeling] Sirius | Reporter: | Pierre-Charles David <pierre-charles.david> |
| Component: | Releng | Assignee: | Pierre-Charles David <pierre-charles.david> |
| Status: | CLOSED FIXED | QA Contact: | Pierre-Charles David <pierre-charles.david> |
| Severity: | normal | ||
| Priority: | P4 | CC: | cedric.brun, laurent.redor, mikael.barbero |
| Version: | unspecified | Keywords: | triaged |
| Target Milestone: | 1.0.0M6 | ||
| Hardware: | All | ||
| OS: | All | ||
| Whiteboard: | |||
|
Description
Pierre-Charles David
I have some comments about the retention policy for nightlies and milestones: For the nightlies, it would be better to keep the last 4 or 5 builds, it gives flexibility and avoid too many disturbance when many nightlies are triggered manually on the same day. For the milestones, it would be better to not keep the milestones forever once the release is done. Moreover, it is more similar to what Eclipse does (http://wiki.eclipse.org/Eclipse/Repository_retention_policy). (In reply to Mikael Barbero from comment #1) > I have some comments about the retention policy for nightlies and milestones: > > For the nightlies, it would be better to keep the last 4 or 5 builds, it > gives flexibility and avoid too many disturbance when many nightlies are > triggered manually on the same day. Agreed. Shoud this be based on the number of builds or in a number of days (e.g. "nightly are kept for a week")? A fixed number of days seem more reliable for consumers, but when we launch several "nightlies" manually during active development, this could cause a lot of nightlies to accumulate, and they are far from cheap: we publish variants for 5 targets (helios to luna) and each build amounts to about 240MB. A fixed number of builds (e.g. 4 or 5) would limit the disk usage but in some cases this would means builds less than a day old would be removed. > For the milestones, it would be better to not keep the milestones forever > once the release is done. Moreover, it is more similar to what Eclipse does > (http://wiki.eclipse.org/Eclipse/Repository_retention_policy). OK to follow the general Eclipse rules on this. For nightlies, let's say best of both worlds: the most 5 recent builds for each day, for the last 5 days? At most, you have 25 builds * 5 targets = 125 update sites * 250Mb ~= 32 Gb for nightlies. This is huge, but you control it (and you may want to change 5 for 3 if needed ;)). What do you think? I have a small concern with the use of composite update site. As they grow they lead to huge delays at build and install time, and when a new dependency is introduced P2 might find that it's better idea to fall back to the previous build (which is aggregated too). If we want to cover the need of "I want to get the latests bits of this stream" we might be better of with a "latest" simlink targetting the latest build of a stream, don't you think ? (In reply to Mikael Barbero from comment #3) > For nightlies, let's say best of both worlds: the most 5 recent builds for > each day, for the last 5 days? At most, you have 25 builds * 5 targets = 125 > update sites * 250Mb ~= 32 Gb for nightlies. This is huge, but you control > it (and you may want to change 5 for 3 if needed ;)). What do you think? Note sure: apparently nightlies should go on build.eclipse.org instead of download.eclipse.org. See http://wiki.eclipse.org/File:Build_infra_layout.png I don't fully understand the constraints here, I'll have to study this a bit more. (In reply to Cedric Brun from comment #4) > If we want to cover the need of "I want to get the latests bits of this > stream" we might be better of with a "latest" simlink targetting the latest > build of a stream, don't you think ? I was under the impression that composite repos were the "standard" Eclipse way (see for example how CDO organizes its repos: http://www.eclipse.org/cdo/downloads/), but a symlink is probably easier to understand (and to implement/automate). I'm moving here the open issue below which was directly on the wiki page. I've added a link to this bugzilla on the page itself. The variants for each target (e.g. helios, indigo, etc.) are build from the same source code but will get different qualifiers. In a way, this is OK because a bundle's name + fully qualified version should be unique. However, how do we decide which timestamp to use for the update-sites? For the first 0.9.0 stable build I chose the start time of the corresponding Hudson job but I did it manually and I'm not sure yet if this is a correct choice and how to automate it. Updated the page to remove helios and indigo from the list of platforms we will publish update sites for (see https://bugs.eclipse.org/bugs/show_bug.cgi?id=422449#c2). Publication of the nightly builds is implemented by the script at releng/org.eclipse.sirius.releng/publish-nightly.sh. *** Bug 417814 has been marked as a duplicate of this bug. *** (In reply to Pierre-Charles David from comment #2) > we publish variants for 5 targets (helios to luna) and each build amounts > to about 240MB. This has been reduced to 3 variants: juno, kepler and luna. Each repo currently takes around 140MB. To be clear on what remains: * the promotion of a nightly as a stable or milestone and from a milestone as release has not been automated. It is a simple matter of "cp -a nightly/THE_NIGHTLY milestone/THE_MILESTONE", and not frequent enough to justify automation for now. * the shortcuts for streams of the form 1.x (not just 1.0.x) have not been implemented. * the removal of old nightlies has not been implemented. Currently I simply log to build.eclipse.org from time to time to manually remove very old ones. Note also that EMF Compare has reused and improved our publication script. We should review their version and see if we want to re-integrate their improvements: http://git.eclipse.org/c/emfcompare/org.eclipse.emf.compare.git/tree/org.eclipse.emf.compare-parent/releng The current situation is not perfect, but seems good enough for now, so I keep the issue open but reduce its importance and priority. Actually I have created the more focused bug 429564 for automating the cleanup of our nightlies and will close this one. The two other remaining tasks I mentioned in my previous comment do not cause any real problem; we will treat them if and when we feel a real need for them. The various update-sites and short-cuts have been working fine for a while. Marking as verified. Available in Sirius 1.0.0M6 (see https://wiki.eclipse.org/Sirius/1.0.0M6). |
