Bug 430536

Description Pierre-Charles David 2014-03-17 11:52:47 EDT

During the Luna M6 integration phase (the last few days), we were hit by a few integration issues which we did not expect. Under the pressure of the M6 deadline we made some mistakes which should have been easy to avoid (see for example the reopening of bug 428322) if we had had more time.

To mitigate the issue we added a sirius_staging.target target platform definition, which gets its bits from releases/staging (updating continuously during the final days) instead of releases/luna (which only exposed the previous milestone, M5, until the very last time). This allowed us to build Sirius 1.0M6 against the actual versions of our dependencies which would be in M6.

However, because we are at offset +3 and depend on several other projects which are at the same offset (e.g. Acceleo), this gives us a very short time window to detect and fix integration isssues. At the worst, if for example Acceleo had contributed its version 3.5 (instead of 3.4 in Luna M5) just 1 hour after the deadline, we would not have been able to build against that version using the current target platforms we use.

The root cause is that 3 offset and a 3-days window to integrate 70+ projects in the train is too little, but we can't do much about that. As an alternative, I propose the following approach, which should help us detect integration issues before the final integration window:
* For each projet we depend on, identify clearly were we can get "the latest bits", which corresponds to what they will contribute to the train *before* they have added it to the simrel. For most well-run project this should be a nigthly build. This step may require some research to identify the proper update-sites, which are not always well-advertised.
* Create a sirius_canary.target[platform] target definition, which gets all our depdendencies from these "bleeding edge" update-sites. Here the term "canary" refers to http://en.wiktionary.org/wiki/canary_in_a_coal_mine as in "Something whose sensitivity to adverse conditions makes it a useful early indicator of such conditions; something which warns of the coming of greater danger or trouble by a deterioration in its health or welfare."
* Have a separate automated build in our HIPP that builds regularly (daily?) against that target. We should not expect all the builds to be green here, some transient integration issues are to be expected. However we should regularly (e.g. weekly) check the status of that job, and take action as soon as we detect a real breaking change.

Of course, such an approach will not work if the projects we depend do actual breaking change in their sources at the end of a cycle (as opposed to simply *contributing* their changes at the end of the cycle).