Bug 339472

Comment 1 Wayne Beaton 2011-03-10 10:46:58 EST

This sounds like something that would require a lot of work upfront with ongoing maintenance. We'll need to get buy-in from Webmaster on the ongoing maintenance part (but we're not quite to a point where we're ready to ask for a call, Denis).

Any thoughts on who can do the initial work?

I wonder if the right approach here is to assemble an RFP. Thoughts?

Comment 2 Miles Parker 2011-03-10 12:52:11 EST

Would any of the current experienced project build engineers / web masters be willing to give a very rough (non-binding :)) time estimate for doing something like this just so we know what the ballpark is? 
CBI/Athena had a lot of the pieces of this rolling already for their PDE based build..one of the things that I lost going to the new infrastructure was the promote scripts. AFA maintenance goes, I think that's a legitimate concern, but my feeling is that a lot of this is getting the basics rolling and working. Perhaps the community could take responsibility for further changes on an on-going basis.

Comment 3 Wayne Beaton 2011-03-10 13:38:10 EST

(In reply to comment #2)
> AFA maintenance goes, I think that's a legitimate concern, but
> my feeling is that a lot of this is getting the basics rolling and working.
> Perhaps the community could take responsibility for further changes on an
> on-going basis.

Further changes is one thing to consider, but if there is an ongoing (hourly, daily, weekly) process, then that process will necessarily be babysat by the Webmaster. We have to consider Webmaster's available resources in this discussion.

Comment 4 Miles Parker 2011-03-10 14:09:10 EST

(In reply to comment #3)

> Further changes is one thing to consider, but if there is an ongoing (hourly,
> daily, weekly) process, then that process will necessarily be babysat by the
> Webmaster. We have to consider Webmaster's available resources in this
> discussion.

Point taken. OTOH, perhaps the webmaster would prefer to babysit a process
rather than a bunch of unruly amateur releng engineers. ;) I know how often
I've had to hassle the webmaster because I needed to get or change permissions
or muck with something else promote related on build-eclipse.org and multiplied
against the number of projects it might be attractive to have a single point of
maintenance and not have us touch that stuff. But that is of course a judgement
for the webmasters.

Comment 5 Denis Roy 2011-03-11 11:01:52 EST

By no means do I consider anyone in this community as either unruly or amateur, but Miles does have a point: managing a common and consistent process/application is much preferred over, say, what we have on Hudson now -- variation and inconsistency from project to project.

Comment 6 Eike Stepper 2012-11-16 02:22:00 EST

Created attachment 223642 [details]
config folders

Comment 7 Eike Stepper 2012-11-16 02:22:36 EST

Created attachment 223643 [details]
main config

Comment 8 Eike Stepper 2012-11-16 02:23:59 EST

Adding some text I wrote to Wayne in July 2011:


Hi Wayne,

I'm not fully done with what I've put on my own agenda, but I can try to outline the general idea. The entire system comrises several infrastructure parts:

- The source repository (SVN in our case)
- Bugzilla to provide input for the generated release notes
- Hudson (2 jobs in our case)
- download.eclipse.org
- our project web space for the generated downloads html page
- a separate web space on eclipse.org (build.eclipse.org?) for a limited number of administrative tasks (copy/rename/delete builds, manage release train staging, etc)
- a working data directory (as of now /tmp/promotion.emf.cdo)
- a "promoter.sh" cron job that can do things under my user id
- a configuration area (I maintain the config data in SVN, too)

The core of the system is the "promoter" job. I've designed a shell script such that it can be started very frequently (e.g. every 1 or 5 minutes). It can recognize very quickly at least one of the following two conditions is true;

- One of the configured Hudson jobs has new builds (i.e. nextBuildNumber has changed)
- An admin task is enqueued from the admin web interface (the entire admin thing is not yet implemented)

If both are false the script ends after milliseconds. In addition the script exits immediately if another instance of the script is already/still running.

If builds are available on Hudson, the script starts a Java process with a two handfull of self-written classes (no other dependencies so far). These classes do the following:

1) The configured Hudson jobs are analyzed in more detail. If new builds are to be promoted (e.g. they're successfully finished, they match configurable criteria like build type etc) they are copied to the configured downloads location (should be a folder called "drops" so that the default mirroring filters exclude N and I builds from mirror distribution)

2) The proper SVN branch is tagged with the build qualifier.

3) The build gets post processed:
    a) content.jar and artifacts.jar are unpacked
    b) from content.xml an additional p2 repository is generated (with xslt) that lists the categories of the original repository under a new unique (i.e. build-specific)  category (e.g. "CDO I20110701-1455"). This is important for a step further down where the build/drop p2 repos get composed into higher level p2 repos (e.g. integration builds, releases, etc). I've learned this from Pascal's blog about category nesting ;-)
    c) p2.mirrorsURL is set to correct values for both repositories (build and categories)
    d) a zip of the p2 repo is created
    e) a zip for the dropins folder is created

4) According to the configuration area a number of p2 composite repositories is generated from the drop repos and the categories repos. Starting with this step all actions happen in a temp folder so that the final result of all the generation can be switched live almost atomically.

5) Release notes get generated according to some configurable rules. The idea is that the release notes show the ID and title of all bugzillas that are mentioned in the SVN commit messages between the last (relevant) build in a composite repo and the current build. Here I'm currently hitting the problem that I can't seem to have web access to Bugzilla from build.eclipse.org. Denis, do you have advice for me?

6) The entire downloads page is statically generated (no php in it!)

7) Finally the temporary generation folder is renamed to the public location so that the repos become accessible and the downloads page visible.

Here is a current example: http://www.eclipse.org/cdo/downloads

Here is a screenshot of one of the genrated p2 composite repos (for weekly integration builds): http://2.bp.blogspot.com/-jHV19LD8hMI/TgHhzBv2fDI/AAAAAAAAAVQ/E25RlI317B4/s1600/promoter.PNG

The configuration area looks like this:
https://bugs.eclipse.org/bugs/attachment.cgi?id=223642

The main promoter.properties looks like:
https://bugs.eclipse.org/bugs/attachment.cgi?id=223643


The entire (single) Eclipse IProject is in our SVN repo: http://dev.eclipse.org/svnroot/modeling/org.eclipse.emf.cdo/infrastructure/org.eclipse.emf.cdo.releng.promotion

The Java classes already use a crude self-made plugin (better: customize) mechanism so that the behaviour can be changed on a per project base.

As I said, although major parts are already working in a pretty stable and predictable way I'm still working on some parts. When I'm done so that the service fulfills our own requirements I'm generally willing to spend some more effort to bring it in a shape so that other projects can check it out, configure it according to their needs and profit from it. Please note that all this is finally a first reaction to my own bugzilla https://bugs.eclipse.org/bugs/show_bug.cgi?id=339472

Documentation (especially of the known issues or limitations) is still missing.

I've attached the promoter.sh shell script and a log of one promoter run (runs can differ majorly depending on the events to be processed and the various states and configurations at that time.

So, it's time to go to bed now ;-)

I hope you can make some sense out of this quick and brief overview. I'm looking forward to getting your feedback...

Cheers
/Eike

----
http://www.esc-net.de
http://thegordian.blogspot.com
http://twitter.com/eikestepper





Am 30.06.2011 20:10, schrieb Wayne Beaton:
> Hi Eike. Mike told me about the work you've been doing on a "promotion
> service". Can you share some details with me?
>
> Thanks,
>
> Wayne
>

Comment 9 Eike Stepper 2012-11-16 02:26:38 EST

Created attachment 223644 [details]
main page of promotion admin service

Comment 10 Chris Aniszczyk 2013-01-18 13:34:21 EST

How about we just use Nexus from Sonatype?

We can either use their current one for open source projects which works great... https://oss.sonatype.org (we use it at Twitter and love it) or roll our own instance...

Comment 11 Eike Stepper 2013-01-19 00:24:31 EST

I don't know Nexus, so it's hard for me to comment without doing a lot of research. Would that be an alternative to having p2 repositories (and maybe zip files) under /home/data/httpd/download.eclipse.org/ ?

Comment 12 Laurent Goubet 2013-04-22 10:45:22 EDT

Eike,

I was looking into replacing the promotion scripts we use for EMF Compare and Acceleo, and would like to get some info on what you currently have regarding your comment #8. Namely, I'd like to know how we could re-use them in another project, and what it would require from that project... A few questions that come to mind:

- Are your promotion scripts stable (or rather: can another project use them)?
- How would we go about re-using your scripts? Do I need to clone cdo.infrastructure, modify it for my needs and commit my fork somewhere else?
- How complicated would it be to drop our current "infrastructure" (really, an ant script that we launch manually when we need a promote) to yours?
- Would this necessitate changing our workflow (specific commit messages for example, since I saw you parse the [xxxxxx] to find a bug id)?

And... What exactly is setting all of this in motion? a cron job that launches one of the shell scripts (I only glanced at cdo.infrastructure on git)?

Comment 13 Eike Stepper 2013-04-23 00:46:36 EDT

(In reply to comment #12)
> - Are your promotion scripts stable (or rather: can another project use
> them)?

Yes, they run for months and years without problems and they should be flexible/configurable so that other projects can use them. If there are places that are not flexible enough I'm interested in these additional requirements.

> - How would we go about re-using your scripts? Do I need to clone
> cdo.infrastructure, modify it for my needs and commit my fork somewhere else?

The repo layout was optimal with SVN. I just checked out these folders:
/org.eclipse.emf.cdo.releng.promotion/promotion/config
/org.eclipse.emf.cdo.releng.promotion/promotion/promoter

Now you have to clone the entire Git repo to your home folder on build.eclipse.org and run /org.eclipse.emf.cdo.releng.promotion/promotion/update.sh, which copies the required folders up into your home folder. Please review update.sh carefully before you run it, you may have to adjust it.

Ideally you copy the config folder into your own Git repo and clone that, too. Then you can configure the promoter through your own Git repo. The promoter folder is considered reusable infrastructure and if you need changes there you should talk to me. If you're happy with the system we could discuss better collaboration options, e.g., a separate repo or project.

> - How complicated would it be to drop our current "infrastructure" (really,
> an ant script that we launch manually when we need a promote) to yours?

We'll know the answer once you've tried it. I think you'd be the first ;-)

> - Would this necessitate changing our workflow (specific commit messages for
> example, since I saw you parse the [xxxxxx] to find a bug id)?

For now, yes. But we should make that configurable. Should be easy to use a regular expression from /org.eclipse.emf.cdo.releng.promotion/promotion/config/promoter.properties

> And... What exactly is setting all of this in motion? a cron job that
> launches one of the shell scripts (I only glanced at cdo.infrastructure on
> git)?

Yes, I have this in my crontab:

* * * * * ~/bin/promote-job.sh

And this in ~/bin/promote-job.sh:

cd ~/promotion/config
../promoter/promoter.sh >> /tmp/promotion.<MY PROJECT>/promoter.log 2>&1

To start the promoter manually (for example after I manually deleted some drops/?? folders) I have ~/bin/promote:

#clear
#../update.sh
../promoter/promoter.sh --force | tee /tmp/promotion.<MY PROJECT>/promoter.log.force
cat /tmp/promotion.<MY PROJECT>/promoter.log.force >> /tmp/promotion.<MY PROJECT>/promoter.log
rm /tmp/promotion.<MY PROJECT>/promoter.log.force
popd > /dev/null

The bootstrap process is certainly a little tricky and Git didn't make it easier. Maybe we find a better way to do this. If you struggle I can help via Skype ;-)

Comment 14 Eclipse Genie 2015-04-15 16:24:40 EDT

This bug hasn't had any activity in quite some time. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet.

If you have further information on the current state of the bug, please add it. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant.

--
The automated Eclipse Genie.

Comment 15 Eclipse Genie 2018-02-09 12:01:55 EST

This bug hasn't had any activity in quite some time. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet.

If you have further information on the current state of the bug, please add it. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant.

--
The automated Eclipse Genie.

Comment 16 Eclipse Genie 2020-02-01 18:08:51 EST

This bug hasn't had any activity in quite some time. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet.

If you have further information on the current state of the bug, please add it. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant.

--
The automated Eclipse Genie.

Comment 17 Eclipse Webmaster 2020-02-13 15:53:05 EST

I'm going to close this as 'wontfix' only because there has been no activity for 7 years.

-M.