Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 365907

Summary:	Simultaneous trigger of many Virgo build jobs lead to failures
Product:	[RT] Virgo	Reporter:	Borislav Kapukaranov <b.kapukaranov>
Component:	unknown	Assignee:	Glyn Normington <glyn.normington>
Status:	CLOSED FIXED	QA Contact:
Severity:	normal
Priority:	P3	CC:	eclipse, glyn.normington, mcculls, milesg78, webmaster
Version:	3.0.1.RELEASE
Target Milestone:	---
Hardware:	PC
OS:	Mac OS X - Carbon (unsup.)
Whiteboard:

Description Borislav Kapukaranov

2011-12-07 10:08:04 EST

Many(almost all) Virgo build jobs rely on having free jmx and http ports when executing their test. 
However when a push in all repositories is performed within very small time interval the respective Hudson build jobs are triggered almost at the same time. An example is a ripple which does its thing and pushes at the end of the operation in every participating repository.
This leads to many false build failures and a rebuild after some time solves the problem. 
This sometimes consumes a lot of time and distracts us for no real reason, depending on the number of failed jobs.

There are a number of solutions we could think of:
1. Somehow serialise the jobs so that the jobs from the Virgo group are always built one at a time
2. Perform the builds on a Hudson slave that can handle only one job at a time
3. Make sure every test group in each repository relies on different ports

Comment 1 Stuart McCulloch

2011-12-07 10:22:10 EST

Sounds like a job for the following plugins:

   http://wiki.hudson-ci.org/display/HUDSON/Locks+and+Latches+plugin
   http://wiki.hudson-ci.org/display/HUDSON/Port+Allocator+Plugin

Comment 2 Glyn Normington

2011-12-07 10:26:48 EST

Stuart McCulloch (on IRC freenode #eclipse-hudson) pointed me at these facilities:

http://wiki.hudson-ci.org/display/HUDSON/Locks+and+Latches+plugin
http://wiki.hudson-ci.org/display/HUDSON/Port+Allocator+Plugin

Comment 3 Glyn Normington

2011-12-07 10:27:20 EST

Whoops - sorry Stuart. You're too helpful. :-)

Comment 4 Stuart McCulloch

2011-12-07 10:35:46 EST

BTW, depending on the number of projects you might find the port-allocator plugin easier to deal with than the locks-and-latches plugin, especially if you don't have access to the global hudson configuration.

Comment 5 Glyn Normington

2011-12-07 10:39:17 EST

(In reply to comment #4)
> BTW, depending on the number of projects you might find the port-allocator
> plugin easier to deal with than the locks-and-latches plugin, especially if you
> don't have access to the global hudson configuration.

We have a lot of projects and a lot of ports configured in various ways, so my next step is to approach the web master about adding a "virgo" lock to the global hudson configuration.

Comment 6 Glyn Normington

2011-12-07 10:40:47 EST

Dear Web Master

Would it be possible to add a "virgo" lock to the global Hudson configuration so we can use it to serialise the Virgo jobs which keep treading on each others toes?

Thanks!

Comment 7 Glyn Normington

2011-12-07 10:43:20 EST

Ahem, that should have been "each other's toes".

Comment 8 Eclipse Webmaster

2011-12-07 10:48:57 EST

(In reply to comment #6)
> Dear Web Master
> 
> Would it be possible to add a "virgo" lock to the global Hudson configuration
> so we can use it to serialise the Virgo jobs which keep treading on each others
> toes?
> 
> Thanks!

Ok I've created the lock: virgo-serialise 

-M.

Comment 9 Glyn Normington

2011-12-07 11:01:27 EST

(In reply to comment #8)
> Ok I've created the lock: virgo-serialise 
> 
> -M.

Perfect. Thanks!

I'd better go and use it now...

Comment 10 Glyn Normington

2011-12-07 11:06:29 EST

I've configured the Virgo jobs which tend to have port clashes along with the sister project Gemini Web to serialise on the lock virgo-serialise. Hopefully that's the end of Virgo and Gemini Web jobs clashing with each other.