Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.
Bug 365907 - Simultaneous trigger of many Virgo build jobs lead to failures
Summary: Simultaneous trigger of many Virgo build jobs lead to failures
Status: CLOSED FIXED
Alias: None
Product: Virgo
Classification: RT
Component: unknown (show other bugs)
Version: 3.0.1.RELEASE   Edit
Hardware: PC Mac OS X - Carbon (unsup.)
: P3 normal (vote)
Target Milestone: ---   Edit
Assignee: Glyn Normington CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-12-07 10:08 EST by Borislav Kapukaranov CLA
Modified: 2011-12-07 11:06 EST (History)
5 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Borislav Kapukaranov CLA 2011-12-07 10:08:04 EST
Many(almost all) Virgo build jobs rely on having free jmx and http ports when executing their test. 
However when a push in all repositories is performed within very small time interval the respective Hudson build jobs are triggered almost at the same time. An example is a ripple which does its thing and pushes at the end of the operation in every participating repository.
This leads to many false build failures and a rebuild after some time solves the problem. 
This sometimes consumes a lot of time and distracts us for no real reason, depending on the number of failed jobs.

There are a number of solutions we could think of:
1. Somehow serialise the jobs so that the jobs from the Virgo group are always built one at a time
2. Perform the builds on a Hudson slave that can handle only one job at a time
3. Make sure every test group in each repository relies on different ports
Comment 2 Glyn Normington CLA 2011-12-07 10:26:48 EST
Stuart McCulloch (on IRC freenode #eclipse-hudson) pointed me at these facilities:

http://wiki.hudson-ci.org/display/HUDSON/Locks+and+Latches+plugin
http://wiki.hudson-ci.org/display/HUDSON/Port+Allocator+Plugin
Comment 3 Glyn Normington CLA 2011-12-07 10:27:20 EST
Whoops - sorry Stuart. You're too helpful. :-)
Comment 4 Stuart McCulloch CLA 2011-12-07 10:35:46 EST
BTW, depending on the number of projects you might find the port-allocator plugin easier to deal with than the locks-and-latches plugin, especially if you don't have access to the global hudson configuration.
Comment 5 Glyn Normington CLA 2011-12-07 10:39:17 EST
(In reply to comment #4)
> BTW, depending on the number of projects you might find the port-allocator
> plugin easier to deal with than the locks-and-latches plugin, especially if you
> don't have access to the global hudson configuration.

We have a lot of projects and a lot of ports configured in various ways, so my next step is to approach the web master about adding a "virgo" lock to the global hudson configuration.
Comment 6 Glyn Normington CLA 2011-12-07 10:40:47 EST
Dear Web Master

Would it be possible to add a "virgo" lock to the global Hudson configuration so we can use it to serialise the Virgo jobs which keep treading on each others toes?

Thanks!
Comment 7 Glyn Normington CLA 2011-12-07 10:43:20 EST
Ahem, that should have been "each other's toes".
Comment 8 Eclipse Webmaster CLA 2011-12-07 10:48:57 EST
(In reply to comment #6)
> Dear Web Master
> 
> Would it be possible to add a "virgo" lock to the global Hudson configuration
> so we can use it to serialise the Virgo jobs which keep treading on each others
> toes?
> 
> Thanks!

Ok I've created the lock: virgo-serialise 

-M.
Comment 9 Glyn Normington CLA 2011-12-07 11:01:27 EST
(In reply to comment #8)
> Ok I've created the lock: virgo-serialise 
> 
> -M.

Perfect. Thanks!

I'd better go and use it now...
Comment 10 Glyn Normington CLA 2011-12-07 11:06:29 EST
I've configured the Virgo jobs which tend to have port clashes along with the sister project Gemini Web to serialise on the lock virgo-serialise. Hopefully that's the end of Virgo and Gemini Web jobs clashing with each other.