Bug 389371

Comment 1 Denis Roy 2012-09-19 13:59:06 EDT

So far we've determined that performance tests on virtualized hardware produce sane results.  Also, UI tests on Windows look very promising.  As soon as we nail down performance tests on Windows, I think we'll all have a better idea of what we're looking for in terms of hardware.

Comment 2 Denis Roy 2012-09-28 16:52:36 EDT

At first glance we're looking at acquiring four servers with the following specs:

- 64G RAM @ 1600
- 2x 1TB SATA _hot swap_ drives
- dual GbE nic
- Dual CPU 6-core Xeons

With 64G of RAM, we could potentially host 16 vservers @ 4GB each, or 10 @ 6G, for a total of 40 - 60 hudson slaves.  These are capable of running various 64-bit Linux and Windows OSses.

Comment 3 David Williams 2012-10-01 14:44:29 EDT

(In reply to comment #0)

> 
> Is Mac support required? We recently acquired a Mac Mini for builds. Is a
> separate machine needed for testing?

I would think so. There is currently 1 64 bit machine (the new one) and 1 32 bit.
 
See bug 354774 for why "we" think 32 bit isn't that useful (in short: rarely used). So to have 1 64 bit mac with 4 executors for unit tests and builds, and another with 1 executor for performance tests would be ideal.

Comment 4 John Arthorne 2012-10-01 15:54:45 EDT

(In reply to comment #3)
> See bug 354774 for why "we" think 32 bit isn't that useful (in short: rarely
> used). So to have 1 64 bit mac with 4 executors for unit tests and builds,
> and another with 1 executor for performance tests would be ideal.

Just a note that we have never run performance tests on Mac in the past. I'm not saying we would never want to do this, but there is likely some work involved in making sure all the perf tests run reasonably on Mac. This doesn't seem high on the priority list, so I wouldn't suggest allocating a Mac perf slave for this until someone is ready to use it.

Comment 5 Mike Milinkovich 2012-10-01 15:58:21 EDT

(In reply to comment #4)

OK, so no Mac required for the moment.

Does everyone agree with the specs that Denis outlined in comment #2 for Linux and Windows? E.g. can we go ahead and order those?

We will have to buy some Windows licenses as well.

Comment 6 David Williams 2012-10-02 00:32:21 EDT

(In reply to comment #4)
> (In reply to comment #3)
 
> Just a note that we have never run performance tests on Mac in the past. I'm
> not saying we would never want to do this, but there is likely some work
> involved in making sure all the perf tests run reasonably on Mac. This
> doesn't seem high on the priority list, so I wouldn't suggest allocating a
> Mac perf slave for this until someone is ready to use it.

Thanks John. I either misunderstood some recent discussions (at status meeting?) or you just have a better memory than others on the call.  [If it was me, I'd be happy to get one platform well tested before moving to the next]. 


(In reply to comment #5)
> (In reply to comment #4)
> 
> Does everyone agree with the specs that Denis outlined in comment #2 for
> Linux and Windows? E.g. can we go ahead and order those?
> 

Sounds reasonable to me. I don't really get the math of it ... how does 10 vservers @ 6G each == 40 slaves? Each of those would have, say, 4 executors? Sounds like each executor would have only a few Gig for their work, with a lot of contention between them when all 4 executors are busy? 

= = rest slightly off topic = = 

Being my usual contrary self, while I think this virtual slave approach can work, with effort, that it is not exactly proven to be working yet. Yes, there was some "stability" in some repeated runs, which John judged  "close enough" but at the same time, when running Linux, there's some tests that seem to "hang" in a particular way that (I'm guessing) makes Hudson web interface unresponsive (half guessing, but half not): partially documented in bug 389365 and bug 389369 comment 11 (and seems like somewhere else I can't find it ... but I offered to run the test over and over so we could test the hypothesis, as long as webmaster on standby to correct things, and Hudson guy on standby to help diagnose). 

Similarly, On Windows, our "resource performance test" seem to hang, or not exit, until forced to after 2 hour time limit where normally the tests take 8 minutes. And on my local windows (not virtual), also running Hudson, this does not happen. 

So ... I think fine to order the equipment, but an investment in understanding these hard problems is also required. Chances are they are fixable ... but I didn't want to leave the impression "all is well and all we need is the equipment".

Comment 7 Denis Roy 2012-10-02 09:33:16 EDT

> Sounds reasonable to me. I don't really get the math of it ... how does 10
> vservers @ 6G each == 40 slaves?

We'd acquire four servers, each capable of hosting between 6 and 10 vservers.


> Each of those would have, say, 4 executors?
> Sounds like each executor would have only a few Gig for their work, with a
> lot of contention between them when all 4 executors are busy? 

Most executors use between 200M and 500M today.  


> Being my usual contrary self, while I think this virtual slave approach can
> work, with effort, that it is not exactly proven to be working yet.

FWIW, slave2 and slave6 are run on bare iron.  In fact, slave2 has 2 CPUs and 2 executors.  You can try running the jobs that hang on those slaves.

I am convinced, however, that any hangs in the jobs are not caused by the virtualized hardware, but instead by Hudson's slave delegation.  I'll propose other solutions in the appropriate bugs.

Comment 8 Matthias Mailänder 2014-05-15 13:56:13 EDT

I am curious. Was this discussion about the details on http://www.h-online.com/open/news/item/Google-donates-to-an-Eclipse-performance-test-lab-update-1703288.html?

Comment 9 Alexander Kurtakov 2019-02-07 11:10:11 EST

So ancient data that the bug is invalid.