Bugzilla – Full Text Bug Listing
| Summary: | need test servers for running JUnit and performance tests | ||
|---|---|---|---|
| Product: | Community | Reporter: | Kim Moir <kim.moir> |
| Component: | CI-Jenkins | Assignee: | CI Admin Inbox <ci.admin-inbox> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | P3 | CC: | bokowski, caniszczyk, denis.roy, dgaff.eclipse, d_a_carver, Ed.Merks, john.arthorne, mike.milinkovich, Mike_Wilson, mknauer, pwebster, webmaster |
| Version: | unspecified | ||
| Target Milestone: | --- | ||
| Hardware: | PC | ||
| OS: | Windows XP | ||
| Whiteboard: | |||
|
Description
Kim Moir
cc'ing committer reps :-) Kim, can you: a) provide the hardware specs for the machines you would like, assuming lots of people would use them b) indicate whether the OS can be virtualized (ie, Windows running virtualized in Xen or VMWare) c) indicate the desired OS versions Hi Denis Just a bit of background, the configuration of the test machines depends on the reference platforms listed in the Eclipse plan (http://www.eclipse.org/projects/project-plan.php?projectid=eclipse#target_environments). As the reference platforms on our plan change, so do the operating systems and VMs of our test machines. I have opened a bug with our sysadmin here to give me access to a virtualized instance of Windows XP to determine if running tests results on this machine gives the same results as on the native OS. I'll let you know what I find out. The current JUnit machines we have are 1U machines with dual core 3Ghz processors and 3GB of RAM. Today we have 2 Windows (XP), 2 Linux (RHEL 5.2), and one Mac (Leopard). The tests take about 6.5 hours to run today. I would be absolutely estastic if they could run in half the time. If we ran the tests on faster hardware, or ran the tests on multiple instances of commodity hardware in parallel, our committers would be much happier. The performance machines can't be virtualized. They need to be run on the native OS. Today these machines are 3GHz dual core machines with 3Gb of RAM. Performance machines should be on par with with what a regular developer would use today so of course they could be faster. Today we have two performance linux machines (SLES 10 and RHEL 5.2). The windows machines are XP. As Windows 7 use becomes more prominent, I'm sure we will need to run performance tests on them. Each of these performance test runs take about 8 hours per machine on the current hardware. It would be nice if the results for these machines were faster, but ultimately the JUnit tests are the higher priority to speed up. During milestone week, these test machines are used 24x7. Even during a regular week, these machines are probably used between 7 and 24 hours a day. It would be nice if there was a way to remotely reboot these machines, and VNC etc into them to see the desktop. Instead of asking what hardware we need, I think we need to look at 1) What level of service do you expect to provide. For example, if a team wants to run tests on a machine X, how long is it reasonable for them to wait for the machine to be free? How long it is reasonable for the x number of tests to complete? 2) What will it cost to provide this level of service? I think I'll need to do some additional testing before we have definitive specs on what we need. Of course, if other Eclipse teams have requirements too, I would encourage them to comment. Also, thank you for setting up the linux box at the Eclipse Foundation to test the Hudson slave configuration, I really appreciate it and will let you know what I find out. (In reply to comment #3) > Hi Denis > > Just a bit of background, the configuration of the test machines > depends on the reference platforms listed in the Eclipse plan > (http://www.eclipse.org/projects/project-plan.php?projectid=eclipse#target_environments). > As the reference platforms on our plan change, so do the operating > systems and VMs of our test machines. That makes sense, but is vague on a subtle yet important point. As a new, more modern platform is added to the list (e.g. Windows 7), is a new machine added for that new platform, or is an existing machine refurbished with the new OS? I can imagine that the long-term support requirements for old versions of the platform would require that new machines are endlessly added to the pool of test machines. (In reply to comment #4) > (In reply to comment #3) > > Hi Denis > > > > Just a bit of background, the configuration of the test machines > > depends on the reference platforms listed in the Eclipse plan > > (http://www.eclipse.org/projects/project-plan.php?projectid=eclipse#target_environments). > > As the reference platforms on our plan change, so do the operating > > systems and VMs of our test machines. > > That makes sense, but is vague on a subtle yet important point. > > As a new, more modern platform is added to the list (e.g. Windows 7), is a new > machine added for that new platform, or is an existing machine refurbished with > the new OS? I can imagine that the long-term support requirements for old > versions of the platform would require that new machines are endlessly added to > the pool of test machines. Or depending on how robust the server is you either add a new virtual machine, or a new cloud instance. Mike, good observation :-) However, it's not really endless. Today at Eclipse, we only run 3.6 and 3.5.2 stream builds. Thus we don't have the hardware configured to run tests with operating systems that were reference platforms for 3.3 or 3.4. They were wiped and installed with the newer versions. I would expect that once Windows 7 becomes the defacto windows platform, we would cease to run tests on Windows XP. This has been what has happened historically. For instance, last year, we stopped running tests on Mac carbon, and instead run tests on Mac cocoa because it's more common. > I would expect that once Windows 7 becomes the defacto windows platform, we
> would cease to run tests on Windows XP.
How about we set up a Windows 7 Professional (if such a thing exists) VM to kick things off?
Sure, thanks Denis. Sounds good. > How about we set up a Windows 7 Professional
$280 CAD! Two hundred and eighty dollars. TWO HUNDRED AND EIGHTY. DOLLARS.
*sigh*
two hundred and eighty dollars
Is 25G large enough for the Windows machine, or do the tests require more space? -M. Our windows machines today have about 200GB of space. They are purged of build artifacts on a weekly basis. Where will the build artifacts be stored, on this machine or a shared filesystem? (In reply to comment #11) > Our windows machines today have about 200GB of space. They are purged of build > artifacts on a weekly basis. Where will the build artifacts be stored, on this > machine or a shared filesystem? I believe the artifacts are archived on the master machine, but don't quote me on this. The workspace will reside on the slave machine that the build occurs. But we're using Windows to test, right? Not build? Yes. Looking at our local build machines, we consume 1.5GB to run the JUnit tests for a single build on a single platform. We have a lot of tests :-) > Looking at our local build machines, we consume 1.5GB to run the JUnit tests
> for a single build on a single platform. We have a lot of tests :-)
Ok, so the Windows box needs 1.5G x How many builds?
I mean, if you say you need 200G on the Windows machine, then we might as well install Windows 7 Pro on the bare iron...
Each of our test runs consume 1.5GB per build run per platform. Could this be reduced on new hardware with a shared filesytem? I think so because we could have a shared p2 repo for running tests on a single build instead of copying it to individual test machine filesystems. I don't necessarily need 200GB. At most, during a milestone week, we run four builds a day. In Hudson you can specify the number of builds you keep anyways so this shouldn't be a problem. What are the requirements of other Eclipse projects that would like to test on Windows? So let's create a 60GB Windows 7 image to start us off. If it's too small, we'll expand it. That will leave room on the server for 2 or 3 other images. Should I be installing the 32bit or 64bit version of Windows? -M. Today we test on a 32 bit version. John, do you see any issues with running our tests on a 64 bit version? I'm not sure of the download numbers these days for 32bit windows versus 64 bit. (In reply to comment #19) > Today we test on a 32 bit version. John, do you see any issues with running > our tests on a 64 bit version? I'm not sure of the download numbers these days > for 32bit windows versus 64 bit. I ran the stats and found for 3.5.2 there were 490,805 downloads of 32-bit and 58,915 of 64-bit. So, 32-bit appears to still be the overwhelming favorite. However 64-bit is certainly going to win out in the long term. So, I would say the preference is 32-bit, but if that is difficult to do we could probably live with the 64-bit version being tested. Ok, I've got 32bit Windows 7 running as a guest on build3. What JDK/JRE version should I be installing? Kim: do you have some time next week to help me test this setup? -M. Yes, I should be able to help you test it next week. Thanks for setting this up! Ok, I've hooked the Windows machine into Hudson via JNLP. I've had no luck at all trying to add Hudson as a service, so perhaps we'll need to find another solution if the JNLP doesn't work out. -M. great - thanks Matt! See the current VMs we use for testing here http://www.eclipse.org/projects/project-plan.php?projectid=eclipse#target_environments Matt, what vms are installed on this machine that I can use for testing? I've installed JDK 6u20 to get hudson running, and the Sun JDK 5 update 22. I can't seem to find the IBM jdk. Any pointers? -M. We just test with Sun VMs on windows, that's fine - thanks! What are the locations of the VMs on the disk of the machine? How do I select the appropriate VM for the windows slave from the Hudson JDK list? (I don't see anything there yet) I've added the 1.5sr22 and 1.6r20 JDKs to the node config. -M. Thanks. What are the actual filesystem locations of the VMs on the windows machine? c:\.... c:\Program Files\Java\ jre6 jre1.5.0_22 jdk1.5.0_22 jdk1.6.0_20 -M. I'm going to boldly close this as fixed, since your jUnit tests are already done on our servers, and we've provisioned a performance tests slave with dedicated resources. I am looking forward to hearing about the results of the performance tests! Boldly closing bugs where webmaster has closed before may result in the Wrath of Kim :-) Thanks, I'll let you know when I have time to test this. I'd like to resolve the issues running the JUnit tests before I tackle the performance tests. Thanks for all your help with this issue. |