Community
Participate
Working Groups
I've been testing running our JUnits on the Windows and Linux Hudson instances at eclipse.org. Things have been going pretty well and I have only 200 failures out of 65,000 tests, mostly due to infrastructure issues or problems in our tests themselves. Today, we also run tests on a Mac. We have an older iMac running Leopard. We would like to continue running tests on the Mac platform once we move the build to the eclipse.org hardware. This is a request for a new Hudson instance on a Mac at eclipse.org.
Mentioned this in the Eclipse architecture call. Wayne that he has a mac mini at home that he will bring in for us to try.
Wayne, is your mac mini x86 or ppc? http://en.wikipedia.org/wiki/Mac_mini We currently run our tests on an x86 iMac running Leopard. We don't even build the ppc fragments anymore.
Intel
Great!
Just a fyi I just talked to Silenio. Currently our test mac machine runs 10.5 (Leopard). It should really run at least 10.6 (Snow Leopard) which is our reference platform. http://www.eclipse.org/eclipse/development/plans/eclipse_project_plan_3_7.xml#target_environments
Until we're done with 3.6.x builds, we should have two machines, one running 10.5 and one running 10.6 for 3.7. If I had a spare desktop machine I would donate it. :-) Otherwise, a Mini should be plenty to run tests.
Scott, I don't have plans to move the 3.6.x builds to run tests at the foundation, only 3.7 builds.
Wayne mentioned at the Ottawa Democamp last week that he had dropped off the mac mini at the foundation. Do you know when it will be installed so I can start testing it. (I know you're both very busy, just asking :-)
> Do you know when it will be installed so I can > start testing it. Ask Santa :) Matt said he's be tinkering with it soon.
Can we do the 'initial' deployment with OSX 10.5.8 and upgrade later? -M.
Sure. It will allow to proceed and test the Hudson install.
Ok the mac is in the rack and I've added it to hudson. If you can tell me what tools you need installed I'll make it happen. -M.
Thanks Matt! Please install unzip, tar, ssh and cvs binaries if they aren't already installed. (I can't login to tell :-)
cvs client only of course. Also, what is the exact location of java on this machine? I assume /System/Library/Frameworks/JavaVM.framework/Versions/1.5.0/Commands/java Could you create a hudson job for me on this machine, job name eclipse-JUnit-Mac so I can run a test. You can copy eclipse-JUnit-Linux. Thanks!
Ok, all of those tools should already be there. The java version is reporting: 1.5.0_19 and it's located in /System/Library/Frameworks/JavaVM.framework/Versions/Current/Commands/java I've created the job. I'm going to close this bug, but if there are any issues you can re-open it(or open a new one as you prefer). -M.
Can you just add this new job to the "Eclipse and Equinox" category. I can't seem to do this anymore.
See <https://hudson.eclipse.org/hudson/job/eclipse-JUnit-mac/1/> ------------------------------------------ Seem to be having problems accessing dev.eclipse.org via pserver on this slave. Started by user kmoir Building remotely on mac-tests [eclipse-JUnit-mac] $ cvs -Q -z3 -d :pserver:anonymous@dev.eclipse.org:/cvsroot/eclipse co -P -r v20110104 org.eclipse.releng.eclipsebuilder org.eclipse.releng.basebuilder cvs [checkout aborted]: connect to dev.eclipse.org(172.25.25.51):2401 failed: Operation timed out FATAL: CVS failed. exit code=1 Recording test results Recording fingerprints
(In reply to comment #16) > Can you just add this new job to the "Eclipse and Equinox" category. I can't > seem to do this anymore. Done. (In reply to comment #17) D'oh. Fixed. -M.
Seem to be getting this error when running JUnit tests requiring the UI. _RegisterApplication(), FAILED TO establish the default connection to the WindowServer, _CGSDefaultConnection() is NULL Does this machine have a UI running? I'm just running the tests like the ones on the Linux box with Xvnc enabled in the hudson config for the job.
Also, could the mac have access to the /shared filesystem that the other eclipse.org servers mount. It doesn't seem to be able to load files from this filesystem.
(In reply to comment #19) > Does this machine have a UI running? I'm just running the tests like the ones > on the Linux box with Xvnc enabled in the hudson config for the job. Yes, or at least it was running when I set it up in the data center. I think the macs use rdesktop(a la windows) for remote management, so perhaps the job config needs to be tweeked? (In reply to comment #20) > Also, could the mac have access to the /shared filesystem that the other > eclipse.org servers mount. It doesn't seem to be able to load files from this > filesystem. The mac is on the same isolated subnet that the windows slave is, so there isn't any 'good' way to get /shared mounted. -M.
The hudson developers suggest using xvnc on the mac to run the ui tests. http://hudson.361315.n4.nabble.com/VNC-plugin-and-Mac-td373540.html#a373541 Can you install and configure it? You can't interact with the desktop via rdesktop in hudson. There isn't a rdesktop plugin.
Note from Matt: So far I haven't found a solution to allow hudson to run a separate display. Scott, do you have any experience this area?
Matt, McQ just walked in my office, and said why are we worrying about multiple displays. Shouldn't hudson just own the display for that machine? Because Hudson is the only purpose for that machine.
Well, because I thought you needed to 'run' the 'test' display via vnc. If you can control the display via your job without vnc then he's probably right. -M.
(In reply to comment #25) > Well, because I thought you needed to 'run' the 'test' display via vnc. If you > can control the display via your job without vnc then he's probably right. If you log in as an administrator (not necessarily root) you don't need VNC or anything like that. An admin user can start UI processes via SSH. I'm not sure about not having a monitor connected, but it shouldn't matter.
Scott, thanks for the suggestion. Matt, does the hudson user run as a regular user or Admin? The vnc plugin is pretty useful for Hudson because it starts and closes the vnc sessions for you. You don't have to worry about displays crashing etc. As an aside, Dave Carver suggestion macports version of xvnc on twitter.
(In reply to comment #27) > Scott, thanks for the suggestion. Matt, does the hudson user run as a regular > user or Admin? It does have admin privileges. > As an aside, Dave Carver suggestion macports version of xvnc on twitter. Cool, well perhaps if you can't the tests running 'directly' we can use this. -M.
What Xvnc binary is currently installed on this machine?
Nick once set up a Mac to have remote UI access for running tests ... perhaps he has a suggestion?
Matt: Another issue I'm seeing is that the tests are running out of heap space halfway through and crashing. How much RAM does the mac mini have? Our mac test machine here has 1GB. http://en.wikipedia.org/wiki/Mac_Mini#Memory_2
(In reply to comment #29) > What Xvnc binary is currently installed on this machine? I'm not quite sure what your asking. Right now I've setup the built in vncserver for remote control. (In reply to comment #31) > Matt: Another issue I'm seeing is that the tests are running out of heap space > halfway through and crashing. How much RAM does the mac mini have? Our mac > test machine here has 1GB. Well it's reporting 2G installed, but top reports a lot in use: PhysMem: 277M wired, 592M active, 695M inactive, 1565M used, 483M free. -M.
The hudson job uses Xvnc to control the display. Where is the location of the Xvnc binary on the filesystem? I use the Xvnc plugin in my mac JUnit job so that UI tests run, but obviously it's not working right now :-) As an aside, I've set up Hudson here on our test machine to try to replicate this issue but I can't muck around with the machine's config right now because the tests are running. Also, I noticed that the Apache foundation has a mac slave so I'm going to ask on their build list for advice regarding ui tests on Mac hudson slaves if we continue to have problems.
Yesterday, I installed hudson on my mac test machine and successfully ran all the tests. The hudson daemon was started as administrator. In the hudson admin, I set the Xvnc Command line to /Library/StartupItems/OSXvnc/OSXvnc-server and the Base display number to 10. The JUnit job was copied from the same one on Hudson at eclipse.org. Xvnc is enabled on the job used to run the UI tests. I'm wondering what the configuration difference is between this machine and the Mac mini that is causing our tests to fail?
Created attachment 186631 [details] hudson vnc configuration
Created attachment 186632 [details] mac hudson config page #1
Created attachment 186633 [details] mac hudson config page #2
Also, I was running a test build on eclipse.org this morning and received this message when trying to run a test on the mac. Does the slave need to be restarted? Started by upstream project "eclipse-equinox-test-N" build number 283 Building remotely on mac-tests [eclipse-JUnit-mac] $ cvs -Q -z3 -d :pserver:anonymous@dev.eclipse.org:/cvsroot/eclipse co -P -r v20110104 org.eclipse.releng.eclipsebuilder org.eclipse.releng.basebuilder $ computing changelog $ pkill Xvnc FATAL: pkill: not found java.io.IOException: pkill: not found at java.lang.UNIXProcess.forkAndExec(Native Method) at java.lang.UNIXProcess.<init>(UNIXProcess.java:52) at java.lang.ProcessImpl.start(ProcessImpl.java:91) at java.lang.ProcessBuilder.start(ProcessBuilder.java:451) at hudson.Proc$LocalProc.<init>(Proc.java:192) at hudson.Proc$LocalProc.<init>(Proc.java:164) at hudson.Launcher$LocalLauncher.launch(Launcher.java:638) at hudson.Launcher$ProcStarter.start(Launcher.java:273) at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:793) at hudson.Launcher$RemoteLaunchCallable.call(Launcher.java:767) at hudson.remoting.UserRequest.perform(UserRequest.java:114) at hudson.remoting.UserRequest.perform(UserRequest.java:48) at hudson.remoting.Request$2.run(Request.java:270) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:417) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:269) at java.util.concurrent.FutureTask.run(FutureTask.java:123) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676) at java.lang.Thread.run(Thread.java:613)
Pkill doesn't seem to exist: Mini-2:~ $ which pkill Mini-2:~ $ man pkill No manual entry for pkill If I look in the same place for vnc I get nothing: Mini-2:~ $ ls /Library/StartupItems/OSXvnc/OSXvnc-server ls: /Library/StartupItems/OSXvnc/OSXvnc-server: No such file or directory Mini-2:~ $ ls /Library/StartupItems/ Mini-2:~ $ Mini-2:~ $ cd /Library Mini-2:Library $ find . -name "*vnc*" Mini-2:Library $ So I'm not sure if the issue is that this system was 'updated' from 10.5 to 10.5.8, or if there is some other software/OS setting that's missing/in the way. I can try re-installing the OS, but I only have the 10.5 disc and I've never had an old OS X install update itself 'nicely' after the implied warranty was up(but that was years ago) And as far as I can tell hudson doesn't support a 'per slave' vnc startup command(it seems to be global for all slaves). -M.
We use http://sourceforge.net/projects/osxvnc/ which is actually not included with the OS install. (Sorry, I didn't realize this before, I didn't install this machine, our sysadmin did). Anyways, could you try installing this and see if this works for the vnc daemon. I realize that specifying different vnc paths on a per slave basis is a limitation of hudson but perhaps we could fake it with soft links on the mac so that the path hudson expects is redirected to the OSvnc binary.
Any luck installing xvnc on the mac? I also asked for help with this issue on stackoverflow http://stackoverflow.com/questions/4728641/running-xvnc-on-multi-platform-hudson-slaves
Sorry it's been a little busy recently. I've installed the OSXvnc server package you listed, and have created a symlink called 'Xvnc' in /usr/local/bin(should be in the path). -M.
Thanks Matt. I appreciate your help and I understand that you've been busy. I'm running a test build now.
It worked! Thanks, now I can run UI tests on the mac :-)
So, it worked one time but now I'm having problems again. Every time I try to start a build with tests using Xvnc, I get the error message "cannot find pkill" and the job dies. Since the mac doesn't come with pkill by default, I installed it by using the following command on my local machine sudo port install proctools and pkill is now an available binary. Could you try that on the mac slave?
This box is very flaky. Now I'm running out of heap space on this machine. The following error occurred while executing this line: /Users/hudsonbuild/workspace/eclipse-JUnit-mac/ws/2011-01-31_13-42-02/eclipse-testing/test-eclipse/eclipse/plugins/org.eclipse.test_3.3.100/library.xml:138: java.lang.OutOfMemoryError: Java heap space at org.apache.tools.ant.ProjectHelper.addLocationToBuildException(ProjectHelper.java:508) at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:418) at org.apache.tools.ant.taskdefs.CallTarget.execute(CallTarget.java:105) at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288) at sun.reflect.GeneratedMethodAccessor109.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106) at org.apache.tools.ant.Task.perform(Task.java:348) at org.apache.tools.ant.Target.execute(Target.java:357) at org.apache.tools.ant.Target.performTasks(Target.java:385) at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1337) at org.apache.tools.ant.Project.executeTarget(Project.java:1306) at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41) at org.eclipse.ant.internal.core.ant.EclipseDefaultExecutor.executeTargets(EclipseDefaultExecutor.java:32) at org.apache.tools.ant.Project.executeTargets(Project.java:1189) at org.eclipse.ant.internal.core.ant.InternalAntRunner.run(InternalAntRunner.java:662) at org.eclipse.ant.internal.core.ant.InternalAntRunner.run(InternalAntRunner.java:534) ... 19 more
Could you please reboot the mac instance and I'll run the tests again and see if the out of memory issues go away? I can't reproduce this issue on my mac test machine.
(In reply to comment #47) > Could you please reboot the mac instance and I'll run the tests again and see > if the out of memory issues go away? I can't reproduce this issue on my mac > test machine. If you are an admin user, 'shutdown -r now' will reboot for you. If not, 'sudo shutdown -r now' will work if you at least have the password.
Scott, I don't have any console/ssh access to the machines. I can only start builds on them via hudson. The webmasters are the only ones with access.
I've restarted the machine. -M.
Is this machine automatically logged into as the hudson user with no screen savers enabled? Now that I have vnc access to the machine, I reran the SWT tests and they passed. No out of heap space issues..very strange.
(In reply to comment #51) > Is this machine automatically logged into as the hudson user with no screen > savers enabled? No. -M.
Yesterday, I logged in via vnc as the hudson user on the mac slave. Then I disconnected my vnc session. I ran the Mac hudson job and all the tests ran as expected except for the p2 ones (separate issue). So I think the machine needs to remain logged in as the Hudson user, no screen saver, as is the case with our test machine here. https://hudson.eclipse.org/hudson/view/Eclipse%20and%20Equinox/job/eclipse-JUnit-mac/57/testReport/
Some of the jdt.debug tests are failing running because the java source is missing on the mac slave Could you download the latest java source that's referenced on the plan (Java for Mac OS X 10.6 Update 3) You install it like this http://tech.puredanger.com/2007/09/21/java-source-mac/ Of course, instead of 1.5.0, we should download 1.6.0 and store the src.jar here /System/Library/Frameworks/JavaVM.framework/Versions/1.6.0/Home You don't need to muck with Eclipse like the blog entry says, you just need to download and install the source jar. I can't download it because of legal reasons, I can only download VMs from our internal site :-) thanks
Also, can you also install proctools on this machine http://proctools.darwinports.com/ I'm getting the message that it can't find pkill when a mac build is launched by the upstream build.
Ok, I've installed the Java 1.5 update 10 pack from Apple(without an OS upgrade I can't install the 10.6 pkg). I've also installed the proctools tools. -M.
I tried a test build this morning and it failed. I think the reason is that you need to link /usr/local/bin/pkill to /usr/bin/pkill ln -s /usr/local/bin/pkill /usr/bin/pkill I can't do this because I'm not an administrator on the mac. I think the reason that hudson fails for on this is that a Linux slave where pkill is in one location trying to execute pkill on another machine were pkill somewhere else.
Done. -M.
Reopening. The existing mac hudson slave can't have Java 1.6 installed because of Apple hardware/software limitations. We need Java 1.6 to run our tests, otherwise thousands fail or do not run. In addition, it's not running 10.6 and that is the reference platform for 3.7. See bug 340602 for details. Please order a new Mac Mini that can run Java 1.6 and Mac OS X 10.6 so we can transition our builds to eclipse.org. Do I need to open a separate bug for the hardware request?
> Please order a new Mac Mini that can run Java 1.6 and Mac OS X 10.6 so we can > transition our builds to eclipse.org. Do I need to open a separate bug for the > hardware request? You can put in a friends of eclipse request if you want to get this done this year. Otherwise, I don't have budget input until 2012. Could this be solved by upgrading to a newer version of Mac OS X?
Possible. The Java updates for OS X 10.5 make it pretty clear that Java 1.6 is only 'supported' on 64bit CPUs. The OS X 10.6 Java update doesn't contain that warning, and the Snow Leopard (10.6) update requirements don't say that it's 64 bit only. -M.
Regarding comment #60 I've opened bug 340991 with a request for FOE disbursement.
(In reply to comment #61) > Possible. The Java updates for OS X 10.5 make it pretty clear that Java 1.6 is > only 'supported' on 64bit CPUs. The OS X 10.6 Java update doesn't contain that > warning, and the Snow Leopard (10.6) update requirements don't say that it's 64 > bit only. You may not need to buy a new Mac just yet. Java 6 on 10.5 is 64-bit only. That is, it's only built for x86_64. If it's a Core 2 Duo you got the 64-bit Java 6 update as well with the last software update. Try 'java -d64 -version' and see what you get. If it tells you 'can't run 64-bit' you should update to Mac OS X 10.6. On OS X 10.6, Java 6 is the only Java version available and is built for i386 and x86_64. A Mac Mini _should_ be able to run 10.6, but I'd double-check first. How old is it? What kind of processor is reported when you choose 'About This Mac' from the Apple menu?
regarding comment #63 regarding running java in 64 bit mode. With java -d64 version, I get "Cannot run Java in 64 bit mode, continuing in 32 bit mode". Regarding the About screen, it lists an Intel Core Duo. My understanding is that Java 6 is not supported on this older hardware from Apple.
Created attachment 191965 [details] about page from current hudson mac slave
(In reply to comment #64) > Regarding the About screen, it lists an Intel Core Duo. My understanding is > that Java 6 is not supported on this older hardware from Apple. That's right -- Core 2 Duo or better is needed to run 64-bit code. But, Java 6 on Mac OS X 10.6 comes in a 32-bit version as well, so you could just update the machine to 10.6 and run everything that way. 10.6 just requires Intel, and it's only $29.
Matt/Denis, could you please update this machine to 10.6 to see if this resolves the problem. Ironically, if we had updated to 10.6 as described in comment 5 we wouldn't have these problems three months later :-) Sometimes spending some money on up to date software and hardware results saves a tremendous amount of people dollars. I wasted a lot of time troubleshooting why Java 6 wasn't working on the Mac slave because it manifested itself as failing tests due to network errors where in fact it was Java 6 executables that were for the wrong architecture.
(In reply to comment #67) > Matt/Denis, could you please update this machine to 10.6 to see if this > resolves the problem. This bug is about provisioning a Mac machine for your tests. We did that months ago. You've opened bug 340602 to discuss the java incompatibility, so I'm not sure why we're discussing the same issue in two bugs. > Ironically, if we had updated to 10.6 as described in comment 5 we wouldn't > have these problems three months later Right. And you might not have had a Mac to tinker with for three months. Since the webmaster is not an infinite pool of resources, we're stuck with the notion of compromise. You agreed to proceeding this way in comment 10.
I realize this that I agreed to this even though it wasn't the right operating system for 3.7 tests. When you don't have anything, you have to proceed with what you have :-) I'm just frustrated about having to continually beg for resources to do my job. Moving the builds to eclipse.org is a 3.7 plan item and yet there is absolutely no funding to make this a reality. It's not personal. I'm just very frustrated how everyone in the community is happy to take the our builds but not willing to contribute more hardware resources or money so committers like me don't have to waste so much time begging for hardware and configuring old machines.
I sympathize with your frustration. I do. But if this move does not include the hardware and IT resources you currently have, then perhaps the community at large should be prepared to compromise? Case in point: when IBM decided to no longer donate language packs for Eclipse releases, the community stepped up to fill in the void -- Aptana with code, IBM and the Foundation with a project called Babel, and companies like Actuate and Adobe with large translation contributions in addition to the numerous individual translations performed by individuals in our community. In the end, the Babel language packs may or may not be of the same quality as those produced by IBM, but for the most part they are 'good enough' and the community has compromised. In this specific case, the webmasters will upgrade the Mac slave and do whatever we can in order to preserve the quality of the tests you have today, but I believe it may be good to realign our expectations to face the reality of this move. Do these tests need to be run here, at the Foundation? Can the community run the tests on their compatible hardware?
Denis, you raise many good points. However I don't think the language packs are quite as important as running JUnit tests. Nobody ever said, let's do less testing to make your software better :-( The Eclipse and Equinox projects have a reputation for excellence and I would hate to see that reputation suffer due to a lack of test resources. In the grand scheme of things, a few servers are cheap compared releasing software with serious bugs in it and tarnishing the Eclipse brand. But I don't have the money to buy test servers and nobody seems to be lining up to buy us some. Yes, IBM has (old) hardware that we use to run the tests. We are trying to determine a way to donate some to the foundation but this process requires accountants and lawyers and I don't know what the end result will be. As I have said before, I get to give away free software every day, but giving away free hardware is not something I can do without prior approval. For many years, other companies have complained that the Eclipse build process is not open and other committers cannot start the build, or look at the test machines. In addition, Mike asked me when he happened to be at our office in November what the timeframe was to have the build completely moved to eclipse.org. So I always understood it to be something that both the community and the foundation felt was a priority. I will raise the issue with our PMC that moving to the build to eclipse.org may require a realignment of expectations.
