This Bugzilla instance is deprecated, and most Eclipse projects now use GitHub or Eclipse GitLab. Please see the deprecation plan for details.
Bug 521639 - Mactests2 is extremely slow
Summary: Mactests2 is extremely slow
Status: CLOSED FIXED
Alias: None
Product: Community
Classification: Eclipse Foundation
Component: CI-Jenkins (show other bugs)
Version: unspecified   Edit
Hardware: PC Mac OS X
: P3 blocker (vote)
Target Milestone: ---   Edit
Assignee: CI Admin Inbox CLA
QA Contact:
URL:
Whiteboard:
Keywords:
: 521771 (view as bug list)
Depends on:
Blocks:
 
Reported: 2017-08-31 00:35 EDT by Jay Arthanareeswaran CLA
Modified: 2017-10-10 09:02 EDT (History)
8 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Jay Arthanareeswaran CLA 2017-08-31 00:35:56 EDT
Mactests2 attached to shared hoop is offline. The tests are unavaibale. Build I20170830-0605 doesn't have the tests on Mac platform.
Comment 1 Jay Arthanareeswaran CLA 2017-08-31 01:52:48 EDT
Looks like it was intermittent. Sravan has started the tests again for the M build. Feel free to close this bug.
Comment 2 Mikaël Barbero CLA 2017-08-31 03:35:57 EDT
AFAICT, there is still something wrong with mac-tests2.  The ethernet interface keeps flapping. We're investigating.
Comment 3 Dani Megert CLA 2017-08-31 03:55:22 EDT
(In reply to Mikaël Barbero from comment #2)
> AFAICT, there is still something wrong with mac-tests2.  The ethernet
> interface keeps flapping. We're investigating.

Is that also the cause for DMG and signing issues? Is the same machine used?
Comment 4 Mikaël Barbero CLA 2017-08-31 03:56:44 EDT
(In reply to Dani Megert from comment #3)
> (In reply to Mikaël Barbero from comment #2)
> > AFAICT, there is still something wrong with mac-tests2.  The ethernet
> > interface keeps flapping. We're investigating.
> 
> Is that also the cause for DMG and signing issues? 

I'm not 100% sure.

> Is the same machine used?

Yes, we only have one intel mac machine (mac-tests2). The other (mac-tests) is an old ppc.
Comment 5 Dani Megert CLA 2017-08-31 09:05:31 EDT
(In reply to Mikaël Barbero from comment #2)
> AFAICT, there is still something wrong with mac-tests2.  The ethernet
> interface keeps flapping. We're investigating.

Any update? This is really becoming a show stopper.
Comment 6 Denis Roy CLA 2017-08-31 09:19:01 EDT
We'll talk about other solutions shortly. We may just buy a new Mac and retire the old POS. Even mac-tests2 is aging.
Comment 7 Jay Arthanareeswaran CLA 2017-09-01 10:24:46 EDT
For the second consecutive time today, the Mac tests failed today. Unfortunately one of them turned out to be our RC3 candidate. Is there any chance this can be fixed before we can promote the build?
Comment 8 Mikaël Barbero CLA 2017-09-01 11:57:38 EDT
It fails in which way? The UI tests or the signing/dmg packaging?

As said in comment 6, we are working on other solutions, but nothing can happen before next week.
Comment 9 Mikaël Barbero CLA 2017-09-01 14:54:40 EDT
It seems that there is no more signing failure, right? From the today's logs, I see 17 over 17 successful signing and 9 over 9 successful dmg creations.
Comment 10 Sravan Kumar Lakkimsetti CLA 2017-09-02 04:49:15 EDT
*** Bug 521771 has been marked as a duplicate of this bug. ***
Comment 11 Sravan Kumar Lakkimsetti CLA 2017-09-02 04:58:07 EDT
(In reply to Mikaël Barbero from comment #9)
> It seems that there is no more signing failure, right? From the today's
> logs, I see 17 over 17 successful signing and 9 over 9 successful dmg
> creations.

There are no signing failures now. We do need this machine for tests. Currently execution of tests is extremely slow. till last week the test execution used to take about 4-5 hours. Now not even 10% tests completed after 6 hours. https://hudson.eclipse.org/shared/job/ep47M-unit-mac64/23/console

We need to get this fixed so that we have a re;ease with proper test results.

We will do a RC4 after this problem is resolved.
Comment 12 Mikaël Barbero CLA 2017-09-04 03:44:22 EDT
Can you please restart a test build. I've found some wrong proxy info in the slave configuration on the shared instance. It may be the root of the slowness. 

In the meantime, I am configuring a new slave. Stay stuned.
Comment 13 Sravan Kumar Lakkimsetti CLA 2017-09-04 03:50:49 EDT
(In reply to Mikaël Barbero from comment #12)
> Can you please restart a test build. I've found some wrong proxy info in the
> slave configuration on the shared instance. It may be the root of the
> slowness. 
> 
> In the meantime, I am configuring a new slave. Stay stuned.

I started a build an hour back. That should complete in an hour. We should get results by that time
Comment 14 Mikaël Barbero CLA 2017-09-04 03:51:46 EDT
I don't see any build running on https://hudson.eclipse.org/shared/
Comment 15 Sravan Kumar Lakkimsetti CLA 2017-09-04 04:09:05 EDT
(In reply to Mikaël Barbero from comment #14)
> I don't see any build running on https://hudson.eclipse.org/shared/

I triggered a test job also now. The one I triggered earlier is full build https://hudson.eclipse.org/releng/job/I-build-Photon/122/console. This will trigger tests after the build is complete
Comment 16 Mikaël Barbero CLA 2017-09-04 05:34:24 EDT
I've noticed a bunch of proxy configuration in the platform build scripts:

production/miscToolsAndNotes/proxyRelated/getSettings.sh
production/miscToolsAndNotes/proxyRelated/platformSpecific.properties
production/testScripts/configuration/sdk.tests/testConfigs/macmini/platformSpecific.properties
production/testScripts/configuration/sdk.tests/testConfigs/macosx/platformSpecific.properties

various in production/sdk/bootstrap/

Now that the mac machine does not need proxies anymore, the proxy should be disabled when something runs on the mac machine. Please check how it could be done in your build configu.
Comment 17 Sravan Kumar Lakkimsetti CLA 2017-09-04 05:55:57 EDT
(In reply to Mikaël Barbero from comment #16)
> I've noticed a bunch of proxy configuration in the platform build scripts:
> 
> production/miscToolsAndNotes/proxyRelated/getSettings.sh
> production/miscToolsAndNotes/proxyRelated/platformSpecific.properties
> production/testScripts/configuration/sdk.tests/testConfigs/macmini/
> platformSpecific.properties
> production/testScripts/configuration/sdk.tests/testConfigs/macosx/
> platformSpecific.properties
> 
> various in production/sdk/bootstrap/
> 
> Now that the mac machine does not need proxies anymore, the proxy should be
> disabled when something runs on the mac machine. Please check how it could
> be done in your build configu.

Ok I will make the necessary changes.
Comment 18 Sravan Kumar Lakkimsetti CLA 2017-09-04 05:59:32 EDT
(In reply to Mikaël Barbero from comment #16)
> I've noticed a bunch of proxy configuration in the platform build scripts:
> 
> production/miscToolsAndNotes/proxyRelated/getSettings.sh
> production/miscToolsAndNotes/proxyRelated/platformSpecific.properties
> production/testScripts/configuration/sdk.tests/testConfigs/macmini/
> platformSpecific.properties
> production/testScripts/configuration/sdk.tests/testConfigs/macosx/
> platformSpecific.properties
> 
> various in production/sdk/bootstrap/
> 
> Now that the mac machine does not need proxies anymore, the proxy should be
> disabled when something runs on the mac machine. Please check how it could
> be done in your build configu.

Can we remove the proxy configuration for all the platforms?
Comment 19 Mikaël Barbero CLA 2017-09-04 06:01:23 EDT
While it's would be possible (as we should have removed proxy requirement from all build-related machines), I don't think it's wise to do it now. Let's fix the issue with the mac first. We can always tackle other changes later on.
Comment 20 Sravan Kumar Lakkimsetti CLA 2017-09-04 06:10:39 EDT
(In reply to Mikaël Barbero from comment #19)
> While it's would be possible (as we should have removed proxy requirement
> from all build-related machines), I don't think it's wise to do it now.
> Let's fix the issue with the mac first. We can always tackle other changes
> later on.

Ok I will make the necessary changes temporarily.
Comment 21 Sravan Kumar Lakkimsetti CLA 2017-09-04 07:17:19 EDT
(In reply to Mikaël Barbero from comment #19)
> While it's would be possible (as we should have removed proxy requirement
> from all build-related machines), I don't think it's wise to do it now.
> Let's fix the issue with the mac first. We can always tackle other changes
> later on.

I have created a eclipse.platform.releng.aggregator-23052293ff20dee25a038e0e17d9c9c2100db637.zip in my home directory on build.eclipse.org. This needs to be copied to /shared/eclipse/builds/4M/siteDir/eclipse/downloads/drops4/M20170901-1220/. and /home/data/httpd/download.eclipse.org/eclipse/downloads/drops4/M20170901-1220/.

Earlier I used do this with my shell account after the project merge it looks like I lost my rights here. Can you please help me in copying this file to both locations? This will remove proxy configurations for mac machine on build M20170901-1220
Comment 22 Mikaël Barbero CLA 2017-09-04 07:53:01 EDT
(In reply to Sravan Kumar Lakkimsetti from comment #21)
> (In reply to Mikaël Barbero from comment #19)
> > While it's would be possible (as we should have removed proxy requirement
> > from all build-related machines), I don't think it's wise to do it now.
> > Let's fix the issue with the mac first. We can always tackle other changes
> > later on.
> 
> I have created a
> eclipse.platform.releng.aggregator-23052293ff20dee25a038e0e17d9c9c2100db637.
> zip in my home directory on build.eclipse.org. This needs to be copied to
> /shared/eclipse/builds/4M/siteDir/eclipse/downloads/drops4/M20170901-1220/.
> and
> /home/data/httpd/download.eclipse.org/eclipse/downloads/drops4/M20170901-
> 1220/.
> 
> Earlier I used do this with my shell account after the project merge it
> looks like I lost my rights here. Can you please help me in copying this
> file to both locations? This will remove proxy configurations for mac
> machine on build M20170901-1220

Done.
Comment 23 Sravan Kumar Lakkimsetti CLA 2017-09-04 07:56:07 EDT
I will retrigger the tests now
Comment 24 Sravan Kumar Lakkimsetti CLA 2017-09-04 07:57:51 EDT
Triggered the tests now https://hudson.eclipse.org/shared/job/ep47M-unit-mac64/25/
Comment 25 Mikaël Barbero CLA 2017-09-04 11:53:47 EDT
I've started a test build on a new machine https://hudson.eclipse.org/shared/job/WEBMASTER-ep47M-unit-mac64 (it's a copy of your job ep47M-unit-mac64 ). I've started it with the same parameters as https://hudson.eclipse.org/shared/job/ep47M-unit-mac64/26/ 

Let's see how it goes there.
Comment 26 Sravan Kumar Lakkimsetti CLA 2017-09-05 02:35:36 EDT
(In reply to Mikaël Barbero from comment #25)
> I've started a test build on a new machine
> https://hudson.eclipse.org/shared/job/WEBMASTER-ep47M-unit-mac64 (it's a
> copy of your job ep47M-unit-mac64 ). I've started it with the same
> parameters as https://hudson.eclipse.org/shared/job/ep47M-unit-mac64/26/ 
> 
> Let's see how it goes there.

The tests looks fine. There still one problem though cvstests.properties file is not created on the new machine. Here is the message 

02:08:43  logNoExistence:
02:08:43       [echo] cvs properties file was not found in this environment. 
02:08:43       [echo]    expected file: /Users/hudsonbuild/cvstest.properties 

This needs to copied from the old machine.

Can you please let us know where this machine is created?
Comment 27 Sravan Kumar Lakkimsetti CLA 2017-09-05 02:39:05 EDT
I also started a full test results with my job. This will run the tests and update the test results page at http://download.eclipse.org/eclipse/downloads/drops4/M20170901-1220/testResults.php

This will give me the information about missing tests. Lets see how it goes.

Thanks
Sravan
Comment 28 Mikaël Barbero CLA 2017-09-05 02:48:50 EDT
(In reply to Sravan Kumar Lakkimsetti from comment #26)
> 02:08:43  logNoExistence:
> 02:08:43       [echo] cvs properties file was not found in this environment. 
> 02:08:43       [echo]    expected file:
> /Users/hudsonbuild/cvstest.properties 
> 
> This needs to copied from the old machine.
> 
> Can you please let us know where this machine is created?

/Users/hudsonbuild folder does not exist. The user running the build is genie.shared and its home folder is /Users/genie.shared. Is it something to be changed in your build script?
Comment 29 Mikaël Barbero CLA 2017-09-05 02:53:15 EDT
(In reply to Mikaël Barbero from comment #28)
> (In reply to Sravan Kumar Lakkimsetti from comment #26)
> > 02:08:43  logNoExistence:
> > 02:08:43       [echo] cvs properties file was not found in this environment. 
> > 02:08:43       [echo]    expected file:
> > /Users/hudsonbuild/cvstest.properties 
> > 
> > This needs to copied from the old machine.
> > 
> > Can you please let us know where this machine is created?
> 
> /Users/hudsonbuild folder does not exist. The user running the build is
> genie.shared and its home folder is /Users/genie.shared. Is it something to
> be changed in your build script?

I've copied the file from the old machine to /Users/genie.shared/cvstest.properties
Comment 30 Sravan Kumar Lakkimsetti CLA 2017-09-05 03:02:03 EDT
(In reply to Mikaël Barbero from comment #29)
> (In reply to Mikaël Barbero from comment #28)
> > (In reply to Sravan Kumar Lakkimsetti from comment #26)
> > > 02:08:43  logNoExistence:
> > > 02:08:43       [echo] cvs properties file was not found in this environment. 
> > > 02:08:43       [echo]    expected file:
> > > /Users/hudsonbuild/cvstest.properties 
> > > 
> > > This needs to copied from the old machine.
> > > 
> > > Can you please let us know where this machine is created?
> > 
> > /Users/hudsonbuild folder does not exist. The user running the build is
> > genie.shared and its home folder is /Users/genie.shared. Is it something to
> > be changed in your build script?
> 
> I've copied the file from the old machine to
> /Users/genie.shared/cvstest.properties

the file name is provided by platformSpecific.properties. I need to change it.
Comment 31 Sravan Kumar Lakkimsetti CLA 2017-09-06 02:40:39 EDT
I ran Mac tests using the new machine, all tests ran but there are about 50 tests failing. Those needs to be investigated. I asked Lakshmi to look into that should hearing from her shortly
Comment 32 Lakshmi P Shanmugam CLA 2017-09-06 03:03:40 EDT
(In reply to Sravan Kumar Lakkimsetti from comment #31)
> I ran Mac tests using the new machine, all tests ran but there are about 50
> tests failing. Those needs to be investigated. I asked Lakshmi to look into
> that should hearing from her shortly

I'm unable to view the test results page as download.eclipse.org is down. Will try again after sometime.
Comment 33 Sravan Kumar Lakkimsetti CLA 2017-09-06 03:48:31 EDT
(In reply to Lakshmi Shanmugam from comment #32)
> (In reply to Sravan Kumar Lakkimsetti from comment #31)
> > I ran Mac tests using the new machine, all tests ran but there are about 50
> > tests failing. Those needs to be investigated. I asked Lakshmi to look into
> > that should hearing from her shortly
> 
> I'm unable to view the test results page as download.eclipse.org is down.
> Will try again after sometime.

You can see the test results here https://hudson.eclipse.org/shared/job/ep47M-unit-mac64/lastCompletedBuild/testReport/

and here
https://hudson.eclipse.org/shared/job/ep48I-unit-mac64/lastCompletedBuild/testReport/
Comment 34 Mikaël Barbero CLA 2017-09-06 03:49:43 EDT
and download.eclipse.org is up again, in case you need it
Comment 35 Lakshmi P Shanmugam CLA 2017-09-06 06:03:28 EDT
From the screenshots [1] & [2], the machine was locked and screensaver was showing while the tests were running. This caused the e4.ui.tests to fail. Not sure what other tests were affected by this.
We need to disable locking & screensaver when the tests are running.

[1] - http://download.eclipse.org/eclipse/downloads/drops4/M20170901-1220/testresults/ep47M-unit-mac64_macosx.cocoa.x86_64_8.0/org.eclipse.jface.tests.internal.databinding.swt.ControlObservableValueTest.testObserveFocus.png
[2] - http://download.eclipse.org/eclipse/downloads/drops4/M20170901-1220/testresults/ep47M-unit-mac64_macosx.cocoa.x86_64_8.0/org.eclipse.swt.tests.junit.Test_org_eclipse_swt_widgets_Display.test_setCursorLocationII.png
Comment 36 Lakshmi P Shanmugam CLA 2017-09-06 06:06:47 EDT
(In reply to Lakshmi Shanmugam from comment #35)

> [2] -
> http://download.eclipse.org/eclipse/downloads/drops4/M20170901-1220/testresults/ep47M-unit-mac64_macosx.cocoa.x86_64_8.0/org.eclipse.swt.tests.junit.Test_org_eclipse_swt_widgets_Display.test_setCursorLocationII.png
> 

We also need to close any open pops & windows before running the tests.
Comment 37 Mikaël Barbero CLA 2017-09-06 06:09:27 EDT
I've deactivated the screensaver and closed all the popups
Comment 38 Sravan Kumar Lakkimsetti CLA 2017-09-07 04:25:29 EDT
The tests are running fine except for CVS.

During the CVS tests we try to contact hipp10.eclipse.org. CVS tests are not able to connect to hipp10 machine. Not sure if whether we can access hipp10 from the new test machine though.
Comment 39 Mikaël Barbero CLA 2017-09-07 04:35:09 EDT
(In reply to Sravan Kumar Lakkimsetti from comment #38)
> The tests are running fine except for CVS.
> 
> During the CVS tests we try to contact hipp10.eclipse.org. CVS tests are not
> able to connect to hipp10 machine. Not sure if whether we can access hipp10
> from the new test machine though.

hipp10 is not reachable from outside of the LAN and the new mac machine is outside. So won't be able to reach out to hipp10. Is it a blocker?
Comment 40 Sravan Kumar Lakkimsetti CLA 2017-09-07 05:22:02 EDT
(In reply to Mikaël Barbero from comment #39)
> (In reply to Sravan Kumar Lakkimsetti from comment #38)
> > The tests are running fine except for CVS.
> > 
> > During the CVS tests we try to contact hipp10.eclipse.org. CVS tests are not
> > able to connect to hipp10 machine. Not sure if whether we can access hipp10
> > from the new test machine though.
> 
> hipp10 is not reachable from outside of the LAN and the new mac machine is
> outside. So won't be able to reach out to hipp10. Is it a blocker?

No its not a blocker. We can release without these tests.
Comment 41 Mikaël Barbero CLA 2017-09-13 09:21:37 EDT
Closing this bug regarding the slowness of mac-tests2. Please open a new bug about the CVS tests. 

Thanks.
Comment 42 Dani Megert CLA 2017-10-10 08:47:49 EDT
(In reply to Mikaël Barbero from comment #41)
> Closing this bug

It's still open ;-)