Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.
Bug 370434 - network problems on hudson-slave5
Summary: network problems on hudson-slave5
Status: RESOLVED FIXED
Alias: None
Product: Community
Classification: Eclipse Foundation
Component: CI-Jenkins (show other bugs)
Version: unspecified   Edit
Hardware: All All
: P3 normal (vote)
Target Milestone: ---   Edit
Assignee: Eclipse Webmaster CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-02-02 09:02 EST by Nicolas Bros CLA
Modified: 2012-02-03 04:05 EST (History)
1 user (show)

See Also:


Attachments
testInstall.sh (992 bytes, text/plain)
2012-02-02 09:02 EST, Nicolas Bros CLA
no flags Details
testInstall.sh (with logging enabled) (1.27 KB, text/plain)
2012-02-02 10:32 EST, Nicolas Bros CLA
no flags Details
log (26.16 KB, text/plain)
2012-02-02 10:34 EST, Nicolas Bros CLA
no flags Details
log hudson-slave5.txt (timeouts) (55.32 KB, text/plain)
2012-02-02 11:27 EST, Nicolas Bros CLA
no flags Details
log hudson-slave6.txt (succeeds) (198.48 KB, text/plain)
2012-02-02 11:28 EST, Nicolas Bros CLA
no flags Details
testInstall.sh (with proxy settings) (1.68 KB, text/plain)
2012-02-03 04:05 EST, Nicolas Bros CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Nicolas Bros CLA 2012-02-02 09:02:35 EST
Created attachment 210444 [details]
testInstall.sh

My Hudson job gets stuck on a p2 install, because it apparently cannot access the network. I get this error logged over and over, with no apparent progress:

-----
Feb 2, 2012 8:44:45 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection timed out
Feb 2, 2012 8:44:45 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry
INFO: Retrying request
-----

To reproduce, you can try to execute the attached script on hudson-slave5.
Comment 1 Eclipse Webmaster CLA 2012-02-02 09:59:44 EST
The only error the proxy is logging is a 403 attempting to reach:  
http://download.cloudsmith.com/buckminster/external-3.7/

Beyond that the network seems to be fine(I can reach Google and other Eclipse.org sites).

Is there anyway to get some better visibility on what network operation(url etc.) failing?

-M.
Comment 2 Nicolas Bros CLA 2012-02-02 10:32:31 EST
Created attachment 210453 [details]
testInstall.sh (with logging enabled)

I have enabled logging in this script.
Comment 3 Nicolas Bros CLA 2012-02-02 10:34:05 EST
Created attachment 210454 [details]
log

Here is the log I get when running the script with logging enabled on hudson-slave5. The error seems to be a "Connection timed out" when accessing "download.cloudsmith.com:80"
Comment 4 Eclipse Webmaster CLA 2012-02-02 10:43:17 EST
I get the same response from Slave5 as I do on my local+remote machines:

>wget download.cloudsmith.com
--2012-02-02 10:42:36--  http://download.cloudsmith.com/
Resolving download.cloudsmith.com... 184.72.237.96
Connecting to download.cloudsmith.com|184.72.237.96|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2012-02-02 10:42:36 ERROR 403: Forbidden.

The network seems to be fine, but Cloudsmith is clearly having problems.

-M.
Comment 5 Nicolas Bros CLA 2012-02-02 11:25:51 EST
I don't think the Error 403 on the update site root is the problem. I get the same error from my browser, and it works when I try to execute the same install script on another slave or on build.eclipse.org. The installation only fails on hudson-slave5.
So, I still think this is an issue with hudson-slave5.
Comment 6 Nicolas Bros CLA 2012-02-02 11:27:08 EST
Created attachment 210458 [details]
log hudson-slave5.txt (timeouts)

Here is the log on hudson-slave5, where it timeouts.
Comment 7 Nicolas Bros CLA 2012-02-02 11:28:31 EST
Created attachment 210460 [details]
log hudson-slave6.txt (succeeds)

Here is the log on hudson-slave6, where the installation succeeds without any timeout.
Comment 8 Eclipse Webmaster CLA 2012-02-02 16:44:52 EST
Interesting.  It seems like slave5 isn't setting the proxy correctly when the director runs.  

I've done my best to 'push' the proxy into the test script (by adding some -D flags) to no effect.

I also cross checked with slave6 and both 'correctly' set the JVM_OPTS and JAVA_ARGS variables(they call the same global script).

If I look in /proc while the test is running, the environment looks correct.

Not really sure where to go from here.  Either Java is ignoring the environment vars(or overriding them) or the director is.

Any suggestions?

-M.
Comment 9 Nicolas Bros CLA 2012-02-03 04:03:56 EST
Yes, you are right, the p2 director is apparently ignoring proxy settings (at least in hudson-slave5, since it works on other slaves).

I have found a way to make it respect the settings is to put them in an Eclipse configuration file. I have documented this on:
http://wiki.eclipse.org/Hudson#Configuring_a_proxy_for_the_p2_director

Thank you for pointing me in the right direction.
Comment 10 Nicolas Bros CLA 2012-02-03 04:05:20 EST
Created attachment 210498 [details]
testInstall.sh (with proxy settings)

For reference, here is the modified script with the proxy settings set in the configuration file.