Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 365921

Summary: Make hudson https urls work reliable
Product: [Technology] Hudson Reporter: Dennis Huebner <dennis.huebner>
Component: CoreAssignee: Winston Prakash <winston.prakash>
Status: RESOLVED WORKSFORME QA Contact:
Severity: major    
Priority: P3 CC: adolfosbh, mygwaymark, pwebster, sbouchet, webmaster
Version: unspecified   
Target Milestone: ---   
Hardware: PC   
OS: Mac OS X - Carbon (unsup.)   
Whiteboard:

Description Dennis Huebner CLA 2011-12-07 11:27:11 EST
As suggested by web masters I changed URL's from:
-Dxpand.p2.repository=file:/opt/users/hudsonbuild/.hudson/jobs/Xpand-nightly-HEAD/lastSuccessful/archive/xpand.p2.repository/
-Dxtext.p2.repository=file:/opt/users/hudsonbuild/.hudson/jobs/Xtext-nightly-HEAD/lastSuccessful/archive/xtext.p2.repository/

to:
-Dxpand.p2.repository=https://hudson.eclipse.org/hudson/job/Xpand-nightly-HEAD/lastSuccessfulBuild/artifact/xpand.p2.repository/
-Dxtext.p2.repository=https://hudson.eclipse.org/hudson/job/Xtext-nightly-HEAD/lastSuccessfulBuild/artifact/xtext.p2.repository/

Running Xtext-nightly-HEAD Job which depends on artifacts produced by MWE and Xpand jobs.
Jobs used as mwe.p2.repository and xpand.p2.repository were disabled whole the time, so were unchanged.

Take one (10:52 am) Xpand repository not found :

+ echo build.id=201112071052
+ :
[Xtext-nightly-HEAD] $ /shared/common/jdk-1.5.0-22.x86_64/bin/java -Dbuckminster.output.root=/opt/users/hudsonbuild/workspace/Xtext-nightly-HEAD/buckminster.output -Dbuckminster.temp.root=/opt/users/hudsonbuild/workspace/Xtext-nightly-HEAD/buckminster.temp -Dcheckout.location=/opt/users/hudsonbuild/workspace/Xtext-nightly-HEAD/org.eclipse.xtext.git -Dmwe.p2.repository=https://hudson.eclipse.org/hudson/job/MWE-nightly-HEAD/468/artifact/mwe.p2.repository/ -Dxpand.p2.repository=https://hudson.eclipse.org/hudson/job/Xpand-nightly-HEAD/lastSuccessfulBuild/artifact/xpand.p2.repository/ -jar /shared/common/buckminster-3.6/plugins/org.eclipse.equinox.launcher_1.1.1.R36x_v20101122_1400.jar -application org.eclipse.buckminster.cmdline.headless -data /opt/users/hudsonbuild/workspace/Xtext-nightly-HEAD/buildroot/buckminster.workspace --loglevel info -S /opt/users/hudsonbuild/workspace/Xtext-nightly-HEAD/commands.txt
INFO:  setpref 'targetPlatformPath=/opt/users/hudsonbuild/workspace/Xtext-nightly-HEAD/buildroot/target.platform'
WARN:  Target platform directory '/opt/users/hudsonbuild/workspace/Xtext-nightly-HEAD/buildroot/target.platform' does not exist and will be created
INFO:  import '-Declipse.download=file:/home/data/httpd/download.eclipse.org' '/opt/users/hudsonbuild/workspace/Xtext-nightly-HEAD/org.eclipse.xtext.git/releng/org.eclipse.xtext.releng/releng/xtext-platform-Galileo.mspec'
ERROR   [0002] : No repository found at https://hudson.eclipse.org/hudson/job/Xpand-nightly-HEAD/lastSuccessfulBuild/artifact/xpand.p2.repository/.
ERROR: No repository found at https://hudson.eclipse.org/hudson/job/Xpand-nightly-HEAD/lastSuccessfulBuild/artifact/xpand.p2.repository/.
ERROR: Errors and Warnings
org.eclipse.core.runtime.CoreException: Errors and Warnings
	at org.eclipse.buckminster.runtime.BuckminsterException.wrap(BuckminsterException.java:96)
	at org.eclipse.buckminster.core.materializer.MaterializationJob.internalRun(MaterializationJob.java:149)
	at org.eclipse.buckminster.core.materializer.MaterializationJob.run(MaterializationJob.java:125)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)


Take two (10:58 am) Nothing changed, just rerun the build. MWE repository not found:

+ echo build.id=201112071058
+ :
[Xtext-nightly-HEAD] $ /shared/common/jdk-1.5.0-22.x86_64/bin/java -Dbuckminster.output.root=/opt/users/hudsonbuild/workspace/Xtext-nightly-HEAD/buckminster.output -Dbuckminster.temp.root=/opt/users/hudsonbuild/workspace/Xtext-nightly-HEAD/buckminster.temp -Dcheckout.location=/opt/users/hudsonbuild/workspace/Xtext-nightly-HEAD/org.eclipse.xtext.git -Dmwe.p2.repository=https://hudson.eclipse.org/hudson/job/MWE-nightly-HEAD/468/artifact/mwe.p2.repository/ -Dxpand.p2.repository=https://hudson.eclipse.org/hudson/job/Xpand-nightly-HEAD/lastSuccessfulBuild/artifact/xpand.p2.repository/ -jar /shared/common/buckminster-3.6/plugins/org.eclipse.equinox.launcher_1.1.1.R36x_v20101122_1400.jar -application org.eclipse.buckminster.cmdline.headless -data /opt/users/hudsonbuild/workspace/Xtext-nightly-HEAD/buildroot/buckminster.workspace --loglevel info -S /opt/users/hudsonbuild/workspace/Xtext-nightly-HEAD/commands.txt
INFO:  setpref 'targetPlatformPath=/opt/users/hudsonbuild/workspace/Xtext-nightly-HEAD/buildroot/target.platform'
WARN:  Target platform directory '/opt/users/hudsonbuild/workspace/Xtext-nightly-HEAD/buildroot/target.platform' does not exist and will be created
INFO:  import '-Declipse.download=file:/home/data/httpd/download.eclipse.org' '/opt/users/hudsonbuild/workspace/Xtext-nightly-HEAD/org.eclipse.xtext.git/releng/org.eclipse.xtext.releng/releng/xtext-platform-Galileo.mspec'
ERROR   [0002] : No repository found at https://hudson.eclipse.org/hudson/job/MWE-nightly-HEAD/468/artifact/mwe.p2.repository/.
ERROR: No repository found at https://hudson.eclipse.org/hudson/job/MWE-nightly-HEAD/468/artifact/mwe.p2.repository/.
ERROR: Errors and Warnings
org.eclipse.core.runtime.CoreException: Errors and Warnings
	at org.eclipse.buckminster.runtime.BuckminsterException.wrap(BuckminsterException.java:96)
	at org.eclipse.buckminster.core.materializer.MaterializationJob.internalRun(MaterializationJob.java:149)
	at org.eclipse.buckminster.core.materializer.MaterializationJob.run(MaterializationJob.java:125)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54)

Take tree (11:00 am)  Nothing changed, just rerun the build. Works!:

+ echo build.id=201112071100
+ :
[Xtext-nightly-HEAD] $ /shared/common/jdk-1.5.0-22.x86_64/bin/java -Dbuckminster.output.root=/opt/users/hudsonbuild/workspace/Xtext-nightly-HEAD/buckminster.output -Dbuckminster.temp.root=/opt/users/hudsonbuild/workspace/Xtext-nightly-HEAD/buckminster.temp -Dcheckout.location=/opt/users/hudsonbuild/workspace/Xtext-nightly-HEAD/org.eclipse.xtext.git -Dmwe.p2.repository=https://hudson.eclipse.org/hudson/job/MWE-nightly-HEAD/468/artifact/mwe.p2.repository/ -Dxpand.p2.repository=https://hudson.eclipse.org/hudson/job/Xpand-nightly-HEAD/lastSuccessfulBuild/artifact/xpand.p2.repository/ -jar /shared/common/buckminster-3.6/plugins/org.eclipse.equinox.launcher_1.1.1.R36x_v20101122_1400.jar -application org.eclipse.buckminster.cmdline.headless -data /opt/users/hudsonbuild/workspace/Xtext-nightly-HEAD/buildroot/buckminster.workspace --loglevel info -S /opt/users/hudsonbuild/workspace/Xtext-nightly-HEAD/commands.txt
INFO:  setpref 'targetPlatformPath=/opt/users/hudsonbuild/workspace/Xtext-nightly-HEAD/buildroot/target.platform'
WARN:  Target platform directory '/opt/users/hudsonbuild/workspace/Xtext-nightly-HEAD/buildroot/target.platform' does not exist and will be created
INFO:  import '-Declipse.download=file:/home/data/httpd/download.eclipse.org' '/opt/users/hudsonbuild/workspace/Xtext-nightly-HEAD/org.eclipse.xtext.git/releng/org.eclipse.xtext.releng/releng/xtext-platform-Galileo.mspec'
INFO:  Resetting target platform Directory /opt/users/hudsonbuild/workspace/Xtext-nightly-HEAD/buildroot/target.platform
INFO:  Import complete.
INFO:  resolve '/opt/users/hudsonbuild/workspace/Xtext-nightly-HEAD/org.eclipse.xtext.git/releng/org.eclipse.xtext.releng/releng/xtext.cquery'
INFO:  Import complete.
INFO:  build

How to set up my Xtext Job to get artifacts from jobs it depends on ? reliable?
Comment 1 Eclipse Webmaster CLA 2011-12-07 14:12:17 EST
I suspect this is really an issue with Hudson itself.  Assigning to the Hudson team for comment.

-M.
Comment 2 Dennis Huebner CLA 2011-12-12 10:11:42 EST
(In reply to comment #1)
> I suspect this is really an issue with Hudson itself.  Assigning to the Hudson
> team for comment.
> 
> -M.

Now I'm some times getting  502 Bad Gateway accessing https URL's :

INFO:  importtargetdefinition '-DuseMirror=false' '-A' '/opt/users/hudsonbuild/.hudson/jobs/Xtext-test/workspace//org.eclipse.xtext.git/releng/org.eclipse.xtext.releng/releng/tests/tests-galileo.target'
INFO:  System property http.proxyHost has been set to proxy.eclipse.org -Dhttp.proxyPort=9898 -Dhttps.proxyHost=proxy.eclipse.org -Dhttps.proxyPort=9898 -Dhttp.nonProxyHosts="*.eclipse.org" -Dhttps.nonProxyHosts="*.eclipse.org" by an external source. This value will be overwritten using the values from the preferences
WARN:  Server returned lastModified <= 0 for https://hudson.eclipse.org/hudson/job/Xpand-nightly-HEAD/lastSuccessfulBuild/artifact/xpand.p2.repository/artifacts.jar
WARN:  Server returned lastModified <= 0 for https://hudson.eclipse.org/hudson/job/Xtext-nightly-Maintenance/lastSuccessfulBuild/artifact/xtext.p2.repository/artifacts.jar
WARN:  Server returned lastModified <= 0 for https://hudson.eclipse.org/hudson/job/MWE-Language-nightly-HEAD/lastSuccessfulBuild/artifact/mwe2lang.p2.repository/artifacts.jar
Unable to load all units for Software Site container of target /opt/users/hudsonbuild/.hudson/jobs/Xtext-test/workspace//org.eclipse.xtext.git/releng/org.eclipse.xtext.releng/releng/tests/tests-galileo.target
[0]An error occurred while collecting items to be installed
[0.2]session context was:(profile=TARGET_DEFINITION:local:1323688989233.target, phase=org.eclipse.equinox.internal.p2.engine.phases.Collect, operand=, action=).
[0.2]HTTP Server 'Bad Gateway' : https://hudson.eclipse.org/hudson/job/Xpand-nightly-HEAD/lastSuccessfulBuild/artifact/xpand.p2.repository/binary/org.eclipse.xpand_root_1.2.1.v201112070721
Caused by: [0.2][0.2]HttpClient connection error response code 502.
Terminating xvnc.
Comment 3 Winston Prakash CLA 2011-12-12 10:50:01 EST
Denis, I will look at this issue and other issues you assigned to me, once I get some breathing time from the Hudson Eclipse move. However, if you want me to look in to any of the bugs urgently, let me know. I will take some time off from the eclipse move work to look in to those urgent bugs.
Comment 4 Bouchet Stéphane CLA 2011-12-12 11:39:09 EST
just my 2 cents, but i think p2 cannot access to any https repository at all without credential.
Comment 5 Dennis Huebner CLA 2011-12-12 11:49:00 EST
(In reply to comment #4)
> just my 2 cents, but i think p2 cannot access to any https repository at all
> without credential.

Hmmm... How can it sometimes work and sometimes not?
Comment 6 Bouchet Stéphane CLA 2011-12-12 12:20:06 EST
(In reply to comment #5)
> (In reply to comment #4)
> > just my 2 cents, but i think p2 cannot access to any https repository at all
> > without credential.
> 
> Hmmm... How can it sometimes work and sometimes not?

i was just looking at the first lines saying : 

(In reply to comment #0)
> As suggested by web masters I changed URL's from:
> -Dxpand.p2.repository=file:/opt/users/hudsonbuild/.hudson/jobs/Xpand-nightly-HEAD/lastSuccessful/archive/xpand.p2.repository/
> -Dxtext.p2.repository=file:/opt/users/hudsonbuild/.hudson/jobs/Xtext-nightly-HEAD/lastSuccessful/archive/xtext.p2.repository/
> 
> to:
> -Dxpand.p2.repository=https://hudson.eclipse.org/hudson/job/Xpand-nightly-HEAD/lastSuccessfulBuild/artifact/xpand.p2.repository/
> -Dxtext.p2.repository=https://hudson.eclipse.org/hudson/job/Xtext-nightly-HEAD/lastSuccessfulBuild/artifact/xtext.p2.repository/
> 

And this is just not working with p2. p2 can read repo from http, or file, but not from https.
Comment 7 Denis Roy CLA 2011-12-12 13:04:32 EST
(In reply to comment #6)
> And this is just not working with p2. p2 can read repo from http, or file, but
> not from https.

I can tweak the rewriterules so that http://hudson.eclipse.org/hudson/job/*/workspace and http://hudson.eclipse.org/hudson/job/*/lastSuccessfulBuild/artifact/ are allowed over plain http.  Would that help?
Comment 8 Dennis Huebner CLA 2011-12-12 16:04:38 EST
(In reply to comment #7)
> (In reply to comment #6)
> > And this is just not working with p2. p2 can read repo from http, or file, but
> > not from https.
> 
> I can tweak the rewriterules so that
> http://hudson.eclipse.org/hudson/job/*/workspace and
> http://hudson.eclipse.org/hudson/job/*/lastSuccessfulBuild/artifact/ are
> allowed over plain http.  Would that help?

Let's give it a try!

@Stephane: https  works too. Just try to use some https update site. Why do you think it wouldn't work?
Comment 9 Denis Roy CLA 2011-12-12 16:17:12 EST
(In reply to comment #8)
> > http://hudson.eclipse.org/hudson/job/*/workspace and
> > http://hudson.eclipse.org/hudson/job/*/lastSuccessfulBuild/artifact/ are
> > allowed over plain http.  Would that help?
> 
> Let's give it a try!

Actually, I've changed the server config to allow all Hudson requests from our internal network to flow on plain http.  Outside access requires https.

Give that a try.
Comment 10 Bouchet Stéphane CLA 2011-12-13 04:23:30 EST
(In reply to comment #8)
> (In reply to comment #7)
> > (In reply to comment #6)
> > > And this is just not working with p2. p2 can read repo from http, or file, but
> > > not from https.
> > 
> > I can tweak the rewriterules so that
> > http://hudson.eclipse.org/hudson/job/*/workspace and
> > http://hudson.eclipse.org/hudson/job/*/lastSuccessfulBuild/artifact/ are
> > allowed over plain http.  Would that help?
> 
> Let's give it a try!
> 
> @Stephane: https  works too. Just try to use some https update site. Why do you
> think it wouldn't work?

because I tried to use some https:// p2 repo in Eclipse and b3 and did not worked because of required credential.

now, as denis said, we can access to any https p2 repo on hudson without problem, so you can ignore my comments :)
Comment 11 Dennis Huebner CLA 2011-12-14 11:29:05 EST
(In reply to comment #9)
> (In reply to comment #8)
> > > http://hudson.eclipse.org/hudson/job/*/workspace and
> > > http://hudson.eclipse.org/hudson/job/*/lastSuccessfulBuild/artifact/ are
> > > allowed over plain http.  Would that help?
> > 
> > Let's give it a try!
> 
> Actually, I've changed the server config to allow all Hudson requests from our
> internal network to flow on plain http.  Outside access requires https.
> 
> Give that a try.

Doesn't work reliable ether :(
It looks like a hudson issue...

INFO:  System property http.proxyHost has been set to proxy.eclipse.org -Dhttp.proxyPort=9898 -Dhttps.proxyHost=proxy.eclipse.org -Dhttps.proxyPort=9898 -Dhttp.nonProxyHosts="*.eclipse.org" -Dhttps.nonProxyHosts="*.eclipse.org" by an external source. This value will be overwritten using the values from the preferences
Unable to load all units for Software Site container of target <https://hudson.eclipse.org/hudson/job/Xtext-test/ws//org.eclipse.xtext.git/releng/org.eclipse.xtext.releng/releng/tests/tests-galileo.target>
[0]Problems loading repositories
[0.2]Unable to read repository at http://hudson.eclipse.org/hudson/job/Xtext-nightly-HEAD/lastSuccessfulBuild/artifact/xtext.p2.repository/.
Caused by: [0.2]java.io.IOException: http://hudson.eclipse.org/hudson/job/Xtext-nightly-HEAD/lastSuccessfulBuild/artifact/xtext.p2.repository/ is not a valid repository location.
Terminating xvnc.
[FINDBUGS] Skipping publisher since build result is FAILURE
Comment 12 Winston Prakash CLA 2013-03-04 19:33:44 EST
Is this still an issue? If so can somebody give me details on how to reproduce this, so that I can take a look at it.

Two thins to consider

- Eclipse Hudson is now on Jetty
- In Hudson 3.0.0 we moved from Winstone server to Jetty.

So my question is, do we still see this with Jetty as the container?
Comment 13 Dennis Huebner CLA 2013-03-05 03:29:15 EST
(In reply to comment #12)
> Is this still an issue? If so can somebody give me details on how to
> reproduce this, so that I can take a look at it.
> 
> Two thins to consider
> 
> - Eclipse Hudson is now on Jetty
> - In Hudson 3.0.0 we moved from Winstone server to Jetty.
> 
> So my question is, do we still see this with Jetty as the container?

It's works better now. We get 503 responses or broken pipe errors sometimes, but I think in those cases that was not a hudson issue.
Comment 14 Winston Prakash CLA 2013-03-05 11:15:22 EST
Dennis, thanks. Marking this as "RESOLVED WORSFORME"