Bugzilla – Full Text Bug Listing
| Summary: | Move job cbi-amp-nightly to hudson.build.org | ||
|---|---|---|---|
| Product: | Community | Reporter: | Miles Parker <milesparker> |
| Component: | CI-Jenkins | Assignee: | CI Admin Inbox <ci.admin-inbox> |
| Status: | CLOSED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | P3 | CC: | d_a_carver, jacek.pospychala, webmaster |
| Version: | unspecified | ||
| Target Milestone: | --- | ||
| Hardware: | All | ||
| OS: | All | ||
| Whiteboard: | |||
| Bug Depends on: | 324121 | ||
| Bug Blocks: | |||
|
Description
Miles Parker
Just to clarify: you want the cbi-amp-nightly build moved from build.eclipse.org to the 'new' hudson instance. You are aware that the new instance is has not been announced as 'ready for general use'? -M. I just followed the link to...I'm going on: http://wiki.eclipse.org/Hudson which I took to mean that it was recommended to make the transition. Hudson build is sometimes crowded but I'm happy to hold off too. My build is currently broken anyway :) so I can stand a few glitches. My only concern moving over would be planned support for SWTBot on multiple platforms and FindBugs, but other than that I just need the basic Buckminster and JUnit support. Miles, I'd like to move your build to get more feedback on the new setup. Is today ok? -M. Sure thing..as I said it's not actually working anyway (at least the JUnit and promote aren't) so I'm not really in a position to complain if it doesn't work on the new machine! And actually since my issues with JUnit seem to be internal it might even be better. cheers, Miles Ok, I've copied the job. Please let me know if you run into problems(that I need to fix) either via email or bugs. -M. Hmm.. should it be showing up n the main page of: https://hudson.eclipse.org/hudson/ I don't see it there. BTW, it doesn't really matter, but the project is using bucky now so it isn't really cbi anymore. Just "amp" for build name would be simpler. Sorry, oversight, it's there now. -M. Thanks. It looks as though we can not use local access for eclipse cvs as on build.eclipse.org/hudon, e.g. : ":local:/cvsroot/modeling"? I switched back to pserver and that does work of course. I think we may be missing some symlinks on the new Hudson servers... Hi, So the build is hanging, and at a point it was getting past in the current build. I'm going to simplify both so that they're equivalent and I get a working build on the old machine so we can do a good a-b compare, but here's the current console log: https://hudson.eclipse.org/hudson/job/cbi-amp-nightly/656/console -Miles The hang is gone, but now the build is just failing with: ... ERROR: null Archiving artifacts https://hudson.eclipse.org/hudson/job/cbi-amp-nightly/658/consoleFull Can you delete the workspace and run the build again? I'm wondering if something from a previous build is getting in the way. -M. Hi Denis, That fixed the current issue -- then I got another issue. See log below, but buckminster is trying to grab a plugin from an external p2 update site. (This is not for inclusion in build but to support the building of an Eclipse hosted feature in GEF3D.) This worked fine on build.eclipse.org. Could there be something (or some policy) preventing access to that external site? thanks, Miles org.lwjgl:osgi.bundle/2.2.1: Rejecting provider p2(http://lwjgl.org/update[http://lwjgl.org/update]): No component match was found org.lwjgl:osgi.bundle/2.2.1: No provider was found that could resolve the request ERROR [0049] : No suitable provider for component org.lwjgl:osgi.bundle/2.2.1 was found in resourceMap file:/opt/users/hudsonbuild/workspace/cbi-amp-nightly/org.eclipse.amp/releng/org.eclipse.amp.releng/releng/amp.rmap ERROR [0049] : No suitable provider for component org.lwjgl:osgi.bundle/2.2.1 was found in searchPath eclipse ERROR [0049] : Rejecting provider p2({0}/releases/helios?importType=binary[http://download.eclipse.org/releases/helios?importType=binary]): No component match was found ERROR [0049] : No suitable provider for component org.lwjgl:osgi.bundle/2.2.1 was found in searchPath platform ERROR [0049] : Rejecting provider p2({0}/eclipse/updates/3.6milestones[http://download.eclipse.org/eclipse/updates/3.6milestones]): No component match was found ERROR [0049] : No suitable provider for component org.lwjgl:osgi.bundle/2.2.1 was found in searchPath lwjgl ERROR [0049] : Rejecting provider p2(http://lwjgl.org/update[http://lwjgl.org/update]): No component match was found ERROR No repository found at http://lwjgl.org/update/. INFO: TAG-ID 0049 = Query for org.eclipse.amp.build:eclipse.feature, path: org.eclipse.amp.build:eclipse.feature$0.8.0.qualifier -> org.eclipse.amp.all:eclipse.feature$0.8.0.qualifier -> org.eclipse.amp.agf3d:eclipse.feature$0.8.0.qualifier -> org.eclipse.amp.escape.ascape.agf3d:osgi.bundle$0.8.0.qualifier Are you using buckminster? -M. (In reply to comment #14) > Are you using buckminster? > > -M. Yep, and the cquery seems to find things correctly on build.eclipse. Is it possible for you to give me a buckminster command I can use to test this? I think this is the same as bug 323579, but I'm not sure why the proxy settings are getting in the way, since they're not required for 'eclipse.org'. -M. (In reply to comment #16) > Is it possible for you to give me a buckminster command I can use to test this? > I think this is the same as bug 323579, but I'm not sure why the proxy > settings are getting in the way, since they're not required for 'eclipse.org'. > > -M. Sure thing -- its pretty simple actually..no ant scripts! If you point to: :pserver:anonymous@dev.eclipse.org:/cvsroot/modeling org.eclipse.amp/releng org.eclipse.amp/org.eclipse.amp/features import -Declipse.download="file:/home/data/httpd/download.eclipse.org" "${checkout.location}org.eclipse.amp/releng/org.eclipse.amp.releng/releng/amp-platform.mspec" ->>Breaks here.. resolve "${checkout.location}org.eclipse.amp/releng/org.eclipse.amp.releng/releng/amp.cquery" build Note that there does seem to be now an issue with a refused connection from eclipse svn here as well. http://dev.eclipse.org/svnroot/technology/org.eclipse.gef3d/trunk/org.eclipse.draw3d/?moduleAfterTag&moduleAfterBranch ERROR org.eclipse.team.svn.core.connector.SVNConnectorException: svn: connection refused by the server` As an addiitonal note, I think there might be a conincidental problem with the update dependency itself..wouldn't that be typical, argh.. I'm looking into that possibility now. Well to me it looks like buckminster wants nothing to do with the http proxy settings. If I run: /shared/common/buckminster-3.6/buckminster -L debug import -Declipse.download="file:/home/data/httpd/download.eclipse.org" "amp-platform.cquery" -vmargs -Dhttp.proxyHost=build.eclipse.org -Dhttp.proxyPort=proxyport as the hudsonbuild user, I can see it trying to connect directly to mirrors: java 14987 hudsonbuild 614u IPv6 5171105 0t0 TCP 172.30.206.141:44996->mirror3.mirror.garr.it:http (SYN_SENT) I've checked that the 'correct' JVM proxy args are set, but they don't seem to have any impact either. One thing to note is that if I add a 'bogus' ftp proxy to the buckminster command line I see connections to the proxy: java 14987 hudsonbuild 107u IPv6 5146732 0t0 TCP 172.30.206.141:44830->build.eclipse.org:proxy (CLOSE_WAIT) I can even see the ftp proxy requests come in: 172.30.206.141 - - [31/Aug/2010:13:18:51 -0400] "GET ftp://mirror.selfnet.de/eclipse/eclipse/updates/3.6/R-3.6-201006080911/plugins/org.eclipse.osgi_3.6.0.v20100517.jar.pack.gz HTTP/1.1" 404 1020 "-" "Java/1.6.0_21" -M. (In reply to comment #19) > Well to me it looks like buckminster wants nothing to do with the http proxy > settings. If I run: I'll freely admit to not understanding how any of this works. :) Is it possible that there is something further that I need to configure? Otherwise, do you think it makes sense to pitch this over to a bucky bug? Well, at this stage it does look like a buckminster bug to me. Based on my reading of the docs with the proxy specified on the command line it should work. -M. I put a dependency on the new Buckminster bug. I'm just reopening this one to make it clear that there isn't a resolution for that issue. Matt/Webmasters, sorry if I'm hijacking this bug, but in another job (cbi-pdt-2.2-helios), where we use Athena, in pure ant we're trying to get 3rd-party lib in Ant: <get src="http://jflex.de/jflex-1.4.1.zip" verbose="true" usetimestamp="true" /> this ends with ConnectionTimedOut exception. So is there any special proxy setup required in Athena, or generally any ant scripts, before accessing the outside world? Hudson sets the following shell variables ANT_OPTS,ANT_ARGS,JVM_OPTS,JAVA_ARGS with the required proxy data. The ant settings are also set within the Hudson users shell, so as long as ant is reading(and using) those variables you shouldn't need to set anything at all. I've been experimenting on one of the slaves with some iptables magic to try and force connections to the proxy. I need to go through the build logs for the cbi and modisco builds I started to see where we're at. -M. (In reply to comment #24) > Hudson sets the following shell variables ANT_OPTS,ANT_ARGS,JVM_OPTS,JAVA_ARGS > with the required proxy data. The ant settings are also set within the Hudson > users shell, so as long as ant is reading(and using) those variables you > shouldn't need to set anything at all. > > I've been experimenting on one of the slaves with some iptables magic to try > and force connections to the proxy. I need to go through the build logs for > the cbi and modisco builds I started to see where we're at. > > -M. Matt one suggestion, you may need to set these as Global Envirnonment variables within Linux itself (not just Hudson's configuration). I think part of the problem here is the way that Buckminster and Athena are launching the builds. They tend to launch Multiple instances of OSGI applications through the use of Ant, and they may not be passing all the environment variables from one JVM to a new instance of another JVM that is created. So they may not even be seeing the environment variables that were set by Hudson in the original jvm that was launched.
> Matt one suggestion, you may need to set these as Global Envirnonment variables
> within Linux itself (not just Hudson's configuration).
Ok, I've added the JVM vars(ANT ones we set already) to the Hudson users .bashrc file, so unless the jobs being forked are running as another user(which would be downright suspicious) they should be set.
-M.
I must say that I am enjoying sharing the joy of clicking that execute button and waiting for the "build failed" message to appear in my in-box 15 minutes later ;D but please let me know if and when it would be helpful to dive back in here. Please see Thomas' comment re: Buckminster and SVN. Any further ideas? The build is working with same configuration on build.eclipse.org. https://bugs.eclipse.org/bugs/show_bug.cgi?id=324121#c1 Can you update the build so that it's using the 'local' files system for SVN instead of trying to reach dev.eclipse.org? -M. (In reply to comment #29) > Can you update the build so that it's using the 'local' files system for SVN > instead of trying to reach dev.eclipse.org? > > -M. So you mean by changing the svn search path URL? If so what's the URL I should use for that? If you mean by using the Hudson VCS integration than no, because this is actually for an Eclipse hosted dependency (not an AMP project) and I'm using CVS to get just my releng project. I'm beginning to think I might have the most complicated dependency chain of any project out there.. Well if you're trying to get 'Eclipse' svn data(which may not be the case) you should be able to use file:///svnroot/path/to/repo and avoid any network calls. -M. Hi,
Still no joy on this. I'm getting:
ERROR [0047] : No suitable provider for component org.lwjgl:osgi.bundle/2.2.1 was found in searchPath lwjgl
ERROR [0047] : Rejecting provider p2(http://lwjgl.org/update[http://lwjgl.org/update]): No component match was found
ERROR No repository found at http://lwjgl.org/update/.
This is a *build-time* dependency on an external p2 site. (It is *not* intended for access or provisioning from Eclipse.org.) I am thinking that it is the external access that is the issue, because my Eclipse hosted p2 dependencies work fine. The dependencies work fine on my local build and did work on build.eclispe.org as well.
Could trying to access an external p2 site be the issue?
I tried that URL from the commandline and it worked fine. I'll check the proxy logs when I get a chance to see what's happening. (In reply to comment #33) > I tried that URL from the commandline and it worked fine. I'll check the proxy > logs when I get a chance to see what's happening. Denis, I know I'm bombarding you today, but I wanted to see if you've had any success finding out what's going on with the proxy issue? I've tried following the instructions on Hudson build wiki, but I'm not sure if I'mm setting that up right. Also had a look at http://www.eclipse.org/forums/index.php?t=tree&goto=628738#page_top -- I'm trying at this point to use Buckminster entirely, i.e. without any ant calls so I don't think that applies to this situation. thanks, miles If I run: /shared/common/buckminster-3.6/buckminster -L debug import -Declipse.download="file:/home/data/httpd/download.eclipse.org" "./releng/org.eclipse.amp.releng/amp-platform.cquery" -vmargs -Dhttp.proxyHost=proxy.eclipse.org -Dhttp.proxyPort=port And then check the proxy logs I see: - - [15/Oct/2010:15:36:37 -0400] "CONNECT dev.eclipse.org:2401 HTTP/1.0" 403 1038 "-" "-" Which is the only 403 being returned that matches the time the command was run. I suspect the answer on the forum is correct(this issue certainly resembles bug 325786). -M. OK, now I'm getting a different proxy error, but it seems like all of the other bucky builds (at least) jsut started having it..please see below. On the original issue, I'm not using ant, so I don't have the issue of trying to pass through the proxy settings. Should I then be able to do something like: -Dhttp.nonProxyHosts="*.eclipse.org,lwjgl.org" ?? [cbi-amp-nightly] $ java -Dbuckminster.output.root=/opt/users/hudsonbuild/workspace/cbi-amp-nightly/buckminster.output -Dbuckminster.temp.root=/opt/users/hudsonbuild/workspace/cbi-amp-nightly/buckminster.temp '-Dhttp.proxyHost=proxy.eclipse.org -Dhttp.proxyPort=9898 -Dhttps.proxyHost=proxy.eclipse.org -Dhttps.proxyPort=9898 -Dhttp.nonProxyHosts="*.eclipse.org" -Dhttps.nonProxyHosts="*.eclipse.org"' '-Dhttp.proxyHost=proxy.eclipse.org -Dhttp.proxyPort=port -Dhttp.nonProxyHosts="*.eclipse.org,lwjgl.org"' -jar /shared/common/buckminster-3.6/plugins/org.eclipse.equinox.launcher_1.1.0.v20100507.jar -application org.eclipse.buckminster.cmdline.headless -data /opt/users/hudsonbuild/workspace/cbi-amp-nightly --loglevel info -S /opt/users/hudsonbuild/workspace/cbi-amp-nightly/commands.txt INFO: import '/opt/users/hudsonbuild/workspace/cbi-amp-nightly/org.eclipse.amp/releng/org.eclipse.amp.releng/amp.cquery' com.jcraft.jsch.JSchException: ProxyHTTP: java.io.IOException: proxy error: Forbidden at com.jcraft.jsch.ProxyHTTP.connect(ProxyHTTP.java:158) at org.eclipse.jsch.internal.core.JSchProvider.connect(JSchProvider.java:141) at org.eclipse.team.internal.ccvs.core.connection.PServerConnection.open(PServerConnection.java:116) at org.eclipse.team.internal.ccvs.core.connection.Connection.open(Connection.java:132) at org.eclipse.team.internal.ccvs.core.connection.CVSRepositoryLocation.createConnection(CVSRepositoryLocation.java:545) at org.eclipse.team.internal.ccvs.core.connection.CVSRepositoryLocation.openConnection(CVSRepositoryLocation.java:806) at org.eclipse.team.internal.ccvs.core.client.Session.open(Session.java:159) at org.eclipse.buckminster.cvspkg.internal.CVSSession.getReaderSession(CVSSession.java:98) at org.eclipse.buckminster.cvspkg.internal.RepositoryMetaData.getMetaData(RepositoryMetaData.java:76) at org.eclipse.buckminster.cvspkg.internal.VersionFinder.getMetaData(VersionFinder.java:97) at org.eclipse.buckminster.cvspkg.internal.VersionFinder.getTrunk(VersionFinder.java:86) at org.eclipse.buckminster.core.version.AbstractSCCSVersionFinder.getBestTrunkMatch(AbstractSCCSVersionFinder.java:320) at org.eclipse.buckminster.core.version.AbstractSCCSVersionFinder.getBestVersion(AbstractSCCSVersionFinder.java:151) at org.eclipse.buckminster.core.rmap.model.Provider.findMatch(Provider.java:220) at org.eclipse.buckminster.core.rmap.model.SearchPath.getProvider(SearchPath.java:100) at org.eclipse.buckminster.core.rmap.model.ResourceMap.resolve(ResourceMap.java:293) at org.eclipse.buckminster.core.rmap.model.ResourceMap.resolve(ResourceMap.java:212) at org.eclipse.buckminster.core.resolver.ResourceMapResolver.innerResolve(ResourceMapResolver.java:202) at org.eclipse.buckminster.core.resolver.ResolverNodeWithJob.resolve(ResolverNodeWithJob.java:190) at org.eclipse.buckminster.core.resolver.ResolverNodeWithJob.run(ResolverNodeWithJob.java:87) at org.eclipse.buckminster.core.resolver.ResolverNodeWithJob$NodeResolutionJob.run(ResolverNodeWithJob.java:44) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:54) Caused by: java.io.IOException: proxy error: Forbidden at com.jcraft.jsch.ProxyHTTP.connect(ProxyHTTP.java:122) ... 21 more
> -Dhttp.nonProxyHosts="*.eclipse.org,lwjgl.org"
This will probably make things 'worse' since you do need the proxy to connect to lwjgl.org .
-M.
(In reply to comment #37) > > -Dhttp.nonProxyHosts="*.eclipse.org,lwjgl.org" > > This will probably make things 'worse' since you do need the proxy to connect > to lwjgl.org . Oh, ok.. hmm. I guess was thinking that the proxy was there for performance and maintenance reasons not security and thought that turning it off would allow things to bypass the proxy altogether, but I guess that isn't possible. So what *should* be in the non proxy hosts rather than the defaults? I'm totally lost here WRT what is needed to contact an external update site. All you should need to do to connect to an external site(via http/s) is to make sure your jobs aren't overriding the proxy settings. In your last block of error code this line: org.eclipse.team.internal.ccvs.core.connection.PServerConnection.open(PServerConnection.java:116) makes me wonder if this isn't the same problem as the one in bug: 327859 -M. (In reply to comment #39) > All you should need to do to connect to an external site(via http/s) is to make > sure your jobs aren't overriding the proxy settings. > > In your last block of error code this line: > > org.eclipse.team.internal.ccvs.core.connection.PServerConnection.open(PServerConnection.java:116) > > makes me wonder if this isn't the same problem as the one in bug: 327859 Yes, that makes sense..I know I've seen that. BTW, I noticed that you triggered a build earlier today. Naturally, the build was broken for a completely different reason then. :) RIght now I'm trying to get my local build aligned back with the hudson build so I can determine exactly the delta and I'm fumbling around a bit with that. If you're ever at a point where you need me to stop mucking with things so there is a stable configuration shoot me an email. Can do. -M. Miles, is your build currently 'stable' so we can try and figure out what's going on? -M. Hi, It's not. :) I was still experimenting with it and have been away from work for a while. Let me get my local build up and running. One of the hangups was on CVS and so I figured I'd plow ahead ahead and made the leap to Git. Depending on how local build goes with current CVS setup I may switch my maps over to Git first. Note that that will not unfortunately take care of the SVN issue because that is form dependency on GEF3D project. I'll ping back as soon as that is working. cheers, Miles OK, just going ahead and dumping CVS overboard seemed to do the trick. :D There is still a long way to go in terms of getting the packaging right, testing and promotion, but most of that is just configuration and hopefully won't implicate Hudson issues per se. In case they do, I'm going to hold this bug open for a bit anyway if you don't mind. Closing. :D Just have to get promotion working but that isn't a Hudson issue. One final question: I'm guessing it would be a bad idea to change the project name, correct? A change on hudson is pretty straight forward. -M. Is that something I could do just by changing the configuration or would you have to do it? If the latter then as it isn't cbi based anymore, I'd rather have the name be simply: amp-nightly And I'd like a separate integration build as well: amp-integration Should I open a separate bug for this? thanks, Miles (In reply to comment #47) > > amp-nightly Done. > > And I'd like a separate integration build as well: > > amp-integration Done. -M. |