Community
Participate
Working Groups
Since Wednesday the egit build on hudson.eclipse.org all of a sudden suffers from download problems when accessing p2 repositories. Since that time we are unable to build egit against the galileo p2 repository when running the build in hudson on hudson.eclipse.org due to download failures when Tycho tries to fetch artifacts. When running the same Maven 3/Tycho based build locally using the same p2 repositories and starting from an empty local Maven repository these local builds succeed. We also observe frequent download failures [1] when the egit build tries to download from the jgit p2 repository via http [2] which typically has been built few minutes before since these two build jobs are chained in Hudson. Usually re-running the egit build fixes this problem. On Hudson we switched p2 mirrors off since we earlier faced problems when p2 decided to pick mirrors on the other end of the planet. Currently we switched off the galileo builds and only run builds against helios. List of p2 repositories the egit build uses: build against galileo: http://download.eclipse.org/jetty/7.1.6.v20100715/repository/ http://download.eclipse.org/releases/galileo http://download.eclipse.org/technology/swtbot/galileo/dev-build/update-site http://download.eclipse.org/tools/mylyn/update/weekly build against helios: http://download.eclipse.org/jetty/7.1.6.v20100715/repository/ http://download.eclipse.org/releases/helios http://download.eclipse.org/technology/swtbot/helios/dev-build/update-site http://download.eclipse.org/tools/mylyn/update/weekly [1] https://hudson.eclipse.org/hudson/job/egit/806/console [2] https://hudson.eclipse.org/hudson/job/jgit/lastSuccessfulBuild/artifact/org.eclipse.jgit.packaging/org.eclipse.jgit.updatesite/target/site/
Is there any way you can provide me with easy steps to reproduce this, or an exact error? Is it "File Not found" ? Saying you can't access a p2 repo is only slightly more precise than saying "The Internet is broken" :-) For instance, this link in your console log is a 404 Not found: https://hudson.eclipse.org/hudson/job/jgit/lastSuccessfulBuild/artifact/org.eclipse.jgit.packaging/org.eclipse.jgit.updatesite/target/site/plugins/org.eclipse.jgit.source_0.10.0.201012092111.jar
I analyzed this problem with our Tycho wizard Jan Sievers and we found that the artifacts the egit build is missing should be in the galileo p2 repository, at least that's what its meta data says. Tycho internally uses p2 to download the artifacts but unfortunately doesn't reveal any error message from p2's download failure. Is there a way to consume the galileo p2 repository via a file:// URL on hudson.eclipse.org ?
We did some further analysis using the p2mirror application to exclude caching effects by maven or tycho and to narrow down the number of involved p2 repositories to the galileo repository and the downloaded artifacts to those we saw in error messages of failing egit builds during the last week. It seems there is some evidence that the galileo p2 repository is corrupted or something goes wildly wrong during download. We checked that the artifacts the egit build complained about should be contained in the galileo repository by downloading and inspecting the corresponding artifacts.xml. Then we ran the p2mirror application with the following mirror Ant script using galileo Eclipse SDK as runtime platform : p2mirror.xml: <project name="" basedir ="."> <!-- SAPs HTTP proxy --> <setproxy proxyhost="proxy" proxyport="8080"/> <p2.mirror> <repository location="file:/c:/tmp/testmirror/" name="A new repository" kind="A" /> <repository location="file:/c:/tmp/testmirror/" name="A new repository" kind="M" /> <source> <repository location="http://download.eclipse.org/releases/galileo" name="gal" /> </source> <!-- these cannot be downloaded, but p2 silently succeeds. These versions are listed in http://download.eclipse.org/releases/galileo/201002260900/aggregate/artifacts.jar Also, if you specify a non-existing version, p2 mirror will fail with an error message complaining that the offending version doesn't exist, this proves that also p2 believes these artifacts should be available --> <iu id="org.easymock" version="2.4.0.v20090202-0900" /> <iu id="org.apache.log4j" version="1.2.13.v200903072027"/> <iu id="org.eclipse.net4j.jms.api" version="1.0.0.v200906160348"/> <!-- downloading this one works --> <iu id="org.junit" version="3.8.2.v20090203-1005"/> </p2.mirror> </project> c:\eclipse\3.5.1_classic\eclipse\eclipsec.exe -data c:\ws\tycho -Declipse.p2.mirrors=false -debug -consolelog -application org.eclipse.ant.core.antRunner -buildfile c:\tmp\p2mirror.xml The result of this mirroring in the download folder is: - artifacts.jar and content.jar are created properly - under the plugin folder we get the junit artifact, this proves that the script does what it promises to do - the other three artifacts (which were also not downloadable by the failing egit builds) are not downloaded but p2 doesn't complain and silently succeeds I will retry the same script using the Helios version of the p2mirror application tomorrow.
Thanks for the analysis so far. I'm not familiar with the inner workings of p2, but if there is anything I do to help please let me know.
We tried many things, it turned out that we get download failures in the egit build with the following bundles which should come from galileo: <iu id="org.easymock" version="2.4.0.v20090202-0900" /> <iu id="org.apache.log4j" version="1.2.13.v200903072027"/> <iu id="org.eclipse.net4j.jms.api" version="1.0.0.v200906160348"/> We first downloaded the artifacts.jar metadata of the galileo repository and successfully checked that it says these bundles should be in the repository. Then we tried to download exactly these bundles directly from the galileo p2 repository using the p2mirror ant task to exclude any maven or tycho problems using the attached p2mirror.xml ant script and the bash script mirror.sh. We found that the p2mirror ant task says it downloaded the artifacts but in fact only the metadata was downloaded but not the corresponding artifacts. In order to check that our script is correct we also included another bundle <iu id="org.junit" version="3.8.2.v20090203-1005"/> this one was downloaded properly including its artifact. We tried with galileo p2mirror ant task on Windows and with helios p2mirror ant task on Mac OS X and got identical results. What is very strange is that we can directly download http://download.eclipse.org/releases/galileo/201002260900/aggregate/plugins/org.easymock_2.4.0.v20090202-0900.jar.pack.gz using a browser So we conclude: - either we miss something - or the galileo p2 repo is broken - or something goes wildly wrong with the download Does this ring any bell for anybody ?
Created attachment 185384 [details] p2mirror ant script
Created attachment 185385 [details] bash script to start mirror task on Mac using helios equinox launcher
Also checked the verbose console log of the p2mirror ant task but this didn't contain any error or warning.
(In reply to comment #5) > We tried many things, it turned out that we get download failures in the egit > build > with the following bundles which should come from galileo: > > <iu id="org.easymock" version="2.4.0.v20090202-0900" /> > <iu id="org.apache.log4j" version="1.2.13.v200903072027"/> > <iu id="org.eclipse.net4j.jms.api" version="1.0.0.v200906160348"/> Net4j has been contributed from http://download.eclipse.org/modeling/emf/cdo/updates/3.0/3.0-SR1 Problems like this one would be easier to analyze if more repos would follow http://thegordian.blogspot.com/2010/05/what-exactly-is-inside-that-p2.html
moved this problem over to p2, hoping that some p2 expert could help here
The script works for me. The artifacts are all available locally. However they are available in the blobstore (galileo/.blobstore/...). This happens because the target repository does not have a mapping rule. The way to set the mapping rules is by pointing to an existing repository that has the format you desire. In your case, you can replace the reference to the target artifact repository by the following line: <repository location="file:/Users/Pascal/tmp/p2/galileo/" name="galileo-mirror" kind="A" format="http://download.eclipse.org/releases/galileo/201002260900/aggregate/"/> I'm not sure that this is what was hitting you in the first place in your tycho problem.
Thanks for this hint, didn't know about that. So it looks like we have to further analyze the tycho part...
I'll close for now. Feel free to reopen if you have more issues. Make sure that Tycho looks for the artifacts using the artifact repository API and does not assume the layout of artifacts on disk.
We wrote a tiny test project [1] which just tries to download the bundles the egit build fails to download and ran this on hudson.eclipse.org, it fails with the same download problems as the egit galileo build [2]. Forwarding to Tycho to get advice how to proceed with this problem [1] https://github.com/msohn/galileoTest [2] https://hudson.eclipse.org/hudson/job/egit.test/16/console
I am unable to reproduce the problem neither locally nor on Sonatype CI GRID [1]. Has anyone seen the problem outside of eclipse.org build infrastructure? Does anyone know if hudson.eclipse.org uses any dns/proxy tricks to shortcut access to http://download.eclipse.org @pascal can you please review Tycho properly uses IArtifactRepository.getArtifacts to download remote artifacts? relevant code is in P2ResolverImpl [2] around line 372. [1] https://grid.sonatype.org/ci/view/Tycho/job/tycho-Bug332342/1/console [2] https://github.com/sonatype/sonatype-tycho/blob/master/tycho-p2-resolver/org.sonatype.tycho.p2.impl/src/main/java/org/sonatype/tycho/p2/impl/resolver/P2ResolverImpl.java
I can also successfully build test project https://github.com/msohn/galileoTest locally so for now it looks like the problem only occurs on eclipse.org build servers.
things you could try out to narrow down the problem source: 1. use file:// URLs instead of http:// for the p2 repos to see if the problem is on the transport level 2. Vary tycho version (0.9.0, 0.10.0, 0.11.0-SNAPSHOT [from maven repo http://repository.sonatype.org/content/groups/forge/ ] ) to see if the used p2/tycho version affects the behavior
Webmaster: is there a way to access the galileo p2 repository via file:// URL from hudson.eclipse.org ?
The Hudson servers mount the shared filesystems as they are on build.eclipse.org downloads: /home/data/httpd/download.eclipse.org archives: /home/data/httpd/archive.eclipse.org /shared: /shared
the download also fails using file:// URL for galileo repository
see https://hudson.eclipse.org/hudson/job/egit.test/25/console
I have a similar problem with Galileo p2 site when building the egit-training tutorial with a Jenkins installation (running behind a company proxy). --- excerpt from console output --- [WARNING] Failed to access p2 repository eclipse-platform (http://download.eclipse.org/releases/galileo), will try to use local cache. Reason: org.eclipse.equinox.p2.core.ProvisionException: Artifact not found: http://download.eclipse.org/releases/galileo/compositeArtifacts.jar. ... [DEBUG] Ignoring Maven repository central (http://repo1.maven.org/maven2) [ERROR] Internal error: java.lang.RuntimeException: Could not download artifacts from any repository [ERROR] osgi.bundle,org.eclipse.swt,3.5.2.v3557f [ERROR] osgi.bundle,org.eclipse.swt.gtk.linux.x86_64,3.5.2.v3557f [ERROR] -> [Help 1] org.apache.maven.InternalErrorException: Internal error: java.lang.RuntimeException: Could not download artifacts from any repository osgi.bundle,org.eclipse.swt,3.5.2.v3557f osgi.bundle,org.eclipse.swt.gtk.linux.x86_64,3.5.2.v3557f --- excerpt from console output --- Is there yet a solution or workaround known?
It is working again now for Eclipse Galileo p2 update site now. Have been any changes made on the update site?
(In reply to comment #23) > It is working again now for Eclipse Galileo p2 update site now. > Have been any changes made on the update site? I suspect the underlying issue may have been p2 bug 345061 . Even when using a file:// URL, this will not stop p2 from reaching out to mirrors via HTTP again (and in case of enough mirrors being down/corrupted, we exceed MAX_RETRY_REQUEST count). @Matthias can you still reproduce the problem with your example job https://hudson.eclipse.org/hudson/job/egit.test/ ? Regards, Jan
(In reply to comment #24) > I suspect the underlying issue may have been p2 bug 345061 . Tycho will update it's p2 version soon. We should see if the problem still occurs once this is done.
tycho 0.13 has been updated to use p2 3.7.0, which we hope solves the problem. please reopen once again if you still run into this.
