Bugzilla – Full Text Bug Listing
| Summary: | Corrupted artifacts served by maven.eclipse.org | ||
|---|---|---|---|
| Product: | [Technology] Dash | Reporter: | Wainer dos Santos Moschetta <wainersm> |
| Component: | Maven | Assignee: | Alex Blewitt <alex.blewitt> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | blocker | ||
| Priority: | P3 | CC: | alex.blewitt, bentmann, bernd.hufmann, dennis.huebner, d_a_carver, hmalphettes, jan.sievers, jesse.mcconnell, matthias.sohn, overholt |
| Version: | unspecified | ||
| Target Milestone: | --- | ||
| Hardware: | PC | ||
| OS: | Linux | ||
| Whiteboard: | |||
|
Description
Wainer dos Santos Moschetta
Fixing component. original artifact http://search.maven.org/remotecontent?filepath=org/eclipse/tycho/tycho/0.13.0/tycho-0.13.0.pom is just fine. Must be corrupted on the way to maven.eclipse.org . Changing component accordingly. Other POMs under http://maven.eclipse.org/nexus/content/groups/public/org/codehaus/plexus/ are corrupted. More corrupted POMs: http://maven.eclipse.org/nexus/content/groups/public/org/apache/maven/plugins/maven-clean-plugin/2.4.1/maven-clean-plugin-2.4.1.pom http://maven.eclipse.org/nexus/content/groups/public/org/apache/apache/7/apache-7.pom I suspect there was a loss of connectivity or something and these are all really html documents of error pages its just a store to reduce traffic to central, how about we nuke the contents and let them repopulate that is the easiest solution for the webmasters alternatively could probably script up something to check for html content where it shouldn't be and smoke the artifact files when that situation is detected Other corrupted file which just killed the Aether CI builds: http://maven.eclipse.org/nexus/content/groups/public/org/sonatype/forge/forge-parent/6/forge-parent-6.pom (In reply to comment #5) > its just a store to reduce traffic to central, how about we nuke the contents > and let them repopulate +1, this should be the easiest and safest approach. At any rate, the configuration option "File content validation" [0] should be enabled for the central proxy repo to prevent this from happening in the future. [0] https://issues.sonatype.org/secure/attachment/24865/2010-09-07_1651_filecontentvalidation.png [1] http://www.sonatype.com/books/nexus-book/reference/config-sect-manage-repo.html Adjusted issue summary to better capture wide scope of issue. Here is a workaround for by-passing eclipse's nexus (credits to Jesse):
Add a settings.xml to the root of the project to build:
<settings>
<!--localRepository>/tmp/jetty-builds/jetty8/localRepo</localRepository-->
<interactiveMode>true</interactiveMode>
<offline>false</offline>
<proxies>
<proxy>
<active>true</active>
<protocol>http</protocol>
<host>proxy.eclipse.org</host>
<port>9898</port>
</proxy>
</proxies>
<mirrors>
<mirror>
<id>central</id>
<name>central</name>
<url>http://repo2.maven.org/maven2/</url>
<mirrorOf>*</mirrorOf>
</mirror>
</mirrors>
</settings>
Then add to the maven's goals: -s settings.xml
For example:
clean verify -X -U -s settings.xml
(In reply to comment #8) > Here is a workaround for by-passing eclipse's nexus (credits to Jesse): From what I gathered so far, maven.eclipse.org is not managed by the EF/webmasters but by the Dash project as an experiment/proof-of-concept. As such I was kinda taken by surprise to notice that maven.eclipse.org gets used by default on at least some Hudson slaves (cf. bug #366336). I would have expected it to be something projects need to explicitly enable on their discretion (as opposed to explicitly workaround). Not meaning any offense here, just saying to me as a (potentially ignorant) newbie it looks like important infrastructure maintained by the webmasters (Hudson) by default depends on less stable infrastructure maintained by volunteers (maven.eclipse.org). loosely managed by the dash folks as we don't have access (afaik) to the disk portion of the instance so we are unable to resolve the issues ourselves The egit builds seem to be affected by corrupted artifacts as well. EGit references http://maven.eclipse.org/nexus/content/groups/public/ in order to use the maven-eclipse-signing-plugin for signing. It seems this is also used for other artifacts which may be corrupt in maven.eclipse.org causing the EGit builds to fail. E.g. see [1], [2]. I can't reproduce these errors when running the builds locally. [1] https://hudson.eclipse.org/hudson/job/egit/1908/console [2] https://hudson.eclipse.org/sandbox/job/egit.gerrit/2145/console (In reply to comment #11) > The egit builds seem to be affected by corrupted artifacts as well. EGit > references http://maven.eclipse.org/nexus/content/groups/public/ in order to > use the maven-eclipse-signing-plugin for signing. It seems this is also used > for other artifacts which may be corrupt in maven.eclipse.org causing the EGit > builds to fail. E.g. see [1], [2]. I can't reproduce these errors when running > the builds locally. > > [1] https://hudson.eclipse.org/hudson/job/egit/1908/console > [2] https://hudson.eclipse.org/sandbox/job/egit.gerrit/2145/console these two build logs show some checksum validation errors for artifacts downloaded from maven.eclipse.org, this is a clear indication that it needs to be cleaned up For building against this repo, please use the following repository: http://maven.eclipse.org/nexus/content/groups/build/ The public URL shouldn't be resolving any Maven central components (we don't want to serve the world from here). It should only be used for accessing Eclipse content. That said, it's not clear where the org/eclipse/tycho/... is coming from. I'll need to understand why/where that's happening first. But please build using the /build/ repository and not the /public/ one, because the /public/ one is not going to resolve transitive dependencies from Maven central for you. Alex (In reply to comment #13) > For building against this repo, please use the following repository: > > http://maven.eclipse.org/nexus/content/groups/build/ > > The public URL shouldn't be resolving any Maven central components (we don't > want to serve the world from here). It should only be used for accessing > Eclipse content. > > That said, it's not clear where the org/eclipse/tycho/... is coming from. I'll > need to understand why/where that's happening first. > > But please build using the /build/ repository and not the /public/ one, because > the /public/ one is not going to resolve transitive dependencies from Maven > central for you. > naming a repository "public" which shouldn't be used is counter intuitive ... wiping the local maven repo and switching from http://maven.eclipse.org/nexus/content/groups/public/ to http://maven.eclipse.org/nexus/content/repositories/milestone-indigo/ in order to download the signing plugin fixed the problems we had with corrupted artifacts. So it looks like http://maven.eclipse.org/nexus/content/groups/public/ has some corrupt content which needs to be cleaned up. This was being caused by a bad xtext proxy which was configured as part of the public API. Unfortunately, it was pointing randomly at a workspace on a build server, which meant that we then propagated bad data (origins unknown) to other users. I have taken the bad location out of service and now the /public/ no longer responds to the org/eclipse/tycho (as expected). Instead, you should use /build/ if you want to build with this as the local cache. (In reply to comment #14) > > But please build using the /build/ repository and not the /public/ one, because > > the /public/ one is not going to resolve transitive dependencies from Maven > > central for you. > > > > naming a repository "public" which shouldn't be used is counter intuitive ... The /public/ one is intended to be used for Eclipse-generated artefacts. So it's not that it shouldn't be built against, it is that it shouldn't be the one used for server-side builds on the Eclipse farm. > wiping the local maven repo and switching from > http://maven.eclipse.org/nexus/content/groups/public/ > to > http://maven.eclipse.org/nexus/content/repositories/milestone-indigo/ > in order to download the signing plugin fixed the problems we had with > corrupted artifacts. > > So it looks like > http://maven.eclipse.org/nexus/content/groups/public/ > has some corrupt content which needs to be cleaned up. It was the 'xtext-proxy' which was serving bad artefacts. I have removed this from the /public/ lookup path. Now the /public/ doesn't respond to maven central hosted artefacts (by design) though you can use /build/ as well for faster local builds on the Eclipse build farm. Using any other repository (.../repo1/, .../repo2/, .../build/) avoided the problem because none of these had the corrupted xtext-proxy in them. It's not clear to me why xtext-proxy was added; I'll add information here if I find out why. (In reply to comment #16) > (In reply to comment #14) > > > But please build using the /build/ repository and not the /public/ one, because > > > the /public/ one is not going to resolve transitive dependencies from Maven > > > central for you. > > > > > > > naming a repository "public" which shouldn't be used is counter intuitive ... > > The /public/ one is intended to be used for Eclipse-generated artefacts. So > it's not that it shouldn't be built against, it is that it shouldn't be the one > used for server-side builds on the Eclipse farm. > > > wiping the local maven repo and switching from > > http://maven.eclipse.org/nexus/content/groups/public/ > > to > > http://maven.eclipse.org/nexus/content/repositories/milestone-indigo/ > > in order to download the signing plugin fixed the problems we had with > > corrupted artifacts. > > > > So it looks like > > http://maven.eclipse.org/nexus/content/groups/public/ > > has some corrupt content which needs to be cleaned up. > > It was the 'xtext-proxy' which was serving bad artefacts. I have removed this > from the /public/ lookup path. Now the /public/ doesn't respond to maven > central hosted artefacts (by design) though you can use /build/ as well for > faster local builds on the Eclipse build farm. > > Using any other repository (.../repo1/, .../repo2/, .../build/) avoided the > problem because none of these had the corrupted xtext-proxy in them. > > It's not clear to me why xtext-proxy was added; I'll add information here if I > find out why. Hi Alex, please see https://bugs.eclipse.org/bugs/show_bug.cgi?id=356202 xtext-proxy should point to http://build.eclipse.org/jobs/Xtext-Maven-Deploy/maven/ where any tycho poms are/were contained. Regards, Dennis. (In reply to comment #17) > Hi Alex, > please see https://bugs.eclipse.org/bugs/show_bug.cgi?id=356202 > xtext-proxy should point to > http://build.eclipse.org/jobs/Xtext-Maven-Deploy/maven/ where any tycho poms > are/were contained. That is absolutely the wrong thing to do. We don't want to delegate out any random repository to the public lookup list; otherwise we could have a situation where any corrupt artefacts could be propagated, which is exactly what happened here. When I visit this link, I get a 'not found' and the build.eclipse.org suggests that I go to hudson.eclipse.org instead. Maybe an environmental change caused the move but I suspect that was the start of it. We should instead enable the Xtext maven jobs to deploy automatically to one of the existing repositories e.g. integration, so that you can push your specific components into the repository instead of us delegating to many. Removed broken repository and ensured that new files are served as expected. Clients may need to clear their maven repository if not done so already to fix the problem. (In reply to comment #18) > (In reply to comment #17) > > Hi Alex, > > please see https://bugs.eclipse.org/bugs/show_bug.cgi?id=356202 > > xtext-proxy should point to > > http://build.eclipse.org/jobs/Xtext-Maven-Deploy/maven/ where any tycho poms > > are/were contained. > > That is absolutely the wrong thing to do. We don't want to delegate out any > random repository to the public lookup list; otherwise we could have a > situation where any corrupt artefacts could be propagated, which is exactly > what happened here. > > When I visit this link, I get a 'not found' and the build.eclipse.org suggests > that I go to hudson.eclipse.org instead. Maybe an environmental change caused > the move but I suspect that was the start of it. > > We should instead enable the Xtext maven jobs to deploy automatically to one of > the existing repositories e.g. integration, so that you can push your specific > components into the repository instead of us delegating to many. The only one possibility to deploy things to nexus, without building them with maven, is to use deploy-file. Deploy-file can only deploy one file per call, correct me if it's wrong. We have a lot of bundles, that means a kind of foreach shell script is needed. That is in IMHO a bad idea. Are there an another possibility to do this? |
