Community
Participate
Working Groups
This week alone I've seen three cases of local builds (on build.eclipse.org and hudson.eclipse.org) that pull Eclipse.org plugins from our mirror sites. That is pure madness. The Eclipse servers (build, dev, git, download) all share a Gigabit LAN, so there is no need to download from mirror sites. In fact, doing so uses our precious Internet bandwidth, and the bandwidth of our mirrors. Please work with me to make this stop. I'd hate to have to impose restrictions on our firewalls to prevent this.
What specific builds had this issue?
It would be nice to know which builds, if possible, to add some focus to the effort .... but, I wouldn't be surprised if webtools was one of them. We added some p2 related aspects to our builds recently (such as, we use the orbit p2 enabled map files) but I never added -Declipse.p2.mirrors=false to any of our java invocations. That'd probably be a good practice. (But, I'll want to make it "variable" based on where build is running, since when running a test build locally, I wouldn't want to insist on getting jars from eclipse.org). (The aggregation builds do have -Declipse.p2.mirrors=false built in ... so if you see it from them, then there is an error somewhere, not just an oversight).
After much thought, the problem likely stems from the mirror list that we give to p2. I'll hack download.php to return only the home site in the mirrors lift if requests come from the inside to see how that helps.
(In reply to comment #3) > After much thought, the problem likely stems from the mirror list that we give > to p2. I'll hack download.php to return only the home site in the mirrors lift > if requests come from the inside to see how that helps. Hi Denis, while at it you might want to keep bug #319155 in mind. It's pretty much the same problem. Thanks Markus
I am launching the Buckminster import with "-Declipse.p2.mirrors=false": $buckminster import -Declipse.download="file:/home/data/httpd/download.eclipse.org" -Declipse.p2.mirrors=false "${WORKSPACE}/org.eclipse.gmt.modisco.releng.buckminster/modisco.mspec" But I notice it still tries to contact mirrors: Connection to http://eclipse.ialto.org/modeling/emf/emf/updates/2.6/plugins/org.eclipse.emf.ecore_2.6.0.v20100614-1136.jar.pack.gz failed on Unable to parse header: </a></div>. Am I doing something wrong, or is p2 ignoring the parameter?
> Am I doing something wrong, or is p2 ignoring the parameter? That file doesn't seem to exist on download.eclipse.org, so perhaps p2 is looking for it elsewhere to avoid failing at all costs. http://download.eclipse.org/modeling/emf/emf/updates/2.6/plugins/org.eclipse.emf.ecore_2.6.0.v20100614-1136.jar.pack.gz I've re-checked download.php and I forgot to include a local subnet to disable mirrors.
(In reply to comment #5) > Am I doing something wrong, or is p2 ignoring the parameter? Also ... A little hard to tell just from this ... more of a buckminster question, since (I think) it depends on how it parses and passes along the arguments on the command line.
I am still getting errors relative to mirrors: !ENTRY org.eclipse.equinox.p2.repository 2 0 2010-09-15 07:47:51.512 !MESSAGE Connection to http://www.mirrorservice.org/sites/download.eclipse.org/eclipseMirror/modeling/emf/emf/updates/2.6/plugins/org.eclipse.emf.exporter_2.6.0.v20100521-1846.jar.pack.gz failed on Connection refused. Retry attempt 0 started This path doesn't exist on download.eclipse.org either. I made Bug 324065 for this, but haven't got an answer from p2 yet.
I found a way to avoid contacting mirrors, by hacking plug-in "org.eclipse.equinox.p2.repository" in the Buckminster install used by the build. This resulted in a much faster build for me. In case anyone finds this useful, here is what I've done: In org.eclipse.equinox.internal.p2.repository.FileReader#sendRetrieveRequest(..), I wrapped the existing code with: if(uri.getHost().endsWith("eclipse.org")) { ... } else { System.out.println("[mirrors disabled] " + uri); throw new CoreException(Status.CANCEL_STATUS); } and then copied the FileReader.class into the "org.eclipse.equinox.p2.repository" jar in the Buckminster install.
Removing dependency because bug is not an issue with p2. See bug 325963 comment 9 for more information.
(In reply to comment #8) > I am still getting errors relative to mirrors: I've also put in a more robust fix for download.php. Anything local to the Eclipse Foundation will not get a mirrors list.
(In reply to comment #11) > I've also put in a more robust fix for download.php. Anything local to the > Eclipse Foundation will not get a mirrors list. Hi Denis, can this fix be adapted for bug #319155? Thanks
(In reply to comment #12) > can this fix be adapted for bug #319155? Absolutely, but I don't want to hardcode IP addresses in download.php (except my own). One thing we can do is leverage our "Internal Mirrors" concept. If I flag a mirror as Internal and provide a host mast (such as .osuosl.olg) the current approach is to place internal mirror first, and the other mirrors afterwards. We could add a field to the database to specify "Use Only Internal Mirror" and hack the SQL statements to only return the internal mirror. http://dev.eclipse.org/viewcvs/index.cgi/www/downloads/download.php?view=annotate&root=Eclipse_Website Since you're the main benefactor for this, feel like giving it a try? :-)
(In reply to comment #13) > (In reply to comment #12) > > can this fix be adapted for bug #319155? > > Absolutely, but I don't want to hardcode IP addresses in download.php (except > my own). One thing we can do is leverage our "Internal Mirrors" concept. If I > flag a mirror as Internal and provide a host mast (such as .osuosl.olg) the > current approach is to place internal mirror first, and the other mirrors > afterwards. We could add a field to the database to specify "Use Only Internal > Mirror" and hack the SQL statements to only return the internal mirror. > > http://dev.eclipse.org/viewcvs/index.cgi/www/downloads/download.php?view=annotate&root=Eclipse_Website > > Since you're the main benefactor for this, feel like giving it a try? :-) Hi Denis, looking at download.php it seems as if the script queries the database three times to aggregate all potential mirrors. Do we want to limit the "Use only internal mirror" functionality to only "EclipseFull" mirrors? Otherwise we will have to define behavior if the "Use only internal mirror" is true but the mirror does not carry the file. E.g. transparently falling back to another mirror might confuse consumers.
The DB is only queried once. The 3 SQL statement you see are unioned. You can add debug=1 to the URL of any mirror list to see additional info (and the big UNIONed query).
Created attachment 179977 [details] Use only internal mirrors (In reply to comment #15) > The DB is only queried once. The 3 SQL statement you see are unioned. You can > add debug=1 to the URL of any mirror list to see additional info (and the big > UNIONed query). I don't really know how to debug download.php locally without access to eclipse.org infrastructure. Anyway, am I right in assuming that for p2 $_redirect is set? If yes, the attached patch should do the trick (requires new column "MIR.internal_only"). :o
> Anyway, am I right in assuming that for p2 $_redirect is set? It doesn't -- otherwise, the internal_mirror hack would already work since internal mirrors are always listed first. p2 gets a list of mirrors for the (feature?) and uses that to download all the artifacts in it (plugins?). Sorry if I'm not aware of all the terminology, but p2 doesn't contact us for each jar.
(In reply to comment #17) > .... p2 gets a list of mirrors for the > (feature?) and uses that to download all the artifacts in it (plugins?). > Sorry > if I'm not aware of all the terminology, but p2 doesn't contact us for each > jar. It gets a list of mirrors for each artifact repository it needs to access. That's the terminology ... but not sure what the question is .... so not sure if that information helps you. For a big install, it normally contacts scores of artifact repositories, each containing hundreds if not thousands of jars/bundles. HTH
(In reply to comment #13) > Since you're the main benefactor for this, feel like giving it a try? :-) Denis, to get started, I have created a mirror site request for our build machine at OSU and the official OSU mirror.
> This week alone I've seen three cases of local builds (on build.eclipse.org and > hudson.eclipse.org) that pull Eclipse.org plugins from our mirror sites. I've been keeping an eye on our proxy logs, and I have sufficient evidence to demonstrate that this is no longer happening. I think the biggest contribution to the solution was to hack download.php to no longer return a list of mirrors when the client I address is on a local eclipse.org subnet.
