Bugzilla – Full Text Bug Listing
| Summary: | Update/Install new software chooses wrong mirror | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Community | Reporter: | Oded Arbel <oded> | ||||
| Component: | Website | Assignee: | phoenix.ui <phoenix.ui-inbox> | ||||
| Status: | RESOLVED FIXED | QA Contact: | |||||
| Severity: | minor | ||||||
| Priority: | P3 | CC: | chris.guindon, denis.roy, d_a_carver, edouard, henrik.lindberg, irbull, jan.sievers, kozakcsabi, markus.tiede, mn, pascal, remy.suen, thomas, tolgaduran1980 | ||||
| Version: | unspecified | ||||||
| Target Milestone: | --- | ||||||
| Hardware: | PC | ||||||
| OS: | All | ||||||
| Whiteboard: | |||||||
| Attachments: |
|
||||||
|
Description
Oded Arbel
This is not so much a p2 issue as it is a en eclipse foundation problem as the selection of "optimal" mirrors is done at the server side. A quick fix is to turn off the use of mirrors. The functionality in p2 (there is an issue logged) that is lacking is that the download speed should be monitored and if below a certain threshold an attempt should be made to use a different mirror. Created attachment 235117 [details]
Screenshot showing progress monitor during download
(In reply to Henrik Lindberg from comment #1) > ... A quick fix is to turn off the use of mirrors. > If only it was possible to turn of mirrors selectively. AFAIK, the only thing you can do is to turn off mirroring completely. We often have the same problem in Sweden. I recently hit a very slow one from Portugal. Takes forever to download a 5.5MB file when the bit rate is 10K per second (see screenshot). If we can come to a general concensus that a mirror is bad, I can simply remove it from the list. Or, should our mirror list detect that a request for mirrors came from p2, and only pull those we know to be awesome (such as OSU OSL, Georgia Tech, others)? Feel free to move this to Community > website and we'll investigate fixing the mirrors issue on the server side. (In reply to Denis Roy from comment #4) > If we can come to a general concensus that a mirror is bad, I can simply > remove it from the list. > > Or, should our mirror list detect that a request for mirrors came from p2, > and only pull those we know to be awesome (such as OSU OSL, Georgia Tech, > others)? > > Feel free to move this to Community > website and we'll investigate fixing > the mirrors issue on the server side. removing mirrors known to be "bad" most of the time and from anywhere is a good first step. You may be able to find this out by actively monitoring them with pingdom or similar. Problem is that "bad" is probably often not a constant in space and time. I.e. a mirror may be very slow or unreachable from certain locations or intermittently only. For reference, the mirror choosing algorithm on the p2 client side is here http://git.eclipse.org/c/equinox/rt.equinox.p2.git/tree/bundles/org.eclipse.equinox.p2.artifact.repository/src/org/eclipse/equinox/internal/p2/artifact/repository/MirrorSelector.java not sure how much this can be influenced from the server side. From my experience, the algorithm often chooses a mirror e.g. in Brazil when the p2 client is in Germany which is not an optimal choice given there are several german mirrors available. > removing mirrors known to be "bad" most of the time and from anywhere is a > good first step. You may be able to find this out by actively monitoring > them with pingdom or similar. We currently do more than that -- we already fetch timestamp files that are peppered throughout the downloads area. Even if the timestamp is fetched successfully, we will remove stale mirrors that are still very responsive but are not up to date with the content. > not sure how much this can be influenced from the server side. > > From my experience, the algorithm often chooses a mirror e.g. in Brazil when > the p2 client is in Germany which is not an optimal choice given there are > several german mirrors available. We perform a geographic lookup based on IP and provide p2 with a list that is sorted according to the location (as is seen on www.eclipse.org/downloads). However, for p2, we could scope the list of mirrors to a specific continent: - North American client = North American mirrors listed - European client = European mirrors only .. and so on. (In reply to Denis Roy from comment #6) > - North American client = North American mirrors listed > - European client = European mirrors only > .. and so on. I like the idea. BTW I played around a little with the p2 mirrors URL for Luna SR2 http://www.eclipse.org/downloads/download.php?format=xml&file=/releases/luna/201409261001/ - I'm getting a slightly different list on every request (I guess that's by design) - but the first ten or so mirrors in the list are always in Germany From [1]: "The mirrors are assumed to be already * sorted geographically with closer mirrors first." So looks to me like the server works as expected. I have a suspicion that the comparator [1] used by the client to decide which mirror to use does not give enough weight to "initial rank" as served by the server and can sometimes get thrown off by its own measurements. So from my point of view simply reducing the number of mirrors served to the ones "closest" (in terms of network) right away is a very pragmatic approach. We don't trust the mirror selection algorithm on the client, so we simply give it the "right" mirrors to choose from :) Also, I like it because I think it could help right now without any p2 client side code change required. [1] http://git.eclipse.org/c/equinox/rt.equinox.p2.git/tree/bundles/org.eclipse.equinox.p2.artifact.repository/src/org/eclipse/equinox/internal/p2/artifact/repository/MirrorSelector.java#n157 (In reply to Jan Sievers from comment #7) > BTW I played around a little with the p2 mirrors URL for Luna SR2 Luna SR1, that is https://git.eclipse.org/r/36663 Changes to the mirror class to be able to determine machine browsers. I need this new function for download.php (patch to follow) Patch to download.php: https://git.eclipse.org/r/#/c/36705/ It can be tested like this: # mirror list for human/browser wget -qO - --header "User-Agent: Mozilla/1.7.1" http://www.eclipse.org/downloads/download-dev.php?file=/technology/epp/downloads/release/luna/SR1/eclipse-java-luna-SR1-linux-gtk.tar.gz\&format=xml # Mirror list for p2/Machine wget -qO - --header "User-Agent: Java/1.7.1" http://www.eclipse.org/downloads/download-dev.php?file=/technology/epp/downloads/release/luna/SR1/eclipse-java-luna-SR1-linux-gtk.tar.gz\&format=xml Moving to Community. (In reply to Oded Arbel from comment #0) > The problem is evident when connecting from Israel, which is for some reason > classified as in Asia and thus gets mirrors from south-east Asia country > even though the network connectivity to south-easy Asia is rather slow and > much worse then to Europe or North America. (In reply to Henrik Lindberg from comment #1) > This is not so much a p2 issue as it is a en eclipse foundation problem as > the selection of "optimal" mirrors is done at the server side. > A quick fix > is to turn off the use of mirrors. Unfortunately for some users, the hints we provide to p2 based on our knowledge of geography will always have some flaws. But the patches herein will help guide p2 along a better path. > From my experience, the algorithm often chooses a mirror e.g. in Brazil when
> the p2 client is in Germany which is not an optimal choice given there are
> several german mirrors available.
This is in place now, so the above scenario should no longer reproduce itself, unless p2 caches the list of mirrors.
Is there anyone out there who can verify this actually makes the situation better?
Typical *nix commands if someone wants to see how we deliver mirror lists to broswers vs. what we consider machines. # mirror list for human/browser wget -qO - --header "User-Agent: Mozilla/1.7.1" http://www.eclipse.org/downloads/download.php?file=/technology/epp/downloads/release/luna/SR1/eclipse-java-luna-SR1-linux-gtk.tar.gz\&format=xml # Mirror list for p2/Machine wget -qO - --header "User-Agent: Java/1.7.1" http://www.eclipse.org/downloads/download.php?file=/technology/epp/downloads/release/luna/SR1/eclipse-java-luna-SR1-linux-gtk.tar.gz\&format=xml (In reply to Denis Roy from comment #13) > Typical *nix commands if someone wants to see how we deliver mirror lists to > broswers vs. what we consider machines. > > # mirror list for human/browser > wget -qO - --header "User-Agent: Mozilla/1.7.1" > http://www.eclipse.org/downloads/download.php?file=/technology/epp/downloads/ > release/luna/SR1/eclipse-java-luna-SR1-linux-gtk.tar.gz\&format=xml > > # Mirror list for p2/Machine > wget -qO - --header "User-Agent: Java/1.7.1" > http://www.eclipse.org/downloads/download.php?file=/technology/epp/downloads/ > release/luna/SR1/eclipse-java-luna-SR1-linux-gtk.tar.gz\&format=xml Executing these commands from my office in Germany, I can confirm I am getting the full list of 102 mirrors worldwide when using a browser User-Agent and a reduced list of only 49 mirrors from Europe and Russia only when using a Java/1.7.1 User-Agent. >This is in place now, so the above scenario should no longer reproduce itself, >unless p2 caches the list of mirrors. maybe someone from the p2 team can confirm the list of mirrors is not being cached? > Is there anyone out there who can verify this actually makes the situation > better? the nature of the problem is sporadic (but when it happens it's quite noticeable). I'll try to do several Luna updates/installs of lots of plugins in a row with p2 HTTP tracing [1] switched on to see if it no longer reaches out to mirrors outside Europe/Russia from Germany. [1] https://wiki.eclipse.org/Equinox/p2/Reporting_Problems I'm getting mirror errors as follows: [INFO] Fetching org.eclipse.jdt.debug_3.7.1.v20111006_r372.jar.pack.gz from http://ftp.osuosl.org/pub/eclipse//eclipse/updates/3.7/R-3.7.2-201202080800/plugins/ (0B of 272.38kB at 0B/s) [ERROR] All attempts to read artifact osgi.bundle,org.eclipse.jdt.debug,3.7.1.v20111006_r372 failed: [ERROR] An error occurred while transferring artifact packed: osgi.bundle,org.eclipse.jdt.debug,3.7.1.v20111006_r372 from repository http://download.eclipse.org/eclipse/updates/3.7/R-3.7.2-201202080800: [ERROR] Problems downloading artifact: osgi.bundle,org.eclipse.jdt.debug,3.7.1.v20111006_r372.: [ERROR] File has invalid content:/tmp/signatureFile3353991001397831433.jar: [ERROR] Invalid content:jdi.jar [ERROR] Invalid content:jdimodel.jar [ERROR] An error occurred while transferring artifact canonical: osgi.bundle,org.eclipse.jdt.debug,3.7.1.v20111006_r372 from repository http://download.eclipse.org/eclipse/updates/3.7/R-3.7.2-201202080800: [ERROR] Retry another mirror: [ERROR] HTTP Server 'Service Unavailable': http://ftp.ussg.iu.edu/eclipse/eclipse/updates/3.7/R-3.7.2-201202080800/plugins/org.eclipse.jdt.debug_3.7.1.v20111006_r372.jar can we blacklist http://ftp.ussg.iu.edu/? (In reply to Denis Roy from comment #12) > Is there anyone out there who can verify this actually makes the situation > better? I did 5 installs from scratch with HTTP tracing as per comment 14 of a large number of plugins provided by the Eclipse Luna p2 repository. From the logs I can see that only mirrors in Europe were used (and most of them in Germany), so the situation w.r.t. preferring geographically closer mirrors has improved for me. Thanks a lot Denis! (In reply to Ricardo Gladwell from comment #15) > [ERROR] HTTP Server 'Service Unavailable': > http://ftp.ussg.iu.edu/eclipse/eclipse/updates/3.7/R-3.7.2-201202080800/ > plugins/org.eclipse.jdt.debug_3.7.1.v20111006_r372.jar > > can we blacklist http://ftp.ussg.iu.edu/? I just tried clicking that link, and was able to download the JAR directly within chrome. The site might have been having issues getting the file or the server at that particular point in time. Ideally if it can't find it at one location, p2 should retry at the next mirror it knows about, but not sure what it does. p2 has always been a magic black box to me. > I just tried clicking that link, and was able to download the JAR directly > within chrome. The site might have been having issues getting the file or > the server at that particular point in time. I'm not sure if modifications to p2 algorithms are outside the scope of this ticket. I think its important to blacklist unreliable/corrupt p2 repos like http://ftp.ussg.iu.edu/ blacklisted ASAP. (In reply to Ricardo Gladwell from comment #15) > [ERROR] HTTP Server 'Service Unavailable': > http://ftp.ussg.iu.edu/eclipse/eclipse/updates/3.7/R-3.7.2-201202080800/ > plugins/org.eclipse.jdt.debug_3.7.1.v20111006_r372.jar > > can we blacklist http://ftp.ussg.iu.edu/? I can fetch that file without problems either. BUT ... you seem to be running a version of Eclipse that is almost 3 years old. Surely p2 has been improved since then, making this type of use case obsolete? (In reply to Ricardo Gladwell from comment #18) > I think its important to blacklist unreliable/corrupt p2 repos like > http://ftp.ussg.iu.edu/ blacklisted ASAP. As mentioned previously, we crawl all our mirrors every hour and fetch between 6 and 24 timestamp files. If the mirror is still listed, it's because we can fetch those files, and those files indicate to us that the mirror is up-to-date. > From the logs I can see that only mirrors in Europe were used (and most of
> them in Germany), so the situation w.r.t. preferring geographically closer
> mirrors has improved for me.
>
> Thanks a lot Denis!
Great, that was the scope of this bug. I'm pretty sure there are other bugs against p2 that pertain to tweaking the algorithm and making it more resilient to failure.
Closing as fixed.
> I can fetch that file without problems either. > > BUT ... you seem to be running a version of Eclipse that is almost 3 years old. Surely p2 has been improved since then, making this type of use case obsolete? I believe Tycho is doing the p2 resolution, we are just running regression tests for old versions of Eclipse. I'm not sure if Tycho would run p2 resolution for an older version for Eclipse when running tests, or use the current version it has access to. > As mentioned previously, we crawl all our mirrors every hour and fetch between 6 and 24 timestamp files. If the mirror is still listed, it's because we can fetch those files, and those files indicate to us that the mirror is up-to-date. That's great, but why am I regularly getting this failures in my builds? Is this because we are using an old p2 implementation? Would this be a bug with Tycho, because the Tycho developers think this is a problem with the p2 mirror list you guys maintain. > That's great, but why am I regularly getting this failures in my builds?
I don't know. p2 should perhaps continue on to another mirror but it doesn't seem to do that. I know many teams disable mirrors to have reliable builds.
> I know many teams disable mirrors to have reliable builds.
Unfortunately, disabling mirrors for p2 turns a 20m build into a 1h+ build so its not a viable option for our downstream project.
If many teams are having to disable mirrors, would this indicate this is a reproducible and important issue with p2 mirrors?
> If many teams are having to disable mirrors, would this indicate this is a > reproducible and important issue with p2 mirrors? Absolutely. The system is inherently flawed, but we get most of our bandwidth for free at the expense of reliability. > Unfortunately, disabling mirrors for p2 turns a 20m build into a 1h+ build > so its not a viable option for our downstream project. And there you have the cost of compromise :) In the end, I don't think there's much more we can do on the server-side. We query our mirrors and remove the ones we can't talk to, but even the best of systems cannot guarantee that every mirror will work for everyone on the planet all the time. As mentioned, p2 should be able to deal with one, or even many bad mirrors more gracefully and, worse case, default to the home site. I'm sure there's a bug opened against p2 for that. > Absolutely. The system is inherently flawed, but we get most of our bandwidth for free at the expense of reliability. So if a p2 mirror is misbehaving regularly there is nothing that can be done about it? > I'm sure there's a bug opened against p2 for that. Who would be the best person to talk to to find out if such a bug exists? Hi, sorry for the late reply - it took me a while to get time to review this. For me the problem is not fixed: living in Israel, I still get only asian mirrors on the short list, most of these are Chinese and Taiwanese sites - which is almost the worst choice for me - only Australia is slower for me. See below for a ping test from my home system (connected with 100MBps DSL), listing all hosts returned by Denis' 1st query from comment #13, with the hosts returned by Denis' 2nd query marked (XXX means no response). Basically the summary is that only *one* host *not* in the short list is slower than any host in the short list. So for me, this problem is really not solved - for Israeli users (and I expect most other countries in the middle east) the current mirror selection algorith selects the *worst* mirrors possible. 76 ftp.wh2.tu-dresden.de 79 ftp.fau.de 79 www.mirrorservice.org 81 ftp.fau.de 85 eclipse.ialto.com 86 ftp.halifax.rwth-aachen.de 88 ftp.snt.utwente.nl 88 mirror.netcologne.de 90 eclipse.mirror.kangaroot.net 93 inf2.uniri.hr 93 www.rcp-vision.com 96 mirrors.linux-bg.org 99 ftp-stud.fht-esslingen.de 100 mirror.selfnet.de 103 eclipse.mirror.dkm.cz 111 ftp.ntua.gr 128 artfiles.org 132 ftp.acc.umu.se 135 ftp.heanet.ie 137 eclipse.mirror.triple-it.nl 143 eclipse.mirror.triple-it.nl 151 mirrors.nic.cz 152 mirror.cc.columbia.edu 152 rm.mirror.garr.it 153 eclipse.mirror.garr.it 158 eclipse.mirror.rafal.ca 159 ftp.linux.org.tr 160 mirrors.nic.cz 164 mirror.switch.ch 165 eclipse.tsl.gr 167 ftp.ussg.iu.edu 173 carroll.aset.psu.edu 178 mirrors.ibiblio.org 180 mirror.csclub.uwaterloo.ca 180 mirror.tspu.ru 183 mirror.cc.vt.edu 189 eclipse.dcc.fc.up.pt 205 mirrors.xmission.com 235 ftp.osuosl.org 236 ftp.osuosl.org 260 ** mirror.bit.edu.cn 264 ** mirrors.yun-idc.com 303 ** mirrors.ustc.edu.cn 322 eclipse.c3sl.ufpr.br 340 ** ftp.jaist.ac.jp 344 ** ftp.yz.yamagata-u.ac.jp 350 ** ftp.kaist.ac.kr 369 ** eclipse.stu.edu.tw 372 ** eclipse.stu.edu.tw 378 ** eclipse.stu.edu.tw 388 ** mirrors.neusoft.edu.cn 404 ** mirror.neu.edu.cn 405 ** ftp.neu.edu.cn 415 ** download.nus.edu.sg XXX ftp.saix.net XXX kambing.ui.ac.id XXX kambing.ui.ac.id XXX mirror.hust.edu.cn XXX mirror.hust.edu.cn XXX mirrors.hustunique.com XXX mirror.ufs.ac.za XXX www.ftp.saix.net XXX www.gtlib.gatech.edu Here is the Bash script I used to generate the results above (sort -n annoyingly sorts any non-number last, so the scripts produces unresposive hosts first, please ignore that anomaly): url='http://www.eclipse.org/downloads/download.php?file=/technology/epp/downloads/release/luna/SR1/eclipse-java-luna-SR1-linux-gtk.tar.gz&format=xml'; hostlist="$(wget -qO - --header "User-Agent: Java/1.7.1" "$url" | perl -nle 'm,https?://([^/]+), and print $1')"; wget -qO - --header "User-Agent: Mozilla/5.0" "$url" | perl -nle 'm,https?://([^/]+), and print $1' | while read host; do ping="$(ping -qn -i 0.5 -c 3 $host | perl -nle 'm,= [\d\.]+/(\d+), and print $1')"; printf "%s\t%s%s\n" "${ping:-XXX}" "$(grep -q $host <<<"$hostlist" && echo '** ')" "$host"; done | sort -n Also, regarding disabling mirrors - I don't think that there are problems with specific mirrors, instead there are problems with specific mirrors for specific users. I'm sure Chinese users love to get Chinese mirrors on the short list and will be very annoyed if those mirrors would be turned off. For this bug, I believe the solution is simple: stop cataloguing by the geo-ip reported continent, and start treating the middle east as part of Europe. [2016-07-12 11:32:38] Executing bootstrap tasks [2016-07-12 11:32:38] Java(TM) SE Runtime Environment 1.8.0_91-b15 [2016-07-12 11:32:38] Product org.eclipse.products.epp.package.java.neon [2016-07-12 11:32:38] Bundle org.eclipse.oomph.setup 1.4.0.v20160530-1355, build=2444, branch=d2a285619d608146f2b29fa5d42db7d6201863c2 [2016-07-12 11:32:38] Bundle org.eclipse.oomph.setup.core 1.4.0.v20160530-1200, build=2444, branch=d2a285619d608146f2b29fa5d42db7d6201863c2 [2016-07-12 11:32:38] Bundle org.eclipse.oomph.setup.p2 1.4.0.v20160530-1200, build=2444, branch=d2a285619d608146f2b29fa5d42db7d6201863c2 [2016-07-12 11:32:38] Performing P2 Director (Eclipse IDE for Java Developers (Neon)) [2016-07-12 11:32:38] Offline = false [2016-07-12 11:32:38] Mirrors = true [2016-07-12 11:32:38] Resolving 23 requirements from 3 repositories to C:\Users\Java_Engineer\eclipse\java-neon\eclipse [2016-07-12 11:32:38] Requirement epp.package.java [4.6.0,4.7.0) [2016-07-12 11:32:38] Requirement org.eclipse.platform.feature.group [4.6.0,4.7.0) [2016-07-12 11:32:38] Requirement org.eclipse.rcp.feature.group [4.6.0,4.7.0) [2016-07-12 11:32:38] Requirement org.eclipse.buildship.feature.group [2016-07-12 11:32:38] Requirement org.eclipse.egit.feature.group [2016-07-12 11:32:38] Requirement org.eclipse.egit.mylyn.feature.group [2016-07-12 11:32:38] Requirement org.eclipse.jdt.feature.group [2016-07-12 11:32:38] Requirement org.eclipse.jgit.feature.group [2016-07-12 11:32:38] Requirement org.eclipse.m2e.feature.feature.group [2016-07-12 11:32:38] Requirement org.eclipse.m2e.logback.feature.feature.group [2016-07-12 11:32:38] Requirement org.eclipse.mylyn.bugzilla_feature.feature.group [2016-07-12 11:32:38] Requirement org.eclipse.mylyn.context_feature.feature.group [2016-07-12 11:32:38] Requirement org.eclipse.mylyn.git.feature.group [2016-07-12 11:32:38] Requirement org.eclipse.mylyn.hudson.feature.group [2016-07-12 11:32:38] Requirement org.eclipse.mylyn.ide_feature.feature.group [2016-07-12 11:32:38] Requirement org.eclipse.mylyn.java_feature.feature.group [2016-07-12 11:32:38] Requirement org.eclipse.mylyn.wikitext_feature.feature.group [2016-07-12 11:32:38] Requirement org.eclipse.mylyn_feature.feature.group [2016-07-12 11:32:38] Requirement org.eclipse.recommenders.mylyn.rcp.feature.feature.group [2016-07-12 11:32:38] Requirement org.eclipse.recommenders.rcp.feature.feature.group [2016-07-12 11:32:38] Requirement org.eclipse.recommenders.snipmatch.rcp.feature.feature.group [2016-07-12 11:32:38] Requirement org.eclipse.wst.xml_ui.feature.feature.group [2016-07-12 11:32:38] Requirement org.eclipse.oomph.setup.feature.group [2016-07-12 11:32:38] Repository http://download.eclipse.org/technology/epp/packages/neon [2016-07-12 11:32:38] Repository http://download.eclipse.org/releases/neon/201606221000 [2016-07-12 11:32:38] Repository http://download.eclipse.org/oomph/updates/milestone/latest [2016-07-12 11:35:49] ERROR: org.eclipse.equinox.p2.transport.ecf code=1002 Unable to read repository at http://download.eclipse.org/oomph/updates/milestone/latest/content.xml. java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.socketRead(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at java.net.SocketInputStream.read(Unknown Source) at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160) at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84) at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140) at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57) at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:260) at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283) at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251) at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197) at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:271) at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:123) at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685) at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487) at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:863) at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) at org.eclipse.ecf.provider.filetransfer.httpclient4.HttpClientFileSystemBrowser.runRequest(HttpClientFileSystemBrowser.java:259) at org.eclipse.ecf.provider.filetransfer.browse.AbstractFileSystemBrowser$DirectoryJob.run(AbstractFileSystemBrowser.java:69) at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55) [2016-07-12 11:35:49] On the server side, we've since made our mirroring heuristics and tracking much more robust, and have implemented transparent mirroring. I have not seen any reports of bad mirrors in a while. This is still an issue for Israeli users - here's the result of asking for the best 20 mirrors from Israel: $ curl -s 'http://www.eclipse.org/downloads/download.php?file=/technology/epp/packages/neon/p2.index&format=xml' | xmllint --xpath '//@label' - | perl -nle 'print $1 while(/\[([^\]]+)\]/g)' | head -n 20 Viet Nam China Philippines China Japan Taiwan Korea, Republic Of Taiwan Korea, Republic Of China Taiwan Japan China China Canada United States Canada United States United States United States From Israel the best connectivity is to Europe and the worst connectivity is to the Asia-Pacific region, but as you can see from the list above, the first 14 options are all Asia-Pacific followed by North America (where, granted connectivity is OK, if these were the only listed I'd have much less to complain about). Currently updating Eclipse from Israel, even for the smallest features, is a at least a half-an-hour affair. As an alternative, I'd like to present Fedora's mirroring system - which used to have the same problem ( https://bugzilla.redhat.com/show_bug.cgi?id=355881 ) - but they fixed it. When I now call the Fedora mirroring system, I get this: $ curl 'https://mirrors.fedoraproject.org/mirrorlist?repo=fedora-26&arch=x86_64' # repo = fedora-26 arch = x86_64 country = BE country = BY country = RU country = RO country = GB country = PL country = ES country = FR country = FI country = NL country = NO country = CZ country = SK country = SE country = DK country = DE country = AT country = IE country = UA https://ftp-stud.hs-esslingen.de/pub/fedora/linux/development/26/Everything/x86_64/os/ https://mirrors.n-ix.net/fedora/linux/development/26/Everything/x86_64/os/ https://mirrors.nic.cz/fedora/linux/development/26/Everything/x86_64/os/ https://www.mirrorservice.org/sites/dl.fedoraproject.org/pub/fedora/linux/development/26/Everything/x86_64/os/ http://mirror.slu.cz/fedora/linux/development/26/Everything/x86_64/os/ http://mirror.datacenter.by/pub/fedoraproject.org/linux/development/26/Everything/x86_64/os/ http://mirror2.hs-esslingen.de/fedora/linux/development/26/Everything/x86_64/os/ https://ftp.halifax.rwth-aachen.de/fedora/linux/development/26/Everything/x86_64/os/ http://fedora.uib.no/fedora/linux/development/26/Everything/x86_64/os/ http://www.nic.funet.fi/pub/mirrors/fedora.redhat.com/pub/fedora/linux/development/26/Everything/x86_64/os/ ... Thanks. From the Fedora bug: "I've remapped .IL into Europe. Because Israel has no mirrors, you will get the mirrorlist for Europe instead." Seems simple enough, it sounds like cheating but we could do that too and solve this now. Oded, do you think moving .il to Europe is the best solution? (In reply to Denis Roy from comment #32) > Seems simple enough, it sounds like cheating but we could do that too and > solve this now. Oded, do you think moving .il to Europe is the best > solution? I wish I had a better solution, but short of running connectivity tests to all endpoints whenever the client wants to update (which has a lot of down sides), I don't see any other way to handle that. It is my expectation that there are going to be a several more cases where that kind of hack is needed - as I've mentioned, outside North America and Europe, a plurality of international interconnections are the exception and not the rule, so GeoIP lookups are usually useless as a way to estimate connectivity. There are even cases (in Asia, Africa and even here in Israel) that two endpoints in the same city are connected to different ISPs and direct connection between them is slower than going out to a different continent. MariaDB [eclipse]> update SYS_countries set continent_code = "eu" where ccode = "il"; Perhaps try now? The mirror list now looks much more reasonable :-) $ curl -s 'http://www.eclipse.org/downloads/download.php?file=/technology/epp/packages/neon/p2.index&format=xml' | xmllint --xpath '//@label' - | perl -nle 'print $1 while(/\[([^\]]+)\]/g)' | head -n 20 Germany Czech Republic Italy Poland United Kingdom Czech Republic Germany Russian Federation Switzerland Croatia (Hrvatska) France Netherlands Germany Germany Germany Ireland Netherlands Turkey Sweden Canada I've ran a plugin install and the behavior is much improved. thanks! Great, thank you. Apologies for taking 6 years to fix this for you :) |