Bugzilla – Full Text Bug Listing
| Summary: | artifacts with 'integration' in the name are not mirrored | ||
|---|---|---|---|
| Product: | Community | Reporter: | Stephan Herrmann <stephan.herrmann> |
| Component: | Servers | Assignee: | Eclipse Webmaster <webmaster> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | P3 | CC: | david_williams, milesparker, thomas, webmaster |
| Version: | unspecified | ||
| Target Milestone: | --- | ||
| Hardware: | PC | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Bug Depends on: | |||
| Bug Blocks: | 345086 | ||
|
Description
Stephan Herrmann
(In reply to comment #0) > Given that the file exists on the server [...] By that I mean the file exists on download.eclipse.org. It's indeed missing from the mirrors. I have confirmed what you said. As of this time (11 PM Eastern, Sunday) there were about 40 mirrors being reported for the indigo M7 site, http://www.eclipse.org/downloads/download.php?file=/releases/indigo/201105060900/aggregate/&protocol=http And none of them contain either org.eclipse.wst.common.emfworkbench.integration_1.2.100.v201011172300.jar.pack.gz or it's "jar" counterpart, org.eclipse.wst.common.emfworkbench.integration_1.2.100.v201011172300.jar That said, p2 is supposed to "fall back" to download.eclipse.org as a last resort, so I'm surprised it would result in a "can not install" problem. So ... not sure what that is about. Do you know ... is your access to download.eclipse.org limited somehow? Say by a firewall or something? I'm currently testing one of the mirror sites exhaustively, to see if there are any others that report "not found" ... but, that's going to take a while longer, so thought I'd write what I've discovered so far. I did exhaustively check http://www.gtlib.gatech.edu (a mirror with a good reputation, I've heard) and found only one other artifact missing: org.eclipse.jst.ws.jaxws.dom.integration_1.0.100.v201101252155.jar.pack.gz and its jar counterpart org.eclipse.jst.ws.jaxws.dom.integration_1.0.100.v201101252155.jar I circled around and checked the previous list of 50 mirrors, and again, none of them had this particular artifact(s). I did not check any of the "features" artifacts ... only the "plugins" artifacts. I guess the thing to do is to "touch" those 4 files on download.eclipse.org, and see if that "forces" all the mirrors to update with those 4 files. But, since this will make it appear, temporarily, that there are zero "current" mirrors (while they get updated) I'll wait and confirm with webmaster. Also, as far as I know, maybe there is something about those artifact (or, more likely, their names?) that are causing them to be "filtered out" and not mirrored? Denis, Matt ... see anything odd about these names or artifacts? (I know we used to have a little issue with files with *.php* in the jar name ... but, even then did not effect each and every mirror. org.eclipse.jst.ws.jaxws.dom.integration org.eclipse.wst.common.emfworkbench.integration If not, I suggest we just "touch" those four files, and see if they propagate. It will be hard to read here in bugzilla, but below are the "ll" for each of the 4 ... nothing seems odd about their timestamps, permissions, or sizes ... so ... sunspots? = = = = = = = $ ll ~/downloads/releases/indigo/201105060900/aggregate/plugins/org.eclipse.jst.ws.jaxws.dom.integration_1.0.100.v201101252155.jar -rw-rw-r-- 1 david_williams callistoadmin 52K 2011-05-05 11:17 /home/data/users/david_williams/downloads/releases/indigo/201105060900/aggregate/plugins/org.eclipse.jst.ws.jaxws.dom.integration_1.0.100.v201101252155.jar [00:40:33] david_williams@build:/home/data/httpd/download.eclipse.org/releases/indigo/201105060900/aggregate/plugins $ ll ~/downloads/releases/indigo/201105060900/aggregate/plugins/org.eclipse.jst.ws.jaxws.dom.integration_1.0.100.v201101252155.jar.pack.gz -rw-rw-r-- 1 david_williams callistoadmin 21K 2011-05-05 11:17 /home/data/users/david_williams/downloads/releases/indigo/201105060900/aggregate/plugins/org.eclipse.jst.ws.jaxws.dom.integration_1.0.100.v201101252155.jar.pack.gz [00:40:33] david_williams@build:/home/data/httpd/download.eclipse.org/releases/indigo/201105060900/aggregate/plugins $ ll ~/downloads/releases/indigo/201105060900/aggregate/plugins/org.eclipse.wst.common.emfworkbench.integration_1.2.100.v201011172300.jar -rw-rw-r-- 1 david_williams callistoadmin 137K 2011-05-05 11:19 /home/data/users/david_williams/downloads/releases/indigo/201105060900/aggregate/plugins/org.eclipse.wst.common.emfworkbench.integration_1.2.100.v201011172300.jar [00:40:33] david_williams@build:/home/data/httpd/download.eclipse.org/releases/indigo/201105060900/aggregate/plugins $ ll ~/downloads/releases/indigo/201105060900/aggregate/plugins/org.eclipse.wst.common.emfworkbench.integration_1.2.100.v201011172300.jar.pack.gz -rw-rw-r-- 1 david_williams callistoadmin 49K 2011-05-05 11:19 /home/data/users/david_williams/downloads/releases/indigo/201105060900/aggregate/plugins/org.eclipse.wst.common.emfworkbench.integration_1.2.100.v201011172300.jar.pack.gz Think I'll move this to "cross project" list, since of general interest, and expertise, even though the 2 artifacts found that are not mirrored anywhere, are originally to "webtools", doesn't seem like a webtools issue. Seems, offhand, there is some infrastructure issue or glitch that caused 2 artifact to not be mirrored? In addition, seems maybe a p2 issue that an install is not falling back to original artifact repo, for which I opened bug 345086. Just a wild guess, but does the mirroring function have a filter that excludes files with the word 'integration' in them? All files reported missing so far has that word in them. (In reply to comment #5) > has that word in them. By that I meant in the file name, not the content. Perhaps that was obvious. (In reply to comment #5) > Just a wild guess, but does the mirroring function have a filter that excludes > files with the word 'integration' in them? All files reported missing so far > has that word in them. Gooood guess! And, wouldn't be a bad exclusion rule, if limited to strictly the "url address" of the resource (stuff before the final '/') but nothing should be filtered out based on the name of the resource itself. FYI, I checked whole "indigo repo" file system, (which does not include platform, or epp stuff) and there were only those 4 artifacts with "integration" in their name. (And these 4, btw, have been delivered by WTP since forever .. so not new).
Previous releases, such as helios, had more, Ganymede even more ... for Helios, it was just also TPTP which had some (pasted below) ... for Ganymede, Birt also had one (as feature, I think).
[03:01:42] david_williams@build:/home/data/httpd/download.eclipse.org/releases/helios
$ find ./ -iname "*integration*"
./201102250900/aggregate/features/org.eclipse.tptp.platform.integration.pde_4.5.2.v201101261720-5417w312212322161.jar
./201102250900/aggregate/plugins/org.eclipse.jst.ws.jaxws.dom.integration_1.0.0.v201004171919.jar
./201102250900/aggregate/plugins/org.eclipse.tptp.platform.integration.pde_4.5.2.v201101211050.jar.pack.gz
./201102250900/aggregate/plugins/org.eclipse.jst.ws.jaxws.dom.integration_1.0.0.v201004171919.jar.pack.gz
./201102250900/aggregate/plugins/org.eclipse.wst.common.emfworkbench.integration_1.2.3.v201012020600.jar.pack.gz
./201102250900/aggregate/plugins/org.eclipse.tptp.platform.integration.pde_4.5.2.v201101211050.jar
./201102250900/aggregate/plugins/org.eclipse.wst.common.emfworkbench.integration_1.2.3.v201012020600.jar
./201006230900/aggregate/features/org.eclipse.tptp.platform.integration.pde_4.5.0.v201005032111-54-7w312116172711.jar
./201006230900/aggregate/plugins/org.eclipse.jst.ws.jaxws.dom.integration_1.0.0.v201004171919.jar
./201006230900/aggregate/plugins/org.eclipse.jst.ws.jaxws.dom.integration_1.0.0.v201004171919.jar.pack.gz
./201006230900/aggregate/plugins/org.eclipse.wst.common.emfworkbench.integration_1.2.0.v201004110600.jar
./201006230900/aggregate/plugins/org.eclipse.tptp.platform.integration.pde_4.5.0.v201005061600.jar.pack.gz
./201006230900/aggregate/plugins/org.eclipse.tptp.platform.integration.pde_4.5.0.v201005061600.jar
./201006230900/aggregate/plugins/org.eclipse.wst.common.emfworkbench.integration_1.2.0.v201004110600.jar.pack.gz
./201009240900/aggregate/features/org.eclipse.tptp.platform.integration.pde_4.5.1.v201009092123-5407w31211A1A3234.jar
./201009240900/aggregate/plugins/org.eclipse.jst.ws.jaxws.dom.integration_1.0.0.v201004171919.jar
./201009240900/aggregate/plugins/org.eclipse.jst.ws.jaxws.dom.integration_1.0.0.v201004171919.jar.pack.gz
./201009240900/aggregate/plugins/org.eclipse.wst.common.emfworkbench.integration_1.2.2.v201008242000.jar.pack.gz
./201009240900/aggregate/plugins/org.eclipse.tptp.platform.integration.pde_4.5.1.v201009092123.jar.pack.gz
./201009240900/aggregate/plugins/org.eclipse.wst.common.emfworkbench.integration_1.2.2.v201008242000.jar
./201009240900/aggregate/plugins/org.eclipse.tptp.platform.integration.pde_4.5.1.v201009092123.jar
I'm going to assume our conclusion is correct, and assign to 'servers' and "webmaster@eclipse.org", and change title to be more specific. Denis, or Matt, just let me know if my assumption is not valid and we'll continue investigation. If there's anyway I can help, let me know. You are correct. We've been filtering out *integration* for mirrors that aren't replicating 'everything' for at least the last 2 years to save space. I'll look into what's not being replicated and how much space we're 'saving'. -M. If anyone's interested ... out of curiosity, I checked Helios SR2 and Indigo M6 and, out of about 60 mirrors, there were 4, and 2, respectively, that had the "org.eclipse.wst.common.emfworkbench.integration" artifact ... so, guess there are a few that "get everything". And, the fact there were _some_ that had it, may be the reason why bug 345086 has not shown up before this. *** Bug 345215 has been marked as a duplicate of this bug. *** If this can't be permanently fixed/settled soon (like today, Tuesday) is there an options to add "exceptions" to the rules? Such as "exclude lines with 'integration' except ... the four artifacts we know about". I ask since I think this is a pretty bad problem. Even if there was not the p2 bug that now cause complete install failure, even if p2 was working as intended ... having these few critical files on just 2 or 4 mirrors (out of 60) is a bottleneck for everyone ... not just when "installing XML" ... because XML is a "core" component, used by scores of other projects ... everything in webtools, for sure, but also PHP, Birt, and I am sure many many others. My guess is this has been a p2-install bottle-neck problem for all of Helios and we simply have no (easy) way to measure or detect such bottle necks. I've updated the exclusion pattern to "*integration*/" which should limit it to just directories. The changes have been pushed out and rsync restarted, so the mirrors should start updating soon. -M. (In reply to comment #14) > ... The changes have been pushed out and rsync restarted, so the > mirrors should start updating soon. Just to be sure ... so "we" need to "touch" anything higher up, so it looks out of date? Probably not, but wanted to be sure. When I "check mirrors" now, there is (still) 54 showing up for /releases/indigo/201105060900/aggregate/ Just wanted to make sure the udpates will happen "behind the scenes", even thought /releases/indigo/201105060900/aggregate/ reports "current" to download.php? Thanks so much, The changes should be transparent, but touching the files you need is cheap insurance. Mirrors are judged to be in-sync based on their ztime file(which is generated ~ by Eclipse.org every hour) so as long as they are within a few hours they are viewed as ok. Once the files have been mirrored it should look the same as it does now(except p2 will be happy). -M. Hmmm, well, now when I try to test ... even a request straight to download.eclipse.org get's redirected to a mirror. Maybe an unrelated issue? Or maybe someone's been playing with the tables too much? (ha ha) Did I cause this? :) wget http://download.eclipse.org/releases/indigo/201105060900/aggregate/plugins/org.eclipse.jst.ws.jaxws.dom.integration_1.0.100.v201101252155.jar --2011-05-10 11:47:15-- http://download.eclipse.org/releases/indigo/201105060900/aggregate/plugins/org.eclipse.jst.ws.jaxws.dom.integration_1.0.100.v201101252155.jar Resolving download.eclipse.org... 206.191.52.47 Connecting to download.eclipse.org|206.191.52.47|:80... connected. HTTP request sent, awaiting response... 302 Found Location: http://www.gtlib.gatech.edu/pub/eclipse/releases/indigo/201105060900/aggregate/plugins/org.eclipse.jst.ws.jaxws.dom.integration_1.0.100.v201101252155.jar [following] --2011-05-10 11:47:15-- http://www.gtlib.gatech.edu/pub/eclipse/releases/indigo/201105060900/aggregate/plugins/org.eclipse.jst.ws.jaxws.dom.integration_1.0.100.v201101252155.jar Resolving www.gtlib.gatech.edu... 128.61.111.9, 128.61.111.10, 128.61.111.11, ... Connecting to www.gtlib.gatech.edu|128.61.111.9|:80... connected. HTTP request sent, awaiting response... 404 Not Found 2011-05-10 11:47:16 ERROR 404: Not Found. (In reply to comment #17) > Hmmm, well, now when I try to test ... even a request straight to > download.eclipse.org get's redirected to a mirror. Ok, it's working now ... not sure if you fixed it ... or something unrelated. But (the reason I was looking) I've done a mini-test, and 6 of the 40 or so sites I tested the other day now have it (just tested one "integration jar") so think I won't touch anything, and assume the mirroring is occurring ... and I'll check later. [I fear if I touch something, it'll update the ztime file, and invalidate the whole site, temporarily, even for those updates/installs which don't need these "integration files" ... and ... after all ... seems to be slowly getting updated]. I'll check again later, to make sure there's more than only 6 sites (that is, that the number of mirrors that have them is indeed increasing). Thank you! I'll move to "resolved fixed", since all appears fine now. The "integration artifacts" I check now on about 12 mirrors, so they are propagating. I also did try the original "install xml editor" scenario that gave rise to the original bug report, and that did install as expected. Thanks again, |