Bugzilla – Full Text Bug Listing
| Summary: | are mirrors and download.php working ok? | ||
|---|---|---|---|
| Product: | Community | Reporter: | David Williams <david_williams> |
| Component: | Cross-Project | Assignee: | Cross-Project issues <cross-project.inbox> |
| Status: | RESOLVED INVALID | QA Contact: | |
| Severity: | normal | ||
| Priority: | P3 | CC: | john.arthorne, richard.gronback |
| Version: | unspecified | ||
| Target Milestone: | --- | ||
| Hardware: | PC | ||
| OS: | Windows XP | ||
| Whiteboard: | |||
|
Description
David Williams
(In reply to comment #0) > The reason I ask is that while trying to do many Galileo monster builds, we got > a LOT of "file not found" exceptions from various mirrors. I don't understand. download.eclipse.org is mere inches away from build.eclipse.org, really, so I'm not sure why you're getting your jars from the States. This is the one instance where it is preferable to use download.eclipse.org directly without mirrors -- they are all connected to the same Gigabit switch. > For one, I thought the mirror system is supposed to avoid quite a bit of that, > with it's system of TIME stamps? It does, but it's not perfect. I just tried accessing all the links below, and they all worked but one (software-mirror). Does java.io.FileNotFound get returned if there is a timeout or any other condition? I've added -Declipse.p2.mirrors=false to the p2.director command that installs features in the base during the build, so hopefully we won't have many mirror issues in Galileo from now on. Disabling mirrors is probably the right answer for this particular case, but we don't want to recommend this to people not running directly on eclipse.org machines. The mirroring always puts the eclipse.org sites *last* in the mirrors list, so it is never contacted unless all mirrors fail. I don't think Denis would be pleased with the result if we put it at the top of the mirror list, but I can make that change if given the word ;) Also, as mentioned on cross-project, p2 was giving up on FileNotFoundException rather than retrying, but this has been fixed recently. Just discussed this Pascal and Matthew "mirror guru" Piggott. Our only theory is that the ganymatic is getting started before the mirroring of those sites has completed. For example if the Ganymatic is kicked off very soon after a project has contributed (or recontributed) their bits to their update site, then that update site may not have completed mirroring yet. We have found it can sometimes take up to 24 hours for mirrors to fully propagate after a large build contribution. Those links are fine now, but it's hard to say if they had content at the time the failure occurred. In any case, disabling mirroring here should solve the problems, as will the recent p2 fix that will exhaustively try every mirror (including eclipse.org) before giving up on a file.
> then that update site may not have completed mirroring yet
But in this case, it shouldn't be returned by download.php right?
Feel free to close this as invalid or works for me.
I was just surprised since I know that in the past, when we (accidentally) touched a file in update site directory, the list of mirrors would immediately drop to zero, and stay at zero for an hour or two (until some had synced up).
So, I was surprised my hacky work-around attempt didn't work. (And Richard has fixed the script so our eclipse based builds won't use the mirrors, so I won't try to fool the mirroring system again :)
So, I'm not worried ... just thought you might want to know in case it was a sign somethign was amiss.
(In reply to comment #4) > Our only theory > is that the ganymatic is getting started before the mirroring of those sites > has completed. Agreed, but judging by the filename (org.eclipse.wst.common.modulecore_1.1.300.v200903120000.jar) I would assume that seven hours from David's bug post to my attempt shouldn't have mattered. Therefore, I'm curious to know why he got a not found error when the file "should" be 8 days old (if the name can be believed). David: I just tried the 'touch' technique, and it still works. http://www.eclipse.org/downloads/download.php?file=/technology/babel/eclipse_properties-R-3.4.tar.gz That file had about 40 mirror sites until I touched it. A refresh of that page showed me zero mirrors. (In reply to comment #6) > (In reply to comment #4) > when the file "should" be 8 days old (if the name can be believed). Yeah, can't believe those names. That's more a name of when it was last changed in cvs and built, as opposed to it's age. I wonder if P2 has a stale mirror list? (i.e. caches it too long) ... but, I know in my little scenario, I started P2 after I did the touch ... so, I'd think it'd get a fresh list of mirrors each invocation? Another thing that can come into play is requesting a mirrorsURL based on a directory instead of a file. For instance, no matter how much you touch this file: 2009-03-20 13:45 eclipse_properties-R-3.4.tar.gz It will not not update the parent directory timestamp: 2009-03-20 11:36 . I've recommended against using a directory for the mirrorsURL for this reason, but JohnA doesn't want to believe me. I may need to bribe him with beer. Mmm, beer. Ah, what you argued previously was sort of the opposite (bug 266280 comment 16). There you suggested against the directory timestamp because it changed too *frequently*. This didn't bother me so much - better too frequent and getting less mirrors than not enough and getting incorrect results. Combining these comments I get: - adding/removing files changes directory timestamp - touching files doesn't change directory timestamp I have entered bug 269570 to address this in p2. There is still no perfect answer because we are downloading hundreds of files, so whatever timestamp we use won't always be accurate. I think from all the discussion, things are working as best they can. Thanks everyone for checking. |