Bugzilla – Full Text Bug Listing
| Summary: | [engine] ProvisioningContext always perform reference following for artifact repositories | ||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Eclipse Project] Equinox | Reporter: | Konstantin Komissarchik <konstantin> | ||||||||||||||||||||||||
| Component: | p2 | Assignee: | Matthew Piggott <matthew> | ||||||||||||||||||||||||
| Status: | VERIFIED FIXED | QA Contact: | |||||||||||||||||||||||||
| Severity: | major | ||||||||||||||||||||||||||
| Priority: | P3 | CC: | john.arthorne, pascal, Silenio_Quarti, susan, tjwatson | ||||||||||||||||||||||||
| Version: | 3.6 | Flags: | john.arthorne:
review+
susan: review+ dj.houghton: review+ tjwatson: review+ |
||||||||||||||||||||||||
| Target Milestone: | 3.6 RC3 | ||||||||||||||||||||||||||
| Hardware: | PC | ||||||||||||||||||||||||||
| OS: | Windows 7 | ||||||||||||||||||||||||||
| Whiteboard: | |||||||||||||||||||||||||||
| Attachments: |
|
||||||||||||||||||||||||||
It's possible that you could be encountering changes surrounding repository reference following. See bug 310206 comment #0. In particular, if a metadata repository references an artifact repository, that repository is now consulted to find artifacts. So, for example, if your local copy of the metadata repository references a remote artifact repository, that repository would be checked for artifacts. In this case, you would want to ensure that the local copy referenced the artifact repository local copy rather than the remote one. My local repositories are mostly zipped p2 repos downloaded from various Eclipse projects. Metadata and artifact repositories are co-located. Would you be able to provide steps to help reproduce this? Marking 3.6 so it does not get forgotten. I am not sure how to setup a portable repro. I think my setup is pretty basic build situation. I use p2repos provided by Eclipse projects such as JDT, PDE, EMF and WTP. I am installing features from those repos. You might be able to repro this if you cut your network connection and call into the director. If there is any trace flags that I can enable that will give you more to go on, let me know. I can run a one-off build with custom tracing too if necessary. I haven't been able to reproduce this problem here, does this happen for you on RC1 or later? The issue still exists with RC1. (In reply to comment #6) > The issue still exists with RC1. We have not been able to reproduce. You will have to provide very detailed steps so we can make progress on this. Pascal, I am not sure how I could provide you with detailed steps. Isn't this situation (complicated setups that are next to impossible to boil down to a standalone repro) pretty typical for p2 bugs? If you guys have tracing that I can enable, I can send back log files. Otherwise I would recommend searching the code base for "Retrying request" and "executeWithRetry". I would imagine that an infinite loop around executeWithRetry method introduced during M7 (problem wasn't there in M6) would be easy to spot. - Konstantin Ok. Ok. I was starting to feel guilty about not spending more time trying to narrow this down... So I spent a few hours this afternoon in hope of creating a portable repro. Still don't have one, but do have new information... The problem seems to be related to the set of repositories that you pass to the director rather than what you are trying to install. What's odd is that ALL repositories that I am passing in are local (either in zip or exploded form). I do not pass in references to any remote repositories. I have a repro setup that tries to install JDT into platform. Pretty basic stuff. If I give the director invocation a reference to just the local JDT repo, the install succeeds. If I instead give it reference to the same local JDT repo together with a bunch of other repos, the install goes into an infinite timeout loop. Actually, it isn't an infinite loop, but it does try whatever it is trying to do several dozen times before giving up. If I wait long enough, the install does succeed. The important thing to note is that there is nothing p2 needs on the network for what I asked it to install. I gave it a local repo with JDT (along with other local repos) and asked to install JDT. I will try a process of elimination to narrow down the repository that is causing p2 to go on unnecessary internet excursion. Created attachment 169242 [details]
Repro (build.xml)
Got to love the process of elimination. I have managed to boil this down to a portable repro. The problematic repository that is causing p2 to go an unnecessary internet excursion appears to be eclipse-junit-tests-*.zip (the unit tests for Eclipse project).
1. Go to Helios build page for platform. You will need eclipse-platform-*.zip for your OS. You will also need org.eclipse.jdt.source-*.zip repo and eclipse-junit-tests-*.zip repo. The second one is a zip inside the unit tests zip that you download. Put these in an empty directory.
2. Make sure you have ant installed and ant-contrib available in its lib directory.
3. Download the attached build.xml file and put it into your repro directory.
4. Cut your network connection.
5. Go to the repro directory in a command prompt.
6. Run "ant -Dversion=I20100511-0800". You will need to substitute the version for the build that you've downloaded. This will succeed fairly quickly.
7. Run "ant -Dversion=I20100511-0800 -Dtests=true". This will go into the timeout loop.
The only difference between (6) and (7) is the presence of the unit tests repository in the list given to p2. I will attach the output that I get.
Created attachment 169244 [details]
Test Case Output (control)
Created attachment 169249 [details]
Test Case Output (internet excursion)
8. Restore internet connection. 9. Repeat step (7). It will complete quickly now. Looking at eclipse-junit-tests repository's content.xml file, I see the following declarations that look suspect: <references size="4"> <repository uri="http://download.eclipse.org/eclipse/updates/3.4" url="http://download.eclipse.org/eclipse/updates/3.4" type="0" options="0" /> <repository uri="http://download.eclipse.org/eclipse/updates/3.4" url="http://download.eclipse.org/eclipse/updates/3.4" type="1" options="0" /> <repository uri="http://download.eclipse.org/eclipse/updates/3.6" url="http://download.eclipse.org/eclipse/updates/3.6" type="0" options="0" /> <repository uri="http://download.eclipse.org/eclipse/updates/3.6" url="http://download.eclipse.org/eclipse/updates/3.6" type="1" options="0" /> </references> Ok. I spent 4 hours narrowing this down. The ball in p2 court now. It seems to me that p2 can be smart enough to look at local repositories first before trying reach across the network. If something like that is not possible, a "don't follow nested repository references" switch for p2 director would allow us to disable the behavior that is causing problems for us. Thanks for the time spent on that and providing all this information. Matt will be investigating that first thing in the morning. Normally p2 contacts the local repositories first. This logic is built in the DownloadManager. Now I wonder if we do not try to go out to the network to get the list of mirrors for some of these sites. We are still investigating, moving to RC3 since the last build for RC2 is today. I've been able to partially reproduce your problem, but with my setup the attempts to contact outside sites timeout immediately without an error message. Would you be able to add the following JVM arguments to the director and capture the output: -Dorg.apache.commons.logging.Log=org.apache.commons.logging.impl.SimpleLog -Dorg.apache.commons.logging.simplelog.showdatetime=true -Dorg.apache.commons.logging.simplelog.log.httpclient.wire.header=debug -Dorg.apache.commons.logging.simplelog.log.org.apache.commons.httpclient=debug -Dorg.apache.commons.logging.simplelog.log.httpclient.wire=debug Where should I look for the output? I added these switches, but am not seeing anything different in the console. No obvious log files in the base directory either. Created attachment 169434 [details]
Repro (build.xml with log debug switches)
I see from Google that SimpleLog should send stuff to System.err, so it's either there is nothing useful coming out or the switches aren't being added correctly. I am attaching the changed script. Let me know if this is not the correct way to set these switches.
Since you are invoking the director using the java.exe (and not eclipse.exe) make sure to pass the arguments using the jvmargs option of the java ant task (http://ant.apache.org/manual/CoreTasks/java.html) The output should occur on the standard output. I don't know what Ant will do with those so it may be necessary for you to run the app directly from a command prompt. We believe we have an explanation for this situation but we would like to confirm what is actually happening in your setup. In short, it is a mix of reference following, weird proxy, usage of local repos that are not recognized as such and more :) But I don't want to spoil it until we have the complete story :) Created attachment 169435 [details]
Test Case Output (with debug)
Using jvmarg element did the trick. Test case output with debug output turned on is attached.
Is this the complete log or just part of it? Could you try unzipping the repositories you are installing from. Don't forget to change your script also make sure to refer to those repositories using an absolute path. thx Created attachment 169445 [details]
Test Case Output (with debug - unzipped repos)
That is the entire output. You should be able to see the start where it is expanding platform zip and the standard ant completion statement. I tried unzipped repos. This time it fails roughly at the same time that the other way finished.
Created attachment 169454 [details] Test Case Output (with debug - unzipped repos) > Don't forget to change your script also make sure to refer to those > repositories using an absolute path. Can't say I wasn't warned. Unzipped repos debug output try 2. Created attachment 169514 [details]
DirectorApplication and ProvisioningContext patch
What occurs here is that the p2 ProvisioningContext follows the references in the junit repository, and as the referenced repo (3.6 updates) is enabled as part of the platform we attempt to load it when looking for artifacts. The repository load attempts to retrieve 6 files (p2.index, compositeAritfacts.jar/.xml, aritfacts.jar/.xml, and site.xml).
When I've reproduced this behaviour both without a lan and with lan and no internet the connection attempts fail very quickly, however from the logs your connection attempts don't fail and timeout after 20 seconds, and retry 6 more times. (6x7x20s = 16.8 minutes) Does the system have a proxy setup, or anything unusual with its network?
The patch I've attached makes two changes: the first is modifying ProvisioningContext to obey the FOLLOW_REPOSITORY_REFERENCES property for artifact repositories as it does for metadata repositories; the second change has the DirectorApplication set the property to false when it creates a ProvisioningContext.
Created attachment 169519 [details]
Updated patch
I've updated the patch to add a flag to the director to enable reference following.
> Does the system have a proxy setup, or anything unusual with its network?
Oracle network is behind a proxy. The first time I came across this is when I connected to the network over VPN and did not set proxy configuration before running the build.
So... behind a proxy with proxy settings not configured in the client.
Context for reviewers. This is a regression that got introduced when we added support to follow repository references. In 3.5 and in 3.6 (until we broke it) the director application would only consult the repositories that were explicitly listed on the command line. However when we added support for following references (enabled by default) we missed the case for artifact repositories which ended up causing up the issue. The core of the fix is in ProvisioningContext. The changes in DirectorApplication are here to be explicit about how we want references to be followed and also add a command line flag to allow for referenced repositories to be followed. +1. Remember to use the proper NON-NLS tags and to update the copyright statments. (In reply to comment #27) > Context for reviewers. This is a regression that got introduced when we added > support to follow repository references. In 3.5 and in 3.6 (until we broke it) > the director application would only consult the repositories that were > explicitly listed on the command line. However when we added support for > following references (enabled by default) we missed the case for artifact > repositories which ended up causing up the issue. This behavior is intentional. It's not so much a regression as it is a change in behavior that was added to fix bug 222838 and bug 278191. See bug 310206 comment 0. Created attachment 169997 [details]
Updated patch
After speaking with Pascal I've updated the patch to add a second property which specifically controls following references to artifact repositories, by default the property is set to follow references.
Created attachment 170002 [details]
Updated patch
Made the property private
This looks good to me. There is no behaviour change in the engine unless the new "follow artifact references" property is set. The only one setting this property is the director. Since the director never followed references in the past this brings us back to the old behaviour in 3.5. Now the question is whether the director should follow artifact references by default. I think the answer is "no". The director has always had the ability to allow the user to control the metadata and artifact repository references separately, so the co-located versus non co-located issues seen by the UI don't come up. In headless contexts such as build where the director is used, the ability to strictly control the source repositories is important so automatically following references is potentially dangerous. I.e., it might find a different version through a reference that it wouldn't find otherwise. And of course in offline situations we don't want to be hitting the network unless the user of the director explicitly asked for it. patch looks good to me. Since the default for this new private property is true, we will retain the M7 behavior for the SDK use cases. This preserves the fixes for bug 222838 and 278191, as verified by all green ProvisioningContextTests. (As added confidence in the test cases, there were three failures with the first patch.) As an added paranoid step, I tested bug 278191 manually, and it is still working as expected. I was able to resolve and install from another eclipse install without [ ] Contact all sites checked. Note that I'm less familiar with the director code, but in principle those changes look correct, and the property must be false in order to deliver the 3.5 "don't follow" behavior expected for build scenarios. I didn't test the actual scenario reported in this bug. I am not that familiar with the code and I believe you do not need my vote. I could not see anything wrong for what is worth. +1 to the patch and fixing this in RC3. Fix released in HEAD. Verified with RC3. Thanks. |
It appears that something has changed for M7 that is making p2 director hit the network when it shouldn't and then get stuck in a loop. We call p2 director during a build to install stuff from local repositories. Haven't seen any issues with this before. Starting with M7, if the computer running the build doesn't have network access (or must use proxy for access), the build hangs with what appears to be an infinite loop of timeout-retry. See this output: [mkdir] Created dir: D:\Oracle\OEPE\Depot\mission\helios\build\target [unzip] Expanding: D:\Oracle\OEPE\Depot\mission\helios\releng\eclipse\eclipse-platform-3.6M7-win32.zip into D:\Oracl e\OEPE\Depot\mission\helios\build\target [java] Installing org.eclipse.jdt.feature.group 3.6.0.v20100427-7z8XFSkFMTdeJGmeBTJvz-kn4502. [java] Installing org.eclipse.pde.feature.group 3.6.0.v20100427-7b7kFHlFEx2XloYdxGBUcUB. [java] Installing org.eclipse.cvs.feature.group 1.2.0.v20100427-7B77FHk8sF7B77MGQ5DMUF. [java] Installing org.eclipse.gef.feature.group 3.6.0.v20100201-1030-777B-81A2Bz-6766395E33747. [java] Installing org.eclipse.emf.feature.group 2.6.0.v201003151457-7a8X-9e7C9iV-xQXul4-DgnI8. [java] Installing org.eclipse.xsd.feature.group 2.6.0.v201003151457-35-7w312114262568. [java] Installing org.eclipse.datatools.common.doc.user.feature.group 1.8.0.v201004271500-26-311A16321A3557. [java] Installing org.eclipse.datatools.connectivity.doc.user.feature.group 1.8.0.v201004271500-37D-7733L3D753L7BBF . [java] Installing org.eclipse.datatools.connectivity.feature.feature.group 1.8.0.v201004271500-7C7b8cEt1ZvqibVXpU5O w4z-u4sU. [java] Installing org.eclipse.datatools.connectivity.oda.designer.feature.feature.group 1.8.0.v201004271500-4107w31 21151A1A53. [java] Installing org.eclipse.datatools.connectivity.oda.feature.feature.group 1.8.0.v201004271500-7H7C4CcNBHBBQKaD _NP. [java] Installing org.eclipse.datatools.doc.user.feature.group 1.8.0.v201004271500-47C08w95ENAK6AFDFK7. [java] Installing org.eclipse.datatools.enablement.apache.derby.feature.feature.group 1.8.0.v201004271500-77778bBmK DMXMfJpJ8Oc7QBR. [java] Installing org.eclipse.datatools.enablement.feature.feature.group 1.8.0.v201004271500-7J8x0BWwUIyw2ynlWaI0iE wFz-LL. [java] Installing org.eclipse.datatools.enablement.hsqldb.feature.feature.group 1.8.0.v201004271500-67C0AqGBJVKbHlH 1GU7QBR. [java] Installing org.eclipse.datatools.enablement.ibm.feature.feature.group 1.8.0.v201004271500-7F47IFC7sRV5SfZ0YE XR. [java] Installing org.eclipse.datatools.enablement.ingres.feature.feature.group 1.8.0.v201004271500-540AkF77g7VBLBP H7. [java] Installing org.eclipse.datatools.enablement.jdbc.feature.feature.group 1.8.0.v201004271500-4-29oB56N5N7L6PAQ . [java] Installing org.eclipse.datatools.enablement.jdt.feature.feature.group 1.8.0.v201004271500-2-07w31211518181A. [java] Installing org.eclipse.datatools.enablement.msft.feature.feature.group 1.8.0.v201004271500-542AkF79P7QCP9SDB . [java] Installing org.eclipse.datatools.enablement.mysql.feature.feature.group 1.8.0.v201004271500-546AkF78Y7R9PAX7 7. [java] Installing org.eclipse.datatools.enablement.oda.designer.feature.feature.group 1.8.0.v201004271500-3328s7353 373J5FB3. [java] Installing org.eclipse.datatools.enablement.oda.feature.feature.group 1.8.0.v201004271500-7A7T78DZRDFuD_KmFc Np. [java] Installing org.eclipse.datatools.enablement.oracle.feature.feature.group 1.8.0.v201004271500-548bAkF78Y7VBJ9 ZEB. [java] Installing org.eclipse.datatools.enablement.postgresql.feature.feature.group 1.8.0.v201004271500-542AkF77g7V 9N9e77. [java] Installing org.eclipse.datatools.enablement.sap.feature.feature.group 1.8.0.v201004271500-540AkF77g7V9N9e77. [java] Installing org.eclipse.datatools.enablement.sqlite.feature.feature.group 1.8.0.v201004271500-541AkF79P7N8NAQ 77. [java] Installing org.eclipse.datatools.enablement.sybase.feature.feature.group 1.8.0.v201004271500-7E45F9NiNWvOOXx PdVT. [java] Installing org.eclipse.datatools.intro.feature.group 1.8.0.v201004271500-26-7w311A17142212. [java] Installing org.eclipse.datatools.modelbase.feature.feature.group 1.8.0.v201004271500-77077CcNBGKBRJaGSUd. [java] Installing org.eclipse.datatools.sqldevtools.data.feature.feature.group 1.8.0.v201004271500-623BgJ9CRAQDRDgJ G. [java] Installing org.eclipse.datatools.sqldevtools.ddl.feature.feature.group 1.8.0.v201004271500-277-78B03G8S_IM7J tPnRwVl. [java] Installing org.eclipse.datatools.sqldevtools.ddlgen.feature.feature.group 1.8.0.v201004271500-7A-2F7RZHL6IsO mQvUk. [java] Installing org.eclipse.datatools.sqldevtools.feature.feature.group 1.8.0.v201004271500-7N7y1FDrfpTXJ8PhuUCC2 CJRz0Ao. [java] Installing org.eclipse.datatools.sqldevtools.parsers.feature.feature.group 1.8.0.v201004271500-621BgJ9Ah9WBT EcIH. [java] Installing org.eclipse.datatools.sqldevtools.results.feature.feature.group 1.8.0.v201004271500-42-9oB57E5J9H 7D59. [java] Installing org.eclipse.datatools.sqldevtools.schemaobjecteditor.feature.feature.group 1.8.0.v201004271500-42 08375LG5BJ93413. [java] Installing org.eclipse.datatools.sqldevtools.sqlbuilder.feature.feature.group 1.8.0.v201004271500-79-0EVVFIx GhLaMvL_. [java] Installing org.eclipse.datatools.sqltools.doc.user.feature.group 1.8.0.v201004271500-37D-7733L3D753L7BBF. [java] Installing org.eclipse.wst.common.fproj.feature.group 3.2.0.v201004210642-377A78s73533B7J3356. [java] Installing org.eclipse.wst.common_core.feature.feature.group 3.2.0.v201004210642-7B7DFEdF7RZHOZIqQvP0QJ. [java] Installing org.eclipse.wst.common_ui.feature.feature.group 3.2.0.v201004210642-7C78FMVEdhOZvYJcwkTez0_JrRwQ. [java] Installing org.eclipse.wst.jsdt.feature.feature.group 1.2.0.v201004150600-7C78FGDF9JgLWLM-cKMz0W9. [java] Installing org.eclipse.wst.server_adapters.feature.feature.group 3.2.0.v201004210628-51EoAkF77g8HBSc. [java] Installing org.eclipse.wst.server_core.feature.feature.group 3.2.0.v201004210628-30F8T8s734C3D5H. [java] Installing org.eclipse.wst.server_ui.feature.feature.group 3.2.0.v201004210628-7B4FADAtJ_TtEKMZJKDPEM866. [java] Installing org.eclipse.wst.server_userdoc.feature.feature.group 3.1.10.v201004210628-20A77w311A21262755. [java] Installing org.eclipse.wst.web_core.feature.feature.group 3.2.0.v201004210707-7E7AFHkAJrn5NLX3lWIGsuHf_6As. [java] Installing org.eclipse.wst.web_ui.feature.feature.group 3.2.0.v201004210707-7O7BFb3EMf6XlkhHH1z00lDAb_nU. [java] Installing org.eclipse.wst.web_userdoc.feature.feature.group 3.1.0.v201004210707-31DF8s733J3E4D56BB. [java] Installing org.eclipse.wst.ws_core.feature.feature.group 3.2.0.v201004210628-7L7RFaJFGtGcvlhm2dv1z08462A. [java] Installing org.eclipse.wst.ws_ui.feature.feature.group 3.2.0.v201004210628-7I78FW2EtEoHoV6ClIL_lIX65TQO. [java] Installing org.eclipse.wst.ws_userdoc.feature.feature.group 3.1.0.v201004210628-44FHl9oB56L5O8L9KAQ. [java] Installing org.eclipse.wst.ws_wsdl15.feature.feature.group 1.5.3.v201004210628-2407w311917141519. [java] Installing org.eclipse.wst.xml_core.feature.feature.group 3.2.0.v201004210707-7C7OFXTF7RZHMqHmTpMpZv. [java] Installing org.eclipse.wst.xml_ui.feature.feature.group 3.2.0.v201004210707-7H7AFRADxumKhLR7fb2JMe9ekqFW. [java] Installing org.eclipse.wst.xml_userdoc.feature.feature.group 3.2.0.v201004210707-50FSgAkF78Y7LDWCIDN. [java] Installing org.eclipse.wst.xsl.feature.feature.group 1.1.0.v200909271729-7S7WF8lFIpRz0sMP89z-hItOOFTI. [java] Installing org.eclipse.jst.common.fproj.enablement.jdt.feature.group 3.2.0.v201004210642-377A78s735339586E8C . [java] Installing org.eclipse.jst.enterprise_core.feature.feature.group 3.2.0.v201004210707-52FMRAkF7B77NBPAUF7. [java] Installing org.eclipse.jst.enterprise_ui.feature.feature.group 3.2.0.v201004210707-7b7GHQSFSK2W4HPRHYiYsMW4T mel. [java] Installing org.eclipse.jst.enterprise_userdoc.feature.feature.group 3.2.0.v201004210707-62FK5BgJ9AfASCeGGNb. [java] Installing org.eclipse.jst.jsf.apache.trinidad.tagsupport.feature.feature.group 2.2.100.v20100421-208Z7w3121 14252964. [java] Installing org.eclipse.jst.server_adapters.ext.feature.feature.group 3.2.0.v201004210628-777HFGPCcNBDiBfJ_4B A8. [java] Installing org.eclipse.jst.server_adapters.feature.feature.group 3.2.0.v201004210628-208Z7w31211419. [java] Installing org.eclipse.jst.server_core.feature.feature.group 3.2.0.v201004210628-31F8U8s734C3F6B. [java] Installing org.eclipse.jst.server_ui.feature.feature.group 3.2.0.v201004210628-7A4FB_9xFaYsCFJOKRKTFF355. [java] Installing org.eclipse.jst.server_userdoc.feature.feature.group 3.2.0.v201004210628-2-DF7w311916261244. [java] Installing org.eclipse.jst.webpageeditor.feature.feature.group 2.3.0.v20100421-46-9oB5855G8M. [java] Installing org.eclipse.jst.web_core.feature.feature.group 3.2.0.v201004210642-7W7LG6sFJmiqPeLLTfB7errin3ew. [java] Installing org.eclipse.jst.web_ui.feature.feature.group 3.2.0.v201004210642-7J7DFJJF7nQl3z-tp28S6aoIZdL5. [java] Installing org.eclipse.jst.web_userdoc.feature.feature.group 3.1.0.v201004210707-20A77w311A17263135. [java] Installing org.eclipse.jst.ws.axis2tools.feature.feature.group 1.1.100.v201004210628-78-FDTDZRDJLDgRJKxIl. [java] Installing org.eclipse.jpt.eclipselink.feature.feature.group 2.3.0.v201004210000-7778BgBgJ9E99PJX9999. [java] Installing org.eclipse.jpt.feature.feature.group 2.3.0.v201004210000-7N7TEwFD3wTd6UMnWYt_i. [java] Installing org.python.pydev.feature.feature.group 1.5.6.2010033101. [java] Installing org.springframework.ide.eclipse.feature.feature.group 2.3.0.200912170948-RELEASE. [java] Installing oracle.eclipse.tools.helios.glassfish.feature.group 2.0.1.201004301326. [java] May 3, 2010 9:44:24 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry [java] INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection timed out: connec t [java] May 3, 2010 9:44:24 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry [java] INFO: Retrying request [java] May 3, 2010 9:44:45 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry [java] INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection timed out: connec t [java] May 3, 2010 9:44:45 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry [java] INFO: Retrying request [java] May 3, 2010 9:45:06 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry [java] INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection timed out: connec t [java] May 3, 2010 9:45:06 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry [java] INFO: Retrying request [java] May 3, 2010 9:45:48 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry [java] INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection timed out: connec t [java] May 3, 2010 9:45:48 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry [java] INFO: Retrying request [java] May 3, 2010 9:46:09 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry [java] INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection timed out: connec t [java] May 3, 2010 9:46:09 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry [java] INFO: Retrying request [java] May 3, 2010 9:46:30 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry [java] INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection timed out: connec t [java] May 3, 2010 9:46:30 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry [java] INFO: Retrying request [java] May 3, 2010 9:47:12 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry [java] INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection timed out: connec t [java] May 3, 2010 9:47:12 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry [java] INFO: Retrying request [java] May 3, 2010 9:47:33 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry [java] INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection timed out: connec t [java] May 3, 2010 9:47:33 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry [java] INFO: Retrying request [java] May 3, 2010 9:47:54 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry [java] INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection timed out: connec t [java] May 3, 2010 9:47:54 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry [java] INFO: Retrying request [java] May 3, 2010 9:48:36 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry [java] INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection timed out: connec t [java] May 3, 2010 9:48:36 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry [java] INFO: Retrying request [java] May 3, 2010 9:48:57 AM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry [java] INFO: I/O exception (java.net.ConnectException) caught when processing request: Connection timed out: connec t