Community
Participate
Working Groups
We discovered that the mirror application runs fairly inconsistently when downloading against remote repositories. Upon further investigation we discovered that the code used to retry failed connections exists in the getArtifact(IArtifactRequest[]) method but not in getArtifact(IArtifactDescriptor) or getRawArtifactDescriptor(IArtifactDescriptor). getArtifact(IArtifactRequest[]) eventually calls getArtifact(IArtifactDescriptor). Would it be possible to move the retry code further down so both methods (and getRawArtifact, which is used by the mirror application) can use it? Alternatively, we could modify the mirror application to use IArtifactRequests instead of simply IArtifactDescriptors.
I took a brief look and I think implementing IAritfactRequest in the mirror application will require a considerable amount of work. Especially since most fo my comparator work relies on the fact that we do all our manipulation and then call for just the download. I think implementing the retry code in the getArtifact and getRawArtifact methods will be slightly simpler. I'm sure I can adapt the existing code and maybe even remove a method from MirrorRequest. I'll release a patch when I have something.
Created attachment 120208 [details] Moves retry code into SimpleArtifactRepository This moves the responsibility of ensure retries to the repository from MirrorRequest. I've run this a few time against the ganymede repository and it seems to work. here is the output I get: Mirroring: org.eclipse.datatools.connectivity.oda.feature/org.eclipse.update.feature/1.6.1.v200809191145-7F27PBcMAAwAtKkESVZ (Descriptor: canonical: org.eclipse.datatools.connectivity.oda.feature/org.eclipse.update.feature/1.6.1.v200809191145-7F27PBcMAAwAtKkESVZ) Dec 10, 2008 4:23:29 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: I/O exception (java.net.SocketException) caught when processing request: Operation timed out: connect:could be due to invalid address Dec 10, 2008 4:23:29 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: Retrying request Dec 10, 2008 4:23:50 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: I/O exception (java.net.SocketException) caught when processing request: Operation timed out: connect:could be due to invalid address Dec 10, 2008 4:23:50 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: Retrying request Dec 10, 2008 4:24:11 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: I/O exception (java.net.SocketException) caught when processing request: Operation timed out: connect:could be due to invalid address Dec 10, 2008 4:24:11 PM org.apache.commons.httpclient.HttpMethodDirector executeWithRetry INFO: Retrying request Mirroring: org.eclipse.jst.servlet.ui.infopop/osgi.bundle/1.0.300.v200805140415 (Descriptor: canonical: org.eclipse.jst.servlet.ui.infopop/osgi.bundle/1.0.300.v200805140415) Please note that the specified file was manually verified to be in the destination I ran some preliminary tests but this needs more testing before it will be releasable. Example: I'm not sure how to force a retry during a jUnit test case.
We have to be really careful about changing this. We had some severe bugs very late in the 3.4 release in this area (bug 233214, bug 236157). This is code that is only heavily tested once we have real mirrors so we tend to not discover problems until late.
(In reply to comment #3) > We have to be really careful about changing this. We had some severe bugs very > late in the 3.4 release in this area (bug 233214, bug 236157). This is code > that is only heavily tested once we have real mirrors so we tend to not > discover problems until late. The way I see it, the mirror application must be able to retry otherwise it's essentially a useless application for mirroring remote repositories. I see 3 scenarios: 1) Modify mirror application to use requests. This however would require some serios work as artifact requests want to perform processing steps (which is why we have getRawArtifact in the first place since getArtifact was unzipping pack200 files). 2) Move the retry into the repositories. Seems fair enough as if a repository wants content, it should be responsible for ensuring it gets it correctly. This is what my patch implements. 3) Put the try in the mirror application. I'm ashamed to say I hadn't thought of this option until right now. this might be viable since getRawArtifact is only is only called twice and the code to do the actual retry is fairly simple (even then, we can create a private wrapper method). I'm going to try this and see how ti turns out but right now my only concern is what about when getArtifact or getRawArtifact are called from outside the Mirror Application? They still won't be able to retry. then again, they couldn't before so this might not be a major loss.
I also just noticed that the above patch may have compilation errors. I am looking into this.
Created attachment 120244 [details] Much better implementation This duplicates the retry code in Mirroring.java. Pretty minor change, don't know why I didn't think of it first. Changes are localized to a single class. Seems to behave the same as my previous patch. I haven't observed the retry message I talked about in my previous patch yet. Both patches would occasionally give me the following error: !ENTRY org.eclipse.equinox.p2.metadata.repository 4 0 2008-12-11 14:21:17.765 !MESSAGE ProvisioningEventBus could not be obtained. Metadata caches may not be cleaned up properly. !ENTRY org.eclipse.osgi 4 0 2008-12-11 14:36:35.000 !MESSAGE Application error !STACK 1 java.lang.NullPointerException at org.apache.commons.httpclient.HttpMethodBase.getStatusCode(HttpMethodBase.java:566) at org.eclipse.ecf.provider.filetransfer.httpclient.HttpClientRetrieveFileTransfer.getResponseCode(HttpClientRetrieveFileTransfer.java:248) at org.eclipse.ecf.provider.filetransfer.httpclient.HttpClientRetrieveFileTransfer.openStreams(HttpClientRetrieveFileTransfer.java:369) at org.eclipse.ecf.provider.filetransfer.retrieve.AbstractRetrieveFileTransfer.sendRetrieveRequest(AbstractRetrieveFileTransfer.java:639) at org.eclipse.ecf.provider.filetransfer.retrieve.AbstractRetrieveFileTransfer.sendRetrieveRequest(AbstractRetrieveFileTransfer.java:381) at org.eclipse.ecf.provider.filetransfer.retrieve.MultiProtocolRetrieveAdapter.sendRetrieveRequest(MultiProtocolRetrieveAdapter.java:95) at org.eclipse.equinox.internal.p2.artifact.repository.ECFTransport.transfer(ECFTransport.java:167) at org.eclipse.equinox.internal.p2.artifact.repository.ECFTransport.performDownload(ECFTransport.java:120) at org.eclipse.equinox.internal.p2.artifact.repository.ECFTransport.download(ECFTransport.java:100) at org.eclipse.equinox.internal.p2.artifact.repository.simple.SimpleArtifactRepository.downloadArtifact(SimpleArtifactRepository.java:460) at org.eclipse.equinox.internal.p2.artifact.repository.simple.SimpleArtifactRepository.downloadArtifact(SimpleArtifactRepository.java:443) at org.eclipse.equinox.internal.p2.artifact.repository.simple.SimpleArtifactRepository.getRawArtifact(SimpleArtifactRepository.java:514) at org.eclipse.equinox.internal.p2.artifact.mirror.Mirroring.downloadArtifact(Mirroring.java:170) at org.eclipse.equinox.internal.p2.artifact.mirror.Mirroring.mirror(Mirroring.java:121) at org.eclipse.equinox.internal.p2.artifact.mirror.Mirroring.run(Mirroring.java:71) at org.eclipse.equinox.internal.p2.artifact.mirror.MirrorApplication.start(MirrorApplication.java:69) at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:193) at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:110) at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:79) at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:366) at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:177) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:618) at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:550) at org.eclipse.equinox.launcher.Main.basicRun(Main.java:505) at org.eclipse.equinox.launcher.Main.run(Main.java:1237) at org.eclipse.equinox.launcher.Main.main(Main.java:1213) !ENTRY org.eclipse.osgi 2 0 2008-12-11 14:36:35.125 !MESSAGE The following is a complete list of bundles which are not resolved, see the prior log entry for the root cause if it exists: !SUBENTRY 1 org.eclipse.osgi 2 0 2008-12-11 14:36:35.125 !MESSAGE Bundle org.eclipse.ecf.identity_2.0.1.qualifier [33] was not resolved. !SUBENTRY 1 org.eclipse.osgi 2 0 2008-12-11 14:36:35.125 !MESSAGE Bundle org.eclipse.jdt.apt.pluggable.core_1.0.200.v20080930-0055 [271] was not resolved. !SUBENTRY 2 org.eclipse.jdt.apt.pluggable.core 2 0 2008-12-11 14:36:35.125 !MESSAGE Missing imported package org.eclipse.jdt.internal.compiler.tool_0.0.0. !SUBENTRY 2 org.eclipse.jdt.apt.pluggable.core 2 0 2008-12-11 14:36:35.125 !MESSAGE Missing imported package org.eclipse.jdt.internal.compiler.apt.dispatch_0.0.0. !SUBENTRY 2 org.eclipse.jdt.apt.pluggable.core 2 0 2008-12-11 14:36:35.125 !MESSAGE Missing imported package org.eclipse.jdt.internal.compiler.apt.model_0.0.0. !SUBENTRY 2 org.eclipse.jdt.apt.pluggable.core 2 0 2008-12-11 14:36:35.125 !MESSAGE Missing imported package org.eclipse.jdt.internal.compiler.apt.util_0.0.0. !SUBENTRY 1 org.eclipse.osgi 2 0 2008-12-11 14:36:35.125 !MESSAGE Bundle org.eclipse.jdt.compiler.apt_1.0.200.v20081209-0315 [273] was not resolved. !SUBENTRY 2 org.eclipse.jdt.compiler.apt 2 0 2008-12-11 14:36:35.125 !MESSAGE Missing imported package org.eclipse.jdt.internal.compiler.tool_0.0.0. !SUBENTRY 1 org.eclipse.osgi 2 0 2008-12-11 14:36:35.125 !MESSAGE Bundle org.eclipse.jdt.compiler.tool_1.0.100.v_929 [274] was not resolved. Any help would be welcome.
Created attachment 120464 [details] Error log file I'm including this log file because I'll occasionally get this error when testing. I was talking to Paul about Bug 258488 and was told that this issue should be fixed in the most recent builds but as of I20081211-1908 I'm still receiving the error.
Mirrored Ganymede with default arguments 3 times (deleting the destination between attempts) and it all seems to work fine as of N20081215-2000. I think this might be ready for release.
Released to HEAD.
