Community
Participate
Working Groups
Under some circumstances it can happen that downloading a file from a remote ssh connection changes the timestamp of the original remote file, or may even truncate it: * Connect to (slow) remote ssh server (I used build.eclipse.org) * Select a directory that contains large files and choose "Show in Table". * Select a reasonably large file (e.g. RSE-runtime-core*.zip) * Copy and Paste to Windows-Local subsystem * Put the transfer into background * Open the Progress View to observe the background file transfer * While the background transfer is ongoing, double click on one or more remote files to open them in the editor * After the background transfer has finished, refresh the transfer's directory to check timestamp and file size. Sometimes, the timestamp changes after the transfer is finished. Rarely, it was also observed that the file was truncated. It definitely does not happen every time; but when opening 3 editor files while the transfer was ongoing, it was observed quite consistently. I'm not sure how to debug this, and how to find out what initiates the "touch". Perhaps the problem is due to multiple parallel access to the ChannelSftp - see bug 149179.
Maybe the timestamp change is due to sending invalid ssh packets to the server due to multiple simultaneous access to the server. Could perhaps be debugged in the Jsch Session by tracing the packets being sent.
The reason for the timestamp change seems to be, that for the copy & paste operation to Local, the file is first downloaded into the RemoteSystemsTempFiles area and moved only afterwards. In the temp area, the file is created with a different timestamp than the original one; so, when the TempFileSynchronizer runs, it thinks that it needs to upload the file back! Problem could be fixed either by doing drag&drop directly to the destination (would not be possible for copy & paste from one remote to a different remote filesystem!), or by making sure that the timestamp in the RemoteTempFiles cache is correct (bug 149150). Since the timestamp problem applies to dstore as well, the download / timestamp issue most probably applies to dstore too. Since Here is a backtrace: Thread [main] (Suspended (breakpoint at line 332 in ChannelSftp)) ChannelSftp.put(String, String, SftpProgressMonitor, int) line: 332 SftpFileService.upload(IProgressMonitor, File, String, String, boolean, String, String) line: 254 FileServiceSubSystem.upload(String, IRemoteFile, String, IProgressMonitor) line: 492 FileServiceSubSystem(RemoteFileSubSystem).uploadUTF8(String, IRemoteFile, IProgressMonitor) line: 1332 FileServiceSubSystem(RemoteFileSubSystem).uploadUTF8(IFile, IRemoteFile, IProgressMonitor) line: 1408 SystemUniversalTempFileListener.upload(IRemoteFileSubSystem, IRemoteFile, IFile, SystemIFileProperties, long, SystemEditableRemoteFile, IProgressMonitor) line: 231 SystemUniversalTempFileListener.doResourceSynchronization(ISubSystem, IFile, String, IProgressMonitor) line: 204 SystemUniversalTempFileListener(SystemTempFileListener).synchronizeTempWithRemote(IFile, IProgressMonitor) line: 578 SystemTempFileListener$RefreshResourcesUIJob.runInUIThread(IProgressMonitor) line: 190 UIJob$1.run() line: 94 RunnableLock.run() line: 35 UISynchronizer(Synchronizer).runAsyncMessages(boolean) line: 123 Display.runAsyncMessages(boolean) line: 3325 Display.readAndDispatch() line: 2971 Workbench.runEventLoop(Window$IExceptionHandler, Display) line: 1914 Workbench.runUI() line: 1878 Workbench.createAndRunWorkbench(Display, WorkbenchAdvisor) line: 419 PlatformUI.createAndRunWorkbench(Display, WorkbenchAdvisor) line: 149 IDEApplication.run(Object) line: 95 PlatformActivator$1.run(Object) line: 78 EclipseAppLauncher.runApplication(Object) line: 92 EclipseAppLauncher.start(Object) line: 68 EclipseStarter.run(Object) line: 400 EclipseStarter.run(String[], Runnable) line: 177 NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method] NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39 DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25 Method.invoke(Object, Object[]) line: 324 Main.invokeFramework(String[], URL[]) line: 336 Main.basicRun(String[]) line: 280 Main.run(String[]) line: 977 Main.main(String[]) line: 952
Fixed timestamps for ssh, the local subsystem still seems to do it wrong: when setting a breakpoint on SftpFileService.upload(), and downloading a few files (partially in background, partially in the editor), at some time RSE tries to upload a file back which SHOULD NOT BE DONE! Perhaps fixing bug 149150 for the Local system would fix the issue.
Hm, after fixing timestamps for Local subsystem, RSE still seems to do it wrong. The following sequence of actions leads to running into a breakpoint on SftpFileService.upload() that should never be called: * Copy & Paste to local: RSE-runtime-core*.zip, put in background * While in background, edit buildNotes.php and directory.txt * After complete, copy&paste to local: RSE-SDK-*.zip, put in background * While in background, edit index.php --> breakpoint is hit while trying to upload RSE-runtime-core*.zip Reassigning to DKM for analysis.
Setting severity Major since original files can be destroyed!
Workaround for avoiding the problem seems to never do any data transfer in background.
Is this problem reproducible when using dstore?
There are some changes to the code. There was a problem with the local time stamp obtained using the Eclipse API, which sometimes returns a cached timestamp. Is the problem still reproducible?
Kushal, can you look at this?
Yes the same problem is reproducable with HEAD as of 7-aug-2006: Windows client, Linux-dstore host (connection tunneled through ssh to make it slow). See also bug 149133
Reproduced with a local reasonably fast dstore connection too, so this should be really easy to reproduce: * Windows client, dstore-linux host (running the latest daemon). * Preparations: * On remote, create a filter "RSETest" (/folk/mober/RSETest). - Below it, create a folder "rsetest" - Copy eclpse-SDK-linux-gtk.tar.gz into "rsetest" (122MB) - Create 4 text files in "rsetest" (file1.txt, file2.txt, ...) * On local, create a filter "RSETest" (e.g. on I:\RSETest) - Below it, create a folder "rsetest" * The testcase: - On remote, select eclipse-SDK*.tar.gz, choose copy - On local, select RSETest/rsetest, choose paste - Put in background, click on lower right corner of Workbench to show progress - On remote, dbl click file1.txt - On remote, dbl click file2.txt, then file1.txt again --> it is important that file1.txt and file2.txt have not been transferred before. --> The background transfer should be stalled by now (bug 149133). If not, try to dbl click on several other text files. - Wait until all background jobs are finished. - Now copy eclipse-SDK*.tar.gz on the remote side, paste into RSETest/rsetest. This time, let the transfer finish completely. - Again, copy eclipse-SDK*.tar.gz from remote, paste into RSETest/rsetest. We expect overwriting the file this time. Put the transfer into background. WHile in background, dbl click on a text file that has been retrieved before. Close the editor. Dbl click it again. --> You will see the message "Synchonizing", then "Overwrite?" --> Remote file will have been truncated, thereby losing data. several text files to show in editor. Make sure to take some files that
Reassign to DaveM since Kushal is sick
The problem was that during the download, when an edited file is opened, a refresh on it's container IFolder is made, causing resource change events for all files including the one that is in mid download. The property information used to prevent an upload has not yet been set at this point in time so the upload ends up happening. I've fixed this by adding the downloaded file to the temp file listener ignore list for the duration of the download.
Verified in RC3 testing
[target cleanup] 1.0 M4 was the original target milestone for this bug
