Community
Participate
Working Groups
A WAR file 200mb+ that is big enough to be copied for 5-10 seconds reproduces the issue. Currently Virgo monitors new files in intervals of 1s for changes of their size in order to determine if they are still being copied. Probably the issue is localised only on Windows as NTFS sometimes updates the file's size in intervals larger than 1s. A better and safer solution is try to open an InputStream to the file - if it's still being copied this will return a FileNotFoundException. How to reproduce: 1. Deploy a big enough WAR(copied in ~10 seconds)
Please ensure the fix is tested on Mac (HFS), Linux (EXT3), and Windows (NTFS) as we suspect that some file systems might allow an input stream to be opened on a file which is being updated.
Actually this is possible on NTFS with shared access flags (OF_SHARE_*): http://msdn.microsoft.com/en-us/library/aa365430%28v=vs.85%29.aspx
I wondere if there is some relatively fast way of checking the completeness of a JAR download without having to start the deployment process? Think jar tvf and check the return code.
The tvf option is interesting. If that will be used to get the entries of the jar file we can do this programatically. The fix was always meant to be in the monitoring step prior deployment.
Opening ZIP file should do the trick for JAR, WAR, PAR. It appears that incomplete archives simply cannot be opened. The end of central directory record (http://en.wikipedia.org/wiki/ZIP_%28file_format%29) should provide us with a way to detect the complete archive. Still there will be problems for big property files or unzipped bundles.
I think we should have parametrized scan interval. This will help with both non-ZIP based artefacts and reducing resource consumption (opening ZIP file every second can cost too much especially for big files).
It would also be a good idea to have parameterized the time which we will wait for a file to be copied on file system.
The thread now scans indefinitely and checks for file size. Some copy operations however set the size to the final value before the file is complete hence the problem. If we have a interval after which the artefact is no longer processed we risk to ignore a file that has been fixed (copied / edited) in the pickup.
One drawback is that a ZIP exception might be thrown because of a corrupted file, not because it is still copying. I tested under Windows that renaming to the same file always returns false while copying, so my proposal is to combine the current solution with this additional check so we will solve the problem under Windows. Wdyt?
Created attachment 203491 [details] Patch for deployment of big archives Hello, We tried a simple patch on windows and it is working. Please take a look and comment. Regards, Assia & Violeta
What about comparing the size of two checks between an interval. E.g. scan at time t0 and t1, if t1-t0>0, then it is still being copied... Just a thought.
Erdal, this is how Virgo currently handles large files copied in pickup. Unfortunatelly that isn't enough as on Windows the size between the intervals is unchanged, actually the file is created with seemingly complete size right from the start although the copy process is still ongoing. That is why we're searching for a better solution for Windows - rename seems to be the best option here. In *nix systems the FS is good enough to update regularly the file size.
I reviewed the rename patch and it looks plausible. I have no idea whether this will improve matters on all Windows platforms though.
(In reply to comment #13) > I reviewed the rename patch and it looks plausible. I have no idea whether this > will improve matters on all Windows platforms though. I tested the solution on Windows XP and Windows 7 and it is working perfect. Assia tested it on Windows Vista and it is working there also. Assia, please could you confirm that you wrote 100% of the code, you have the right to contribute the code to Eclipse. Thanks Violeta
I confirm that I wrote 100% of the code and I have the right to contribute the code to Eclipse.
Change is committed, tested and pushed. Commit ID: cf8fc0b9fb2562f292de6269fa5c5ef47a6a5e46
Comment #6 -> Bug 359124 - Parameterize hot deployment's FS scan interval Comment #7 -> Bug 359125 - Parameterize the time that hot deployer will wait file to be copied to pickup folder
