Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 340949

Summary: Hot deploy of large .war files let JVM crash (libzip.so)
Product: [RT] Jetty Reporter: stephan
Component: serverAssignee: Greg Wilkins <gregw>
Status: RESOLVED FIXED QA Contact:
Severity: critical    
Priority: P3 CC: gregw, jesse.mcconnell, jetty-inbox, valentino
Version: unspecified   
Target Milestone: 7.2.x   
Hardware: All   
OS: Linux   
Whiteboard:
Attachments:
Description Flags
JVM Crash Log none

Description stephan CLA 2011-03-25 08:24:42 EDT 
Build Identifier: jetty-7.0.2.v20100331

When hot-deploying large .war files with a low bandwidth connection, the JVM crashes. This is because of an overlapping scanInterval of the WebAppProvider. If it takes longer to "copy" the .war file in the scanned folder as the time the folder is scanned again, the server tries to unpack the .war file that is not fully loaded. This causes the JVM to crash. This may be a failure in the libzip.so, that does not throw an exception when accessing the corrupt archiv, but furthermore lets the JVM crash.
If Jetty checks if the file is writable when scanning the directory, then it can be noticed that the file is not fully transfered. With this information the file can be skipped and the error can be avoided.

Reproducible: Always

Steps to Reproduce:
1. Start Jetty with configured "org.eclipse.jetty.deploy.providers.WebAppProvider" (scanInterval 5 seconds)
2. For deploying a new .war file use a connection with a small bandwidth
3. If the upload of the .war file in the scanned directory takes longer as the scanInterval, the server (in particular the JVM) crashes.
Comment 1 stephan CLA 2011-03-25 08:28:13 EDT 
Created attachment 191904 [details]
JVM Crash Log
Comment 2 Jesse McConnell CLA 2011-03-29 11:36:52 EDT 
"If Jetty checks if the file is writable when scanning the directory, then it
can be noticed that the file is not fully transfered. With this information the
file can be skipped and the error can be avoided."

something to consider...traditionally I always figured we ignored this deployment approach in favor of context.xml files that you 'touch' when you are ready to update the war file.  I was a bit shocked to find things automatically deploying in webapps directory the other day...some assumption of mine proved wrong I suppose.

anyway, I would recommend using that approach with the context file in general personally but I'll assign this to Jan for _comment_ (not assigning to you to fix it jan..just for your expert comment :)

cheers
Comment 3 Greg Wilkins CLA 2011-03-31 18:32:50 EDT 
I do think we need to update our scanner to only notify when a file has been stable for an entire scan interval - ie not notify on first detected change, but after stability after the change.

I'll see if I can get this in for 7.4
Comment 4 Greg Wilkins CLA 2011-03-31 22:17:50 EDT 
I've updated the file scanner to check both the lastModified date and the file size when checking if a file has changed.  Then for all notifications (add, remove and change), it waits until the file has been stable for 2 scans before reporting.

This means that if a file is growing (even if it's modification date is set the same) then it will not be reported to the deployer.

r2948