Bug 347130

Comment 1 Greg Wilkins 2011-05-25 22:42:03 EDT

I can't replicate this with a simple test.

I have added getContextPath().getResourcePaths() to the Dump servlet (in trunk and will be in 7.4.2), so a request like /dump?resource=/  will do various getResourcePaths calls.

I have tried this with various deployments and redeployments and cannot see a problem.   

Can you be more precise about the context paths etc being used and exactly the code that apache click is using ?

Comment 2 josvazg 2011-05-26 02:53:29 EDT

On org.apache.click.service.XmlConfigService:

/**
     * Return the list of templates within the web application.
     *
     * @return list of all templates within the web application
     */
    private List getTemplateFiles() {
        List fileList = new ArrayList();

        Set resources = servletContext.getResourcePaths("/");
        if (onGoogleAppEngine) {
            // resources could be immutable so create copy
            Set tempResources = new HashSet();

            // Load the two GAE preconfigured automapped folders
            tempResources.addAll(servletContext.getResourcePaths("/page"));
            tempResources.addAll(servletContext.getResourcePaths("/pages"));
            tempResources.addAll(resources);

            // assign copy to resources
            resources = Collections.unmodifiableSet(tempResources);
        }

        // Add all resources within web application
        for (Iterator i = resources.iterator(); i.hasNext();) {
            String resource = (String) i.next();

            if (isTemplate(resource)) {
                fileList.add(resource);

            } else if (resource.endsWith("/")) {
                if (!resource.equalsIgnoreCase("/WEB-INF/")) {
                    processDirectory(resource, fileList);
                }
            }
        }

        Collections.sort(fileList);

        return fileList;
    }

This call:
Set resources = servletContext.getResourcePaths("/");
Return resources = [ ] // empty 
when you redeploy after deploying different war file.

The servletContext object is defined as:
private ServletContext servletContext; // javax.servlet.ServletContext

An If I check its runtime type it is:
class org.eclipse.jetty.webapp.WebAppContext$Context

Comment 3 Jan Bartel 2011-07-13 03:02:48 EDT

Created attachment 199542 [details]
Tiny webapp to do a getResourcePaths

To use, deploy as example.war then do:

http://localhost:8080/example/foo

Comment 4 Jan Bartel 2011-07-13 03:04:08 EDT

Hi josvazg,

I haven't been able to reproduce this problem using jetty-7.4.4.

I'm attaching a very tiny webapp that has a servlet that does the getResourcePaths() call.

Can you use it to reproduce the problem in your environment? If so, please post the exact sequence of steps performed. Please also clarify if when you say "redeploy" you mean that you update the existing war, or do you do an undeploy followed by a deploy (ie remove the war, then copy it back again later)?

thanks
Jan

Comment 5 Jan Bartel 2011-07-28 22:41:20 EDT

josvazg,

Have you tried the test webapp yet? Unless you can provide some more info on this one, I will have to close it as a "works for me" ....

thanks
Jan

Comment 6 josvazg 2011-07-29 02:30:55 EDT

Sorry I did not read previos messages.

I can't reproduce the problem with your sample program either. I tried redeploying as a war and as a directory and it allways gets the correct resource paths:

GET
Resourcepath: /stuff/
Resourcepath: /WEB-INF/
Resourcepath: /META-INF/

The problem seems to affect Apache Click apps using resource path info for page automapping.


(In reply to comment #5)
> josvazg,
> 
> Have you tried the test webapp yet? Unless you can provide some more info on
> this one, I will have to close it as a "works for me" ....
> 
> thanks
> Jan

Comment 7 josvazg 2011-07-29 02:47:57 EDT

I am going to write or find a sample apache click webapp that can reproduce the problem and attach it here.
(I tried to send one app that was failing but it was too large)

Comment 8 josvazg 2011-07-29 02:55:19 EDT

Created attachment 200579 [details]
Sample webapp to reproduce the problem on redeployments

Comment 9 josvazg 2011-07-29 03:13:49 EDT

Try to deploy an redeploy the clicktest application I sent and check the jetty logs.

On (correct) deployment you should see this:

...
automapped pages:
[Click] [debug] /border-template.html -> CLASS NOT FOUND
[Click] [debug] /click/error.htm -> CLASS NOT FOUND
[Click] [debug] /click/not-found.htm -> CLASS NOT FOUND
[Click] [debug] /home.html -> com.rfranco.clicktest.page.HomePage
[Click] [debug] initializing FileLoadService: org.apache.click.service.CommonsFileUploadService
...

On a failed redeployment you would get this:

...
[Click] [debug] automapped pages:
[Click] [debug] initializing FileLoadService: org.apache.click.service.CommonsFileUploadService
...

On JBoss it NEVER fails.

VERY IMPORTANT: TO REPRODUCE the problem it seems you have to follow this steps:
1) Deploy clicktest (properly) (copy it to webapps/)
2) Deploy ANOTHER webapp.
3) Redeploy clicktest again (copy it again on top of itself or touch it)
=> You get the failed deployment!
4) From that moment the only way to get it to deploy right is to kill jetty and restart it

Comment 10 josvazg 2011-07-29 03:28:42 EDT

MORE INFO:

It seems that not just (re)deploying ANY other application will make it fail, it ONLY fails IF I deploy some apps of mine (struts2 based, but not all of them)

For instance, if I deploy clicktest then your test webapp (or clicktest with another name) and then clicktest again IT DOES NOT FAIL.

But if I use some strust2 based app to deploy between deployment and redeployment of clicktest IT FAILS.

Very difficult to reproduce as I don't know what difference between two strust2 based webapps I have one that that makes it fail and the other that doesn't.

Anyway, there is some hidden problem on Jetty because THIS DOES NOT happen on JBoss at all but more importantly, because applications are independent and, no matter what crap does any previously deployed app have, Jetty deployer should NOT get confused so that it can't redeploy another webapp properly, specially when those applications are totally independent and unaware of each other.

Comment 11 Jan Bartel 2011-07-31 21:37:19 EDT

josvazg,


I have tried to reproduce the problem without success.

I have deployed your clicktest war and 2 of the struts2 example webapps:struts2-showcase-2.0.14.war and struts2-mailreader-2.0.14.war. 

Redeploying the clicktest war always produces the same result:

/click/not-found.htm is Template
/click/error.htm is Template
/border-template.html is Template
/home.html is Template
[Click] [debug] automapped pages:
[Click] [debug] /border-template.html -> CLASS NOT FOUND
[Click] [debug] /click/error.htm -> CLASS NOT FOUND
[Click] [debug] /click/not-found.htm -> CLASS NOT FOUND
[Click] [debug] /home.html -> com.rfranco.clicktest.page.HomePage
[Click] [debug] initializing FileLoadService: org.apache.click.service.CommonsFileUploadService
[Click] [debug] initializing TemplateService: org.apache.click.service.VelocityTemplateService
[Click] [debug] initializing ResourceService: org.apache.click.service.ClickResourceService
[Click] [info ] initialized in debug mode


Can you attach the struts2 webapp that causes the redeployment to fail? 

thanks
Jan



(In reply to comment #10)
> MORE INFO:
> 
> It seems that not just (re)deploying ANY other application will make it fail,
> it ONLY fails IF I deploy some apps of mine (struts2 based, but not all of
> them)
> 
> For instance, if I deploy clicktest then your test webapp (or clicktest with
> another name) and then clicktest again IT DOES NOT FAIL.
> 
> But if I use some strust2 based app to deploy between deployment and
> redeployment of clicktest IT FAILS.
> 
> Very difficult to reproduce as I don't know what difference between two strust2
> based webapps I have one that that makes it fail and the other that doesn't.
> 
> Anyway, there is some hidden problem on Jetty because THIS DOES NOT happen on
> JBoss at all but more importantly, because applications are independent and, no
> matter what crap does any previously deployed app have, Jetty deployer should
> NOT get confused so that it can't redeploy another webapp properly, specially
> when those applications are totally independent and unaware of each other.

Comment 12 josvazg 2011-08-03 01:14:40 EDT

How?
It weights 14Mb or more and I have to send it in "binary" form as it is a NON opensource application that is part of our company products.

Comment 13 josvazg 2011-08-03 01:18:36 EDT

Even if you get the copy it will probably be very difficult to debug with a binary webapp.

I will try to isolate the problem deactivating parts of the application till I make the problem disappear. If I manage to do that I will try to send you a much smaller sample webapp in source code form.

Comment 14 josvazg 2011-08-03 02:29:20 EDT

Created attachment 200771 [details]
evil-webapp sample that screws up jetty deployment

I think I found it.

This small webapp is an updated version from your original attachment: example-resourcepath-webapp.

The way to reproduce:
1) Deploy clicktest and check it deploys properly (automapped pages: finds pages)
2) Deploy evil-webapp.
3) Re-deploy or touch clicktest and check automapped pages returns nothing.

This evil-webapp has a context loader listener and the offending line of code seems to be:
PrintServiceLookup.lookupDefaultPrintService();

But, why do PrinterService screws up jetty deployment beats me, to be honest.

Comment 15 Jan Bartel 2011-08-10 20:21:01 EDT

Hi josvazg,

Thanks for the evil-webapp sample. This is the wierdest thing I've ever seen. Even if the evil-webapp is deployed FIRST, subsequently deploying the click webapp will fail!

It turns out the cause is this:

java.lang.IllegalStateException: zip file closed
	at java.util.zip.ZipFile.ensureOpen(ZipFile.java:416)
	at java.util.zip.ZipFile.entries(ZipFile.java:311)
	at java.util.jar.JarFile.entries(JarFile.java:219)
	at org.eclipse.jetty.util.resource.JarFileResource.list(JarFileResource.java:261)
	at org.eclipse.jetty.util.resource.ResourceCollection.list(ResourceCollection.java:375)
	at org.eclipse.jetty.server.handler.ContextHandler.getResourcePaths(ContextHandler.java:1439)
	at org.eclipse.jetty.server.handler.ContextHandler$Context.getResourcePaths(ContextHandler.java:1722)
	at org.apache.click.service.XmlConfigService.getTemplateFiles(XmlConfigService.java:1629)
	at org.apache.click.service.XmlConfigService.loadPages(XmlConfigService.java:916)
	at org.apache.click.service.XmlConfigService.onInit(XmlConfigService.java:234)
	at org.apache.click.ClickServlet.initConfigService(ClickServlet.java:1626)
	at org.apache.click.ClickServlet.init(ClickServlet.java:190)

The jar file in question is this one:
  /tmp/jetty-0.0.0.0-8080-clicktest.war-_clicktest-any-/webapp/WEB-INF/lib/click-extras-2.1.0.jar


The reason that this is being examined for resource paths is because jetty-7 implements some features of the servlet-3.0 specification, whereby you must examine the META-INF dir of any jar file for a resources/ directory. So the resource base (the places that are examined when asking for resource paths) for the click app are:

 file:/tmp/jetty-0.0.0.0-8080-clicktest.war-_clicktest-any-/webapp/, 
 jar:file:/tmp/jetty-0.0.0.0-8080-clicktest.war-_clicktest-any-/webapp/WEB-INF/lib/click-extras-2.1.0.jar!/META-INF/resources/,
 jar:file:/tmp/jetty-0.0.0.0-8080-clicktest.war-_clicktest-any-/webapp/WEB-INF/lib/click-nodeps-2.1.0.jar!/META-INF/resources/,
 jar:file:/tmp/jetty-0.0.0.0-8080-clicktest.war-_clicktest-any-/webapp/WEB-INF/lib/click-2.1.0.jar!/META-INF/resources/]

What I cannot explain is why running the evil-webapp, and in particular asking for the javax.print.PrintServices should affect a jar file in a) a different webapp and b) a webapp that was deployed AFTERWARDS.

However, I have 2 possible workarounds for you:

1. by default jetty uses jar url connection caching. Disable this by putting the following line into jetty.xml near the top:

   <Set class="org.eclipse.jetty.util.resource.Resource"  name="DefaultUseCaches">false</Set>

or 

2. prevent jetty from examining META-INF of web-inf jars by changing the Configurations that are applied to each and every webapp and leaving out the MetaInfConfiguration. For example, if you're using the web app deployer to automatically deploy every webapp in $JETTY_HOME/webapps, edit $JETTY_HOME/etc/jetty-webapps.xml:

<Configure id="Server" class="org.eclipse.jetty.server.Server">
  <Ref id="DeploymentManager">
    <Call id="webappprovider" name="addAppProvider">
      <Arg>
        <New class="org.eclipse.jetty.deploy.providers.WebAppProvider">
          <Set name="monitoredDirName"><Property name="jetty.home" default="." />/webapps</Set>
          <Set name="defaultsDescriptor"><Property name="jetty.home" default="."/>/etc/webdefault.xml</Set>
          <Set name="scanInterval">1</Set>
          <Set name="contextXmlDir"><Property name="jetty.home" default="." />/contexts</Set>
          <Set name="extractWars">true</Set>
          <!-- add these lines: -->
          <Set name="configurationClasses">
            <Array type="String">
              <Item>org.eclipse.jetty.webapp.WebInfConfiguration</Item>
              <Item>org.eclipse.jetty.webapp.WebXmlConfiguration</Item>
              <Item>org.eclipse.jetty.webapp.FragmentConfiguration</Item>
              <Item>org.eclipse.jetty.webapp.JettyWebXmlConfiguration</Item>
              <Item>org.eclipse.jetty.webapp.TagLibConfiguration</Item>
            </Array>
          </Set>
        </New>
      </Arg>
    </Call>
  </Ref>
</Configure>


I'm continuing to investigate why the evil-webapp causes this problem, but it may take quite a while, so I recommend you use one of the work-arounds above in the meanwhile.

thanks,
Jan

Comment 16 Joakim Erdfelt 2011-09-13 15:00:33 EDT

Just a thought, but couldn't this issue and Bug 294531 be combined into a undeploy binding on the DeploymentManager to perform strict webapp cleanup?

Comment 17 Jan Bartel 2011-09-13 18:53:10 EDT

Joakim,

This problem occurs whether the files are deployed as wars or as deployed unpacked, so not sure that it relates to Bug 294531?

Jan

(In reply to comment #16)
> Just a thought, but couldn't this issue and Bug 294531 be combined into a
> undeploy binding on the DeploymentManager to perform strict webapp cleanup?

Comment 18 Jan Bartel 2012-07-31 04:39:14 EDT

josvazg,

It gives me immense please to tell you I finally figured this one out. To understand what is going on, here's some salient facts:

* Sun's PrintServiceLookup impl starts a background thread to check on the health of the printers and regularly calls HttpURLConnection.setDefaultUseCaches(false) while doing it. This then changes the default for all URLs subsequently created in the jvm to be marked as not cacheable.

* Sun's jar: protocol implementation by default caches JarURLConnections and JarFiles. However, the JarURLConnection.connect() method will call JarFile.close() if setDefaultUseCaches(false) has been called.

* Jetty's JarFileResource class remembers a JarURLConnection and the JarFile related to it, so it can reuse them

* JarURLConnections to distinct urls can actually share a JarFile. Consider jar:file:/tmp/foo.jar!/some/dir/file.txt, and jar:file:/tmp/foo.jar!/other/dir/other.jpg. Both of these urls have the same JarFile (ie jar:file:/tmp/foo.jar!/).

The upshot of all this is that if the printer service background thread kicks in and sets defaultUseCaches to false, it can wind up closing the JarFile that the various JarFileResource instances have remembered, leading to the ZipFileClosedException.

This seems to mostly affect the JarFileResource.list() method, as that operates on the JarFile to list its entries.

I put a fix into JarFileResource.list() method to retry the list() operation on an exception, after first releasing the connection and jar file. This seems to fix the problem.

This fix will be in 7.6.6/8.1.6.

Heaving a sigh of relief and closing this bug,

Jan

Comment 19 josvazg 2012-07-31 05:20:54 EDT

Thanks Jan,

Tell me when 7.6.6 is out or how can I get an automatic notification when it is. 
I will give it a try and see if I can use instead of JBoss, at least for my development machine at first. 

I remember having that exception:
java.lang.IllegalStateException: zip file closed
	at java.util.zip.ZipFile.ensureOpen(ZipFile.java:416)
...

But I didn't know nor why neither that it was related to this same bug, so its possible this will "kill two birds with one stone"

Comment 20 Jan Bartel 2012-07-31 18:37:10 EDT

josvazg,

Our continuous build system publishes snapshots nightly to here:
https://oss.sonatype.org/content/groups/jetty/org/eclipse/jetty

You should be able to find a snapshot distro with the fix in it here:
https://oss.sonatype.org/content/groups/jetty/org/eclipse/jetty/jetty-distribution/7.6.6-SNAPSHOT/

cheers
Jan


(In reply to comment #19)
> Thanks Jan,
> 
> Tell me when 7.6.6 is out or how can I get an automatic notification when it
> is. 
> I will give it a try and see if I can use instead of JBoss, at least for my
> development machine at first. 
> 
> I remember having that exception:
> java.lang.IllegalStateException: zip file closed
> 	at java.util.zip.ZipFile.ensureOpen(ZipFile.java:416)
> ...
> 
> But I didn't know nor why neither that it was related to this same bug, so
> its possible this will "kill two birds with one stone"

Comment 21 Jesse McConnell 2012-08-01 09:55:33 EDT

ideally 7.1.6 will be out 2 months after 7.1.5 which was released july 16th...so on or about september 16th