Bugzilla – Full Text Bug Listing
| Summary: | HTTP 413 "FULL HEAD" error and servlets gradualling stop responding | ||
|---|---|---|---|
| Product: | [RT] Jetty | Reporter: | Soveran <yxzhong> |
| Component: | server | Assignee: | Greg Wilkins <gregw> |
| Status: | RESOLVED WONTFIX | QA Contact: | |
| Severity: | critical | ||
| Priority: | P3 | CC: | cgold, curtis.windatt.public, esirois, jesse.mcconnell, jetty-inbox |
| Version: | unspecified | ||
| Target Milestone: | --- | ||
| Hardware: | PC | ||
| OS: | Windows XP | ||
| Whiteboard: | |||
|
Description
Soveran
Below attach the communication mail with Chris Goldthorpe: Dear Chris, Thanks again for the quick response, please see my reply in red fonts at former mail Thank you and best regards, Soveran Zhong(钟翼翔） User Technologies, IBM China Development Lab, Shanghai 5/F, Building 10, 399 Keyuan Road, Zhangjiang Hi-Tech Park, Pudong New District, Shanghai 201203, China Tel: (86)-13661779171 E-mail: yxzhong@cn.ibm.com Notes ID: Yi Xiang Zhong/China/IBMCN Chris Goldthorpe <cgold@us.ibm.com> 2011-05-31 23:44 To Yi Xiang Zhong/China/IBM@IBMCN cc Brett Johnson <johnsobr@us.ibm.com>, Darel Benysh <benysh@us.ibm.com>, Doug Foulds <dfoulds@ca.ibm.com>, Yu Chi Li/China/IBM@IBMCN, Mei Yang/China/IBM@IBMCN, Peng P Wei/China/IBM@IBMCN, Hao HH Zhang/China/IBM@IBMCN Subject Re: Bug 345091 - HTTP 413 "FULL HEAD" error and servlets gradualling stop responding It is true that every browser has a limit on the size of cookies, however I believe that the Jetty server has a lower limit than IE or Firefox does. The error message is coming from the server which implies that the browser did not hit it's limit. The browser was able to send a large amount of cookies to the server but the server could not handle them. To answer your three questions 1. You will need to mail webmaster@eclipse.org to request the deletion of a comment in a Bugzilla entry. It may be easiest to request deletion of the entire bug and open a new bug against project:eclipse, product:jetty, since that is where the problem lies. Soveran-->thanks a lot 2. Check to see what server IEHS 3.4.x is running on. If it is running on Tomcat or any other server that is not Jetty that would completely explain why 3.4.x does not have the problem. Soveran-->like I said, IEHS 3.4.X and EHS 3.4.2 also use Jetty,that's why we feel so strange about it,so if you could find out the reason would be fantastic 3. I'm not sure how to gracefully recover. It would be better to focus on making the problem not show up in the first place. Soveran-->I believe what user really meant is if we cannot fix the 413 error due to the size limit, at least we need to let it fail gracefully, let user know what's going on rather than a blank page been showed Chris From: Yi Xiang Zhong <yxzhong@cn.ibm.com> To: Chris Goldthorpe/Beaverton/IBM@IBMUS Cc: Darel Benysh/Rochester/IBM@IBMUS, Chris Goldthorpe/Beaverton/IBM@IBMUS, Hao HH Zhang <zhhaohh@cn.ibm.com>, Mei Yang <meiyang@cn.ibm.com>, Yu Chi Li <liyuchi@cn.ibm.com>, Brett Johnson/Silicon Valley/IBM@IBMUS, Doug Foulds <dfoulds@ca.ibm.com>, Peng P Wei <weipsh@cn.ibm.com> Date: 05/30/2011 10:51 PM Subject: Re: Bug 345091 - HTTP 413 "FULL HEAD" error and servlets gradualling stop responding Dear Chris, Thank you for the information, and I just got the reply from our customer,quoted as following, there are three actions might need your support : 05/30/2011 - Brett Johnson Please edit that Eclipse bug and remove the IBM confidential information from that bug report. This includes the internal host names of helios.svl.ibm.com as well as the all of the details of my cookies that could include both my user names as well as possibly other encoded but not encrypted information that could uniquely identify me or my system. This information was shared internally and only expected to be used internally and should not have been posted externally without careful consideration of the contents. With regards to the bug, cookie sizes are controlled by the browser and each browser has different limitations as to how many characters can be set. Internet Explorer 8+ allows for cookies of 5196bytes and most other browsers of 4096 bytes. In my original testing, the cookie problem actually didn't appear to be due to a single cookie, but rather all of the cookies for a domain. Browsers also have size restrictions on total cookie storage per domain. All of the different cookies in sum are sent to the server. Even if Jetty can handle the single cookie, it might run into problems with multiple cookies that are large but don't exceed the 3500 character limit. I think its very important to know why IEHS 3.4.x does not have this problem? That is still a Jetty based IEHS. More importantly, yes the web servers have their own limits on cookies and browsers due to their differences might exceed those limits. The problem here is that the application does not fail gracefully, it fails rather painfully and does not give a user an indication of a fix. Ideally, when this problem occurs, the server would identify it and possibly try to clear out the cookies that it controls to free up space. IEHSc is now setting many more cookies than it previously did mostly due to the authentication services. Our ICs not only get publib.boulder.ibm.com cookies, but they also inherit cookies from ibm.com too. It is quite likely that marketing cookies or cookies being set on other ibm.com sites could cause this problem for our users even if we were to limit the sizes within our own sites. Actions list as below: 1>I failed to remove the confidential information in the PMR which I opened in your database, maybe you or the administrator would help to do that? 2>We need to figure out the reason why IEHS3.4.X and EHS3.4.2 is OK while using the same Jetty server as EHS3.6. 3>We need to come up a solution to let the application fail gracefully when encounter this full head 413 error. Your earliest response is highly appreciated! Thank you and best regards, Soveran Zhong(钟翼翔） User Technologies, IBM China Development Lab, Shanghai 5/F, Building 10, 399 Keyuan Road, Zhangjiang Hi-Tech Park, Pudong New District, Shanghai 201203, China Tel: (86)-13661779171 E-mail: yxzhong@cn.ibm.com Notes ID: Yi Xiang Zhong/China/IBMCN Chris Goldthorpe <cgold@us.ibm.com> 2011-05-28 06:03 To Chris Goldthorpe <cgold@us.ibm.com> cc Yi Xiang Zhong/China/IBM@IBMCN, Darel Benysh <benysh@us.ibm.com>, Yu Chi Li/China/IBM@IBMCN, Mei Yang/China/IBM@IBMCN, Hao HH Zhang/China/IBM@IBMCN Subject Re: Bug 345091 - HTTP 413 "FULL HEAD" error and servlets gradualling stop responding Here is a link which has more information about the limit on the size of the header http://blog.richeton.com/2010/03/11/set-jetty-buffer-size-maven/#more-897 From: Chris Goldthorpe/Beaverton/IBM To: Yi Xiang Zhong <yxzhong@cn.ibm.com> Cc: Darel Benysh/Rochester/IBM@IBMUS, Yu Chi Li <liyuchi@cn.ibm.com>, Mei Yang <meiyang@cn.ibm.com>, Hao HH Zhang/China/IBM@IBMCN Date: 05/20/2011 11:56 AM Subject: Re: Bug 345091 - HTTP 413 "FULL HEAD" error and servlets gradualling stop responding Here's the problem as I see it. The Jetty Server has a limit on the size of cookies that can be passed in the request header. I am guessing that this limit is larger on other servers. Anyhow if you create enough large cookies any application hosted on Jetty will fail. If you create some large cookies which almost cause the maximum size to be hit many but not all Jetty based applications will fail because there is no space to write more cookies. I tried this experiment and Eclipse 3.5, 3.6 and 3.7 will all fail in the same way if you create a large cookie using a Firefox add-on and try to open the help system in Firefox. Eclipse 3.3 which uses Tomcat in place of Jetty does not fail in this way. It appears that Eclipse 3.4 which also uses Jetty does not fail in the this way but I don't have an explanation for that anomaly. The Eclipse help system and the Jetty server are separate components. If we conclude that the restriction on the cookie size is too severe then the problem which affects the Eclipse Help System is actually a problem in the Jetty server and not in the Eclipse help system. For IEHS you have the option of using a different server and I would be interested to know if the problem goes away on a different server. The other question is why so many large cookies are being created by other webapps running on the same server Vivian, I had assigned this bug to you because it was reported against IEHS but I don't know if you have time to look at it. In any case it seems that the problem is not so much in the help code but in the server. So far no Eclipse customers have reported this issue. Chris From: Yi Xiang Zhong <yxzhong@cn.ibm.com> To: Chris Goldthorpe/Beaverton/IBM@IBMUS Cc: Mei Yang <meiyang@cn.ibm.com>, Yu Chi Li <liyuchi@cn.ibm.com>, Darel Benysh/Rochester/IBM@IBMUS Date: 05/19/2011 11:53 PM Subject: Re: Bug 345091 - HTTP 413 "FULL HEAD" error and servlets gradualling stop responding Dear Chris, Please be noted that this issue is exposed in EHS 3.6 not just IEHS 3.6(It's OK in EHS 3.4.2 though), and it's not packaged as War file,just a win package, I believe the server should be using Jetty Thank you and best regards, Soveran Zhong(钟翼翔） User Technologies, IBM China Development Lab, Shanghai 5/F, Building 10, 399 Keyuan Road, Zhangjiang Hi-Tech Park, Pudong New District, Shanghai 201203, China Tel: (86)-13661779171 E-mail: yxzhong@cn.ibm.com Notes ID: Yi Xiang Zhong/China/IBMCN Chris Goldthorpe <cgold@us.ibm.com> 2011-05-20 06:05 To Yi Xiang Zhong/China/IBM@IBMCN cc Subject Bug 345091 - HTTP 413 "FULL HEAD" error and servlets gradualling stop responding Hi Soveran, I've been looking into this. The long cookie that you are adding is around 3500 characters in size and the Jetty web server, which Eclipse uses has a limit on the size of the header of around 4K bytes.It is not surprising that adding a cookie of this size causes the server to return a 413 error. You said that you saw the problem in IEHS, which is packaged as a war file and has no server. It's possible that using a different web server would make the problem go away, I'm not sure if all servers have the 4K limit. I don't see this as being purely a problem with the help system - if there are large cookies which were not written by the help system which are consuming almost all of the available cookie header size then it is not surprising that a 413 error would be thrown. From your side there are two things to investigate - one is why so many large cookies are being written by by this host. The other is whether the problem would go away if you used a different web server. As far as I can tell the error is coming from the server and not the help system, so any fix to allow for a larger cookie header would have to be made in the Jetty Server and not the help system. Chris So, we need to focus on below part(extracted from the mail): Chris -->2.IEHS 3.4.x is running on. If it is running on Tomcat or any other server that is not Jetty that would completely explain why 3.4.x does not have the problem. Soveran-->like I said, IEHS 3.4.X and EHS 3.4.2 also use Jetty,that's why we feel so strange about it,so if you could find out the reason would be fantastic Chris--> 3. I'm not sure how to gracefully recover. It would be better to focus on making the problem not show up in the first place. Soveran-->I believe what user really meant is if we cannot fix the 413 error due to the size limit, at least we need to let it fail gracefully, let user know what's going on rather than a blank page been showed Moving to Jetty. I don't see a version of jetty in this issue.. I believe the version is 6.1.23 - 20100421 well, if I had to take a stab at this I would say that if your dealing with such large cookies then you should increase the size of the header buffer being used. So where ever the connector is being created, call the setRequestHeaderSize() on it and set it to a size that will be enough to handle the increased header size. Below quote our user's comment: Can we prevent the cookies from being written too large when they are being created? For example, any code that is adding a cookie should perhaps verify that the content that is going to be written does not surpass the length that Jetty can handle. I know you don't have control over all cookies that might be set for a given domain (or its parent domains), but catching and ensuring cookies that IEHS/IEHSc sets might prevent some occurences of this problem. I am unfortunately hearing about this problem more and more often from our information developers on their test infocenters and also from their developers when reviewing the documentation. The appearance to them is that the pages seem to be blank and they don't know why or how to fix it. Asking them to clear all their cookies periodically is not a great solution and the problem will likely return before too long because of the intranet authentication and single-sign-on and related cookes. It is making it difficult for us to reliably use IEHSc 3.6.x for internal reviews. Any kind of workarounds that can be implemented without needing users to have to manually clear cookies would be very beneficial. Older jetty releases had a default header size of 4k, and then 6k. This is now 8k to allow for multiple large cookies. I suggest you configured the header buffer size of your connectors(aka listeners) to 8k. It looks like the following property is the one to use: http.headerbuffersize. I'm taking a wild guess based on - http://confluence.atlassian.com/display/FISHEYE/Setting+JVM+System+Properties How can I set that property somewhere within Eclipse to force Jetty to make use of a lager header buffer size? I was thinking updating JettyHttpServer.java with the following code: //Increase the buffer size for the buffer from 4k to 16K d.put("http.headerbuffersize ", "16384"); //$NON-NLS-1$ //$NON-NLS-2$ And the following to JettyConstants.java /** * name="http.headerbuffersize" type="Interger" (default: 4096) */ public static final String HTTP_HEADERBUFFERSIZE = "http.headerbuffersize"; //$NON-NLS-1$ That said, (1) I don't know how to rebuild the org.eclipse.equinox.http.jetty_ <version>.jar file. (2) There has to be an easier way to update the headersize other than rebuilding the Java source. What's the best way to get this updated for use in Eclipse? Kind regards, Eric yes, that is how I would do it. Hmm.. Would prefer not having to update the base Java code. That make's it harder to move to newer versions of Eclipse. Ideally, it would an extra parameter from the command line. Does anyone know if, with EHS, starting the IC from the command line and passing the Jetty parameter value via a JVM command line property like -Dorg.eclipse.equinox.http.jetty.http.headerbuffersize=16384 would work as expected? Or is property functionality that would need to be added? Kind regards, Eric |