Bugzilla – Full Text Bug Listing
| Summary: | JVM crashes when I startup tomcat server using profile option within eclipse. | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | z_Archived | Reporter: | yousuff_mohamed | ||||||||||
| Component: | TPTP | Assignee: | Mike Reid <mikereid> | ||||||||||
| Status: | CLOSED WORKSFORME | QA Contact: | Kathy Chan <kathy> | ||||||||||
| Severity: | major | ||||||||||||
| Priority: | P2 | CC: | jgwest, michael, mikereid, mpcarl, sorin.marinescu-ext, yousuff_mohamed | ||||||||||
| Version: | unspecified | ||||||||||||
| Target Milestone: | --- | ||||||||||||
| Hardware: | PC | ||||||||||||
| OS: | Windows Vista | ||||||||||||
| Whiteboard: | |||||||||||||
| Attachments: |
|
||||||||||||
|
Description
yousuff_mohamed
Hi Yousuff, I was unable to reproduce using Tomcat 6 and Sun Java 1.6.0_20, can you upgrade your version of Java and let us know if you continue to see the issue? Hi Jonathan, Thanks for the update. I tried with Sun Java 1.6.0_21 again and found the issue again. After more analysis, I see that I missed a point to reproduce this issue. The issue occurs when I use tomcat server along with a third party web application deployed in it. The application is called webtop and is provided by emc corporation. Here is what I did to come to that conclusion. First I created a new server in 'servers view' with the existing java version and started the server by right-click and choose profile --> memory analysis --> finish. I found the server started fine with no issues. I stopped the server and added webtop application as an external module and restarted in profile mode again. I see the JVM crashed now. I tried the same steps again with Sun Java 1.6.0_21 and found the issue still remains. So the issue is most likely not with java version. Rather I guess it has got something to do with Profiler when used with this webapplication. Let me know, if there is any logs or any additional information I can provide to help fix this issue. Is there a freely available version of Webtop? Would you be able to provide me a link? I haven't been able to find one. This is a commercial product and unfortunately trial version is also not available. And I am not authorized to handover it to anybody. Luckily there is a free version of another software that bundles this webtop software in it. You may find that software at https://developer-content.emc.com/downloads/dctm_dev_edition_process.htm?product=dcm_dev Its quite big download but once you download and install, you will find this software installed along with it. I am committed to help and support you with this. what do you think about it ? Hi Jonathan, Are you looking at the issue, do you need any further help from me, is there anything I could do to support you further. Hi Youssuff, I'm not sure if its temporary but I was unable to download that package, the web page is giving me an error. However, in light of the findings on bug 319310 I wonder if what you are seeing is the same issue. To help determine that can you try the following: - Enable the profiler's logging mechanism by setting the following environment variables: MARTINI_LOGGER_LOG_LEVEL=5 MARTINI_LOGGER_DIRECTORY=C:\temp (or wherever you prefer) You can set the variables globally by right-clicking "My Computer" selecting Properties > Advanced > Environment Variables - Once the env vars have been set, restart Eclipse - Now try reproducing the problem. You should find a file called MLogNNNNN.log is produced in the directory you specified for MARTINI_LOGGER_DIRECTORY. Now here's the fun part. After Tomcat crashes, check the log file and see if it ends with a line similar to: Informative 10-08-19 08:22:55 PM 10485 5 CJVMTIInterface [ClassFileLoadHookHandler] org/eclipse/ajdt/core/builder/AJBuilder, class loader = 0x93486fc If it does, could you go about extracting that .class from the webtop software and attaching it to the bug? If you don't feel comfortable attaching the .class file then I can try building you a .dll with the patch from bug 319310 for you to try, if you'd prefer that. /mike. Created attachment 177175 [details]
JVM Crash Log file
Created attachment 177176 [details]
Martini Log file
Hi Mike, I tried your suggestions. I have attached the relevant log files for your reference. You were looking for the .class file as per the last line in the log file. However the software is very huge, around 180 mb that it's very hard for me to search and locate it, moreover it's third party licensed software and so I am concerned with playing around with it. Would you kindly go ahead and provide me that dll, so I try your suggestions again. I am having same issue with another EMC WDK Application (WDK is the framework that Webtop is built upon). I was able to work around by installing a 1.4 JVM in Eclipse (I just pointed to a 1.4 jre, I am not using it). This seemed to enable profiling of Pre1.5 to the profile menu. Choosing this option (even though I am running in 1.6 seems to allow things to work. The Hotspot crash appeared when trying to Profile for Time or Memory Usage, thread analysis worked. I tested this issue with 1.5.0_12 and 1.6.0_03 JDK and JRE. Unable to test with newer runtimes, but hope to be able to look into this over next couple of days. If there is something you would like me to check for in the reported problem class (I pasted below from Mohamed's logs), I might be able to get that done as I have engineering contacts at EMC. I would also be interested in knowing if there are other EMC/Documentum problems reported on the buglist that I could possibly help with. I am very busy on project work, but will do whatever I can to help out. Informative 08/22/10 04:38:54 1092 5 CJVMTIInterface [ClassFileLoadHookHandler] com/documentum/fc/client/attributehandler/DfReadOnlyAttributeHandler, class loader = 0E29B704 Mohamed, can you try configuration an 1.4 JRE and see if that enables the Pre1.5 profiling for you adn see if that works for you? It is at least a workaround for the moment (you do not have to run things under 1.4, just have a jre defined in Eclipse should be enough). Created attachment 177871 [details] Binary with tentative fix for bug 319310 Hi, Apologies for the delay, I meant to get back to this sooner. Attached is JIE.dll. Its a debug build with my current fix for the aforementioned bug 319310. Additionally since we now know the class causing the trouble, you can check if this is due to the same issue by examining the Java byte code using javap: javap -c -private -classpath <path/to/DfReadOnlyAttributeHandler> com.documentum.fc.client.attributehandler.DfReadOnlyAttributeHandler If this is the same problem as bug 319310, you should be able to find a method with the following characteristic: One of the "target" fields in the exception table does not correspond to the beginning of a line as laid out by the line number table. For example: Exception table: from to target type 264 271 274 Class org/osgi/service/prefs/BackingStoreException 12 20 334 Class org/eclipse/core/runtime/CoreException 23 311 314 Class org/eclipse/core/runtime/CoreException 12 20 378 Class org/osgi/service/prefs/BackingStoreException 23 311 358 Class org/osgi/service/prefs/BackingStoreException LineNumberTable: line 1213: 0 ... line 1269: 333 <-- Ex. table entry with target 334 does not match the beginning of a source line line 1270: 377 ... If you find this to be true of one (or more) of the methods in the bytecode listing of the problematic class, this would indicate the same problem. In that case I believe the attached JIE.dll should solve it. Note that this is not very well tested, it is hot off the presses on my own dev environment, so use at your own risk. However, it does correct the issue as far as my testing has gone so far. To install: - Find the location of the org.eclipse.tptp.platform.jvmti.runtime plugin directory (e.g. eclipse\plugins\org.eclipse.tptp.platform.jvmti.runtime_4.6.0.v201006071900) - Navigate into the agent_files\win_ia32 sub-directory - Rename/backup the existing JIE.dll and copy the patched one in place of it Looking forward to hearing your findings. /mike. Hello Michael McCollough, Thanks for the help. I tried to change JRE as you mentioned but then tomcat failed to start with a message that it requires atleast jre 5.0 or later; so I could not perform the tests as you asked. Hi Mike Reid, After long search locate the jar file containing the class file, I managed to run the javap command as you specified. I did not find any of the method with the characteristics you specified. Additionally, I also tried the JIE.dll you have provided. First I renamed the existing JIE.dll and moved it to the folder one level above its current folder. Then, I put the new JIE.dll in that folder. I have also set the environment variables to capture the martini log files similar to my preivous exercise and now started eclipse with -clean option. Trying to startup tomcat in profile mode failed again with a JVM crash. I have attached the martini log file, javap output and jvm crash log file for further analysis. I appreciate all the help provided and look forward for more help in resolving this issue. Created attachment 178235 [details]
All logs after JIE.dll change
Hi Yousuff, I had a glance at the javap output. I think this is indeed related to bug 319310. This .class file does not have any line number information at all, which is sort of a worst-case scenario since in that case none of the exception table entries map to an existing line number. It would seem my fix is not quite robust yet so I'll be working on improving it. Now that you've located the .class do you think you would be able to attach it to the bug? In the meantime if you are willing to use Tomcat 5.5 for the profiling work you can get a JDK 1.4 compat package which will allow you to run Tomcat 5.5 with a 1.4 JVM. See "JDK 1.4 Compatability Package": http://tomcat.apache.org/download-55.cgi#5.5.30 /mike. Deferring to 4.7.2. (In reply to comment #14) Hi Mike Reid, You mentioned that none of the exception table entries map to an existing line number. When I open the file name javap_322212.txt file, I could see that all of the line numbers in the exception table entries point to an existing line number. For example, see the below lines (from file line number 60 to 62 in that javap output file) Exception table: from to target type 6 63 63 Class java/lang/Throwable I assume the numbers 6 and 63 refer to existing numbers in the code column (which I assume you mentioned as line number); 6 points to '6: invokestatic' and 63 points to '63: astore_2'. Could you kindly double check the contents javap output file, I just want to make sure we did not miss anything. Actually, I am not much familiar with javap tool and its output, so if I am wrong in understanding it, kindly excuse my above comments and ignore it. And I would like to highlight one more thing. I see that the last line in the latest martini log file MLog04712.log points to a different class file (com/documentum/fc/client/DfSysObject) this time. I wonder if your fix resolved the previous issue but then hit upon some other new issue. As for the suggestion to use different version of tomcat, I cannot do that since I have some enhancements to the application which is already tested and working fine with the current setup/environment and would prefer not to alter it to prevent any other issues popping up in the application. But thanks for your suggestion, I can bear with this issue for some time and hope that a future release of TPTP might fix these issues. I can wait and test any updated fix from you and continue this cycle until we can find a solution. As mentioned before, I am bit concerned about playing with a licensed software, would you let me know if the .class file is absolutely necessary or crucial to resolve this issue. If that is the case, then I would strongly suggest that you try to download the full product from the below link which I provided you earlier, https://developer-content.emc.com/downloads/dctm_dev_edition_process.htm?product=dcm_dev I found the reason why you faced trouble last time with the above link. Actually you need to be logged in to that site with a valid username/password in order for that link to work, my sincere apologies for not mentioning it last time. Since I am always logged on to that website, I did not face trouble with that link. You can obtain a valid username/password by registration which is free from the site https://community.emc.com/community/edn. Once you got a valid credential and logged in to https://community.emc.com/community/edn, you can use the above download link to get the product for your installation. (In reply to comment #10) Michael McCollough, Would you do a favor please, check with your engineering contacts at emc if it is ok to extract the .class files from dfc.jar and release it in a public domain to resolve this issue, and if this would be within their licensing terms. (In reply to comment #12) > Hello Michael McCollough, > Thanks for the help. > I tried to change JRE as you mentioned but then tomcat failed to start with a > message that it requires atleast jre 5.0 or later; so I could not perform the > tests as you asked. You do not need to change your JRE that Tomcat runs under, with my 3.6 Eclipse and TPTP, having a 1.4 JRE defined allowed me to choose the Pre1.5 Profiles to run. Initial tests show them slower than I expect (could be my env), but they all worked without exception. I hope to be able to spend some time on this by Thursday. I thought responding via the email woudl post my comments back here so sorry this response is late in coming, I have not worked with Bugzilla much. I reponded to the email as soon as it came to my Blackberry, that response must have gone to /dev/null? :)
> Michael McCollough,
> Would you do a favor please, check with your engineering contacts at emc if it
> is ok to extract the .class files from dfc.jar and release it in a public
> domain to resolve this issue, and if this would be within their licensing
> terms.
We could likely expedite this by me doing a WebEx (or similar software) with someone as I have a full env setup. Let me know what tools you want me to have available before hand so I can prep the machine (this will save you a few hours of setting up a dev env for Documentum. Sunday evenings after 7pm EST are usually a pretty open time for me. Except for tonight, 7pm EST shoudl be good for Thu or Fri as well. I can also do early mornings around 5 or 6am EST if desired. email: eclipsebug322212:at:mccollough.us (replace :at:) to coordinate
Hi, Is this still a problem? Based on where we were last time the profiler now crashes on the class: com/documentum/fc/client/DfSysObject I am not really seeing a good way for me to be able to understand the cause of this well enough to provide a fix without being able to see the bytecode that is causing it. I've not had success in getting the package installed and it is difficult to find time to continue working on getting it installed. Please attach the offending .class if possible; if not we'll have to close this with the possibility of re-opening if we are ever able to find some bytecode which can reproduce this. /mike. I'm going to go ahead and close this. Will re-open if some offending bytecode can be found. (In reply to comment #20) > I'm going to go ahead and close this. Will re-open if some offending bytecode > can be found. Hello, I ran into the same issue today. The Martini log file ended with DfReadOnlyAttributeHandler, I applied your patched JIE.dll and the profiler now crashes at DfSysObject. I use XP SP3, Eclipse Galileo 3.5.2 with the latest TPTP (4.7.2) and jdk1.6.0_26. Have you made any progress on this issue? Is there anything I can do to help you? |