Community
Participate
Working Groups
Please see bug 202546 comment #29 for detail (https://bugs.eclipse.org/bugs/show_bug.cgi?id=202546#c29) In Profile on Server use case, after server is started up with JVMTI agent, auto attach failed with an error message in console view and server process changes status to Terminated in profiling monitor view.
The problem happens when RAC is running before server starts in attach mode. And server process is never started if RAC/IAC is not started before server process.
The same error is thrown when either RAC or IAC is run before server process is started.
Is this issue specific to the IBM JVM? Could it be a (yet another) manifestation of bug 202550?
It's working for SUN JVM only for execution and memory analysis, the JVM crashs when Thread Analysis is selected. and the error with IBM JVM is different from bug 202550, this bug happens at attach time, but bug 202550 happens at detach time.
(In reply to comment #4) > It's working for SUN JVM only for execution and memory analysis, the JVM crashs > when Thread Analysis is selected. This is definitely not related to bug 202550, since Thread Analysis is not using instrumentation at all. Can you attach a stack-dump of the crash? > and the error with IBM JVM is different from bug 202550, this bug happens at > attach time, but bug 202550 happens at detach time. Both attach and detach operations require dynamic byte-code instrumentation, which is not supported on IBM JVM. Therefore, in the case of the execution and heap profilers, it is the same error as bug 202550.
Asaf, I don't see any dump file in the eclipse folder. I reproduce the problem by pointing eclipse to java_sun.1.4.2_13 and setup a Tomcat server, and start profile. note that I put the latest patch from bug 202546 in the testing driver. Thanks.
(In reply to comment #6) > I reproduce the problem by pointing eclipse to java_sun.1.4.2_13 and setup a > Tomcat server, and start profile. note that I put the latest patch from bug > 202546 in the testing driver. Thanks. Are you sure about the Java version (1.4.2)? This is not a supported version for the JVMTI profiler...
(In reply to comment #7) > Are you sure about the Java version (1.4.2)? This is not a supported version > for the JVMTI profiler... > sorry. I meant java_sun_1.5.0_11
Could you give more details about run, some things isn't clear. Especially regarding Profile on Server use case, what configuration did you use and how did configure. Thanks, Alex.
Can't reproduce, please provide details.
With the TPTP 4.5m3 build, I can reproduce the problem. What I did is setup a Tomcat 5.5.23 server under server view, start server in profile mode with Execution Analysis, and error is thrown in console. Note that I did not start RAC, but let IAC kick in with test connection on host. Similar problem occurs even if I start RAC before profiling the server. capture of console output: Nov 23, 2007 12:03:30 AM org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO: The Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: D:\java_ibm_1.5.0_SR4\jre\bin;.;D:\ibm\4500m3\eclipse\plugins\org.eclipse.tptp.platform.ac.win_ia32_4.4.0.v200706020100\agent_controller\bin;D:\ibm\4500m3\eclipse\plugins\org.eclipse.tptp.platform.ac.win_ia32_4.4.0.v200706020100\agent_controller\lib;D:\ibm\4500m3\eclipse\plugins\org.eclipse.tptp.platform.jvmti.runtime_4.4.100.v200709060100\agent_files\win_ia32;D:/java_ibm_1.5.0_SR4/bin/../jre/bin/j9vm;D:/java_ibm_1.5.0_SR4/bin/../jre/bin;D:\ibm\AgentController/plugins/org.eclipse.hyades.probekit/lib;D:\ibm\AgentController\bin;D:\java_ibm_1.5.0_SR4\bin;C:\Program Files\ThinkPad\Utilities;E:\Program Files\Unixdos;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\Intel\Wireless\Bin\;C:\Program Files\ThinkPad\ConnectUtilities;C:\Program Files\IBM\Personal Communications\;C:\Program Files\IBM\Trace Facility\;C:\Program Files\ATI Technologies\ATI.ACE\;E:\Program Files\UltraEdit-32;C:\Program Files\QuickTime\QTSystem\;C:\Program Files\Common Files\Teleca Shared Nov 23, 2007 12:03:31 AM org.apache.coyote.http11.Http11BaseProtocol init INFO: Initializing Coyote HTTP/1.1 on http-8080 Nov 23, 2007 12:03:31 AM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 3844 ms Nov 23, 2007 12:03:31 AM org.apache.catalina.core.StandardService start INFO: Starting service Catalina Nov 23, 2007 12:03:31 AM org.apache.catalina.core.StandardEngine start INFO: Starting Servlet Engine: Apache Tomcat/5.5.23 Nov 23, 2007 12:03:31 AM org.apache.catalina.core.StandardHost start INFO: XML validation disabled Nov 23, 2007 12:03:34 AM org.apache.coyote.http11.Http11BaseProtocol start INFO: Starting Coyote HTTP/1.1 on http-8080 Nov 23, 2007 12:03:35 AM org.apache.jk.common.ChannelSocket init INFO: JK: ajp13 listening on /0.0.0.0:8009 Nov 23, 2007 12:03:35 AM org.apache.jk.server.JkMain start INFO: Jk running ID=0 time=0/297 config=null Nov 23, 2007 12:03:35 AM org.apache.catalina.storeconfig.StoreLoader load INFO: Find registry server-registry.xml at classpath resource Nov 23, 2007 12:03:36 AM org.apache.catalina.startup.Catalina start INFO: Server startup in 4859 ms [Error: Attach failed]
Eugene, could you post list of all components with versions which you used for execution (e.g. WTP, GEF, or something other that required)? Thanks, Alex.
Here they are: tptp.runtime.allInOne.win32.win32.x86-TPTP-4.5.0M3.zip agntctrl.win_ia32-TPTP-4.5.0M3.zip dtp_1.6M3.zip GEF-runtime-3.4M3.zip wtp-S-3.0M3-20071114232332.zip Thanks.
Eugene, I can't reproduce it anyway in described environment. I was able to run server in profiling mode, and collect some data.
Hi Alex, Please make sure you don't have your system environment variable setup up and point to the JVMTI libraries before you run the profile on sever. In actual scenario, users are not supposed to setup any environment but instead profile on sever will do it for user.
Eugene, I've checked environment, and it couldn't affect results. Did you obtain this issue at the first attempt after workbench execution? Could you verify (via SampleClient.exe) that AC is executed and responsible during server startup? Please set this environment variables and post Martini log file to Bugzilla MARTINI_LOGGER_LOG_LEVEL=5 MARTINI_LOGGER_DIRECTORY=<path for log saving> Thanks, Alex.
I don't see log file produced. This is what I did before starting Eclipse D:\4500m3\eclipse>set MARTINI_LOGGER_LOG_LEVEL=5 D:\4500m3\eclipse>set MARTINI_LOGGER_DIRECTORY=D:\ibm\4500m3\eclipse\MARTINI_log D:\4500m3\eclipse>eclipse.exe Also the profile seems to be waiting unless I explicitly start IAC by going to preferences > Hosts > Test Connection, then profile starts with console error as stated in my last comment. BTW, are you using IBM JVM 1.5?
(In reply to comment #17) > I don't see log file produced. If the log file is not produced, it means that the JVM cannot successfully load the JPIBootLoader JVMTI Agent (which in-turn loads the entire JVMTI runtime into the JVM process). This usually happens when the environment (PATH, required variables) is not properly defined. Another possible cause is if you try to mix-and-match 32 bit and 64 bit binaries on an EM64T platform (e.g., a 32-bit JVM cannot load 64-bit agents, and vice versa). Is there some debugging option in the JVM that can be turned on to produce a detailed log of the JVM initialization? This can shed some light on what is happening.
It sounds like it could be a problem on the programmatic setup of the environment variable in the profile on server code. I suspect it works on your environment because you have proper setup of environment variables on your machine by default. Could you point me to the list of environment variables that is required for profile to work? Please reference with the IAC libraries because they are the only libraries that I can point to on runtime. I cannot get any RAC information within workbench itself. Thanks a lot.
The necessary environment settings are specified in the jvmti.runtime Readme file (relevant quotes below). With regard to IAC, the binaries are located in the following directories (according to platforms): Win/IA32: plugins\org.eclipse.tptp.platform.jvmti.runtime_4.4.100.v200709060100\agent_files\win_ia32 Win/EM64T: plugins\org.eclipse.tptp.platform.jvmti.runtime_4.4.100.v200709060100\agent_files\win_em64t Linux/IA32: plugins/org.eclipse.tptp.platform.jvmti.runtime_4.4.100.v200709060100/agent_files/linux_ia32 Linux/EM64T: plugins/org.eclipse.tptp.platform.jvmti.runtime_4.4.100.v200709060100/agent_files/linux_em64t --- JVMTI Runtime Environment Configuration (from Readme file) --- Windows: a) Set TPTP_AC_HOME=<<Agent Controller Home>> b) Set JAVA_PROFILER_HOME=<IAC Profiler binaries> c) Set PATH=%JAVA_PROFILER_HOME%;%PATH%;%TPTP_AC_HOME%\bin d) Set PATH=%PATH%;%JAVA_HOME%\bin Linux: a) Set TPTP_AC_HOME=<<Agent Controller Home>> b) Set JAVA_PROFILER_HOME=<IAC Profiler binaries> c) Add the following directories to LD_LIBRARY_PATH: $JAVA_PROFILER_HOME:$TPTP_AC_HOME/lib --- end of quote ---
Can I assume <<Agent Controller Home>> and <<IAC Profiler binaries>> as plugins\org.eclipse.tptp.platform.jvmti.runtime_4.4.100.v200709060100\agent_files\%arch_folder%?
<<Agent Controller Home>> represents the IAC binaries (not the profiler). The TPTP_AC_HOME variable (which uses this value) must point to the AC binaries. <<IAC Profiler binaries>> represents plugins\org.eclipse.tptp.platform.jvmti.runtime_4.4.100.v200709060100\agent_files\%arch_folder%
Hi, From provided log is seen that server is started successfully. INFO: Server startup in 4859 ms ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Doesn't that mean that all required libraries were found and well linked with JVM. One another possible cause of such problem can be that server is executed but without profiling configuration at all (don't use option of JVM TI agent loading), just in usual configuration. I found that it can happen if you press Cancel in "Profile on server" dialog (Bug 211903), maybe it is something similar. I validate that server executed with "-agentlib:..." option thru tool "Process Explorer" from sinternals.com which provide command line for any executed process in Windows, maybe simpler way exists. Also this tool shows full environment used for process execution, you can check it for any strange paths. Thanks, Alex.
(In reply to comment #23) > I validate that server executed with "-agentlib:..." option thru tool "Process > Explorer" from sinternals.com which provide command line for any executed > process in Windows, maybe simpler way exists. > Also this tool shows full environment used for process execution, you can check > it for any strange paths. This tool can also show the DLLs mapped into the address space of a process. You can use this to check whether the JVMTI Runtime DLLs (JPIBootLoader.dll, JPI.dll, et al) where loaded into the JVM process.
Hi Alex, The server did started okay, but it failed when the it's tried to be attached. Looking into the process information, I can see the proper vm arg is inserted in the server process by profile on server code. D:\java_ibm_1.5.0_SR5\bin\javaw.exe -agentlib:JPIBootLoader=JPIAgent:server=enabled;CGProf:execdetails=true -Dcatalina.base=D:\ibm\4500-1213\eclipse\workspace-server\.metadata\.plugins\org.eclipse.wst.server.core\tmp0 -Dcatalina.home=D:\servers\apache-tomcat-5.5.23 -Dwtp.deploy=D:\ibm\4500-1213\eclipse\workspace-server\.metadata\.plugins\org.eclipse.wst.server.core\tmp0\wtpwebapps -Djava.endorsed.dirs=D:\servers\apache-tomcat-5.5.23\common\endorsed -classpath D:\servers\apache-tomcat-5.5.23\bin\bootstrap.jar;D:\java_ibm_1.5.0_SR5\lib\tools.jar org.apache.catalina.startup.Bootstrap start
I tried again with TPTP-4.5.0-200712130940 in 2 scenarios: 1. RAC is started before Workbench starts > Server started with the following command line captured in debug view: D:\java_ibm_1.5.0_SR5\bin\javaw.exe -agentlib:JPIBootLoader=JPIAgent:server=enabled;CGProf:execdetails=true -Dcatalina.base=D:\ibm\4500-1213\eclipse\workspace-server\.metadata\.plugins\org.eclipse.wst.server.core\tmp0 -Dcatalina.home=D:\servers\apache-tomcat-5.5.23 -Dwtp.deploy=D:\ibm\4500-1213\eclipse\workspace-server\.metadata\.plugins\org.eclipse.wst.server.core\tmp0\wtpwebapps -Djava.endorsed.dirs=D:\servers\apache-tomcat-5.5.23\common\endorsed -Dfile.encoding=Cp1252 -classpath D:\servers\apache-tomcat-5.5.23\bin\bootstrap.jar;D:\java_ibm_1.5.0_SR5\lib\tools.jar org.apache.catalina.startup.Bootstrap start > Server started okay, but failed and terminated itself as soon as the profile tool try to attach to the agent: INFO: Find registry server-registry.xml at classpath resource Dec 14, 2007 11:47:23 AM org.apache.catalina.startup.Catalina start INFO: Server startup in 16307 ms [Error: Attach failed] Could not start Agent Controller. Empty serviceconfig.xml file Make sure you have created a valid serviceconfig.xml file by running the SetConfig script. > Server is shown as terminated when console showed the [Error:..] message. > It looks like the referencing to IAC libraries is failing. 2. RAC is never started, and IAC is assumed to be used. > Server showed as 'Staring..' but there is no console output and it never start. Failed eventually due to time out. Debugging the code. The profile on Server code appends the following environment variables as part of the server launch configuration: JAVA_PROFILER_HOME=D:\ibm\4500-1213\eclipse\plugins\org.eclipse.tptp.platform.jvmti.runtime_4.4.100.v200712110400\agent_files\win_ia32 PATH=D:\ibm\4500-1213\eclipse\plugins\org.eclipse.tptp.platform.ac.win_ia32_4.4.0.v200706020100\agent_controller\bin;D:\ibm\4500-1213 \eclipse\plugins\org.eclipse.tptp.platform.ac.win_ia32_4.4.0.v200706020100\agent_controller\lib;D:\ibm\4500-1213 \eclipse\plugins\org.eclipse.tptp.platform.jvmti.runtime_4.4.100.v200712110400\agent_files\win_ia32;D:\java_ibm_1.5.0_SR5 \jre\bin;D:/java_ibm_1.5.0_SR5/bin/../jre/bin/j9vm;D:/java_ibm_1.5.0_SR5/bin/../jre/bin;D:\ibm\AgentController/plugins/org.eclipse.hyades.probekit/lib;D:\ibm\AgentControlle r\bin;D:\java_ibm_1.5.0_SR5\bin;C:\Program Files\ThinkPad\Utilities;E:\Program Files\Unixdos;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\Program Files\Intel\Wireless\Bin\;C:\Program Files\ThinkPad\ConnectUtilities;C:\Program Files\IBM\Personal Communications\;C:\Program Files\IBM\Trace Facility\;C:\Program Files\ATI Technologies\ATI.ACE\;E:\Program Files\UltraEdit-32;C:\Program Files\QuickTime\QTSystem\;C:\Program Files\Common Files\Teleca Shared RASERVER=D:\ibm\AgentController AC_HOME=D:\ibm\AgentController TPTP_AC_HOME=D:\ibm\4500-1213\eclipse\plugins\org.eclipse.tptp.platform.ac.win_ia32_4.4.0.v200706020100\agent_controller
Hi, Eugene. I can't reproduce it again. Why are you using TPTP-4.5.0-200712130940 if this record targeted for 4.4.1? RAC case: Could you send list of environment variables with values for RAC case? And please check that AC have been initialized and serviceconfig.xml for executed ACServer contains meaningful values. IAC: The case with IAC similar to mentioned Bug 211903, please verify that AC already started then you select "Profile" item from server's context menu. Also try to remover any reference to D:\ibm\AgentController for IAC case. Thanks, Alex.
Hi Alex, I don't have any environment variable for RAC case. All required variables are supposed to be generated within the profile on server code on demand and inserted into the server launch configuration in both RAC and IAC scenarios. The variables inserted are as listed in my last comment. RAC was initialized with setConfig.bat. The error on configuration seems to complain on IAC configuration actually. For the IAC case, there is no AC running when I checked under task manager, neither IAC or RAC. btw, I was using 4.5 as I thought it should contains all the latest code. Thanks, Eugene
(In reply to comment #26) > 1. RAC is started before Workbench starts > > Server started with the following command line captured in debug view: > D:\java_ibm_1.5.0_SR5\bin\javaw.exe > -agentlib:JPIBootLoader=JPIAgent:server=enabled;CGProf:execdetails=true > -Dcatalina.base=D:\ibm\4500-1213\eclipse\workspace-server\.metadata\.plugins\org.eclipse.wst.server.core\tmp0 > -Dcatalina.home=D:\servers\apache-tomcat-5.5.23 > -Dwtp.deploy=D:\ibm\4500-1213\eclipse\workspace-server\.metadata\.plugins\org.eclipse.wst.server.core\tmp0\wtpwebapps > -Djava.endorsed.dirs=D:\servers\apache-tomcat-5.5.23\common\endorsed > -Dfile.encoding=Cp1252 -classpath > D:\servers\apache-tomcat-5.5.23\bin\bootstrap.jar;D:\java_ibm_1.5.0_SR5\lib\tools.jar > org.apache.catalina.startup.Bootstrap start > > Server started okay, but failed and terminated itself as soon as the > profile tool try to attach to the agent: > INFO: Find registry server-registry.xml at classpath resource > Dec 14, 2007 11:47:23 AM org.apache.catalina.startup.Catalina start > INFO: Server startup in 16307 ms > [Error: Attach failed] This case is most likely related to bug 202550. As can be seen from the data above, IBM JVM 1.5 is being used (we haven't validated 202550 with SR6 yet...) and the crash happens during attach, when classes are being redefined. You can enable Martini logging (the MARTINI_LOGGER_DIRECTORY and MARTINI_LOGGER_LOG_LEVEL environment variables), reproduce the error and check the log files, and I am pretty sure you will see an error during the RedefineClasses process. As for the IAC case (number 2): this is the issue we currently trying to debug, right? Do you have any information on the loaded processes and the DLLs mapped into their address space (see comments 23 and 24)?
For IAC case. Eugene, from your previous comments seems that ACServer wasn't run then you are trying attach. So, could you define problem more precise. Is it that ACServer isn't started automatically then you start server use case profiling and hence can't later attach to it or in that after ACServer normally started but attaching is failed? Also there is a bug regarding it Bug 202078. Eugene, I reassign it to you since it looks like IAC/WTP misconfiguration or direct demonstration of Bug 202078. Please verify it. On our site we can't reproduce it. Thanks, Alex.
Eugene, I need to close down 441. Would you please see if this defect is reproducible? Thanks.
This is reproducible with the latest TPTP 4.4.1. and the basic scenario to show the same problem is reported in bug 215437.
Eugene, please resolve the defect if the fix for 215437 resolves the problem.
Joanna, 215437 resolved the problem in attach action. However the pre-requisite of a running AC still apply. I would like to look for a solution on automatic AC startup for profile on server use case. I would like to lower this to critical instead.
(In reply to comment #34) > Joanna, > 215437 resolved the problem in attach action. However the pre-requisite of a > running AC still apply. I would like to look for a solution on automatic AC > startup for profile on server use case. > I would like to lower this to critical instead. Hi Eugene, Please check if #186080 resolves the problem. With it, AC can be started later, together with workbench.
Igor, I tried the dll in 186080, but it doesn't solve the problem. I was using the dll in 4.4.1 driver and I'm not 100% sure if it's compatible with it. Also the workbench is running when profile on server is started from a workbench action.
(In reply to comment #36) > Igor, I tried the dll in 186080, but it doesn't solve the problem. I was using > the dll in 4.4.1 driver and I'm not 100% sure if it's compatible with it. Also > the workbench is running when profile on server is started from a workbench > action. I meant that with #186080 resolved, you can start the profiled application in enabled mode without AC. AC might be started later, jast before attaching. It might be IAC as well as AC.
(In reply to comment #36) > Igor, I tried the dll in 186080, but it doesn't solve the problem. Eugene, library attached to Bug 186080 can't be used separately, this fix also touched ACCollector. It is worth to wait 4.5 build or I can send you updated libraries. Alex.
(In reply to comment #38) > (In reply to comment #36) > > Igor, I tried the dll in 186080, but it doesn't solve the problem. > Eugene, library attached to Bug 186080 can't be used separately, this fix also > touched ACCollector. It is worth to wait 4.5 build or I can send you updated > libraries. > > Alex. > Alex, I see. That may explains why the single dll doesn't solve the problem in 4.4.1 build. Would you be able to create a patch for 4.4.1 level such that I can test it for my use case? Thanks a lot.
> Alex, I see. That may explains why the single dll doesn't solve the problem in > 4.4.1 build. Would you be able to create a patch for 4.4.1 level such that I > can test it for my use case? Thanks a lot. > Patch attached to Bug 186080 can be directly applied to 4.4.1 branch.
I have debug further on this bug and to resolve the problem an AC is required to be running before the attach action happens. TI agent is waiting for an running AC before anything can start. I have updated the profile on server code to detect a running AC and start on demand to complete the use case.
Created attachment 88810 [details] patch Joanna, would you please review the patch? The patch is smoke tested with TPTP 4.4.1 tptp.sdk-TPTP-4.4.1-200802030100.
Created attachment 88811 [details] patched plug-in jar
Hi Eugene, patch is good.
Thanks Jonathan.
patch applied to 441 and Head
Patch committed
*** Bug 202078 has been marked as a duplicate of this bug. ***
CLOSE BUG
