Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 243474

Summary: profiling, linux, IAC, agent library not loading, JPIBootLoader
Product: z_Archived Reporter: Aubrey Barnard <barnard>
Component: TPTPAssignee: Bing Xu <xubing>
Status: CLOSED FIXED QA Contact:
Severity: blocker    
Priority: P1 CC: ewchan, jcayne, jgwest, jkubasta, tom.gl
Version: unspecified   
Target Milestone: ---   
Hardware: PC   
OS: Linux   
Whiteboard: closed461
Attachments:
Description Flags
Workspace log for failing IAC none

Description Aubrey Barnard CLA 2008-08-07 13:37:23 EDT
Summary:
Profiling an application doesn't work using the integrated agent controller (Linux, x86-64).

Problem:
I installed the TPTP from the Software Updates for the Ganymede release of Eclipse. Then, following the Popescu tutorial, I tried profiling a JUnit test run. However, I was not successful and received this error:
Error occurred during initialization of VM
Could not find agent library on the library path or in the local directory: JPIBootLoader

The same error occurs with a simple "Hello World" application. The error happens after I accept all the defaults in the launch configuration dialog and click to start profiling.

Although this appears to be a path issue, the documentation implies that profiling should "just work" when using the Integrated Agent Controller to profile a local Java application; set up is only needed when installing the Agent Controller.

I have encountered a number of other errors as well, apparently later in the profiling start up sequence. So, unfortunately, the symptoms vary, and I don't know how to reproduce the results exactly yet. (I probably need a fresh Eclipse and TPTP install each time.) I can do more investigation as needed.

Environment:
OS: Red Hat Enterprise Linux 5 x86-64
Eclipse: 3.4.0 (Ganymede release) x86-64
Java: Sun's JDK 1.6.0-06 x86-64
TPTP (Platform Project): 4.5.0.v200804180100
TPTP (Tracing and Profiling Tools Project): 4.3.100.v200802291616

Reference inquiry:
http://www.eclipse.org/newsportal/article.php?id=6487&group=eclipse.tptp#6487
Comment 1 Eugene Chan CLA 2008-08-07 14:01:21 EDT
I can also reproduce a similar problem on a downstream product of TPTP.
Comment 2 Bing Xu CLA 2008-08-07 14:25:51 EDT
Can you point me to the Popescu tutorial?  It might be outdated.  Did you set the PATH and LD_LIBRARY_PATH?
Comment 3 Aubrey Barnard CLA 2008-08-07 16:25:02 EDT
The Popescu tutorial can be found here:
http://www.eclipse.org/articles/Article-TPTP-Profiling-Tool/tptpProfilingArticle.html
Or you can get to it by going to the front page (http://www.eclipse.org/tptp/), clicking the "Articles & Tutorials" link in the "Discover TPTP" box on the upper right. On the resulting page the Popescu tutorial is the third entry in the first section ("Articles about TPTP") entitled "Java Application Profiling using TPTP (Eclipse corner) by Valentina Popescu", February 2006.

I think that the tutorial is a bit outdated. However, I didn't follow it very closely. Therefore, it is likely more of a distraction than a help. I am working on writing a set of reproducible steps for the problem which will be a better starting point.

I did not set either of the paths you mentioned because the documentation (http://www.eclipse.org/tptp/home/downloads/4.5.0/documents/installguide/InstallGuide45.html, http://dev.eclipse.org/viewcvs/index.cgi/platform/org.eclipse.tptp.platform.agentcontroller/src-native-new/packaging_md/linux/getting_started.html?root=TPTP_Project&view=co) says that you don't need to do anything to get the Integrated Agent Controller to work. Furthermore, I still have problems when setting PATH and LD_LIBRARY_PATH to point to the agent controller installed in <eclipse-base>/plugins/org.eclipse.tptp.platform.ac.<...>/agent_controller/{bin|lib}. I think it is more than just a path issue.
Comment 4 Eugene Chan CLA 2008-08-08 10:19:34 EDT
The following error is seen when profile in a downstream product on Linux RHEL5 32bit machine, with latest TPTP 4.5.1 build. (0801)

JVMJ9TI001E Agent library JPIBootLoader could not be opened (libJPIBootLoader.so: cannot open shared object file: No such file or directory)
JVMJ9VM015W Initialization error for library j9jvmti23(-3): JVMJ9VM009E J9VMDllMain failed
Could not create the Java virtual machine.
JVMJ9TI001E Agent library JPIBootLoader could not be opened (libJPIBootLoader.so: cannot open shared object file: No such file or directory)
JVMJ9VM015W Initialization error for library j9jvmti23(-3): JVMJ9VM009E J9VMDllMain failed
Could not create the Java virtual machine.
Comment 5 Jonathan West CLA 2008-08-08 11:10:44 EDT
(In reply to comment #4)
> The following error is seen when profile in a downstream product on Linux RHEL5
> 32bit machine, with latest TPTP 4.5.1 build. (0801)
> JVMJ9TI001E Agent library JPIBootLoader could not be opened
> (libJPIBootLoader.so: cannot open shared object file: No such file or
> directory)
> JVMJ9VM015W Initialization error for library j9jvmti23(-3): JVMJ9VM009E
> J9VMDllMain failed
> Could not create the Java virtual machine.
> JVMJ9TI001E Agent library JPIBootLoader could not be opened
> (libJPIBootLoader.so: cannot open shared object file: No such file or
> directory)
> JVMJ9VM015W Initialization error for library j9jvmti23(-3): JVMJ9VM009E
> J9VMDllMain failed
> Could not create the Java virtual machine.

WRT 4.5.1 probloem, Joel tells me that JVMTI profiling was broken on Linux until 08-06 build.
Comment 6 jkubasta CLA 2008-08-08 11:47:16 EDT
Aubrey, would you please see if the 08/06 driver (or later) fixes the problem you are seeing?
Comment 7 Eugene Chan CLA 2008-08-08 13:28:52 EDT
On RHEL5.0 32 bit machine with 0806 all-in-one 4.5.1 build, I can reproduce the same problem.
Comment 8 Eugene Chan CLA 2008-08-08 13:30:56 EDT
In the 0806 all-in-one build, I get 
FATAL ERROR: JPIAgent can't load ACCollector
for JVMTI agent standalone use case, and
JVMJ9VM011W Unable to load piAgent: libpiAgent.so: cannot open shared object file: No such file or direcory
for JVMPI agent standalone use case.
Comment 9 Bing Xu CLA 2008-08-08 13:58:49 EDT
(In reply to comment #8)
> In the 0806 all-in-one build, I get 
> FATAL ERROR: JPIAgent can't load ACCollector
> for JVMTI agent standalone use case, and
> JVMJ9VM011W Unable to load piAgent: libpiAgent.so: cannot open shared object
> file: No such file or direcory
> for JVMPI agent standalone use case.

These errors were most likely caused by 243213.  0807 build should contains the fix which should be available soon.
Comment 10 Aubrey Barnard CLA 2008-08-11 12:55:56 EDT
I've tried to reproduce the original problem in a variety of setups. However, I now encounter bug 243784 and cannot reproduce the original problem. So, for the time being, I only see in the original problem in my development environment, which is unsuitable for detailed investigation. Any suggestions?
Comment 11 Bing Xu CLA 2008-08-11 13:35:17 EDT
0809 build should contains the fix for 243213.   The only problem so far with that build is IAC can't be started but stand-alone AC should work.

Download from http://www.eclipse.org/tptp/home/downloads/?buildId=TPTP-4.5.1-200808090100.
Comment 12 Bing Xu CLA 2008-08-13 11:49:26 EDT
Aubrey, can you try 0812 build and see if IAC works for you?
Comment 13 Bing Xu CLA 2008-08-14 14:44:44 EDT
Aubrey, any luck with IAC?   This is marked as a blocker and may impact TPTP 4.5.1 release.
Comment 14 jkubasta CLA 2008-08-14 23:19:39 EDT
Bing, I had someone try to reproduce with the 20080813_1936 driver. Here are the results...

couldn't get the same error as before, but this time profiling a server got timed out, the workspace log file is as follows:
!SESSION 2008-08-14 16:42:35.908 -----------------------------------------------
eclipse.buildId=unknown
java.fullversion=J2RE 1.6.0 IBM J9 2.4 Linux x86-32 jvmxi3260-20080806_21795 (JIT enabled, AOT enabled)
J9VM - 20080806_021795_lHdSMr
JIT  - r9_20080721_1330ifx2
GC   - 20080724_AA
BootLoader constants: OS=linux, ARCH=x86, WS=gtk, NL=en_US
Command-line arguments:  -os linux -ws gtk -arch x86 -clean

!ENTRY org.eclipse.wst.common.project.facet.core 4 0 2008-08-14 16:48:54.960
!MESSAGE Project facet jsf.base has not been defined. It is used in plugin com.businessobjects.integration.eclipse.enterprise.facet.

!ENTRY org.eclipse.wst.server.core 4 0 2008-08-14 16:54:25.806
!MESSAGE Server WebSphere Application Server v6.1 at localhost was unable to start within 300 seconds. If the server requires more time, try increasing the timeout in the server editor.



The log file from base_v61 is as follows:

<JIT: FullSpeedDebug: ignoring countString>
Unhandled exception
Type=Segmentation error vmState=0x00080002
J9Generic_Signal_Number=00000004 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000001
Handler1=0089952B Handler2=001298BC InaccessibleAddress=FFFFFFFC
EDI=0019EF92 ESI=FFFFFFFC EAX=FFFFFFFC EBX=00000006
ECX=00000005 EDX=0019EF92
EIP=0018A2A0 ES=C040007B DS=0000007B ESP=01BE703C
EFlags=00210246 CS=00000073 SS=0000007B EBP=01BE7064
Module=/opt/IBM/RadRC2/SDP75Beta/runtimes/base_v61/java/jre/bin/libj9shr23.so
Module_base_address=00183000
Target=2_30_20080625_20583_lHdSMr (Linux 2.6.18-8.el5)
CPU=x86 (2 logical CPUs) (0x7e97e000 RAM)
JVMDUMP006I Processing Dump Event "gpf", detail "" - Please Wait.
JVMDUMP007I JVM Requesting System Dump using '/opt/IBM/RadRC2/SDP75Beta/runtimes/base_v61/profiles/was61profile1/core.20080814.164940.23993.0001.dmp'


but couldn't find the dmp file mentioned above
Comment 15 Bing Xu CLA 2008-08-15 12:50:30 EDT
(In reply to comment #14)
> Bing, I had someone try to reproduce with the 20080813_1936 driver. Here are
> the results...

Was this done on Eugene's Linux machine?
Comment 16 Aubrey Barnard CLA 2008-08-15 14:13:47 EDT
Created attachment 110114 [details]
Workspace log for failing IAC

I tried the 200808150100 All-In-One build just now. No success. There appear to be an improvement, however.

Steps:
1. Unzip the All-In-One distribution.
2. Run Eclipse.
3. Create Java project.
4. Create Java class with main.
5. Profile as application.
6. Launch configuration dialog comes up with the Monitors tab showing.
7. Tries to find monitors, but fails with:
"IWAT0435E An error occured [sic] when connecting to the host."
shown in red above where the list of monitors would be.
8. The improvement (?) is that the progress for "Fetching children o...ector:" increases until about 50% and then the message goes away, rather than hanging at 0%.

The log in the workspace lists many problems. I have attached the whole thing since I don't know what snippet would be relevant.
Comment 17 Jonathan West CLA 2008-08-15 14:29:06 EDT
Looks like the same problem as bug 240677. The library that needs to be installed to fix the problem is listed in comment #7 of that bug, and can be installed through the standard RHEL package GUI.
Comment 18 jkubasta CLA 2008-08-18 14:49:23 EDT
Bing, please confirm this is caused by 240677 so we can close the defect.
Comment 19 Bing Xu CLA 2008-08-18 15:18:08 EDT
Hi Aubrey,

+++++++++++++++++

ENTRY org.eclipse.tptp.platform.iac.administrator 4 0 2008-08-15 12:58:53.930
!MESSAGE RAServer generated the following output:  [Error Stream]:ACServer: error while loading shared libraries: libstdc++-libc6.2-2.so.3: cannot open shared object file: No such file or directory
++++++++++++++++


From these errors in your log file, we think it's the same problem that causes 240667.  240667 has been retargeted to 4.5.2 release.  If you want to fix the problem before that, on RHEL5, the following package can be installed: compat-libstdc++-296 - 2.96-138.i386.rpm.





*** This bug has been marked as a duplicate of bug 240667 ***
Comment 20 Aubrey Barnard CLA 2008-08-18 15:28:13 EDT
Bing, you've made a typo and connected this bug to bug 240667 instead of bug
240677. (I noticed since you added me to the CC list on bug 240667, which seems
to have no relevance to this bug.)
Comment 21 Bing Xu CLA 2008-08-18 16:11:07 EDT

*** This bug has been marked as a duplicate of bug 240677 ***
Comment 22 Aubrey Barnard CLA 2008-08-18 16:29:52 EDT
I had our system administrator install the library in question, and I no longer see errors regarding the library.

However, the IAC still does not work. The log is not all that helpful. Here is the relevant part from the end of the log:
!ENTRY org.eclipse.tptp.platform.iac.administrator 4 0 2008-08-18 15:08:01.283
!MESSAGE RAServer generated the following output:  [Output Stream]:Creating default Agent Controller configuration file.

!ENTRY org.eclipse.tptp.platform.iac.administrator 4 0 2008-08-18 15:08:01.284
!MESSAGE RAServer generated the following output:  [Output Stream]:Security is turned off.  Access is set to Local.

!ENTRY org.eclipse.tptp.platform.iac.administrator 4 0 2008-08-18 15:08:01.284
!MESSAGE RAServer generated the following output:  [Output Stream]:Run the SetConfig script to change the default settings.

!ENTRY org.eclipse.tptp.platform.iac.administrator 4 0 2008-08-18 15:08:02.562
!MESSAGE RAServer generated the following output:  [Output Stream]:Error reading the Agent Controller configuration file. Please check for valid configuration.Could not start Agent Controller.

The last 3 messages repeat 4 more times.

On the GUI, the symptoms are the same as those I mentioned in comment 16.

In my opinion, this bug should be reopened. A problem still exists that is different than that of bug 240677.

(Note: I used build 200808150100 to investigate this.)
Comment 23 Bing Xu CLA 2008-08-18 20:12:09 EDT
Aubrey, can you attach your serviceconfig.xml?
Comment 24 Bing Xu CLA 2008-08-18 20:43:49 EDT
Are you invoking IAC from Eclipse workbench?   Can you describe how to reproduce your error, including <install dir> and <workspace dir>?
Comment 25 Aubrey Barnard CLA 2008-08-19 16:03:57 EDT
I apologize for making a mistake in comment 22. I was sure I was using the 200808150100 build when I wrote comment 22, but after investigating your questions, I realized I must have been using the 200808090100 build. (An oversight in my automated test setup.)

In the 200808150100 build, the IAC works for me.

If you still want me to provide additional information, I would be glad to do so.
Comment 26 Bing Xu CLA 2008-08-19 16:54:54 EDT
Thanks Aubrey.  Just be aware that IAC stil has some issue (244265 and 243214) which should be fixed very soon.

Closing now. Reopen if appears again.
Comment 27 Paul Slauenwhite CLA 2009-09-24 14:33:25 EDT
As of TPTP 4.6.0, TPTP is in maintenance mode and focusing on improving quality by resolving relevant enhancements/defects and increasing test coverage through test creation, automation, Build Verification Tests (BVTs), and expanded run-time execution. As part of the TPTP Bugzilla housecleaning process (see http://wiki.eclipse.org/Bugzilla_Housecleaning_Processes), this enhancement/defect is verified/closed by the Project Lead since this enhancement/defect has been resolved and unverified for more than 1 year and considered to be fixed. If this enhancement/defect is still unresolved and reproducible in the latest TPTP release (http://www.eclipse.org/tptp/home/downloads/), please re-open.