Community
Participate
Working Groups
Command line RPT 8.0 with 20080509_1309 (using IAC) "works" but displays this stack trace: java.lang.UnsatisfiedLinkError: hcclco (Not found in java.library.path) at java.lang.ClassLoader.loadLibraryWithPath(ClassLoader.java:993) at java.lang.ClassLoader.loadLibraryWithClassLoader(ClassLoader.java:962) at java.lang.System.loadLibrary(System.java:465) at org.eclipse.hyades.internal.execution.remote.RemoteComponentSkeleton.<clinit>(RemoteComponentSkeleton.java:139) at java.lang.J9VMInternals.initializeImpl(Native Method) at java.lang.J9VMInternals.initialize(J9VMInternals.java:200) at org.eclipse.hyades.logging.core.LoggingAgent.<init>(LoggingAgent.java:118) at org.eclipse.hyades.logging.core.LoggingAgent.<init>(LoggingAgent.java:93) at org.eclipse.hyades.logging.java.LoggingAgentHandler.publish(LoggingAgentHandler.java:280) at java.util.logging.Logger.log(Unknown Source) at com.ibm.etools.common.logging.listeners.CommonLoggingListener$1.run(CommonLoggingListener.java:107)
The Agent Controller installation (TPTP-4.5.0-200805150100) does not contain the /bin/hcclco.dll/.so library, which is required by: org.eclipse.hyades.execution.core.impl.ExecutionEnvironmentImpl.java org.eclipse.hyades.execution.core.impl.ProcessExecutorImpl.java org.eclipse.hyades.internal.execution.remote.RemoteComponentSkeleton.java
The same problem exists for the IAC and Agent Controller.
(In reply to comment #1) > The Agent Controller installation (TPTP-4.5.0-200805150100) does not contain > the /bin/hcclco.dll/.so library, which is required by: > > org.eclipse.hyades.execution.core.impl.ExecutionEnvironmentImpl.java > org.eclipse.hyades.execution.core.impl.ProcessExecutorImpl.java > org.eclipse.hyades.internal.execution.remote.RemoteComponentSkeleton.java > Correction, the /bin/hcclco.dll/.so library is installed with the IAC/Agent Controller in the TPTP-4.5.0-200805150100 driver.
Kevin, what TPTP build are you testing?
4.5 M6
Hi, Just to jump in on this, I have also seen the error several times using a runtime workbench using a consumming product install with an M6 workbench and M6 AC. Note that this error does not necessarily cause the test execution to fail. So, this problem has been around for a while. Others besides Kevin and myself have made comments about seeing the same error with M6 and later installs. DuWayne
(In reply to comment #6) > Hi, > > Just to jump in on this, I have also seen the error several times using a > runtime workbench using a consumming product install with an M6 workbench and > M6 AC. Note that this error does not necessarily cause the test execution to > fail. So, this problem has been around for a while. > > Others besides Kevin and myself have made comments about seeing the same error > with M6 and later installs. > > DuWayne > Duwayne/Kevin: Can you test using I7 or the I8 candidate drivers?
Kevin, DuWayne: any update please?
Sorry, we have had issues with our product install bits in i8 for the last several days. I understood that late yesterday afternoon, a work-around was put together. So, we will see if we can reproduce the error in i8 today.
The issue remains in the consumming product using TPTP 4.5 i8.
When running any agent that uses RemoteComponentSkeleton (e.g. a Java Hyades agent), all of the Hyades native libraries (hcclco, hcbnd, hcjbnd, etc) need to be available, in order for the agent to properly register and communicate with the agent controller. These binaries need to be in the PATH on windows, or the LD_LIBRARY_PATH on Linux. Generally, the error message you're seeing indicates that the above is not the case. Paul, have you by chance seen this message before using TPTP-only components? Kevin/Duwayne, any idea how to reproduce without using RPT? At what point in the process does you see this message, and which agent is failing to load?
(In reply to comment #11) > When running any agent that uses RemoteComponentSkeleton (e.g. a Java Hyades > agent), all of the Hyades native libraries (hcclco, hcbnd, hcjbnd, etc) need to > be available, in order for the agent to properly register and communicate with > the agent controller. These binaries need to be in the PATH on windows, or the > LD_LIBRARY_PATH on Linux. Generally, the error message you're seeing indicates > that the above is not the case. > > Paul, have you by chance seen this message before using TPTP-only components? Yes, see https://bugs.eclipse.org/bugs/show_bug.cgi?id=228873. Kevin/Duwayne: Please confirm that defect 228873 is not the root cause.
Hi Johnathan, 1. I do not recall seeing this error in any TPTP command line or other use case. Only in consumming product tests. 2. Comment #12, 228873 was an entirely different defect that was contained in tptp-automation-client.jar. That is not at all related to this one. 3. Note in comment #6 that this error does not necessarily prevent success in running a test. 4. Note that the error is intermittant. Thus, it is never a case that the library is not in the install. 5. The error is not only through ASF, it is also seen when running a test from a runtime workbench. Given the above, would you expect the test execution to succeed with execution history, etc., if loadLibrary really failed? Could this be directly or indirectly related to the previous defect where the environment strings were being mangled/missing such that loadLibrary truly failed because of incorrect path? Note that both deployment and environment strings are significantly more involved in the consumming product than in a TPTP URL test.
(In reply to comment #13) > 2. Comment #12, 228873 was an entirely different defect that was contained in > tptp-automation-client.jar. That is not at all related to this one. Correct. I was incorrect since Mark's comments (https://bugs.eclipse.org/bugs/show_bug.cgi?id=228873#c8) apply to this defect (see https://bugs.eclipse.org/bugs/show_bug.cgi?id=228873#c11).
A few more questions, I see that the bug description states the the IAC is being used, even though this is a command line operation... is this a standard configuration? And what is the agent itself doing... "com.ibm.etools.common.logging.listeners.CommonLoggingListener", how is it used by RPT? If the test execution is succeeding, complete with execution history, what's this agent's job? >> "Given the above, would you expect the test execution to succeed with execution history, etc., if loadLibrary really failed?" Well loadLibrary IS failing, that is all that we know about this bug so far :). That's the unsatisfied link error. This does, however, go to my question above. >> Could this be directly or indirectly related to the previous defect where the environment strings were being mangled/missing such that loadLibrary truly failed because of incorrect path? I had considered this, however the changes that caused the environment variable problems were in the I7/M7 timeframe. The first environment variable code change was April 18th as part of bug 224134. I6/M6 build was released April 7th, and this is the build the bug was originally reported against.
Hi Johnathan, I will look into the issues you have asked about in more detail. We do open a separate port passed in to the test runner to communicate with the workbench. Among other things, the agent's job includes starting the session JVM, deployment, setting the environment, and starting the test runner. These questions are relavent to how the test completes despite the failure. However, I am curious as to how these details could be related to the loadLibrary failure issue? The reason I brought up mangled/missing path is because of the fact that this issue is intermittant. The fact that there was a problem found and fixed does not preclude the possibility of another similar problem that has not yet been found. The real question in my mind is, what would be the reason for intermittant failure to loadLibrary?
Hi DuWayne, > > I will look into the issues you have asked about in more detail. We do open a > separate port passed in to the test runner to communicate with the workbench. > Among other things, the agent's job includes starting the session JVM, > deployment, setting the environment, and starting the test runner. These > questions are relavent to how the test completes despite the failure. However, > I am curious as to how these details could be related to the loadLibrary > failure issue? At some point in the process somebody is starting a LoggingAgent without having the necessary environment variables in their path to do so. Because, based on the exception, the kick off point seems to be the CommonLoggingListener class, I think this is the first point that needs to be investigated. It looks the LoggingAgentHandler.publish() method is somehow being called by the java.util.logging.Logger.log() method, but you'd have to look at the CommonLoggingListener source to determine why. Paul, are you familiar with LoggingAgentHandler class, and how it interfaces with the java.util.logging package? Is it possible that either TPTP or the consuming product are kicking this off accidentally? (Logging libraries such as log4j or commons logging, for instance, have a tendency to search the class path for log setting files, and then use them when they aren't actually relevant to the component being used) As far as I can tell by searching the TPTP code base, none of the TPTP code references or uses the LoggingAgentHandler (except ARM logging, which I presume RPT doesn't use). > The real question in my mind is, what would be the reason for > intermittant failure to loadLibrary? > That's a very good question ;). I'm not sure... without source code to a test case that reproduces the problem it's difficult to say so far. Ohh, and one other question to add to the pile... who is starting the CommonLoggingListener? Is that launched through the agent controller, or by a process that is launched by the agent controller, or by the workbench?
(In reply to comment #17) > Hi DuWayne, > > > > > I will look into the issues you have asked about in more detail. We do open a > > separate port passed in to the test runner to communicate with the workbench. > > Among other things, the agent's job includes starting the session JVM, > > deployment, setting the environment, and starting the test runner. These > > questions are relavent to how the test completes despite the failure. However, > > I am curious as to how these details could be related to the loadLibrary > > failure issue? > At some point in the process somebody is starting a LoggingAgent without having > the necessary environment variables in their path to do so. Because, based on > the exception, the kick off point seems to be the CommonLoggingListener class, > I think this is the first point that needs to be investigated. It looks the > LoggingAgentHandler.publish() method is somehow being called by the > java.util.logging.Logger.log() method, but you'd have to look at the > CommonLoggingListener source to determine why. > > Paul, are you familiar with LoggingAgentHandler class, and how it interfaces > with the java.util.logging package? Is it possible that either TPTP or the > consuming product are kicking this off accidentally? (Logging libraries such > as log4j or commons logging, for instance, have a tendency to search the class > path for log setting files, and then use them when they aren't actually > relevant to the component being used) The Logging Agent handleris an implementation of a Java Logging handler for writing logged events to a Logging Agent. Applications that log events to the Java Logging APIs can be configured to write these events to various handlers, including file handler (e.g. log file), Logging Agent handler, etc. The same type of handler (although called appender and logger) is provided for Apache Log4J and Apache Commons Logging. > As far as I can tell by searching the TPTP code base, none of the TPTP code > references or uses the LoggingAgentHandler (except ARM logging, which I presume > RPT doesn't use). TPTP does not use Java Logging as our logging API (Eclipse Logging). In this case, the consuming product is probably writing Common Base Events to Java Logging, configured to use the Logging Agent handler. Alex Nan now owns the logging components in TPTP. > > The real question in my mind is, what would be the reason for > > intermittant failure to loadLibrary? > > > That's a very good question ;). I'm not sure... without source code to a test > case that reproduces the problem it's difficult to say so far. > > Ohh, and one other question to add to the pile... who is starting the > CommonLoggingListener? Is that launched through the agent controller, or by a > process that is launched by the agent controller, or by the workbench? CommonLoggingListener is provided by IBM in the consuming product, independent of TPTP. It is started early when the workbench is started to listen for Eclipse Logging events.
Note, the Logging Agent handler wraps a Logging Agent which wraps a RemoteComponentSkelton.
Created attachment 103805 [details] Test Plug-in. This test plug-in contains a similar configuration to that used in the consuming product but only using TPTP code (e.g. Logging Agent). Duwayne, can you add this test plug-in to the consuming product and restart with the -clean option to see if the behavior can be reproduced. NOTE: The Agent Controller cannot be installed or configured (see #2 at http://dev.eclipse.org/viewcvs/index.cgi/platform/org.eclipse.tptp.platform.agentcontroller/src-native-new/packaging_md/windows/getting_started.html?root=TPTP_Project&view=co#Configure_Agent_Controller).
I have not been able to get Paul's plugin to repro the problem yet, I will try over the weekend. I don't think the activator is getting called.
In working on another defect, I found that Paul's plugin is working to create this error in TPTP. Using the runtime workbench, the RemoteComponentSkeleton is activated on the workbench when starting an HTTP recording (or running a test for example). The AC_LIBRARY_PATH is not set, which I believe is by design when it is the workbench process rather than the session JVM. Then, since the library path is null, the following is executed: try { if (!nativesLoadable) { // Bug 112749 begins System.loadLibrary("hcclco");//$NON-NLS-1$ System.loadLibrary("hccls");//$NON-NLS-1$ System.loadLibrary("hcbnd");//$NON-NLS-1$ System.loadLibrary("hcclsm");//$NON-NLS-1$ System.loadLibrary("hccldt");//$NON-NLS-1$ // Bug 112749 ends System.loadLibrary("hcjbnd");//$NON-NLS-1$ nativesLoadable = true; } } catch (Throwable e) { e.printStackTrace(System.err); } This always fails because the IAC dll's are not on the system path. It seems to me that if the static initializer is called while running workbench code, we have no need to load these librarties. The reason the test executes ok is because the agent actually initialized properly, it is the workbench that throws the exception that is apparently harmless. Given these findings, someone should be able to fix this today while I work on my other defect.
(In reply to comment #22) > In working on another defect, I found that Paul's plugin is working to create > this error in TPTP. Using the runtime workbench, the RemoteComponentSkeleton > is activated on the workbench when starting an HTTP recording (or running a > test for example). The AC_LIBRARY_PATH is not set, which I believe is by > design when it is the workbench process rather than the session JVM. Yep, that's correct. Generally speaking, there's no need to include the AC path in the PATH or AC_LIBRARY_PATH when starting the workbench. When running IAC from inside Eclipse the necessary paths are inserted into the ACServer process' environment variables when it is automatically started by the TPTP IAC workbench plugins, from inside the workbench. As you noted, because the workbench doesn't have these values in its path when it is run, it would be unable to start any logging agents on it's own, thus the error we are seeing here. > > The reason the test executes ok is because the agent actually initialized > properly, it is the workbench that throws the exception that is apparently > harmless. Given these findings, someone should be able to fix this today while > I work on my other defect. Because, AFAIK, the problem is on the workbench side and neither the agent controller nor the Hyades execution framework is involved in this bug, I am reassigning to test project lead for monitoring. Reassign back to me if either AC or Hyades agent/client framework are implicated by further investigation, thanks!
Bing, pls handle.
This problem is caused whenever the /org.eclipse.hyades.execution/src-remote/org/eclipse/hyades/internal/execution/remote/RemoteComponentSkeleton.java (e.g. /org.eclipse.hyades.logging.core/src/org/eclipse/hyades/logging/core/LoggingAgent.java, /org.eclipse.hyades.logging.core/src.log4j/org/eclipse/hyades/logging/log4j/LoggingAgentAppender.java, /org.eclipse.hyades.logging.core/src.java14/org/eclipse/hyades/logging/java/LoggingAgentHandler.java, etc.) is used AND the Agent Controller is not installed/configured (see #2 at http://dev.eclipse.org/viewcvs/index.cgi/platform/org.eclipse.tptp.platform.agentcontroller/src-native-new/packaging_md/windows/getting_started.html?root=TPTP_Project&view=co#Configure_Agent_Controller) AND the IAC is not running. Since the IAC emulates an installation of the Agent Controller (see #2 at http://dev.eclipse.org/viewcvs/index.cgi/platform/org.eclipse.tptp.platform.agentcontroller/src-native-new/packaging_md/windows/getting_started.html?root=TPTP_Project&view=co#Configure_Agent_Controller), I would think that it it should include the AC path in the PATH or AC_LIBRARY_PATH when starting the workbench so the RemoteComponentSkeleton can be used without starting the IAC. Jonathan and Bing, comments?
There were some comments in https://bugs.eclipse.org/bugs/show_bug.cgi?id=112749 that I ran across regarding the IAC library path. Worth a read in making decisions. Particularly comment #5, item 5.
Created attachment 104260 [details] Update library path when launch IAC Jonathan, can you review the patch. Thanks.
Paul and Duwayne, how do I reproduce this problem using TPTP only?
(In reply to comment #28) > Paul and Duwayne, how do I reproduce this problem using TPTP only? > Unzip the 'Test Plug-in' to your /dropins/eclipse/plugins/ directory and restart Eclipse with the -clean option.
I downloaded the Test plugin and unzipped it. Then I start Eclipse with -clean. I tried to do a HTTP recording or run a manual test but couldn't see the exception stacktrace in cosole or eclipse log or IAC log. Any extra steps needed?
Use a runtime workbench to start a recording and you will see the exception in the console window of your IDE workbench. Make sure the target platform plugins is showing that you picked up the plugin.
Created attachment 104375 [details] Test Plug-in V2 Exported the package in the plug-in manifest and added some logging statements (IStatus log records going to the .log file) for the constructor/startUp/earlyStartUp/stop methods in the activator. Please unzip in your workspace, import, and then export as a deployable plug-in to the /dropins/eclipse/plugins directory.
(In reply to comment #32) > Created an attachment (id=104375) [details] > Test Plug-in V2 > > Exported the package in the plug-in manifest and added some logging statements > (IStatus log records going to the .log file) for the > constructor/startUp/earlyStartUp/stop methods in the activator. > > Please unzip in your workspace, import, and then export as a deployable plug-in > to the /dropins/eclipse/plugins directory. > This linking error can be reproduced by unzipping/importing this plug-in into your workspace, starting a runtime workbench, and run a test suite from the runtime workbench UI.
Created attachment 104535 [details] Patch.
After meeting with Bing/Duwayne/Joanna, we have come to the following conclusion: -If the Agent Controller is not installed/configured (see #2 at http://dev.eclipse.org/viewcvs/index.cgi/platform/org.eclipse.tptp.platform.agentcontroller/src-native-new/packaging_md/windows/getting_started.html?root=TPTP_Project&view=co#Configure_Agent_Controller), the ac.library.path environment variable is not defined/set and the Agent Controller's bin directory (containing the native libraries) is not on the system path. -Until the IAC is started, plug-ins (for example, the test plug-in) using the LoggingAgent class (wrapping the RemoteComponentSkeleton class) will cause the UnsatisfiedLinkError exception. -Changing the IAC to add the IAC's bin directory to the system path or AC_LIBRARY_PATH when starting the workbench is not the correct fix since plug-ins (for example, the test plug-in) implementing the org.eclipse.ui.startup extension point will start before the IAC plug-in starts, resulting in the same symptom. -This UnsatisfiedLinkError exception is printed to the console when the native libraries are not on the system path due to a code change under defect #148461 (TPTP 4.4). However, there is no need to report this error since the RemoteComponentSkeleton reports the error when the native libraries cannot be located on the ac.library.path (error condition since the Agent Controller is installed/configured but the native libraries are not on the ac.library.path) and it gracefully handles the case when the native libraries are not available on the system path. -There is no negative impact on TPTP or the consuming product (confirmed with Duwayne) when the natives libraries are not available (e.g. UnsatisfiedLinkError exception) since the RemoteComponentSkeleton gracefully handles the case. The LoggingAgent (see the test plug-in) does handle this case (AgentControllerUnavailableException when initializing the RemoteComponentSkeleton). However, the LoggingAgent does not handle reinitializing the RemoteComponentSkeleton, if the IAC is started or the Agent Controller is installed/configured. The lowest risk fix is to remove the exception print statement when the native libraries are not available on the system path (see attached patch). This fix does not change the behavior RemoteComponentSkeleton, which already gracefully handles the case when the native libraries are not available on the system path, and revert the code back to the TPTP 4.4.0 level. In addition, defect #236710 has been opened to address the scenario in the LoggingAgent when the native libraries are not available.
Jonathan/Duwayne/Bing, please review the enclosed patch.
Reviewed, +1.
Patch is good given that new defect would be raised to address the library path issue.
Hours worked triaging and creating/testing the patch.
Looks good Paul, +1.
Patch PMC-approved and checked into CVS (HEAD).
Hi Paul, was the copyright year updated?
(In reply to comment #42) > Hi Paul, was the copyright year updated? Please ignore this. It was updated.
As of TPTP 4.6.0, TPTP is in maintenance mode and focusing on improving quality by resolving relevant enhancements/defects and increasing test coverage through test creation, automation, Build Verification Tests (BVTs), and expanded run-time execution. As part of the TPTP Bugzilla housecleaning process (see http://wiki.eclipse.org/Bugzilla_Housecleaning_Processes), this enhancement/defect is verified/closed by the Project Lead since this enhancement/defect has been resolved and unverified for more than 1 year and considered to be fixed. If this enhancement/defect is still unresolved and reproducible in the latest TPTP release (http://www.eclipse.org/tptp/home/downloads/), please re-open.
