Community
Participate
Working Groups
Driver: tptp.sdk-TPTP-4.4.0-200705080100B Using JVMPI, with IAC and mentioned driver. 1.Launch an Java application outside of workbench and attach to it from workbench 'Attach to Agent' launch configuration 2.Start profile, verify data is shown in view of agent(a1). Detach from agent(a1) 3.Reattach to process, a new Profiling agent(a2) is shown and continue profile. Note profile data are sent to the a1 instead of a2.
same problem with RAC
problem exists in both PI and TI attach.
I think we should try to resolved this bug in 4.4 as it's a regression.
Marius, The attach and reattach agent are basically with the same attribute (including the runtimeID). Would the loader load everything to the same Agent in such case? example org.eclipse.hyades.models.hierarchy.impl.TRCAgentProxyImpl@69246924 (name: Java Profiling Agent, type: Profiler, runtimeId: e271cff5-f09d-4513-bf7a-188b783537d2, startTime: 1.181161789241E9, stopTime: 0.0, collectionMode: EXECUTION_STATISTICS_ONLY, active: true, attached: true, collectionData: false, monitored: false, profileFile: null, deltaTime: 0.0) org.eclipse.hyades.models.hierarchy.impl.TRCAgentProxyImpl@361c361c (name: Java Profiling Agent, type: Profiler, runtimeId: e271cff5-f09d-4513-bf7a-188b783537d2, startTime: 1.181161829769E9, stopTime: 0.0, collectionMode: EXECUTION_STATISTICS_ONLY, active: true, attached: true, collectionData: false, monitored: false, profileFile: null, deltaTime: 0.0)
I captured the last example in the AttachListener.AgentActive method. Assuming the attach and detach is working properly, attached: collectionData: and Monitored: attributes should be updated correspondingly . Would the loader be able to load to the agent that is attach=true and monitored=true among all agent of the same runtimeId?
Why is this defect a critical. This ought to be a blocker.
Created attachment 70621 [details] Team Project set to debug the problem
Created attachment 70622 [details] Patch to debug the connection/XMLTraceDataProcessor reuse problem I suppose on detach the data connection should be closed and XMLTraceDataProcessor should be cleaned-up, see the next attachement for debug output that showcases the problem.
Created attachment 70623 [details] stdout from debug session The output is from a session where I attached, collected some events, detached, reattach, collected some events (on the original connection/XMLTraceDataProcessor), terminated the process, started the process again and repeated the attach/reattach 3 times again with some event generated/collected between each operation (the same XMLTraceDataProcessor was used). This problem might be the reason for the memory leaks described in bug 186711 and the thread leaks in bug 179157 (because XMLTraceDataProcessor.cleanup() is not called).
Ali and Eugene please take a look at what changes were done in 4.4 in the attach/lunch configurations that triggers this behavior. I'll also continue to investigate and try to find a fix for this problem (which might fix indirectly the other 2 bugs mentioned above).
Created attachment 70627 [details] stdout from TPTP 4.2.2 debug session As I suspected the behavior in attach/detach/reattach has been changed since TPTP 4.2.2. In TPTP 4.2.2 the connection was closed on detach (so the XMLTraceDataProcessor has been cleaned up correctly and all the required deregistration was invoked) and on reattach the TRACE/node/processCreate/agentCreate events were resent. One other thing that I've noticed is that the old agent client framework uses the loadEvent by stream (where the connection input stream is wrapped with a filter stream that will strip out of the data message headers and then is passed to the XMLLoader parser, the XML parsing is done in the same thread) where the new agent client framework uses the old method loadEvent by event (one by one and the XML parsing is done in the parserThread). Guru please take a look at this and if is not possible to use the TCPDataProcessorNew input stream mode (which is fairly old and stable) in 4.4 we should follow up on this post 4.4.
Created attachment 70680 [details] Agent.detach behavior in 4.2.2 I've run a profiling session in TPTP 4.2.2 (using JVMPI) on PerformanceExample with Attach->Detach->ReAttach(from Profiling Monitor->Terminate the application from outside operations. See how the data connection stream was closed (Connection closed!) after the AgentImpl.detach method was called.
Created attachment 70681 [details] Attach.debug behavior in TPTP 4.4 I've run the same scenario as above in TPTP 4.4 (using JVMPI) on PerformanceExample. See how the data connection stream was closed (Connection closed!) only when I terminated the application (after second attach), and the same XMLTraceDataProcessor was used in both attach operation (in TPTP 4.2.2 a different XMLTraceDataProcessor was used in each attach operation).
Created attachment 70682 [details] Attach.debug behavior in TPTP 4.4 I've run the same scenario as above in TPTP 4.4 (using JVMPI) on PerformanceExample. See how the data connection stream was closed (Connection closed!) only when I terminated the application (after second attach), and the same XMLTraceDataProcessor was used in both attach operation (in TPTP 4.2.2 a different XMLTraceDataProcessor was used in each attach operation).
Guru, I'll transfer this defect to you for further investigation, as I explained above the AgentImpl.detach behavior changed in 4.4 vs 4.2.2 (the data connection input stream is not closed on detach). Althogh a data connection stream is created for each attach operation (search for XMLTraceDataProcessor.incommingStream) the data is sent through the first data connection in both cases. Please let me know if you need any help.
Marius, I am explicitly calling DataProcessor.cleanUp but despite this on re-attach the XMLLoader is loading into the old data processor. I am using the new Execution framework. In the Agent.StopMonitoring (equiv of Detach) the data server connection is closed and the data listeners are removed. Despite this the loader is not loading data into the new registered data processor. Pls investigate
Marius I think I found the issue. I have the patch being tested now. Will attach it for review. I have the attach scenario working fine now.
Created attachment 71220 [details] Patch for Execution Framework and the TI Client to resolve the dataprocessor loader issue Test case is to profile the Eclipse Workbench using MemoryANalysis. with the current build. Following the 6/13 candidate build you can test with all the types of collectors
Marius, is there someone else who can review this patch while you work on your defect fixes?
Guru, It seems that the patch it still doesn't fix the problem, I applied the patch on the build where I tested initially (0607). I'll try again with the latest TPTP build. Looking at the patch I'm not sure if by calling the dataProcessor.cleanup() you solve the problem, instead of calling that directly it should be called from DataServerListener.dataServerExited which should be called when the data connection is closed. Regarding the testcase I've being using was to start PerformanceExample with: java -Xrunpiagent:server=controlled PerformanceExample then attach from the TPTP workbench (using JVMPI execution time analysis with graphical details enabled) and then I did a detach and reattach, then clicked Add button in PerformanceExample. I'm not familiar with the agent client code, beside the TCPDataServer class, and I don't know how different is the new client code, but it might be easier to take a look at how this was done in 4.2 (or 4.3), it should follow the same approach. The way I see it is that the detach method should trigger the close of data connection input stream which would trigger the dataServerExited (search for ".dataServerExited()" in org.eclipse.hyades.internal.execution.local.common.TCPDataServer class) which would trigger the cleanup. You need to match the sequence from TPTP 4.2.2 detach behavior output (https://bugs.eclipse.org/bugs/attachment.cgi?id=70680): java.lang.Throwable: AgentImpl.detach() = org.eclipse.hyades.internal.execution.local.control.AgentImpl@50145014 at org.eclipse.hyades.internal.execution.local.control.AgentImpl.detach(AgentImpl.java:199) at org.eclipse.hyades.trace.ui.internal.util.PDCoreUtil.detachAgent(PDCoreUtil.java:308) ... TCPDataServerNew.done(): entered - Thread[TCPDataFiller,6,main] java.lang.Throwable: TCPDataServerNew.done(): Connection closed ! Thread[TCPDataFiller,6,main] at org.eclipse.hyades.internal.execution.local.common.TCPDataServer$TCPDataProcessorNew.done(TCPDataServer.java:848) at org.eclipse.hyades.internal.execution.local.common.TCPDataServer$TCPDataProcessorNew.access$1(TCPDataServer.java:839) at org.eclipse.hyades.internal.execution.local.common.TCPDataServer$TCPDataProcessorNew$1.read(TCPDataServer.java:622) at org.eclipse.hyades.trace.ui.internal.piclient.XMLTraceDataProcessor$4.read(XMLTraceDataProcessor.java:1035) ... at org.eclipse.hyades.loaders.util.XMLLoader.cleanUp(XMLLoader.java:250) at org.eclipse.hyades.trace.ui.internal.piclient.XMLTraceDataProcessor.cleanUp(XMLTraceDataProcessor.java:1135) at org.eclipse.hyades.trace.ui.internal.piclient.XMLTraceDataProcessor.dataServerExited(XMLTraceDataProcessor.java:327) at org.eclipse.hyades.internal.execution.local.common.TCPDataServer$TCPDataProcessorNew.run(TCPDataServer.java:772)
Marius I have changes in two areas a) In the execution framework since we are using ephemeral ports the dataserver is explicitly shutdown and memory is released. (These are the changes in the execution framework part of the patch) b) Secondly, by forcing the DataProcessor.cleanup - I am releasing the processor resources to address the data server shutdown delay. If I do not have the clean up - the data is not loaded by the new processor even if the dataserver is shutdown. Try this with the MemoryAnalysis since that this the best barometer of this. I have tested it substantially and it works fine with the patch attached. Makesure to refresh your CVS sources and test against atleast the 6/11 AC.
BTW the data server code is in the new execution framework and this differs slightly with the old (hyades) execution framework/
*** Bug 192555 has been marked as a duplicate of this bug. ***
Guru, you need to call dataServerExited (avoid calling cleanup directly) in order to make sure all the cleanup and deregistration is done. Make sure the JVMPI works also, it didn't work for me.
Created attachment 71321 [details] Updated patch and clean up of errors Ruslan Pls use this patch for the testing. Also look at the Execution Stat - not sending data on the attach. The Memory and Thread are working good.
Created attachment 71432 [details] Proposed fix These fix extends patch #71321 to enable exec data collection: 1. collection mode to be copied from old to new tiAgent (which affect displaying of data) 2. added extra polling calls on attach in order to enable data update in the view Review these patch please.
Patch reviewed and tested and it looks good. The attach functionality is now working well.
Please submit approval request as patched build will be done this afternoon
Guru/Ruslan, Would you please attach a patched plug-in jar for testing with the 0614 build? Thanks a lot!
Joanna, You mean to attach a jar of the plugins in context.
I think that is a question for Eugene...
Marius, please confirm this patch fixes 186711
(In reply to comment #31) > I think that is a question for Eugene... > that's correct. Also I try to verify the patch with my development workbench. problem stills shown with 1) PI reattach of an detached process that is started outside of workbench, and 2) TI launch from WB and reattach within same WB.
Eugene I am unable to recreate the issue called out in the last comment reg. TI. We just went thru the cases for TI and we were not able to recreate it. Please check if you are running the workbench with the latest patch applied. Thanks
(In reply to comment #32) > Marius, please confirm this patch fixes 186711 > I'm still testing the patch, please see my note on the platform mailing list. This defect is for both PI and TI and if it doesn't both we should split it into two defects, although we should fix both problems at once.
(In reply to comment #34) > Eugene I am unable to recreate the issue called out in the last comment reg. > TI. We just went thru the cases for TI and we were not able to recreate it. > Please check if you are running the workbench with the latest patch applied. > > Thanks > We have tested on CoreDuo or Core2Duo systems, I'll test also on a single CPU and update the status.
Created attachment 71517 [details] Here is the output from the Test I did to validate 187090. Here is the output from the Test I did to validate 187090. I do not see any issues. Eugene, Joanna – Let me know if you have any questions.
Hi Guru, You are right, If I wait long enough, I will see data refresh with values. However, I see some problem in your screen captures, In the second Attach, why do main() and remove() events show in table as they are not invoked since the second attach, Also I see problem of bug 192300 when I test this scenario which I believe is not a UI problem at all. The execution framework does not respect the stop monitoring event from the workbench side at all.
(In reply to comment #37) Guru, please can you specify the CPU type also (solo, hyper-threaded, coreduo or core2duo). Please add the following line before first line in the XMLTraceDataProcessor.cleanUp() method: ModelDebugger.log("XMLTraceDataProcessor.cleanUp - before, this="+this+", scanner="+scanner); and after last line in the same method add: ModelDebugger.log("XMLTraceDataProcessor.cleanUp - after, this="+this+", scanner="+scanner); also change the org.eclipse.hyades.models.util.ModelDebugger.exceptionDepth to 10. run your scenario and attach the output here.
(In addiction to comment #39) Please profile execution and heap in both statistical and full graphical details modes and provide the debug output.
(In reply to comment #38) > Hi Guru, > You are right, If I wait long enough, I will see data refresh with values. > However, I see some problem in your screen captures, In the second Attach, why > do main() and remove() events show in table as they are not invoked since the > second attach, Also I see problem of bug 192300 when I test this scenario which > I believe is not a UI problem at all. The execution framework does not respect > the stop monitoring event from the workbench side at all. > I've seen the same delay when testing on a T40 (single CPU). Another problem is that Eclipse attach/launch is not working, I've waited a few minutes and no data was coming from TI agent, PI seems to work.
Guru and Ruslan please add the info mentioned in comment #39, I will try to add today my debug output also for that scenario. Use PerformanceExample if possible. Make sure you add also the following line in XMLLoader.startElement before running the profiling sessions: public void startElement(String elementName, boolean hasAttributes, boolean isEmpty) { System.out.println("XMLLoader.startElement() "+this.toString()+" = "+elementName); As I said on the mailing list if cleanup is not called (in both cases PI and TI) on detach we will see memory and thread leaks (already mentioned in other critical bugs). Before 4.4 dataServerExited was called (as you can easily see in https://bugs.eclipse.org/bugs/attachment.cgi?id=70680) when there was no data expected to come from the agent, without that call I wouldn't know when to stop the XML parser and refresh threads.
Marius, I am calling DataProcessor.cleanUp method. This is called on AgentDetach. See the patch section =================================================================== RCS file: /cvsroot/tptp/platform/org.eclipse.tptp.platform.jvmti.client/src/org/eclipse/tptp/platform/jvmti/client/internal/TIUtility.java,v retrieving revision 1.3 diff -u -r1.3 TIUtility.java --- src/org/eclipse/tptp/platform/jvmti/client/internal/TIUtility.java 27 Feb 2007 06:05:25 -0000 1.3 +++ src/org/eclipse/tptp/platform/jvmti/client/internal/TIUtility.java 15 Jun 2007 10:57:24 -0000 @@ -87,7 +87,13 @@ { if (trcAgentProxy == null || !trcAgentProxy.isToProfileFile()) return; + final ACDataProcessorAdapter dataProcessor = (ACDataProcessorAdapter) LoadersUtils.locateDataProcessor(trcAgentProxy); + if (dataProcessor != null && dataProcessor.getProfileFileName() == null) + { + dataProcessor.cleanUp(); + } + if (dataProcessor != null && dataProcessor.getProfileFileName() != null) { /* Start a thread that will periodically check the activity of the processor. If it does not detect any activity
(In reply to comment #43) > Marius, > I am calling DataProcessor.cleanUp method. > This is called on AgentDetach. Guru, I've seen that you call it in the patch, but I've haven't seen the method being called, I think somehow that code is not touched. Please attach the debug output requested above, that way you can confirm that the method is called and the XMLLoader.scanner is made null.
Fixed
Can anyone provide a list of files that are affected by this defect? I see 3 patches and some files are modified in more than one patch. I need this list to patch the candidate build.
Sorin, The patch to be applied is https://bugs.eclipse.org/bugs/attachment.cgi?id=71432.
verified and close bug
