Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 201408

Summary: [ThreadProf] Many redundant, short-living threads reported when data collection is resumed
Product: z_Archived Reporter: Asaf Yaffe <asaf.yaffe>
Component: TPTPAssignee: Igor Alelekov <igor.alelekov>
Status: CLOSED FIXED QA Contact:
Severity: major    
Priority: P1 CC: analexee, asaf.yaffe, guru.nagarajan, jkubasta, stanislav.v.polevic
Version: unspecifiedKeywords: plan
Target Milestone: ---   
Hardware: All   
OS: All   
Whiteboard: closed460

Description Asaf Yaffe CLA 2007-08-28 07:04:40 EDT
Driver: TPTP-4.4.1-200708280100

Steps to reproduce:
1. Start the attached Java test application in "enabled" mode:
java -cp THREAD -agentlib:JPIBootLoader=JPIAgent:server=enabled;ThreadProf TestMain
2. From Workbench, create an "attach" launch configuration, configure it to "automatically start monitoring when application is launched", and start profiling.
3. In the Thread Statistics view, observe that approx. 13 threads (named Thread-0 to Thread-12) appear in "stop" state with a running time of 0. These threads were not created by the test application.

Root-cause analysis:

When data collection starts, ThreadProf queries Martini for all active threads, and then repeatedly queries Martini for information about the thread objects and the monitors held by each thread. All this activity is done in the context of the "command handler" thread of JVMTI Agent (JPIAgent). This thread is external to the JVM, so Martini needs to attach it to the JVM each time it calls a data request operation. Each time the thread is attached, the JVM generates a ThreadStart JVMTI event for the thread. These events are reported by Martini to ThreadProf.

Possible solutions:

Fixing this behavior in Martini is hard. I am not aware of any way to distinguish legitimate ThreadStart JVMTI events from ThreadStart events generated as a result of attaching a thread to the JVM.

Here are two other approaches for fixing this issue:
1. In ThreadProf: temporarily disable the ThreadStart and ThreadEnd event observers while processing the attach/detach/pause/resume commands. The downside of this approach is that some thread events may be missed at that time.

2. Move the responsibility of attaching external threads from Martini to its clients (ThreadProf, CGProf, etc). This can be done by exposing two new Martini APIs: AttachCurrentThread and DetachCurrentThread, and by modifying all other APIs to return errors when called from an unattached thread. This way, the Martini client can attach all its external threads to the JVM once, and detach them when they terminate.
Comment 1 Alexander N. Alexeev CLA 2007-08-28 08:57:09 EDT
Second approch is preferable, because it is allows to control time and conditions then attach/detach happens.
But probably existed functionality with automatic attach then required should remain. It allows solve problem without breakage of current functionality.
One possible approche is add second set of function which can  be called from already attached thread.
Comment 2 Stanislav Polevic CLA 2007-10-08 06:50:28 EDT
Asaf, since this solution requires changes in Martini API, could you estimate it, please?
From profiler side I estimate it in 2 days to implement and 1 day to test.
Comment 3 Asaf Yaffe CLA 2007-10-08 07:02:09 EDT
Martini changes should take no longer than 1 day (implementation and testing)
Comment 4 Stanislav Polevic CLA 2008-01-09 05:08:46 EST
Added to Igor's queue.
Comment 5 Igor Alelekov CLA 2008-02-08 07:10:17 EST
Hi Asaf,
Regarding to the second solution:
Don't you think that instead of set of short-living threads we will get the same threads, but long-living? JVM creates a new thread each time of attachement. And, if clients will keep their attached state, these threads will not die and will be reported to the user as before.

As another option, can we just filter out these redundant threads based on some their characteristics? I hoped we can use thread's state (JVMTI_THREAD_STATE_IN_NATIVE field), but in my sample jvm returns the same state (5) for all threads. 

Thanks, Igor.
Comment 6 Asaf Yaffe CLA 2008-02-10 03:57:10 EST
(In reply to comment #5)
> Don't you think that instead of set of short-living threads we will get the
> same threads, but long-living? 

You'll get one such thread each time you attach. 
Best case scenario: one thread per profiling session
Worst case scenario: one thread each time data collection is attached, paused or resumed.

Both options are better than the current behavior.

> 
> As another option, can we just filter out these redundant threads based on some
> their characteristics? I hoped we can use thread's state
> (JVMTI_THREAD_STATE_IN_NATIVE field), but in my sample jvm returns the same
> state (5) for all threads. 

I couldn't find such a characteristic. When you attach an external thread to the JVM, the JVM creates a java.lang.Thread for the attached thread, and this triggers the JVMTI ThreadStart event. The same process happens with regular application threads.

Comment 7 Igor Alelekov CLA 2008-02-13 09:56:38 EST
(In reply to comment #0)
> 2. Move the responsibility of attaching external threads from Martini to its
> clients (ThreadProf, CGProf, etc). This can be done by exposing two new Martini
> APIs: AttachCurrentThread and DetachCurrentThread, and by modifying all other
> APIs to return errors when called from an unattached thread. This way, the
> Martini client can attach all its external threads to the JVM once, and detach
> them when they terminate.

Asaf, as a version of the (2), Can we consider the following changes?
- All attachements in the Martini remain the same to ensure that a thread is attached. If the thread was already attached before, the call will not have any effect.
- Remove calls to DetachCurrentThread() from Martini
- expose DetachCurrentThread() call to delegate responsibility of detaching from Martini to its clients.

It seems this might be more simple to implement and no additional checks for unattached threads are required.


Comment 8 Asaf Yaffe CLA 2008-02-17 03:13:08 EST
(In reply to comment #7)
> Asaf, as a version of the (2), Can we consider the following changes?
> - All attachements in the Martini remain the same to ensure that a thread is
> attached. If the thread was already attached before, the call will not have any
> effect.
> - Remove calls to DetachCurrentThread() from Martini
> - expose DetachCurrentThread() call to delegate responsibility of detaching
> from Martini to its clients.
> 
> It seems this might be more simple to implement and no additional checks for
> unattached threads are required.
> 

This suggestion will make the API behavior less predictable (e.g., if a tool didn't call an "attach" API, why should it call a "detach" API?). 

How about enhancing option 2 to behave as follows:
1) Expose Martini Attach and Detach APIs
2) Keep auto-attach functionality intact, but only in cases when the thread was unattached prior to the API calls which require attach.

This will result in a consistent behavior:
1) If the tool didn't call Attach, Martini attaches the thread for the duration of the API execution, and then automatically detaches
2) If the tool attached the thread prior to invoking an API, Martini will not detach the thread upon API completion.

What do you think?
Comment 9 Igor Alelekov CLA 2008-02-18 02:31:50 EST
Agree. I like it.
Do you think we should open new bugzilla to track required changes in Martini?
Comment 10 Asaf Yaffe CLA 2008-02-18 02:47:01 EST
Actually, thread attach/detach APIs are already informally exposed by Martini, but not through the MPI interface. 

The JPI library exports two functions:

JPI_API TResult JPI_AttachCurrentThread(JNIEnv **ppJNIEnv, bool *bAttached);
JPI_API void JPI_DetachCurrentThread();

**ppJNIEnv is an output variable for the thread's JNI environment pointer.
*bAttached is an output variable that tells you whether the thread was attached (true) or was already attached prior to the call (false). This allows you to write safe code like this:

<code>

bool bAttached;
JNIEnv *pNewEnv;
JPI_AttachCurrentThread(&pNewEnv, &bAttached);

// do something while attached...

if (bAttached) JPIDetachCurrentThread();

</code>


These functions are currently used by the Dynamic Probekit Agent. Please see the "CProbekitAgent::BindJpiFunctions()" function in src-native\src\ProbekitAgent\ProbekitAgent.cpp for an example on how to bind these two APIs.
Comment 11 Asaf Yaffe CLA 2008-02-18 02:49:43 EST
In reply to comment #10)
Also note that Martini itself is using these two function to attach/detach threads as necessary. 

Bottom line: use these APIs. No further changes are required from Martini.
Comment 12 Igor Alelekov CLA 2008-02-18 03:01:27 EST
(In reply to comment #11)
> In reply to comment #10)
> Also note that Martini itself is using these two function to attach/detach
> threads as necessary. 
> Bottom line: use these APIs. No further changes are required from Martini.

Thanks, I'll use these APIs to attach/detach.
Comment 13 Igor Alelekov CLA 2008-03-21 11:18:56 EDT
resolving as fixed
Comment 14 Paul Slauenwhite CLA 2009-06-30 13:28:24 EDT
As of TPTP 4.6.0, TPTP is in maintenance mode and focusing on improving quality by resolving relevant enhancements/defects and increasing test coverage through test creation, automation, Build Verification Tests (BVTs), and expanded run-time execution. As part of the TPTP Bugzilla housecleaning process (see http://wiki.eclipse.org/Bugzilla_Housecleaning_Processes), this enhancement/defect is verified/closed by the Project Lead since this enhancement/defect has been resolved and unverified for more than 1 year and considered to be fixed. If this enhancement/defect is still unresolved and reproducible in the latest TPTP release (http://www.eclipse.org/tptp/home/downloads/), please re-open.
Comment 15 Paul Slauenwhite CLA 2009-06-30 13:48:18 EDT
As of TPTP 4.6.0, TPTP is in maintenance mode and focusing on improving quality by resolving relevant enhancements/defects and increasing test coverage through test creation, automation, Build Verification Tests (BVTs), and expanded run-time execution. As part of the TPTP Bugzilla housecleaning process (see http://wiki.eclipse.org/Bugzilla_Housecleaning_Processes), this enhancement/defect is verified/closed by the Project Lead since this enhancement/defect has been resolved and unverified for more than 1 year and considered to be fixed. If this enhancement/defect is still unresolved and reproducible in the latest TPTP release (http://www.eclipse.org/tptp/home/downloads/), please re-open.