Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 344422

Summary: Java Tracing
Product: [Eclipse Project] JDT Reporter: Martin Robertson <mchr3k>
Component: DebugAssignee: JDT-Debug-Inbox <jdt-debug-inbox>
Status: RESOLVED WONTFIX QA Contact:
Severity: enhancement    
Priority: P3 CC: curtis.windatt.public, mchr3k, Michael_Rennie, ndsilva, pwebster, remy.suen, swanj
Version: 3.7   
Target Milestone: ---   
Hardware: All   
OS: All   
Whiteboard:

Description Martin Robertson CLA 2011-05-02 05:20:09 EDT
I am a full time Java developer and I work with Java code using Eclipse almost every day in my job. Most of the time the existing Java debug features are suitable for helping me to track down problems. However, there is one use case where I think the existing tools are insufficient.

When I am working with multi threaded code I find that breakpoints become less useful as the order of execution of threads is important and so having individual theads stopped by a breakpoint can change the behaviour which I am trying to investigate. In this case I tend to rely on letting the code execute uninterrupted and writing out trace logs which show what code was executed and in what order. However, the process of adding this tracing to my code is time consuming and an obvious candidate for automation.

To this end I have created the InTrace tool (https://github.com/mchr3k/org.intrace/wiki) which uses a Java Agent to add instrumentation to selected classes so that trace output can be generated. The really nice aspect of doing this through a Java agent is that I can add instrumentation to any application class even if I don't have the source code.

InTrace is currently a standalone tool. I think it would be much more useful if this functionality was added to the Eclipse Java Debug project. I am imagining something like the following:

- EITHER
-- A new launch option to enable loading the InTrace agent when a Java program is launched.
-- A new post launch action which uses the Attach API (http://download.oracle.com/javase/6/docs/jdk/api/attach/spec/com/sun/tools/attach/VirtualMachine.html) to load the InTrace agent into an active program.
- A new View similar to the existing Console but showing the collected Trace output.

I would love to hear any feedback about this idea or about my InTrace tool. I expect that doing the Eclipse integration well would take a reasonable amount of time so I don't want to start it until I get a sense of whether it would be accepted. I am not keen to produce a standalone plugin as I think this useful feature should be available to as many people as possible as part of the Eclipse Java Debug project.
Comment 1 Remy Suen CLA 2011-05-02 06:08:50 EDT
(In reply to comment #0)
> -- A new post launch action which uses the Attach API
> (http://download.oracle.com/javase/6/docs/jdk/api/attach/spec/com/sun/tools/attach/VirtualMachine.html)

So this feature would only work for the HotSpot JVM, is this correct?
Comment 2 Martin Robertson CLA 2011-05-02 06:54:58 EDT
(In reply to comment #1)
> (In reply to comment #0)
> > -- A new post launch action which uses the Attach API
> > (http://download.oracle.com/javase/6/docs/jdk/api/attach/spec/com/sun/tools/attach/VirtualMachine.html)
> 
> So this feature would only work for the HotSpot JVM, is this correct?

I assume from the package name that is correct. However the actual Java Agent API (java.lang.instrument) is presumably available across all JVMs so the idea of a new launch option should work across all JVMs. 

I have to admit that I do all my work with the Sun JVM so I haven't tested with anything else.
Comment 3 Martin Robertson CLA 2011-05-02 06:57:26 EDT
(In reply to comment #2)
> (In reply to comment #1)
> > (In reply to comment #0)
> > > -- A new post launch action which uses the Attach API
> > > (http://download.oracle.com/javase/6/docs/jdk/api/attach/spec/com/sun/tools/attach/VirtualMachine.html)
> > 
> > So this feature would only work for the HotSpot JVM, is this correct?
> 
> I assume from the package name that is correct. However the actual Java Agent
> API (java.lang.instrument) is presumably available across all JVMs so the idea
> of a new launch option should work across all JVMs. 
> 
> I have to admit that I do all my work with the Sun JVM so I haven't tested with
> anything else.

Actually a quick google suggests that at least the IBM JVM has some level of support in its own platform specific package: http://publib.boulder.ibm.com/infocenter/javasdk/v6r0/topic/com.ibm.java.doc.user.aix64.60/user/attachapi.html
Comment 4 Martin Robertson CLA 2011-05-02 11:30:21 EDT
I am starting to think of the code changes which would be necessary for the "at launch" approach. There are two approaches which I have identified.

1) Add a new "Trace" launch mode to the existing "Run", "Debug" etc. 

This would use extension point org.eclipse.debug.core.launchModes to define the new mode. org.eclipse.debug.core.launchDelegates would be used to define how to launch appropriate existing launch configuration types (e.g. Java Application) in this new mode. 

I'm not sure how the actual VM argument would be injected.

2) Add a new config option to appropriate existing launch configuration types (e.g. Java Application).

I'm not sure what extension point would be used to expose UI for the new config option.

I'm not sure how the actual VM argument would be injected.
Comment 5 Michael Rennie CLA 2011-05-02 14:21:17 EDT
(In reply to comment #4)
> I am starting to think of the code changes which would be necessary for the "at
> launch" approach. There are two approaches which I have identified.
> 
> 1) Add a new "Trace" launch mode to the existing "Run", "Debug" etc. 
> 

This would be a good place to start. That way you can play with the integration and work out any kinks, then in the future we could look at supporting mixed-mode launching, which would allow you to just extend the normal Java Application launch configuration.

Another option might be to just add a new launch configuration type (for run and debug) that reused the Java Application stuff but added support for tracing - this approach would give you full control over the UI for the tab group and the shape / values in the launch configuration

> 
> I'm not sure what extension point would be used to expose UI for the new config
> option.
> 

If you wanted to extend the Java Application tab group you can use the org.eclipse.debug.ui.launchConfigurationTabs extension point to add your own custom tab to the tab group - you could add a new 'Tracing' tab to the Java Application tab group for example.

If you wanted to go the route of a new launch configuration type you would want to use the following extension points:

org.eclipse.debug.core.launchConfigurationTypes
org.eclipse.debug.core.launchDelegates
org.eclipse.debug.ui.launchConfigurationTabGroups

> I'm not sure how the actual VM argument would be injected.

Provided you extended / reused the code for a Java Application launch configuration, you could add them manually on the VM arguments block for the time being while you play around with it, and work on an automated scheme later.

You could also add any entries to the IJavaLaunchConfigurationConstants.ATTR_VM_ARGUMENTS value in the launch configuration and they will automatically be added to the VM arguments during launching.
Comment 6 Martin Robertson CLA 2011-05-02 18:46:16 EDT
Thanks for this feedback. 

My next concern is how to package this as a plugin. In the past I have written Eclipse plugins and packaged them by pressing "Build All" on a feature project. However, this produces a jar for the plugin. 

This plugin will need to add a -javaagent VM argument which points at an intrace-agent.jar. I know some eclipse plugins are installed as an unzipped directory. How do I make this happen for my own plugin? Can you point me at the right docs to generate a path to the jar in the install directory of a plugin?
Comment 7 Martin Robertson CLA 2011-05-08 19:02:24 EDT
OK so I have ported my existing UI to Eclipse. You can try out the new functionality by downloading the following archived update site.

https://github.com/downloads/mchr3k/org.intrace/intrace.ecl.site.zip

This is a pretty rough implementation but should hopefully work for you. This plugin includes the following.

- New launch mode "InTrace"
- Supported launch types: Java Application, Junit Test
- An "InTrace" launch is a modified "Run" launch
-- InTrace agent is loaded using a JVM argument
-- New InTrace editor is opened to connect to the agen
-- Launched program is initially paused until you press a "Start" button in the InTrace editor
-- Once the InTrace editor has been closed there is currently no way to re-open it

How to use:
1) Install the plugin
2) Launch an application or junit test in InTrace mode
3) Click "Select Classes..." and enter some include/exclude patterns to define which classes you want to trace
4) Click "Start Program"
5) Trace should be written to the Output tab

Let me know what you think. I know there is a LOT of polish which could still be added.
Comment 8 Martin Robertson CLA 2011-05-09 07:54:47 EDT
This page contains the documentation for the Eclipse UI.

https://github.com/mchr3k/org.intrace/wiki/Docs:-Eclipse-UI-%28Alpha%29
Comment 9 Martin Robertson CLA 2011-05-13 12:27:08 EDT
The target milestone has been set to Eclipse 3.8. Does that mean this feature will definitely be included with Eclipse 3.8? If so, when will I realistically be able to get my design/code reviewed? I'm sure you are very busy with the Eclipse 3.7 release right now.
Comment 10 Michael Rennie CLA 2011-05-13 13:01:51 EDT
(In reply to comment #9)
> The target milestone has been set to Eclipse 3.8. Does that mean this feature
> will definitely be included with Eclipse 3.8? 

Not exactly. It means we/I will review / work on it for inclusion in 3.8. I tagged it more-so so I would not forget about it.

> If so, when will I realistically be able to get my design/code reviewed? 

I am planning to do it in the coming weeks.

> I'm sure you are very busy with the Eclipse 3.7 release right now.

Yes.
Comment 11 Martin Robertson CLA 2011-05-20 08:38:53 EDT
I have now released V0.0.0.2 of my plugin with the following enhancements:

- Open a new InTrace editor from an existing launch - right click the launch and select "Open InTrace Output".
- Lots of code cleanup.
- Proper error handling - any errors should now be properly logged/shown to the user.

You can download this update and all future updates using this update site URL: 

http://intracesite.appspot.com/
Comment 12 Martin Robertson CLA 2011-05-21 03:48:01 EDT
The URL for the docs on GitHub is now https://github.com/mchr3k/org.intrace/wiki/Docs:-Eclipse-UI
Comment 13 Martin Robertson CLA 2011-05-21 04:10:12 EDT
Sorry about the spam, the correct link is actually https://github.com/mchr3k/org.intrace/wiki/Docs:-InTrace-Eclipse-Plugin
Comment 14 Michael Rennie CLA 2011-08-22 15:13:51 EDT
(In reply to comment #13)
> Sorry about the spam, the correct link is actually
> https://github.com/mchr3k/org.intrace/wiki/Docs:-InTrace-Eclipse-Plugin

I had a chance to play with the tracing support, and while I think it is pretty cool, I also think it is better left as a 3rd party add-on (something like JaCoCo, etc).

My biggest fear (and reason for not wanting to make it part of JDT debug) is that it is a lot of code and we currently do not have a lot of committer resources to spend on support / bug fixing for this new addition.

Thoughts?
Comment 15 Martin Robertson CLA 2011-08-23 12:21:55 EDT
Thanks for taking the time to look at this.

In an ideal world I think it would be great if JDT supported all of the following "out of the box":
- Code Coverage
- Profiling
- Tracing

However, given you have limited resources I agree that it makes sense to keep InTrace separate. The only thing I would suggest is that you add the integration of these features into core JDT onto a wishlist document for future work. In the event that you find some resources in the future it would be nice to think you might reconsider integrating InTrace. 

Please let me know if you have any feedback about InTrace itself.
Comment 16 Michael Rennie CLA 2011-08-23 15:04:53 EDT
(In reply to comment #15)

> The only thing I would suggest is that you add the
> integration of these features into core JDT onto a wishlist document for future
> work. In the event that you find some resources in the future it would be nice
> to think you might reconsider integrating InTrace. 
> 

I created a wiki page for us to keep track of such items: http://wiki.eclipse.org/Debug/Plan/Future

> Please let me know if you have any feedback about InTrace itself.

The tooling was simple to use and the integration was good, I thought it just needed a bit more polishing in the presentation. It would also be good if there was some way to organize the tracing(s) that you do get, to help with the analysis of them.

Marking as wontfix, since there are no immediate plans to add this to JDT debug.
Comment 17 Martin Robertson CLA 2011-08-24 18:34:52 EDT
(In reply to comment #16)
> (In reply to comment #15)
> 
> > The only thing I would suggest is that you add the
> > integration of these features into core JDT onto a wishlist document for future
> > work. In the event that you find some resources in the future it would be nice
> > to think you might reconsider integrating InTrace. 
> > 
> 
> I created a wiki page for us to keep track of such items:
> http://wiki.eclipse.org/Debug/Plan/Future
>

I have linked to some related projects here: http://mchr3k.github.com/org.intrace/related.html

Some of these tools aren't free but it would still be good to get some inspiration from them. In particular, I would love to see functionality like this in JDT: http://www.chrononsystems.com/

You should probably add a link to Eclipse TPTP to the Future plans page - perhaps parts of TPTP should become part of the default JDT project. However, I would personally suggest that TPTP needs some work to simplify it and make it easier to get started with.
 
> > Please let me know if you have any feedback about InTrace itself.
> 
> The tooling was simple to use and the integration was good, I thought it just
> needed a bit more polishing in the presentation. 

Do you have any specific suggestions about polishing the UI?

I have already been through several iterations with the UI but would love to get some more ideas about how to make it simpler.

http://mchr3k.github.com/org.intrace/img/connectedgui-v4.jpg
http://mchr3k.github.com/org.intrace/img/connectedgui-v3.jpg
http://mchr3k.github.com/org.intrace/img/connectedgui-v2.jpg
http://mchr3k.github.com/org.intrace/img/connectedgui.jpg

> It would also be good if there was some way to organize the tracing(s) 
> that you do get, to help with the analysis of them.

The UI lets you save the trace output as a text file - are you imagining a View to provide quick access to historical traces?
Comment 18 Michael Rennie CLA 2011-08-30 09:24:24 EDT
(In reply to comment #17)

> Do you have any specific suggestions about polishing the UI?
> 

Specific, no. It just felt like there was a lot going on all at once, especially once the trace starts filling the central text area. Perhaps it would be better to run the trace as a job and just present the user with the end result, rather than run it all in a view? Maybe even present the results as a search result (or something similar)?

> The UI lets you save the trace output as a text file - are you imagining a View
> to provide quick access to historical traces?

Yes, and some way to group / reorganize a given trace - i.e. a tree view representation of the text file (or something similar).