Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 188796

Summary: [jsr199] Using JSR199 to extend ECJ
Product: [Eclipse Project] JDT Reporter: talper
Component: CoreAssignee: Kenneth Olson <olsok>
Status: VERIFIED FIXED QA Contact: Stephan Herrmann <stephan.herrmann>
Severity: normal    
Priority: P3 CC: daniel_megert, dh_tue, duemir, jarthana, jerome_lanneluc, jordy.potman.recore, Olivier_Thomann, olsok, philippe_mulet, shankhba, stephan.herrmann
Version: 3.3Flags: jarthana: review+
Target Milestone: 4.5   
Hardware: All   
OS: All   
See Also: https://git.eclipse.org/r/43921
https://git.eclipse.org/r/43949
Whiteboard:
Attachments:
Description Flags
Testcase that works with javac but fails with ecj
none
Sample implementation
jarthana: iplog+
Make EclipseCompiler support non-'file' URI schemes
jarthana: iplog+
Removed unused field from ClasspathJsr199
jarthana: iplog+
Allow (re)use of Archive after 'flush' call
jarthana: iplog+
Allow (re)use of EclipseFileManager after 'close' call
jarthana: iplog+
Proposed patch
none
Proposed patch none

Description talper CLA 2007-05-23 21:23:56 EDT
Build ID: S-3.3RC1-200705171700

Steps To Reproduce:
1. Take the interfaces defined in JSR199 (see the javax.tools package)
2. Attempt to implement classes which provide an implementation of the interfaces providing all the IO needed by the compiler.
3. See that the JavaFileManager passed in as parameter to javax.tools.JavaCompiler.getTask is not used by ECJ to load classes or sources, but instead the setPaths method on org.eclipse.jdt.internal.compiler.tool.EclipseCompiler assumes that it is dealing with implementations that are not abstracted away from the file system.

More information:
Quoting from section 2.1 of JSR199
see: http://jcp.org/en/jsr/detail?id=199#2
---------------
The JavaTM Compiler API is a set of interfaces that describes the functions provided by a JavaTM Language Compiler, and a service provider framework so vendors can provide implementations of these interfaces.

The interfaces abstract the way a compiler interacts with its environment. While the existing command-line versions of compiler receive their inputs from the file systems and deposit their outputs there, reporting errors in a single output stream, the new compiler API will allow a compiler to interact with an abstraction of the file system. ...
---------------

While it is true that the JSR199 specification is somewhat vague about how exactly a compiler is supposed to interact with the interfaces defined, it seems that it should interact in a way which enables the aforementioned goals.
Comment 1 Olivier Thomann CLA 2007-05-24 09:10:47 EDT
If you have a specific test case that shows obvious problems, please attach it.
Comment 2 talper CLA 2007-05-24 15:02:22 EDT
Created attachment 68640 [details]
Testcase that works with javac but fails with ecj

This class can be run and will perform a simple compilation with a JavaFileManager that extends ForwardingJavaFileManager<StandardJavaFileManager> and logs all calls, then forwards the calls on to the StandardJavaFileManager returned from the JavaCompiler, and finally logs return values.

My tests show that this works with javac and fails with ecj.
Comment 3 Philipe Mulet CLA 2007-05-25 05:02:18 EDT
If a fix becomes ready for RC3, I think we should consider it.
Comment 4 Olivier Thomann CLA 2007-07-31 16:28:17 EDT
For now we have a implementation of the jsr199 that is a wrapper of the batch compiler. By definition a batch compiler is file-based.
We should define an implementation of a INameEnvironment that would wrap a java file manager.
Then we should end up with something closer to what you want. Note that using the system compiler I get call to inferBinaryName and there is no garantee that you would end up with the same trace.
I don't see this for 3.3.1, but rather for 3.4.
Comment 5 talper CLA 2007-08-07 14:38:38 EDT
That's fine if the trace is not precisely the same. I'm hoping to be able to override the mechanism for retrieving sources and classes used for compile as well as the mechanism for writing out compiled classes.

I'm looking forward to 3.4. :)
Comment 6 Philipe Mulet CLA 2008-01-30 04:41:11 EST
Maxime - pls interact with Olivier
Comment 7 Olivier Thomann CLA 2008-02-01 10:18:58 EST
To summarize our discussion, we should retrieve source file and class files through the java file manager API and we should write the .class file using the getJavaFileForOutput method.

So we need a way to map JavaFileObject to NameEnvironmentAnswer.
Comment 8 Philipe Mulet CLA 2008-03-12 12:08:06 EDT
To be clear, only EclipseCompilerImpl should be using the tool API, not the original batch (Main).
Comment 9 Maxime Daniel CLA 2008-04-16 08:28:28 EDT
Released org.eclipse.jdt.compiler.tool.tests.CompilerInvocationTests#12-16 to survey our results in that area. Inactive tests #13 and 14 fail, because we happen to fetch source and class files directly from the file system under certain conditions. Will investigate a fix in the direction set by Olivier's comment #7.
Comment 10 Jerome Lanneluc CLA 2008-05-02 04:54:31 EDT
We ran out of time for 3.4. Deferring to 3.5
Comment 11 Jerome Lanneluc CLA 2009-04-28 11:32:28 EDT
Not for 3.5
Comment 12 Dennis Hendriks CLA 2012-10-16 09:13:59 EDT
I'm also running into trouble with the EclipseCompiler not properly using the JavaFileManager for input/output/dependencies. See the discussion at http://www.eclipse.org/forums/index.php?t=rview&goto=946656

Is the any progress on this issue? It would be really nice if the EclipseCompiler could be used for run-time in-memory compilation.
Comment 13 Stephan Herrmann CLA 2012-10-25 13:09:41 EDT
@Olivier, did you have any draft patch for this issue that you could share?
Comment 14 Olivier Thomann CLA 2012-10-26 08:04:40 EDT
No, unfortunately. What needs to be done is that as soon as the batch compiler needs to access the file system, we redirect the jsr199 implementation to go through the JavaFileManager.
This sounds easy, but it requires some work and we never had the time to work on this.
Comment 15 Stephan Herrmann CLA 2012-10-28 12:05:24 EDT
While JDT/Core may not have the resources to implement this in the near future, I believe this would be an excellent candidate for a contribution from the community!

If I understand correctly, the task is to

- refactor org.eclipse.jdt.internal.compiler.batch.Main so that all File
  operations are extracted to overridable methods (no change of behavior here).

- override these hooks in 
  org.eclipse.jdt.internal.compiler.tool.EclipseCompilerImpl
  as to map all operations to the fileManager instance

- also consider usage of classes like
  org.eclipse.jdt.internal.compiler.util.Util which are used for some
  file access, too. I see two alternatives here:
  - also wrap calls to Util methods in overridable methods, or
  - change relevant Util methods to non-static and use a Util instance
    for dynamic dispatch (more widespread impact on the code, but one
    less indirection).

Marking as helpwanted to encourage a contribution.

If s.o. wants to work on this, please announce here and keep us informed about your design and progress, thanks!
Comment 16 Denys Digtiar CLA 2013-05-20 19:17:09 EDT
I am trying to fix this bug. Here is my first small step:

https://git.eclipse.org/r/#/c/12993/
Comment 17 Stephan Herrmann CLA 2013-08-29 08:18:56 EDT
OMG, I complete forgot about this review request. Sorry!

Denys, are you still available for follow-up work after a review?
Comment 18 Denys Digtiar CLA 2013-08-29 09:19:36 EDT
(In reply to comment #17)
> OMG, I complete forgot about this review request. Sorry!
> 
> Denys, are you still available for follow-up work after a review?

Since I started working on this bug, I found a fulltime job and enrolled on a bunch of Coursera courses. I would love to contribute, but at the moment I do not have time to do this.
Comment 19 Stephan Herrmann CLA 2013-08-29 10:43:57 EDT
(In reply to comment #18)
> (In reply to comment #17)
> > OMG, I complete forgot about this review request. Sorry!
> > 
> > Denys, are you still available for follow-up work after a review?
> 
> Since I started working on this bug, I found a fulltime job and enrolled on
> a bunch of Coursera courses. I would love to contribute, but at the moment I
> do not have time to do this.

I understand. My apologies for not reacting sooner.

Please ping me here in the bug if your situation changes allowing you to spend some time on continuing this work. Thanks!
Comment 20 Stephan Herrmann CLA 2014-08-31 14:13:07 EDT
Denys, are you still considering to contribute?
Comment 21 Denys Digtiar CLA 2014-08-31 18:39:56 EDT
(In reply to Stephan Herrmann from comment #20)
> Denys, are you still considering to contribute?

I am afraid not. At least, not by fixing this defect. I felt that I bit more than I can chew. Code complexity and API compatibility constraints seem to much for me to handle in those few hours of spare time I have. 

P.S. If you have any minor defect on mind, I still may consider giving it a crack.
Comment 22 Olivier Thomann CLA 2014-09-02 11:39:01 EDT
I'll try to find some time on my spare time (when available :-)) to fix this one.
Comment 23 Kenneth Olson CLA 2014-10-07 09:46:45 EDT
An approach that I used for my application was to modify EclipseCompilerImpl and create a new subclass of ClasspathLocation (ClasspathJsr199).

In EclipseCompilerImpl.setPaths() it also checks for an instance of JavaFileManager (after not finding an EclipseFileManager or StandardFileManager) and then creates an instance of ClasspathJsr199 for PLATFORM_CLASS_PATH, SOURCE_PATH and CLASS_PATH. The handleBootclasspath() and handleClasspath() methods will only be called if ClasspathJsr199 instances were not created.

The new ClasspathJsr199 class implements the various ClasspathLocation methods using the appropriate JavaFileManager methods.

The challenge is to get reasonable performance from the isPackage() method which requires using using JavaFileManager.list() with a recursive list operation. This proved unsatisfactory for my needs and so I introduced "knowledge" of my specific JavaFileManager implementation where I created an optimized isPackage() method to meet my performance requirements.
Comment 24 Olivier Thomann CLA 2014-10-07 09:53:39 EDT
(In reply to Kenneth Olson from comment #23)
> An approach that I used for my application was to modify EclipseCompilerImpl
> and create a new subclass of ClasspathLocation (ClasspathJsr199).
Hi Kenneth,

Would it be possible for you to provide what you have done? I was about to work on this one in my spare time to get it done. So I could start from where you are.

Thanks.
Comment 25 Kenneth Olson CLA 2014-10-07 10:21:18 EDT
Created attachment 247682 [details]
Sample implementation

The code I used is attached with the non-JSR-199 optimizations removed. Hope this gives you a reasonable starting point. Code was based on the 4.4.1 version of ECJ.
Comment 26 Kenneth Olson CLA 2014-10-07 10:38:18 EDT
(In reply to Olivier Thomann from comment #24)
> (In reply to Kenneth Olson from comment #23)
> > An approach that I used for my application was to modify EclipseCompilerImpl
> > and create a new subclass of ClasspathLocation (ClasspathJsr199).
> Hi Kenneth,
> 
> Would it be possible for you to provide what you have done? I was about to
> work on this one in my spare time to get it done. So I could start from
> where you are.
> 
> Thanks.

One other issue you might encounter (or want to fix) is that the EclipseFileManager implementation of getDefaultBootclasspath() is inadequate for IBM JVMs. It is missing the logic from Util.collectRunningVMBootclasspath() that uses the running JVM's system properties to find all the relevant jars and misses the vm.jar under the bin/compressedrefs directory which results in problems looking up java.lang.String (among others).
Comment 27 Dani Megert CLA 2014-10-08 04:36:42 EDT
(In reply to Kenneth Olson from comment #25)
> Created attachment 247682 [details]
> Sample implementation

Kenneth, can you please sign the CLA and confirm the following in this bug report:
"This contribution complies with http://www.eclipse.org/legal/CoO.php"

Thanks.
Comment 28 Kenneth Olson CLA 2014-10-08 08:04:52 EDT
(In reply to Dani Megert from comment #27)
> (In reply to Kenneth Olson from comment #25)
> > Created attachment 247682 [details]
> > Sample implementation
> 
> Kenneth, can you please sign the CLA and confirm the following in this bug
> report:
> "This contribution complies with http://www.eclipse.org/legal/CoO.php"
> 
> Thanks.

I confirm that the sample implementation contribution complies with http://www.eclipse.org/legal/CoO.php.
Comment 29 Jay Arthanareeswaran CLA 2014-10-10 05:21:19 EDT
Stephan, let me know if you are not finding time for the review. I won't mind taking this one up.
Comment 30 Kenneth Olson CLA 2015-01-09 14:25:24 EST
(In reply to Jay Arthanareeswaran from comment #29)
> Stephan, let me know if you are not finding time for the review. I won't
> mind taking this one up.

I have not received any feedback on the sample implementation I attached to this bug item. Is there some action I should be taking to move this forward?
Comment 31 Olivier Thomann CLA 2015-01-09 14:30:09 EST
I started to take a look at this, but I didn't get enough time to go very far. I plan to come back to it very soon.
Comment 32 Dennis Hendriks CLA 2015-03-11 08:28:19 EDT
Recently I've been experimenting with the 'sample implementation' provided by Kenneth in comment 25, to use it for runtime in-memory compilation and execution of runtime generated Java code. Previously, I used the compiler provided by the JDK, but I hoped switching to the Eclipse JDT Java compiler would make this faster. Indeed, the compile time was reduced from 31 seconds to 7 seconds! To me, this is very significant for compilation during runtime of the application, as the user of the application is waiting for the compilation to finish.

However, I had a few issues with the 'Sample implementation' of comment 25:

1) The 'Sample implementation' can only handle file paths/URIs (URIs with a 'file' scheme). I made a patch to allow other URIs (URI schemes), as I use my own custom 'memory://...' URIs.

2) The 'Sample implementation' has a 'ClasspathJsr199' class, which has an unused field, that I think is left over from the custom optimization Kenneth mentioned he used, but that he removed when he contributed the code. I made a patch to remove the field.

3) With the JDK compiler, I could compile some sources, and then do a second compile call to compile some more sources. With the Eclipse JDT compiler, this was not possible, and resulted in NPE exceptions. I made two patches, one for Archive.java and one for EclipseFileManager.java, to allow reuse.

I hope that me testing the 'Sample implementation' can speed up inclusion of the 'Sample implementation' in the Eclipse sources. Currently, I have to ship a custom version of the entire 'org.eclipse.jdt.compiler.tool' plug-in fragment, use a different fragment name, different package names (due to signed vs unsigned code), etc, which means I have to update this for every new version of Eclipse. It would be really nice if this bugzilla could be resolved for Eclipse Mars.

I also hope my additional patches can be included. If you have any question regarding them, please let me know. 

I'm willing to help to speed up inclusion of the 'Sample implementation', my patches, and any other changes that are deemed necessary, into the Eclipse code base. Just let me know how I can help.
Comment 33 Dennis Hendriks CLA 2015-03-11 08:29:58 EDT
Created attachment 251465 [details]
Make EclipseCompiler support non-'file' URI schemes

See comment 32 for details.
Comment 34 Dennis Hendriks CLA 2015-03-11 08:30:44 EDT
Created attachment 251466 [details]
Removed unused field from ClasspathJsr199

See comment 32 for details.
Comment 35 Dennis Hendriks CLA 2015-03-11 08:31:31 EDT
Created attachment 251467 [details]
Allow (re)use of Archive after 'flush' call

See comment 32 for details.
Comment 36 Dennis Hendriks CLA 2015-03-11 08:32:11 EDT
Created attachment 251468 [details]
Allow (re)use of EclipseFileManager after 'close' call

See comment 32 for details.
Comment 37 Olivier Thomann CLA 2015-03-13 21:02:44 EDT
Created attachment 251547 [details]
Proposed patch

Here is a patch that reuses all given patches. It passes all tools and apt tests. I'd like somebody else to do a full review.
Jay, are you up to it?
Thanks.
Comment 38 Olivier Thomann CLA 2015-03-13 21:05:32 EDT
Stephan, could you please take a look since you are the QA contact for this bug report?
There might be some performance issues for the isPackage(..) calls, but at least it goes now properly through the java file object for read and write. I also fixed an issue with the eclipse compiler implementation that didn't properly initialized the bootclasspath inside the tools package.
Comment 39 Stephan Herrmann CLA 2015-03-14 11:29:05 EDT
My sincere apologies for lack of communication in this bug in the past!
It looks like I completely and repeatedly blew this one, outch.

I'll coordinate with Jay to ensure the latest patch gets proper review.
Comment 40 Jay Arthanareeswaran CLA 2015-03-16 01:45:15 EDT
(In reply to Stephan Herrmann from comment #39)
> I'll coordinate with Jay to ensure the latest patch gets proper review.

I will review this today.
Comment 41 Dennis Hendriks CLA 2015-03-16 04:47:33 EDT
(In reply to Olivier Thomann from comment #37)
> Created attachment 251547 [details]
> Proposed patch
> 
> Here is a patch that reuses all given patches. It passes all tools and apt
> tests. I'd like somebody else to do a full review.
> Jay, are you up to it?
> Thanks.

Just looked into your path. It seems to contain a lot more than just the previous patches... I have a few small questions:

org/eclipse/jdt/internal/compiler/apt/util/EclipseFileManager.java: line 202 and 203 both contain '}' with the same indentation. Do you agree that this is an indentation issue?

org/eclipse/jdt/internal/compiler/batch/ClasspathJsr199.java: at several places exception are caught, and printed using 'printStackTrace()'. To me, this doesn't seem like a proper way to report/handle exceptions. Do you agree?
Comment 42 Eclipse Genie CLA 2015-03-16 07:35:06 EDT
New Gerrit change created: https://git.eclipse.org/r/43921

WARNING: this patchset contains 1053 new lines of code and may require a Contribution Questionnaire (CQ) if the author is not a committer on the project. Please see:https://wiki.eclipse.org/Project_Management_Infrastructure/Creating_A_Contribution_Questionnaire
Comment 43 Jay Arthanareeswaran CLA 2015-03-16 08:55:07 EDT
(In reply to Dennis Hendriks from comment #41)
> Just looked into your path. It seems to contain a lot more than just the
> previous patches... I have a few small questions:

Thanks for taking the time to go through the patch.
 
> org/eclipse/jdt/internal/compiler/apt/util/EclipseFileManager.java: line 202
> and 203 both contain '}' with the same indentation. Do you agree that this
> is an indentation issue?

I don't see the issue. Indentation seems alright to me. Perhaps you are looking at a different place?

> org/eclipse/jdt/internal/compiler/batch/ClasspathJsr199.java: at several
> places exception are caught, and printed using 'printStackTrace()'. To me,
> this doesn't seem like a proper way to report/handle exceptions. Do you
> agree?

Agree. This is not a best practice, but we do this in other places too, refer too ClasspathJar. Anyway, in this case, Olivier has added the part to print the exception too apart from returning null. I think that's okay considering that a log service hasn't been made available in that part of the code.

I have made couple of minor changes (a copyright update and a generification and pushed this in Gerrit. Let's take it from there.
Comment 44 Dennis Hendriks CLA 2015-03-16 09:17:57 EDT
(In reply to Jay Arthanareeswaran from comment #43)
> (In reply to Dennis Hendriks from comment #41)
> > org/eclipse/jdt/internal/compiler/apt/util/EclipseFileManager.java: line 202
> > and 203 both contain '}' with the same indentation. Do you agree that this
> > is an indentation issue?
> 
> I don't see the issue. Indentation seems alright to me. Perhaps you are
> looking at a different place?

I added the same comments to Gerrit. It's my first time using Gerrit, so I hope I did it correctly.
Comment 45 Olivier Thomann CLA 2015-03-16 10:00:14 EDT
(In reply to Dennis Hendriks from comment #41)
> Just looked into your path. It seems to contain a lot more than just the
> previous patches... I have a few small questions:
Yes, it contains more code, but it includes all previous small patches.I fixed some issues of not being able to use a eclipse file manager with the system compiler or the eclipse compiler with the standard file manager from the system compiler. I think now this works as expected. Correct me if I am wrong.

> org/eclipse/jdt/internal/compiler/apt/util/EclipseFileManager.java: line 202
> and 203 both contain '}' with the same indentation. Do you agree that this
> is an indentation issue?
I think this is related to the fact that this lines combines tabulation and spaces. It looks good in the editor.

> org/eclipse/jdt/internal/compiler/batch/ClasspathJsr199.java: at several
> places exception are caught, and printed using 'printStackTrace()'. To me,
> this doesn't seem like a proper way to report/handle exceptions. Do you
> agree?
This was mostly to find out what could be wrong since I didn't see a proper logging mechanism in place in this code.
Comment 46 Dennis Hendriks CLA 2015-03-16 10:54:31 EDT
(In reply to Olivier Thomann from comment #45)
> (In reply to Dennis Hendriks from comment #41)
> > Just looked into your path. It seems to contain a lot more than just the
> > previous patches... I have a few small questions:
> Yes, it contains more code, but it includes all previous small patches.I
> fixed some issues of not being able to use a eclipse file manager with the
> system compiler or the eclipse compiler with the standard file manager from
> the system compiler. I think now this works as expected. Correct me if I am
> wrong.

OK, thanks for the clarification.

> > org/eclipse/jdt/internal/compiler/apt/util/EclipseFileManager.java: line 202
> > and 203 both contain '}' with the same indentation. Do you agree that this
> > is an indentation issue?
> I think this is related to the fact that this lines combines tabulation and
> spaces. It looks good in the editor.

I guess the solution would be to only use tabs then?

> > org/eclipse/jdt/internal/compiler/batch/ClasspathJsr199.java: at several
> > places exception are caught, and printed using 'printStackTrace()'. To me,
> > this doesn't seem like a proper way to report/handle exceptions. Do you
> > agree?
> This was mostly to find out what could be wrong since I didn't see a proper
> logging mechanism in place in this code.

If such an exception occurs, you currently get output on the console, and there is no way to handle the exception or to prevent printing of the text. I think it should instead be reported, for instance by not catching the exception and letting the user code invoking the compiler handle it. Alternatively, it could be ignored, but only if it won't hide bugs/errors. I would personally let the compiler crash if an exception occurs that it can't properly handle itself.
Comment 47 Olivier Thomann CLA 2015-03-16 11:51:12 EDT
(In reply to Dennis Hendriks from comment #46)
> I guess the solution would be to only use tabs then?
yes

> If such an exception occurs, you currently get output on the console, and
> there is no way to handle the exception or to prevent printing of the text.
> I think it should instead be reported, for instance by not catching the
> exception and letting the user code invoking the compiler handle it.
> Alternatively, it could be ignored, but only if it won't hide bugs/errors. I
> would personally let the compiler crash if an exception occurs that it can't
> properly handle itself.
In this case, I think they should be ignored. I'll attach a new patch with all issues fixed.
Comment 48 Olivier Thomann CLA 2015-03-16 11:52:33 EDT
Created attachment 251601 [details]
Proposed patch

Jay, could you please handle gerrit with this patch for me? I am not used to gerrit at all. Thx.
Comment 49 Jay Arthanareeswaran CLA 2015-03-16 12:12:45 EDT
I have reviewed the patch and have no issues. Just couple of questions/points:

1. I don't understand the change and the motivation behind for AllTests.

2. Not a question, just a point to note that we have two versions of ArchiveFileObject, both being almost identical and they even implement the same API. I don't know the history, likely done for some good reason. But just curious. Same for Archive as well.
Comment 50 Eclipse Genie CLA 2015-03-16 12:20:24 EDT
New Gerrit change created: https://git.eclipse.org/r/43949

WARNING: this patchset contains 1066 new lines of code and may require a Contribution Questionnaire (CQ) if the author is not a committer on the project. Please see:https://wiki.eclipse.org/Project_Management_Infrastructure/Creating_A_Contribution_Questionnaire
Comment 51 Jay Arthanareeswaran CLA 2015-03-16 12:31:14 EDT
No time today for Hudson job. Assuming it's all non-functional changes and Hudson will still stand by its approval, I have directly pushed the change:

http://git.eclipse.org/c/jdt/eclipse.jdt.core.git/commit/?id=30e98d6cc084dfe1b207ffb7493b2de7b6cec8c4
Comment 52 Olivier Thomann CLA 2015-03-16 14:29:42 EDT
(In reply to Jay Arthanareeswaran from comment #49)
> I have reviewed the patch and have no issues. Just couple of
> questions/points:
> 
> 1. I don't understand the change and the motivation behind for AllTests.
Test tools tests were ordered dependant and one test was disabled. I think I found the root cause of the problem and there is no reason to keep the test disabled anymore.
 
> 2. Not a question, just a point to note that we have two versions of
> ArchiveFileObject, both being almost identical and they even implement the
> same API. I don't know the history, likely done for some good reason. But
> just curious. Same for Archive as well.
Yes, I know. Not ideal. The code is exactly the same. The reason behind it was time constraint at the time the code was introduced.... meaning years ago without time to revisit that afterwards.
When we introduced the two fragments (apt and tool), I could not find the best layout for them. Ideally what I wanted to do was:
jdt-core (main bundble)
  fragment tools-common
    fragment tools-apt (annotation processor)
    fragment tools-tools (jsr199)

We wanted both apt and tools to be able to exist independently of each other, but I could not find a way to have a fragment depending on a fragment. Maybe the best way would have been to add a new bundle with a dependency on jdt.core and then two fragments of that new bundle. I had no time to experiment this solution at the time we had to do it.
I am open to any better solution.

Let me know if you have more questions.
Comment 53 Dennis Hendriks CLA 2015-03-17 05:49:39 EDT
(In reply to Olivier Thomann from comment #52)
> (In reply to Jay Arthanareeswaran from comment #49)
> > 2. Not a question, just a point to note that we have two versions of
> > ArchiveFileObject, both being almost identical and they even implement the
> > same API. I don't know the history, likely done for some good reason. But
> > just curious. Same for Archive as well.
> Yes, I know. Not ideal. The code is exactly the same. The reason behind it
> was time constraint at the time the code was introduced.... meaning years
> ago without time to revisit that afterwards.
> When we introduced the two fragments (apt and tool), I could not find the
> best layout for them. Ideally what I wanted to do was:
> jdt-core (main bundble)
>   fragment tools-common
>     fragment tools-apt (annotation processor)
>     fragment tools-tools (jsr199)
> 
> We wanted both apt and tools to be able to exist independently of each
> other, but I could not find a way to have a fragment depending on a
> fragment. Maybe the best way would have been to add a new bundle with a
> dependency on jdt.core and then two fragments of that new bundle. I had no
> time to experiment this solution at the time we had to do it.
> I am open to any better solution.

I like the 'tools-common' idea. Code duplication tends to cause a lot of issues, especially in the long run, as it is nearly impossible to keep them identical/consistent. I was wondering though, why they are all fragments? Why can't 'tools-common', 'tools-apt', and 'tools-tools' (weird name) all be plug-ins, instead of fragments?
Comment 54 Olivier Thomann CLA 2015-03-17 09:00:30 EDT
(In reply to Dennis Hendriks from comment #53)
> (In reply to Olivier Thomann from comment #52)
> > (In reply to Jay Arthanareeswaran from comment #49)
> > > 2. Not a question, just a point to note that we have two versions of
> > > ArchiveFileObject, both being almost identical and they even implement the
> > > same API. I don't know the history, likely done for some good reason. But
> > > just curious. Same for Archive as well.
> > Yes, I know. Not ideal. The code is exactly the same. The reason behind it
> > was time constraint at the time the code was introduced.... meaning years
> > ago without time to revisit that afterwards.
> > When we introduced the two fragments (apt and tool), I could not find the
> > best layout for them. Ideally what I wanted to do was:
> > jdt-core (main bundble)
> >   fragment tools-common
> >     fragment tools-apt (annotation processor)
> >     fragment tools-tools (jsr199)
> > 
> > We wanted both apt and tools to be able to exist independently of each
> > other, but I could not find a way to have a fragment depending on a
> > fragment. Maybe the best way would have been to add a new bundle with a
> > dependency on jdt.core and then two fragments of that new bundle. I had no
> > time to experiment this solution at the time we had to do it.
> > I am open to any better solution.
> 
> I like the 'tools-common' idea. Code duplication tends to cause a lot of
> issues, especially in the long run, as it is nearly impossible to keep them
> identical/consistent. I was wondering though, why they are all fragments?
> Why can't 'tools-common', 'tools-apt', and 'tools-tools' (weird name) all be
> plug-ins, instead of fragments?
'tools-common', 'tools-apt', and 'tools-tools': these names are not the actual
names of the fragments. The actual names are:
org.eclipse.jdt.compiler.apt
org.eclipse.jdt.compiler.tool

Ideally we would need org.eclipse.jdt.compiler.common.

They are fragments, because this was seen as a contribution to an actual plugin (org.eclipse.jdt.core). If someone has time to reorganize these fragments in a way that works at runtime, that would be awesome. I was not sure how to make a fragment depending on a fragment. Maybe we don't need fragment and we should use plug-ins with x-friends clause.
Jay, Stephan, any thoughts? This should of course be moved to a new bug report as it is not related anymore with this one.
Comment 55 Jay Arthanareeswaran CLA 2015-03-17 09:39:38 EDT
Alright, I may have made a mistake in that I put Olivier's name as author in the commit, even though he mentioned his patch was derived from both Kenneth's and Dennis'.

I think we still have the time in M6 to correct this. Olivier, your thoughts?
Comment 56 Dani Megert CLA 2015-03-17 09:42:48 EDT
(In reply to Jay Arthanareeswaran from comment #55)
> Alright, I may have made a mistake in that I put Olivier's name as author in
> the commit, even though he mentioned his patch was derived from both
> Kenneth's and Dennis'.
> 
> I think we still have the time in M6 to correct this. Olivier, your thoughts?

We need numbers on who contributed how much code. If the CQ limit is exceeded (by a contributor) we have to revert the change, file a CQ, get it approved, and then commit the changes via Gerrit where we can list all authors.
Comment 57 Dennis Hendriks CLA 2015-03-17 09:51:52 EDT
(In reply to Dani Megert from comment #56)
> We need numbers on who contributed how much code. If the CQ limit is
> exceeded (by a contributor) we have to revert the change, file a CQ, get it
> approved, and then commit the changes via Gerrit where we can list all
> authors.

Here are the numbers for the 4 patches I contributed:
 - 1+0+0+1 = 2 changed lines
 - 1+3+0+5 = 9 added lines
 - 0+0+1+0 = 1 removed line
Comment 58 Olivier Thomann CLA 2015-03-17 11:12:57 EDT
Stephan, what should be done for the new method org.eclipse.jdt.internal.compiler.batch.ClasspathJsr199.hasAnnotationFileFor(String)? I was not sure what to do so I provided a default implementation that returns null. Please take a look and feel free to modify it.
Comment 59 Kenneth Olson CLA 2015-03-17 11:28:12 EDT
(In reply to Dani Megert from comment #56)
> We need numbers on who contributed how much code. If the CQ limit is
> exceeded (by a contributor) we have to revert the change, file a CQ, get it
> approved, and then commit the changes via Gerrit where we can list all
> authors.
Based on my original sample implementation, the line counts are:
In EclipseCompilerImpl:
     new lines = 28
     changed lines = 10
     removed lines = 8
For the new file ClasspathJSR199:
     total lines = 190

By the time the code was incorporated into the final patch these may have changed from my original submission.
Comment 60 Jay Arthanareeswaran CLA 2015-03-17 11:52:18 EDT
Since there is contribution from several authors and mostly below 200 lines of code, Olivier and I don't think there's a need for a CQ. I have updated the copyright with the correct contributors names, though.

Sorry about the lapse.
Comment 61 Dani Megert CLA 2015-03-17 12:05:13 EDT
(In reply to Jay Arthanareeswaran from comment #60)
> Since there is contribution from several authors and mostly below 200 lines
> of code, Olivier and I don't think there's a need for a CQ.

+1. However, we need to make sure the names appear in the IP log (copyright notice is not enough for that). You have two choices: mark the relevant patches with iplog+ or revert the commit and submit the change via Gerrit where you can list all authors.
Comment 62 Stephan Herrmann CLA 2015-03-17 12:13:47 EDT
(In reply to Olivier Thomann from comment #58)
> Stephan, what should be done for the new method
> org.eclipse.jdt.internal.compiler.batch.ClasspathJsr199.
> hasAnnotationFileFor(String)? I was not sure what to do so I provided a
> default implementation that returns null. Please take a look and feel free
> to modify it.

Thanks for the heads-up! I've filed bug 462387 to follow up (after Mars).
Comment 63 Jay Arthanareeswaran CLA 2015-03-18 23:42:10 EDT
Comment on attachment 247682 [details]
Sample implementation

This is not a patch but modified/new files zipped. Not sure if this is okay to have iplog+. Dani, let me know if this has to be uploaded as a patch.
Comment 64 Jay Arthanareeswaran CLA 2015-03-19 05:00:26 EDT
I have set the iplog+ for all the patches from both Dennis and Kenneth.
Comment 65 Sasikanth Bharadwaj CLA 2015-05-19 04:13:28 EDT
Verified for 4.5 RC1 using 20150514-1000 build