Community
Participate
Working Groups
Also anonymous classes are not supported by the debug infrastructure.
I can't see how this will ever work without either hefty agent or dynamic class loading into the debugee, or through explicit JVM support? Are you suggesting any such tricks, or is this bug mainly raised to document the shortcomings of the current situation?
I guess the only possible solution is: 1. If you see the lambda extract method body and add this method to target class (https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-6.html#jvms-6.5.invokedynamic) 2. Then generate invokedynamic targeting to generated method. In current evaluation infrastructure I found it tricky. More over checking my target jvm on canAddMethod flag it returns false. (https://docs.oracle.com/javase/7/docs/platform/jpda/jdwp/jdwp-protocol.html#JDWP_VirtualMachine_RedefineClasses)
(In reply to Andrei Sviridenko from comment #2) > I guess the only possible solution is: > > 1. If you see the lambda extract method body and add this method to target > class > (https://docs.oracle.com/javase/specs/jvms/se7/html/jvms-6.html#jvms-6.5. > invokedynamic) > > In current evaluation infrastructure I found it tricky. > More over checking my target jvm on canAddMethod flag it returns false. > > (https://docs.oracle.com/javase/7/docs/platform/jpda/jdwp/jdwp-protocol. > html#JDWP_VirtualMachine_RedefineClasses) Exactly. Tricky is an understatement, but we'd basically have to compile and redefine the class, because whenever there's lambdas and invokeDynamic, there's new stuff in the constant pool as well. There is a JEP open for this kind of functionality (http://openjdk.java.net/jeps/159), by bringing in the DCEVM project: https://github.com/dcevm/dcevm > 2. Then generate invokedynamic targeting to generated method. That's a problem, too, since there's no JDWP command for invokeDynamic, as far as I can tell. So to support that, we'd need another new (temporary) method as well, just to make the actual invokeDynamic call to generate the lambda or method reference.
(In reply to Jesper Moller from comment #3) > That's a problem, too, since there's no JDWP command for invokeDynamic, as > far as I can tell. So to support that, we'd need another new (temporary) > method as well, just to make the actual invokeDynamic call to generate the > lambda or method reference. I just thought about this alternative approach: We could just use the LambdaMetafactory API through JDWP instead. See http://stackoverflow.com/questions/23595136/get-a-list-of-classes-lambdas/23606688#23606688 We should be able to make method references and constructor references quite easily. Simple lambdas containing no loops, conditionals or try/catches should be doable, too, but it is very non-trivial, essentially combining the expression from MethodHandles. It will be quite slow because of all the JWDP round trips to the MethodHandles.* methods, but it should be doable, and some of the common building blocks could be cached (such as references to often-used MethodType instances, etc.) For Java 9, they're introducing MethodHandles for loops and try/catch: http://openjdk.java.net/jeps/274 This approach would not give us anonymous classes, we'd still need classfile generation and dynamic loading for that, with the limitations that would bring us, e.g. lack of accessor methods for the containing class' private fields. In fact, the same limitations may exist for the lambda expressions as well, depending on the security manager situation of the debugee. Who would like to give it a shot?
I(In reply to Jesper Moller from comment #4) > (In reply to Jesper Moller from comment #3) > > That's a problem, too, since there's no JDWP command for invokeDynamic, as > > far as I can tell. So to support that, we'd need another new (temporary) > > method as well, just to make the actual invokeDynamic call to generate the > > lambda or method reference. > > I just thought about this alternative approach: We could just use the > LambdaMetafactory API through JDWP instead. See > http://stackoverflow.com/questions/23595136/get-a-list-of-classes-lambdas/ > 23606688#23606688 > > We should be able to make method references and constructor references quite > easily. > > Simple lambdas containing no loops, conditionals or try/catches should be > doable, too, but it is very non-trivial, essentially combining the > expression from MethodHandles. > > It will be quite slow because of all the JWDP round trips to the > MethodHandles.* methods, but it should be doable, and some of the common > building blocks could be cached (such as references to often-used MethodType > instances, etc.) > > For Java 9, they're introducing MethodHandles for loops and try/catch: > http://openjdk.java.net/jeps/274 > > This approach would not give us anonymous classes, we'd still need classfile > generation and dynamic loading for that, with the limitations that would > bring us, e.g. lack of accessor methods for the containing class' private > fields. In fact, the same limitations may exist for the lambda expressions > as well, depending on the security manager situation of the debugee. > > Who would like to give it a shot? Seams reasonable for method references, but how about injection lambda body to debugee to create a method handle? Did you think about Unsafe approach?(not good one indeed) I made sample based on EvalTest18 test class and looks like it may work converting lambda to anonymous class, compiling to bytecode and creating it via unsafe: public static void main(String[] args) throws Exception { List<String> strings = Arrays.asList("One", "Two", "Three"); strings.parallelStream().filter(new Predicate<String>() { @Override public boolean test(String s) { return s.equals("One"); } }).findAny(); byte[] lbda = { -54, -2, -70, -66, 0, 0, 0, 52, 0, 41, 7, 0, 2, 1, 0, 45, 111, 114, 103, 47, 101, 99, 108, 105, 112, 115, 101, 47, 106, 100, 116, 47, 100, 101, 98, 117, 103, 47, 116, 101, 115, 116, 115, 47, 101, 118, 97, 108, 47, 106, 56, 116, 101, 115, 116, 47, 100, 115, 103, 36, 49, 7, 0, 4, 1, 0, 16, 106, 97, 118, 97, 47, 108, 97, 110, 103, 47, 79, 98, 106, 101, 99, 116, 7, 0, 6, 1, 0, 28, 106, 97, 118, 97, 47, 117, 116, 105, 108, 47, 102, 117, 110, 99, 116, 105, 111, 110, 47, 80, 114, 101, 100, 105, 99, 97, 116, 101, 1, 0, 6, 60, 105, 110, 105, 116, 62, 1, 0, 3, 40, 41, 86, 1, 0, 4, 67, 111, 100, 101, 10, 0, 3, 0, 11, 12, 0, 7, 0, 8, 1, 0, 15, 76, 105, 110, 101, 78, 117, 109, 98, 101, 114, 84, 97, 98, 108, 101, 1, 0, 18, 76, 111, 99, 97, 108, 86, 97, 114, 105, 97, 98, 108, 101, 84, 97, 98, 108, 101, 1, 0, 4, 116, 104, 105, 115, 1, 0, 47, 76, 111, 114, 103, 47, 101, 99, 108, 105, 112, 115, 101, 47, 106, 100, 116, 47, 100, 101, 98, 117, 103, 47, 116, 101, 115, 116, 115, 47, 101, 118, 97, 108, 47, 106, 56, 116, 101, 115, 116, 47, 100, 115, 103, 36, 49, 59, 1, 0, 4, 116, 101, 115, 116, 1, 0, 21, 40, 76, 106, 97, 118, 97, 47, 108, 97, 110, 103, 47, 83, 116, 114, 105, 110, 103, 59, 41, 90, 8, 0, 19, 1, 0, 3, 79, 110, 101, 10, 0, 21, 0, 23, 7, 0, 22, 1, 0, 16, 106, 97, 118, 97, 47, 108, 97, 110, 103, 47, 83, 116, 114, 105, 110, 103, 12, 0, 24, 0, 25, 1, 0, 6, 101, 113, 117, 97, 108, 115, 1, 0, 21, 40, 76, 106, 97, 118, 97, 47, 108, 97, 110, 103, 47, 79, 98, 106, 101, 99, 116, 59, 41, 90, 1, 0, 1, 115, 1, 0, 18, 76, 106, 97, 118, 97, 47, 108, 97, 110, 103, 47, 83, 116, 114, 105, 110, 103, 59, 10, 0, 1, 0, 29, 12, 0, 16, 0, 17, 1, 0, 10, 83, 111, 117, 114, 99, 101, 70, 105, 108, 101, 1, 0, 8, 100, 115, 103, 46, 106, 97, 118, 97, 1, 0, 9, 83, 105, 103, 110, 97, 116, 117, 114, 101, 1, 0, 68, 76, 106, 97, 118, 97, 47, 108, 97, 110, 103, 47, 79, 98, 106, 101, 99, 116, 59, 76, 106, 97, 118, 97, 47, 117, 116, 105, 108, 47, 102, 117, 110, 99, 116, 105, 111, 110, 47, 80, 114, 101, 100, 105, 99, 97, 116, 101, 60, 76, 106, 97, 118, 97, 47, 108, 97, 110, 103, 47, 83, 116, 114, 105, 110, 103, 59, 62, 59, 1, 0, 15, 69, 110, 99, 108, 111, 115, 105, 110, 103, 77, 101, 116, 104, 111, 100, 7, 0, 36, 1, 0, 43, 111, 114, 103, 47, 101, 99, 108, 105, 112, 115, 101, 47, 106, 100, 116, 47, 100, 101, 98, 117, 103, 47, 116, 101, 115, 116, 115, 47, 101, 118, 97, 108, 47, 106, 56, 116, 101, 115, 116, 47, 100, 115, 103, 12, 0, 38, 0, 39, 1, 0, 4, 109, 97, 105, 110, 1, 0, 22, 40, 91, 76, 106, 97, 118, 97, 47, 108, 97, 110, 103, 47, 83, 116, 114, 105, 110, 103, 59, 41, 86, 1, 0, 12, 73, 110, 110, 101, 114, 67, 108, 97, 115, 115, 101, 115, 0, 32, 0, 1, 0, 3, 0, 1, 0, 5, 0, 0, 0, 3, 0, 0, 0, 7, 0, 8, 0, 1, 0, 9, 0, 0, 0, 51, 0, 1, 0, 1, 0, 0, 0, 5, 42, -73, 0, 10, -79, 0, 0, 0, 2, 0, 12, 0, 0, 0, 10, 0, 2, 0, 0, 0, 14, 0, 4, 0, 1, 0, 13, 0, 0, 0, 12, 0, 1, 0, 0, 0, 5, 0, 14, 0, 15, 0, 0, 0, 1, 0, 16, 0, 17, 0, 1, 0, 9, 0, 0, 0, 59, 0, 2, 0, 2, 0, 0, 0, 7, 43, 18, 18, -74, 0, 20, -84, 0, 0, 0, 2, 0, 12, 0, 0, 0, 6, 0, 1, 0, 0, 0, 18, 0, 13, 0, 0, 0, 22, 0, 2, 0, 0, 0, 7, 0, 14, 0, 15, 0, 0, 0, 0, 0, 7, 0, 26, 0, 27, 0, 1, 16, 65, 0, 16, 0, 25, 0, 1, 0, 9, 0, 0, 0, 41, 0, 2, 0, 2, 0, 0, 0, 9, 42, 43, -64, 0, 21, -74, 0, 28, -84, 0, 0, 0, 2, 0, 12, 0, 0, 0, 6, 0, 1, 0, 0, 0, 1, 0, 13, 0, 0, 0, 2, 0, 0, 0, 4, 0, 30, 0, 0, 0, 2, 0, 31, 0, 32, 0, 0, 0, 2, 0, 33, 0, 34, 0, 0, 0, 4, 0, 35, 0, 37, 0, 40, 0, 0, 0, 10, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0 }; Field f = Unsafe.class.getDeclaredField("theUnsafe"); f.setAccessible(true); Unsafe unsafe = (Unsafe) f.get(null); Class<?> clazz = unsafe.defineAnonymousClass(dsg.class, lbda, null); Object newInstance = clazz.newInstance(); System.err.println(strings.parallelStream().filter((Predicate<String>) newInstance).findAny()); }
(In reply to Andrei Sviridenko from comment #5) > Seams reasonable for method references, but how about injection lambda body > to debugee to create a method handle? There are subtle differences in scope rules. It would still be far better to be able to inject the lambda method into the enclosing class. > Did you think about Unsafe approach?(not good one indeed) I agree it's probably a lot easier to reuse the compiler as-is rather than to reinvent the compiler through MethodHandles (and JDWP). But then again it'd be Unsafe, and I'm thinking @dr_Deprecator has his sights on anything Unsafe. Of course, we could supply fall-backs. Up for that challenge, then?
(In reply to Jesper Moller from comment #6) > (In reply to Andrei Sviridenko from comment #5) > > > Seams reasonable for method references, but how about injection lambda body > > to debugee to create a method handle? > > There are subtle differences in scope rules. It would still be far better to > be able to inject the lambda method into the enclosing class. > > > Did you think about Unsafe approach?(not good one indeed) > > I agree it's probably a lot easier to reuse the compiler as-is rather than > to reinvent the compiler through MethodHandles (and JDWP). But then again > it'd be Unsafe, and I'm thinking @dr_Deprecator has his sights on anything > Unsafe. Of course, we could supply fall-backs. > > Up for that challenge, then? Not sure if I able to complete with lambdas as compiler infrastructure looks complicated to noob in eclipse :)
*** Bug 481334 has been marked as a duplicate of this bug. ***
This bug blocks: https://github.com/Microsoft/vscode-java-debug/issues/281
(In reply to Brian Bonner from comment #9) > This bug blocks: https://github.com/Microsoft/vscode-java-debug/issues/281 Hint: we accept high quality patches.
(In reply to Andrey Loskutov from comment #10) > (In reply to Brian Bonner from comment #9) > > This bug blocks: https://github.com/Microsoft/vscode-java-debug/issues/281 > > Hint: we accept high quality patches. Also convincing Oracle to resolve http://openjdk.java.net/jeps/159 would help immensely ;p
I'm novice in this area, but why it is impossible for Eclipse but possible for IntelliJ ? I mean they supported this starting with Java8 it self i think, now Java 14 is going to be release and still Eclipse lacks this feature. This will be really handy for developers who are using Eclipse with Java8+.
Even netbeans has the support now https://netbeans.org/bugzilla/show_bug.cgi?id=253248 Why not Eclipse support it :( ?
(In reply to Gayan Perera from comment #13) > Even netbeans has the support now > https://netbeans.org/bugzilla/show_bug.cgi?id=253248 > > Why not Eclipse support it :( ? Due to lack of incentivised developers, unfortunately. :-( There are several hints and suggestions in the preceding thread as to how implement this. It looks as if NetBeans solves this by loading a new class containing a method holding the body of the lambda function. If I'm not mistaken, that would preclude access to private members of the enclosing scope -- I don't have NetBeans installed, so I can't check. IntelliJ also does this by introducing a separate class -- they somehow support accessing the private members of the enclosing scope, likely using reflection. I.e. they compile the lambda used in the breakpoint expression differently than the normal compiler does. In other words: Yes, it is not impossible, but it is a lot of work. If there are any volunteers, I'd gladly outline how I'd solve it.
I'm also often facing this issue as I'm using lambdas more and more frequently and automatically, and am quite frustrated when I start writing some lambda expression in the Debug Shell and get JDT declining the execution and I'm forced to rewrite it with loops.
Even vscode java plugins has a bug which depends on this problem https://github.com/microsoft/vscode-java-debug/issues/281 If someone can work on this issue that will bring great value to the IDE.
@Roland will you be interested to support on solving this issue since this might be a very needed feature with new java changes and lot of product adapting to java8 and above.
It seems the team has started J14 development, but no one is even looking at this bug :(
Hi All JDT developers, can we consider this for 4.14 ?
Gayan, if I recall the mailing list, I remember you volunteering to look into this, what happened to that? :)
(In reply to Pierre-Yves B. from comment #21) > Gayan, if I recall the mailing list, I remember you volunteering to look > into this, what happened to that? :) I'll volunteer to outline an overall approach to how to go about it, but I can't commit to implementing all of it.
I thought i could contribute but it seems to fix this one should have in deep knowledge in eclipse compiler and debugger protocol. So i think it might be much faster if a JDT developer can take this.
(In reply to Gayan Perera from comment #23) > I thought i could contribute but it seems to fix this one should have in > deep knowledge in eclipse compiler and debugger protocol. So i think it > might be much faster if a JDT developer can take this. If you hope that others implement this, then you won't have much influence on the schedule, as nobody seems to have sufficient time for this. It's your choice. :)
(In reply to Jesper Moller from comment #22) > (In reply to Pierre-Yves B. from comment #21) > > Gayan, if I recall the mailing list, I remember you volunteering to look > > into this, what happened to that? :) > > I'll volunteer to outline an overall approach to how to go about it, but I > can't commit to implementing all of it. Thanks Jesper!
A little progress report: TL;DR: This is just as hard as I expected. My initial attempt is to keep using the instruction-based evaluator, and only change the case where a lambda expression or anonymous class is used in the AST, and then replace it with an instantiation of an object of a class which is compiled and injected into the debuggee's JVM (through JDWP). It turns out that there is a functionality for something quite similar already, in the form of the LocalEvaluationEngine, which is presently used for the scrapbook functionality. Pros: - It takes care of setting up the compiler infra (in fact, org.eclipse.jdt.internal.core.eval.EvaluationContextWrapper#evaluateCodeSnippet does this) and handles the bound variables. Cons: - It does this by having filesystem access to the path of the classloader of the debugee. This isn't possible in the general case, so we'd have to inject the class using JDWP. - It doesn't work when injecting generic types, since it uses a mixture of source generation from JDI type-info and hand-rolled parser for type references (in CodeSnippetParser) - It relies on actual stack frame information to inject variables, which won't fly for a breakpoint, which we must be able to parse and compile without a running program. At the moment, the pros outweigh the cons. We might still be able to use org.eclipse.jdt.internal.core.eval.EvaluationContextWrapper (if the problems in CodeSnippetParser are fixable) -- otherwise, we have to implement a *third* way of compiling a snippet for execution in a running JVM. Unsolved yet: For stateless lambdas, this would be super easy. Imagine: class Debuggee { public void breakHere(List<Integer> someInts) { something(); // BP here and evaluate this: someInts.stream().anyMatch(i -> i < 100) } } this should translate into this class: class Lambda1 implement Predicate<Integer> { public boolean test(Integer i) { return i < 100; } } When the AST is first prepared (or first executed), the above source is generated, compiled and injected into the process. Then the resulting classId is found and instantiated, and pushed onto the interpreter stack (in this case, the lambda is stateless, so we'd only have to make a single one, but that's just an optimization for another day) As soon as the lambdas refer to variables, things get a little trickier: Local variables must be captured from the debugger stack frame and passed into the constructor, just like for normal lambda compilation. class Debuggee2 { public void breakHere(List<Integer> someInts, int someLimit) { something(); // BP here and evaluate this: someInts.stream().anyMatch(i -> i < someLimit) } } This should translate into this class: class Lambda2 implement Predicate<Integer> { public Lambda2(int someLimit) { this.bind$someLimit = someLimit; } public boolean test(Integer i) { return i < this.bind$someLimit; } } When we start referring to an enclosing instance, it gets really tricky: Yes, the 'this' reference can be captured from the debugger stack frame. However, if we refer to any non-public members, there's a problem: class Debuggee3 { private int x = 42; public void breakHere(List<Integer> someInts) { something(); // BP here and evaluate this: someInts.stream().anyMatch(i -> i < f) } } This must be somehow be translated into: class Lambda3 implement Predicate<Integer> { private Debuggee3 bind$this; public Lambda3(Debuggee context$this) { this.bind$this = context$this; } public boolean test(Integer i) { return i < this.bind$this.f; // KABOOM, Lambda3 can't use Debuggee3's private fields. } } A horrible workaround is to use reflection here, but that has a deep impact on the generated source. I can't decide whether it's best to parse and type-check the expression, and then splice that into a new class as AST nodes, or to re-generate source from the AST instead, using a specialized ASTVisitor. Any opinions?
Just a note: I'll try to keep the AST-juggling inside jdt.core and then call into that from jdt.debug.
(In reply to Jesper Moller from comment #26) > A little progress report: > > > > I can't decide whether it's best to parse and type-check the expression, and > then splice that into a new class as AST nodes, or to re-generate source > from the AST instead, using a specialized ASTVisitor. > > Any opinions? I guess ASTVisitor approach will me more cleaner.
A little progress: This very simple test now runs, demonstrating the "generate lambda source, compile, and inject into running debugee using JDWP using Unsafe" approach (done in remoteEval(), using a remote variant of the LocalEvaluationEngine): public void testEvalStatictMethod_remote() throws Exception { IJavaThread thread = null; try { String type = "EvalTest18"; createLineBreakpoint(26, type); thread = launchToBreakpoint(type); assertNotNull("The program did not suspend", thread); String snippet = "(java.util.function.IntSupplier)(() -> 42)"; IJavaObject lambda = (IJavaObject) doRemoteEval(thread, snippet); IValue result = lambda.sendMessage("getAsInt", "()I", null, thread, false); assertEquals("42", result.getValueString()); } finally { removeAllBreakpoints(); terminateAndRemove(thread); } } I have not yet solved the parser problems encountered above, merely simplified my test case a bit. I would still prefer to use a nimbler API than IEvaluationContext, so that I could fold different lambda expressions into one class and simply call one method instead of having to to multiple roundtrips across JDWP, but I'm not keep on increasing the API footprint for something that specialized. If combined with the ASTVisitor to ferret out and rewrite any escaping name bindings, this could be brought to work. I've got some DOM issues re. type information for method references, but that could be my fault.
Any progress in this issue ?
*** Bug 553061 has been marked as a duplicate of this bug. ***
(In reply to Gayan Perera from comment #30) > Any progress in this issue ? Yes, a bit. I’ve hooked the eval-stuff up with the normal eval logic, but the SnippetCompiler-whatever is still a problem, in that it doesn’t support generics. So I still have to suggest a fix for that. And I still haven’t addressed the issue of private member access (yet).
Moving to 4.15 as per Jesper's comment.
*** Bug 553747 has been marked as a duplicate of this bug. ***
Jesper, will it be feasible in 4.15?
(In reply to Sarika Sinha from comment #35) > Jesper, will it be feasible in 4.15? Yes - the basics are working. I think I can complete *some* support, but maybe not all variations of outer captured variables.
(In reply to Jesper Moller from comment #36) > (In reply to Sarika Sinha from comment #35) > > Jesper, will it be feasible in 4.15? > > Yes - the basics are working. > > I think I can complete *some* support, but maybe not all variations of outer > captured variables. That will be great!
(In reply to Sarika Sinha from comment #37) > (In reply to Jesper Moller from comment #36) > > Yes - the basics are working. > > > > I think I can complete *some* support, but maybe not all variations of outer > > captured variables. > > That will be great! +1, some support would be already awesome!
@Jasper can we expect yours changes in 4.15 ?
Created attachment 281711 [details] Patch for people to try Patch against eclipse.jdt.debug. This patch is the "Work In Progress" that I have now. The basic mechanism is this: When compiling a functional expression, a new SnippetClass is generated, that, when instantiated and called, generates a resultValue member with the value of the functional expression. When generating the snippet, free variables are 'captured' and added to the SnippetClass. The bytes of the generated classes are stored in a "FunctionalOperator" which is an operator like the other compiled AST instructions, and the first time it is run in the target, the class is injected using Unsafe, an instance is instantiated, then the free (bound) variables are initialized, the 'run' method is called, and the 'resultValue' field is fetched (containing the functional expression). This is then pushed onto the evaluator stack just like other operators do it. The free variable name handling is still very crude, but it can grab local variables as demonstrated in the test case. Also, I discovered an unprotected global mutable without protection: org.eclipse.jdt.internal.eval.EvaluationContext.CODE_SNIPPET_COUNTER I'll have to raise an issue for that separately. In the meanwhile, I put the unprotected evaluateCodeSnippet call in a synchronized-block :-( Could do with some more tests, and performance considerations, too: When a watch expression is executed (e.g. in the "Expressions"-view in the JDT Debugger), it appears to be re-compiled for every evaluation. For lambdas, that means a compilation run and a lass initialization in the debugee, FOR EACH BREAKPOINT. This is what I have now, and as for 4.15, I don't have a lot of time between now and the M3 deadline. So, somebody should try this out, try a few conditional breakpoints, a few evaluation expressions, and see how bad it is. Any takers?
I did some basic tests and its working awesome. May be its good to add this as a MVP solution to 4.15 and handle the edge cases later. We should inform user that the edge cases are not supported yet without giving errors. What do you all think ? I really love when things are getting fixed like this in Eclipse.
Jesper, please consider to create Gerrit patch.
Thank you for the encouragement, Gayan and Andrey. I'm working on more tests to ensure that we know how good the current version is. I'll make the patch tomorrow, and see if I can get anyone to review a concurrency fix for JDT Core.
(In reply to Jesper Moller from comment #43) > Thank you for the encouragement, Gayan and Andrey. I'm working on more tests > to ensure that we know how good the current version is. > > I'll make the patch tomorrow, and see if I can get anyone to review a > concurrency fix for JDT Core. Thanks Jesper! We can work on putting it in master for 4.15 M3 by 14th Feb.
Well, this is embarrassing. It seems I managed to mess up my Gerrit push, and ended up pushing to master instead (I didn't think I could do that, I'm not a committer on Debug, I think). Anyway, I reverted the commit right away. I managed to push to Gerrit as https://git.eclipse.org/r/#/c/157396/ but then I noticed that I've forgotten to fix the copyright notices for the non-new files. I hope this can be ignored, or somebody else is able to fix it. For best overview of what works, look at FunctionalCaptureTest18.java which is run in the test case org.eclipse.jdt.debug.tests.eval.Java8Tests.testContextEvaluations. Each line with 'assertFunctionalExpression' is evaluated in the debugger and asserts that the method can be called without exception, EXCEPT for the lines marked /*SKIP*/, which is quite a few of them. This is because the generated snippet class is loaded in isolation, and so can't bootstrap if they refer to the surrounding code. I have NO chance of improving this patch before M3, but will continue progress next Monday. I hope we can make this work.
(In reply to Jesper Moller from comment #45) > (I didn't think I could do that, I'm not a committer on Debug, I think). You are a committer for JDT, which nowadays accounts for Core, UI and Debug. As long as s.o. with experience in this particular area is involved, all is fine.
There are a number of improvements which need to be made to make the feature really nice: 1. The the remote class using the thread's classloader instead of whatever is used by Unsafe.defineClass. This could be done by making and loading a "ByteArrayClassloader" which then is instantiated and fed the new classes, using the Thread's classloader as a parent. Then, the classes can be loaded and used, and they should be able to see the project's classes. Perhaps it should use the context classloader as a parent. From Java 9 on, 2. Cache the loaded classes and support, so repeated compiles of the same expression (like in the "Expressions" view) need not recompile or reload. Perhaps by debugger target, perhaps even across those using a small LRU cache. 3. Figure out a way to access private members of the surrounding class from inside the lambda, using MethodHandles or reflection. 4. Make specific errors for the cases which aren't handlet yet. 5. Test nested class-like constructs, like lambdas in lambdas. 6. Improve syntax error handling, but maybe that's a more general thing in the various UI elements.
New Gerrit change created: https://git.eclipse.org/r/157396
(In reply to Eclipse Genie from comment #48) > New Gerrit change created: https://git.eclipse.org/r/157396 Unable to comment on gerrit due to jenkins issue, so posting my comment here. I am getting this error when trying to use Lambda expression in Conditional editor or debug Shell : Evaluation failed - unable to instantiate code snippet class.
(In reply to Sarika Sinha from comment #49) > (In reply to Eclipse Genie from comment #48) > > New Gerrit change created: https://git.eclipse.org/r/157396 > > Unable to comment on gerrit due to jenkins issue, so posting my comment here. > > I am getting this error when trying to use Lambda expression in Conditional > editor or debug Shell : > Evaluation failed - unable to instantiate code snippet class. Did you reference any non-system classes? Which Java version? Can you run the Java8 debug test class?
(In reply to Jesper Moller from comment #50) > > Did you reference any non-system classes? > Which Java version? > > Can you run the Java8 debug test class? Looks like the error is limited to Modular Java Projects. It is not throwing any error With Java 8 or Java 11 Non modular project.
(In reply to Sarika Sinha from comment #51) > (In reply to Jesper Moller from comment #50) > > > > Did you reference any non-system classes? > > Which Java version? > > > > Can you run the Java8 debug test class? > > Looks like the error is limited to Modular Java Projects. > It is not throwing any error With Java 8 or Java 11 Non modular project. Basically, this fails in org.eclipse.jdt.internal.debug.eval.RemoteEvaluator.loadTheClasses(IJavaThread): IJavaClassType unsafeClass = (IJavaClassType) findType("sun.misc.Unsafe", debugTarget);
Ok, that makes sense under module constraints. I haven’t done experiments yet re. useing the Thread’s classloader methods directly. Or perhaps goung through the lookup.
Ok, So I plan to release this change for M3 and close this bug. Create a new bug for Modular and other enhancements and add a note in ReadMe pointing to the new bug. Any objection?
(In reply to Sarika Sinha from comment #54) > Ok, So I plan to release this change for M3 and close this bug. > Create a new bug for Modular and other enhancements and add a note in ReadMe > pointing to the new bug. > > Any objection? No, but the patch will hopefully address possible "modular" problems in the sense it will not throw exceptions etc, it should just "not work" for Java 9+?
(In reply to Andrey Loskutov from comment #55) > (In reply to Sarika Sinha from comment #54) > > Ok, So I plan to release this change for M3 and close this bug. > > Create a new bug for Modular and other enhancements and add a note in ReadMe > > pointing to the new bug. > > > > Any objection? > > No, but the patch will hopefully address possible "modular" problems in the > sense it will not throw exceptions etc, it should just "not work" for Java > 9+? Ok, So now for all the cases it gives error: Has Compilations error -> Reference expression cannot be used in an evaluation expression. And with the patch, it will give following error for modular projects: Evaluation failed - unable to instantiate code snippet class.
(In reply to Sarika Sinha from comment #56) > Ok, So now for all the cases it gives error: > Has Compilations error -> Reference expression cannot be used in an > evaluation expression. > > And with the patch, it will give following error for modular projects: > Evaluation failed - unable to instantiate code snippet class. We should move this discussion on the patch. To differentiate between "usual" debugger bugs and Java 9+ weirdness I would wish we would make it clear, that the evaluation is not working on 9+ - either directly saying that "missing "Unsafe" class" or pointing to the new bug you are going to create.
(In reply to Andrey Loskutov from comment #57) > (In reply to Sarika Sinha from comment #56) > > Ok, So now for all the cases it gives error: > > Has Compilations error -> Reference expression cannot be used in an > > evaluation expression. > > > > And with the patch, it will give following error for modular projects: > > Evaluation failed - unable to instantiate code snippet class. > > We should move this discussion on the patch. To differentiate between > "usual" debugger bugs and Java 9+ weirdness I would wish we would make it > clear, that the evaluation is not working on 9+ - either directly saying > that "missing "Unsafe" class" or pointing to the new bug you are going to > create. +1 for handling jdk9 later. What if we can handle the situation by trying to load the UnSafe class and fail. If we do that one could enable access to internal apis to get it working using the jdk options. Not perfect but could make the feature usable until we provide a proper fix for jdk9+
Note: It’s not JDK9+, it’s with or without modules, that’s the issue. Maybe that’s easier to explain. Also, I can try to fix it on Monday.
(In reply to Jesper Moller from comment #59) > Note: It’s not JDK9+, it’s with or without modules, that’s the issue. Maybe > that’s easier to explain. > > Also, I can try to fix it on Monday. In that case, we can release this version today so that it gets into build with the message changed to "Lambda expression evaluation is not yet supported in modular projects." And Monday we can release the fix for modular?
(In reply to Sarika Sinha from comment #60) > And Monday we can release the fix for modular? I can’t promise that I can fix it (as I haven’t used the modularity features at all), but I’ll try.
Gerrit change https://git.eclipse.org/r/157396 was merged to [master]. Commit: http://git.eclipse.org/c/jdt/eclipse.jdt.debug.git/commit/?id=d9744f28d9ca87d7973aa4ac4266bf497c280e46
New Gerrit change created: https://git.eclipse.org/r/157684
(In reply to Jesper Moller from comment #61) > (In reply to Sarika Sinha from comment #60) > > And Monday we can release the fix for modular? > > I can’t promise that I can fix it (as I haven’t used the modularity features > at all), but I’ll try. Sure, I will leave this bug open till Monday. If we don't have a fix, I will create a new bug for remaining tasks.
Gerrit change https://git.eclipse.org/r/157684 was merged to [master]. Commit: http://git.eclipse.org/c/jdt/eclipse.jdt.debug.git/commit/?id=9aa5cc88979fc3970948fe5509d876e7702938ae
(In reply to Eclipse Genie from comment #65) > Gerrit change https://git.eclipse.org/r/157684 was merged to [master]. > Commit: > http://git.eclipse.org/c/jdt/eclipse.jdt.debug.git/commit/ > ?id=9aa5cc88979fc3970948fe5509d876e7702938ae The change makes the message clear for unsupported feature.
Does this means we will not be able use this feature with jvm flags to remove module restrictions on a jdk which has modular support?
(In reply to Gayan Perera from comment #67) > Does this means we will not be able use this feature with jvm flags to > remove module restrictions on a jdk which has modular support? It means you cannot use this feature for a modular project (i.e a project with module-info.java).
(In reply to Sarika Sinha from comment #68) > (In reply to Gayan Perera from comment #67) > > Does this means we will not be able use this feature with jvm flags to > > remove module restrictions on a jdk which has modular support? > > It means you cannot use this feature for a modular project (i.e a project > with module-info.java). Thats too bad if we cannot fix this issue for 4.15, I thought we could provide a different strategy for modular JDKs 1.9 and above with vm arguments to open up the modular restrictions. May be we could check if a particular vm argument is present in a module jdk and try to load the class to test if we could execute this feature.
(In reply to Gayan Perera from comment #69) > Thats too bad if we cannot fix this issue for 4.15, I thought we could > provide a different strategy for modular JDKs 1.9 and above with vm > arguments to open up the modular restrictions. May be we could check if a > particular vm argument is present in a module jdk and try to load the class > to test if we could execute this feature. As I wrote above, I’ll try to adress this as soon as I get back to civilization. I’ve not used the module stuff much, for me that was one of the edge cases not covered by the MVP as you framed it in comment #41. :-) Are there any debugger tests which run under module path?
I haven't *used* JPMS much either, but from messages flying by on jigsaw-dev I think the core question will indeed be: should this feature rely on command line switches (e.g., to enable working with Unsafe), or should we go directly to using new API (perhaps because classloading is inherently restricted even with command line tweaks applied). Here MethodHandles have already been mentioned. Also class Lookup comes to mind. JDT/Debug has precedents for synthesizing JPMS options behind the scenes, so if we knew which --add-exports etc. are needed, this shouldn't be hard. Would this be sufficient to make it work? What exactly needs to be opened up? I don't think JDT has precedents regarding new JPMS-related API. I think on mailing lists there was a bias towards using new API rather then command line switches, but perhaps this was mainly to spare end users from having to write complex command lines. In the case of JDT/Debug we could probably hide this from our users, so that's less of an argument.
(In reply to Jesper Moller from comment #70) > Are there any debugger tests which run under module path? Very few : ClasspathShortenerTests LongModulePathTests ModuleOptionsTests
New Gerrit change created: https://git.eclipse.org/r/157864
I dropped the use of Unsafe, instead getting the classloader of the surrounding class, and then use ClassLoader.defineClass to load the classes directly (the JDI debug interface conveniently bypasses access checks). I did not have time to make regression tests running under JPMS, so it's only tested manually, but works fine. Keeping my fingers crossed for the Jenkins build.
FWIW: I've created bug 560247 to track the missing features.
Moved to RC1.
I've uploaded a new patch with Andrei's review comment handled, and now also access to fields and methods of the enclosing class (public only for now) And better error handling as well.
Gerrit change https://git.eclipse.org/r/157864 was merged to [master]. Commit: http://git.eclipse.org/c/jdt/eclipse.jdt.debug.git/commit/?id=7d64f8892969bc3e50aee121bc6a18a04a24e8e1
(In reply to Jesper Moller from comment #77) > I've uploaded a new patch with Andrei's review comment handled, and now also > access to fields and methods of the enclosing class (public only for now) > And better error handling as well. Changes have been released, thanks Jesper! Can you add N&N entry by Wednesday 26th ?
@Jasper will this https://bugs.eclipse.org/bugs/show_bug.cgi?id=560392 as well. Or will the knowledge you have now can help to see if its fixable easily ?
(In reply to Sarika Sinha from comment #79) > Changes have been released, thanks Jesper! You're welcome. > Can you add N&N entry by Wednesday 26th ? Sure - but how? (Haven't done this in a while...)
Thanks also from my side, I feel I'm in debt with you as I wanted this feature for a while! Please ping me whenever we have the opportunity to meet, as I'll gladly sponsor this work by offering you a few drinks! > > Can you add N&N entry by Wednesday 26th ? > Sure - but how? (Haven't done this in a while...) You can push Gerrit patches just like for JDT, using repo www.eclipse.org/eclipse/news.git ( https://git.eclipse.org/c/www.eclipse.org/eclipse/news.git/ for a view of contents and actual Git links)
(In reply to Gayan Perera from comment #80) > @Jasper will this https://bugs.eclipse.org/bugs/show_bug.cgi?id=560392 as > well. Or will the knowledge you have now can help to see if its fixable > easily ? No, this bug doesn't change that, unfortunately. There are some other bugs related to debugging inside a lambda expression, but the general problem is that it's very implementation defined, both w.r.t. the compiler used and the LambdaMetafactory.
(In reply to Mickael Istria from comment #82) > Thanks also from my side, I feel I'm in debt with you as I wanted this > feature for a while! Please ping me whenever we have the opportunity to > meet, as I'll gladly sponsor this work by offering you a few drinks! :-) I'll take you up on your offer, if I run in to you some day. Don't know which tect conf I'll attend this year, though.
(In reply to Jesper Moller from comment #84) > (In reply to Mickael Istria from comment #82) > > Thanks also from my side, I feel I'm in debt with you as I wanted this > > feature for a while! Please ping me whenever we have the opportunity to > > meet, as I'll gladly sponsor this work by offering you a few drinks! > > :-) I'll take you up on your offer, if I run in to you some day. Don't know > which tect conf I'll attend this year, though. I definitely think the time has come for your visit at EclipseCon :)
Verified using Eclipse SDK Version: 2020-03 (4.15) Build id: I20200226-1800 @Jesper, when are you planning to add N&N entry?
New Gerrit change created: https://git.eclipse.org/r/158516
Gerrit change https://git.eclipse.org/r/158516 was merged to [master]. Commit: http://git.eclipse.org/c/www.eclipse.org/eclipse/news.git/commit/?id=3709599b36019c6ca1129048db019ee3e54fdc20
Thanks for working on this Jesper, looking forward to using this new feature! :)
(In reply to Pierre-Yves B. from comment #89) > Thanks for working on this Jesper, looking forward to using this new > feature! :) Thanks. Keep in mind the feature currently will not support all contexts, notably: - System classes (because the bootstrap classloader is ‘null’, and for module path, contains package restrictions) - Inner classes (because JDT Core tries to generate a class referencing the binary name (eg some.Class$Nested) - Generic clases - when the functional expression relies on a typevar (because we reuse the typevar name instead og the relevant bound, and so snippet class compilation fails, but also because of erasure, we just can’t tell which type the variable refers to) - Access to non-public members of surrounding context (beacuse of missing bridge methods or nest-mate mechanisms for newly loaded code) - Performance could be better (!!) - Anonymous inner classes are not there yet Also, the error messages could be better in these cases. I’m working on fixes for some of these issues, but reproducible bug reports will be very helpful.
There's a forum post[1] reporting issues with strs.stream().forEach(System.out::println) Is that one of the known limitations? Or is that even independent of the work done here? [1] https://www.eclipse.org/forums/index.php?t=rview&goto=1824303#msg_1824303
