Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 545327

Summary: [tests] Fix broken performance tests
Product: [Eclipse Project] JDT Reporter: Paul Pazderski <paul-eclipse>
Component: DebugAssignee: Paul Pazderski <paul-eclipse>
Status: RESOLVED FIXED QA Contact:
Severity: minor    
Priority: P3 CC: sarika.sinha
Version: 4.11   
Target Milestone: 4.12 M1   
Hardware: All   
OS: All   
See Also: https://git.eclipse.org/r/138598
https://git.eclipse.org/r/138599
https://git.eclipse.org/c/jdt/eclipse.jdt.debug.git/commit/?id=63d0fa90cfbcf48872b74720c68e7611344b412c
https://git.eclipse.org/c/jdt/eclipse.jdt.debug.git/commit/?id=fefdf3218e30d555278df1fb51918ebf43377eef
Whiteboard:

Description Paul Pazderski CLA 2019-03-12 18:17:23 EDT 
As you can see at https://download.eclipse.org/eclipse/downloads/drops4/I20190307-0500/performance/html/org.eclipse.jdt.debug.tests_ep411I-perf-lin64_linux.gtk.x86_64_8.0.html almost half of the performance tests fail.

I tests seem to fail for two distinct reasons.
First as consequence of bug 535802 comment 96 some tests fail due to wrong breakpoint positions. 
Actually this affects more performance tests as the log shows since some by chance found a new breakpoint position but do not work as intended anymore. E.g. the testConditionalBreakpoints place the breakpoint outside of it's test loop which results in a much faster test run.

The second fail variant are the console tests which actual work but cause a failed assertion on cleanup.

btw, also affected by bug 535802 are the not used refactoring tests (see disabled tests in AutomatedSuite) but most of those breakpoint lines seems already broken before the license update.
Comment 1 Eclipse Genie CLA 2019-03-12 18:18:46 EDT 
New Gerrit change created: https://git.eclipse.org/r/138598
Comment 2 Paul Pazderski CLA 2019-03-12 18:19:11 EDT 
(In reply to Eclipse Genie from comment #1)
> New Gerrit change created: https://git.eclipse.org/r/138598

This change fixes the breakpoint positions for all jdt debug performance tests.

Remember: some of those tests finished successfully before but were not executed as intended. That's why there may be some performance regression reported since after this change those tests actually measure performance again.
E.g. testConditionalBreakpoints was on my test six times slower after the breakpoint is at its intended position again.
Comment 3 Eclipse Genie CLA 2019-03-12 18:19:29 EDT 
New Gerrit change created: https://git.eclipse.org/r/138599
Comment 4 Paul Pazderski CLA 2019-03-12 18:20:49 EDT 
(In reply to Eclipse Genie from comment #3)
> New Gerrit change created: https://git.eclipse.org/r/138599

The second problem is less clear for me.
Apparently DebugUITools.launch registers the launch in ILaunchManager but does not remove it (at least not the way it is used).
Inspired from ConsoleInputTests the change fix this problem by explicit removing the terminated launch.

Alternatively you may simply remove all launches on test end like TestUtil.tearDown does.
Comment 5 Sarika Sinha CLA 2019-03-12 21:39:52 EDT 
I will be able to look at these after Java 12 release.

For the failing gerrit builds, increase the last segment of version by 100 in manifest and pom for the plugin as these are the first changes for 4.12.
Comment 8 Sarika Sinha CLA 2019-04-03 08:34:35 EDT 
Thanks Paul.
I have not seen Performance test results last week. If we get a good result this week, we can verify this bug.
Comment 9 Paul Pazderski CLA 2019-04-03 10:43:25 EDT 
Great. And please remember some previous successful tests where actual wrong so don't blame me prematurely for performance regressions. :)