Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 442455

Summary:	We should not have empty performance targets
Product:	[Eclipse Project] Platform	Reporter:	David Williams <david_williams>
Component:	Releng	Assignee:	Platform-Releng-Inbox <platform-releng-inbox>
Status:	VERIFIED FIXED	QA Contact:
Severity:	normal
Priority:	P3	CC:	daniel_megert, Lars.Vogel
Version:	4.5
Target Milestone:	4.5 M2
Hardware:	All
OS:	All
Whiteboard:
Bug Depends on:	442456, 442528, 442529, 442530, 442531, 442532, 442533
Bug Blocks:	374441

Description David Williams

2014-08-25 01:51:04 EDT

As mention in bug 374441 comment 41, there are 41 test suites identified as having "performance targets". This I determined by looking in "test.properties", which is created during a build, by a custom CBI maven task (that, someone wrote for us ... Thanh? Igor?) ... and it finds 41. I'll be going through the list, and see which ones work, and which ones don't. 

I made my first major discovery tonight ... so thought I'd document it here ... and use this bug to document other findings. 

The major finding was that 'org.eclipse.ant.tests.core' is one of the suite identified as having a performance target, yet in preliminary runs, it seemed that "nothing ever happened", so I'm thinking, great, some thing is broken. 

But, finally looked at the test.xml for that 'org.eclipse.ant.tests.core' bundle, and saw 

  <!-- This target runs the performance test suites. -->
  <target name="performance">
  </target>

That is, it has a performance target, that is completely empty! Not sure if this was changed at some point in history, or left in as a reminder to add some performance tests some day. But, bottom line, I think we should not allow empty performance targets. (Not sure if there are others, or not) but, this sort of thing is nothing but confusing unless someone looks closely, and that's a waste of time.

Comment 1 David Williams

2014-08-25 02:24:43 EDT

Another "finding" was that several of the "jdt" tests run out of memory when its set to 500m. I don't know what I was thinking ... but, will set to 2G to avoid "minor" memory issues, but suspect that's about as high as we want to go?

Comment 2 David Williams

2014-08-25 15:37:49 EDT

By manually "inspecting" the test.xml file, determined there are 19 other tests that have empty performance targets (for 20 total, leaving 21 that do have performance targets). [No wonder I thought so many were not working :) ] 

eclipse.jdt.core/org.eclipse.jdt.apt.pluggable.tests/test.xml
eclipse.jdt.core/org.eclipse.jdt.compiler.apt.tests/test.xml
eclipse.jdt.core/org.eclipse.jdt.compiler.tool.tests/test.xml
eclipse.jdt.core/org.eclipse.jdt.core.tests.builder/test.xml
eclipse.jdt.core/org.eclipse.jdt.core.tests.compiler/test.xml
eclipse.jdt.core/org.eclipse.jdt.core.tests.model/test.xml

eclipse.jdt.ui/org.eclipse.ltk.core.refactoring.tests
eclipse.jdt.ui/org.eclipse.ltk.ui.refactoring.tests

eclipse.pde.build/org.eclipse.pde.build.tests
eclipse.pde.ui/ds/org.eclipse.pde.ds.tests

eclipse.platform.runtime/tests/org.eclipse.core.expressions.tests/test.xml

eclipse.platform.team/tests/org.eclipse.core.tests.net/test.xml
eclipse.platform.team/tests/org.eclipse.team.tests.core

eclipse.platform.text/org.eclipse.core.filebuffers.tests/test.xml
eclipse.platform.text/org.eclipse.jface.text.tests
eclipse.platform.text/org.eclipse.search.tests
eclipse.platform.text/org.eclipse.text.tests
eclipse.platform.text/org.eclipse.ui.editors.tests
eclipse.platform.text/org.eclipse.ui.workbench.texteditor.tests

eclipse.platform.ua/org.eclipse.ua.tests.doc

Comment 3 David Williams

2014-08-25 16:07:16 EDT

Bugs open on components with empty targets. 

These "bugs" don't prevent progress from being made, but, it makes the results hard to interpret the console comes back and says, basically "no results found for 'xyx performance tests' " ... impossible to distinguish that from a test that just "did run" for other reasons. 

My guess is historically a lot of the tools that "analyze" the data just ignores those cases, so do not so up as "failed" (or "DNF") by the time it gets to that point ... but for someone new to the performance tests, it's very confusing.

Comment 4 David Williams

2014-08-25 16:41:44 EDT

My counts are adding up, so I may have "lost" one ... but sure if "empty" one, or one with a valid target, but the list of tests I currently have to run, are the following 19: 

    org.eclipse.ant.tests.ui
    org.eclipse.compare.tests
    org.eclipse.core.tests.runtime
    org.eclipse.core.tests.resources
    org.eclipse.equinox.p2.tests.ui 
    org.eclipse.jdt.core.tests.performance
    org.eclipse.jdt.debug.tests
    org.eclipse.jdt.text.tests
    org.eclipse.jdt.ui.tests
    org.eclipse.jdt.ui.tests.refactoring
    org.eclipse.osgi.tests
    org.eclipse.pde.api.tools.tests
    org.eclipse.pde.ui.tests
    org.eclipse.swt.tests
    org.eclipse.team.tests.cvs.core
    org.eclipse.ua.tests
    org.eclipse.ui.tests.forms
    org.eclipse.ui.tests.performance
    org.eclipse.ui.tests.rcp

Let me know if anyone sees any obvious ones missing. (Though, eventually we'll find it, when I no longer have to "manually" manage the list, but have accurate targets).

Comment 5 Dani Megert

2014-09-01 08:02:58 EDT

(In reply to David Williams from comment #0)
> But, finally looked at the test.xml for that 'org.eclipse.ant.tests.core'
> bundle, and saw 
> 
>   <!-- This target runs the performance test suites. -->
>   <target name="performance">
>   </target>

Fixed with http://git.eclipse.org/c/platform/eclipse.platform.git/commit/?id=9769a843ecc2089a3ad55b149402893e4d5a5ee8

Comment 6 Dani Megert

2014-09-01 08:11:43 EDT

All dependent bugs are fixed.

Comment 7 David Williams

2014-09-02 12:17:41 EDT

Confirmed there are 19 tests bundles (in test.properties) now that "empty ones' have been removed: 

org.eclipse.ant.tests.ui
org.eclipse.compare.tests
org.eclipse.core.tests.resources
org.eclipse.core.tests.runtime
org.eclipse.equinox.p2.tests.ui
org.eclipse.jdt.core.tests.performance
org.eclipse.jdt.debug.tests
org.eclipse.jdt.text.tests
org.eclipse.jdt.ui.tests
org.eclipse.jdt.ui.tests.refactoring
org.eclipse.osgi.tests
org.eclipse.pde.api.tools.tests
org.eclipse.pde.ui.tests
org.eclipse.swt.tests
org.eclipse.team.tests.cvs.core
org.eclipse.ua.tests
org.eclipse.ui.tests.forms
org.eclipse.ui.tests.performance
org.eclipse.ui.tests.rcp