Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 360044

Summary:	Add new performance tests for target and model loading
Product:	[Eclipse Project] PDE	Reporter:	Curtis Windatt <curtis.windatt.public>
Component:	UI	Assignee:	Curtis Windatt <curtis.windatt.public>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	P3
Version:	4.1
Target Milestone:	3.8 M3
Hardware:	PC
OS:	Linux
Whiteboard:

Description Curtis Windatt

2011-10-05 16:50:57 EDT

Currently PDE only has tests to open the manifest editor and the schema editor. We do not have tests around the PDE model, which is the most performance sensitive part of PDE.

We would like to add new performance tests to cover the following:

1) Resolving of a target definition / setting as target platform
Resolving targets takes some time as metadata (bundles.info) must be read then the correct files opened and some filters applied. Performance of this determines how long it takes to open the target definition editor. To set a target definition as the target platform, preferences must be set which takes additional time (a model reset must also be performed, but this is covered by (2)).

2) Creating the PDE state and models
The first time a user performs an operation involving PDE, the model manager must be initialized. This involves loading the manifest files of all the plug-ins, creating an OSGi state, reading additional information from build.properties and plugin.xml files, and storing all of this into a manager for easy lookup.

3) Finding specific models or specific information from within the models
The PluginRegistry provides an API way to access the PDE models of plugins from the target and workspace. Plug-in lookups are performed extensively throughout PDE so their performance affects many actions. In addition, the returned model contains the most useful information, but additional steps might be required to get build.properties or plugin.xml data from the plug-in, impacting performance.

It is not clear if these operations are long enough running to produce useful results (that do not fluctuate by more than 10%).

Comment 1 Curtis Windatt

2011-10-07 15:54:57 EDT

Interesting, there is already a performance test in our test plug-in that measures the time to create a PDEState for the current target platform.  The test was disabled at least 5 years ago because it is inconsistent.  Since the default target is used as test data, any time the content of the SDK changed the performance result could be affected.

The code from InitializeModelsPerfTest will be incorporated into the new tests then the class deleted.

Comment 2 Curtis Windatt

2011-10-13 13:55:27 EDT

Added the new tests to master and 3_7_maintenance.

Comment 3 Curtis Windatt

2011-10-13 13:56:54 EDT

Note that InitializeModelsPerfTest was left intact.  It still does not participate in the PDE performance test suite, but as its code checks the performance of PDEState more closely than the new target tests.

Comment 4 Curtis Windatt

2011-10-14 12:04:23 EDT

Putting in some example performance tests from my machine for reference

!SESSION 2011-10-14 09:48:53.520 -----------------------------------------------
eclipse.buildId=I20111011-2000
java.version=1.5.0_22
java.vendor=Sun Microsystems Inc.

Scenario 'org.eclipse.pde.ui.tests.performance.parts.TargetPlatformPerfTest#testResolveTargetDefinition()' (average over 200 samples):
  System Time:             234ms        (95% in [230ms, 237ms])        Measurable effect: 7ms (0.3 SDs)
  Used Java Heap:         1.02M         (95% in [-1.34M, 3.38M])       Measurable effect: 4.76M (0.3 SDs) (required sample size for an effect of 5% of stdev: 6401)
  Working Set:             1.1M         (95% in [996.48K, 1.22M])      Measurable effect: 252.57K (0.3 SDs) (required sample size for an effect of 5% of mean: 4056)
  Elapsed Process:         234ms        (95% in [230ms, 237ms])        Measurable effect: 7ms (0.3 SDs)
  Kernel time:              29ms        (95% in [27ms, 30ms])          Measurable effect: 2ms (0.3 SDs) (required sample size for an effect of 5% of mean: 789)
  CPU Time:                255ms        (95% in [246ms, 264ms])        Measurable effect: 18ms (0.3 SDs) (required sample size for an effect of 5% of mean: 427)
  Hard Page Faults:          0          (95% in [0, 0])               
  Soft Page Faults:       1.32K         (95% in [1.28K, 1.36K])        Measurable effect: 84 (0.3 SDs) (required sample size for an effect of 5% of mean: 326)
  Text Size:                 0          (95% in [0, 0])               
  Data Size:                 0          (95% in [0, 0])               
  Library Size:          35.04K         (95% in [-90.09K, 160.17K])    Measurable effect: 252.28K (0.3 SDs) (required sample size for an effect of 5% of stdev: 6400)

Scenario 'org.eclipse.pde.ui.tests.performance.parts.TargetPlatformPerfTest#testSearchModelRegistry()' (average over 100 samples):
  System Time:             155ms        (95% in [154ms, 156ms])        Measurable effect: 1ms (0.4 SDs)
  Used Java Heap:       219.01K         (95% in [-317.1K, 755.12K])    Measurable effect: 1.06M (0.4 SDs) (required sample size for an effect of 5% of stdev: 6400)
  Working Set:            7.88K         (95% in [-7.52K, 23.28K])      Measurable effect: 31.04K (0.4 SDs) (required sample size for an effect of 5% of stdev: 6401)
  Elapsed Process:         155ms        (95% in [154ms, 156ms])        Measurable effect: 1ms (0.4 SDs)
  Kernel time:               0ms        (95% in [0ms, 1ms])            Measurable effect: 1ms (0.4 SDs) (required sample size for an effect of 5% of stdev: 6400)
  CPU Time:                160ms        (95% in [157ms, 162ms])        Measurable effect: 5ms (0.4 SDs)
  Hard Page Faults:          0          (95% in [0, 0])               
  Soft Page Faults:         58          (95% in [-38, 154])            Measurable effect: 195 (0.4 SDs) (required sample size for an effect of 5% of stdev: 6400)
  Text Size:                 0          (95% in [0, 0])               
  Data Size:                 0          (95% in [0, 0])               
  Library Size:              0          (95% in [0, 0])               

Scenario 'org.eclipse.pde.ui.tests.performance.parts.TargetPlatformPerfTest#testChangeTargetPlatform()' (average over 150 samples):
  System Time:            4.06s         (95% in [3.97s, 4.14s])        Measurable effect: 174ms (0.3 SDs)
  Used Java Heap:      -801.08K         (95% in [-21.18M, 19.61M])     Measurable effect: 41.12M (0.3 SDs) (required sample size for an effect of 5% of stdev: 6400)
  Working Set:           34.88K         (95% in [-65.74K, 135.5K])     Measurable effect: 202.85K (0.3 SDs) (required sample size for an effect of 5% of stdev: 6401)
  Elapsed Process:        4.06s         (95% in [3.97s, 4.14s])        Measurable effect: 174ms (0.3 SDs)
  Kernel time:              65ms        (95% in [59ms, 72ms])          Measurable effect: 13ms (0.3 SDs) (required sample size for an effect of 5% of mean: 2589)
  CPU Time:               4.84s         (95% in [4.76s, 4.92s])        Measurable effect: 164ms (0.3 SDs)
  Hard Page Faults:          0          (95% in [0, 0])               
  Soft Page Faults:       1.13K         (95% in [877, 1.39K])          Measurable effect: 514 (0.3 SDs) (required sample size for an effect of 5% of stdev: 6400)
  Text Size:                 0          (95% in [0, 0])               
  Data Size:                 0          (95% in [0, 0])               
  Library Size:           4.27K         (95% in [-4.2K, 12.73K])       Measurable effect: 17.07K (0.3 SDs) (required sample size for an effect of 5% of stdev: 6400)

Comment 5 Curtis Windatt

2011-11-14 15:43:28 EST

The testChangeTargetPlatform() has been failing locally on multiple machines, even with the iterations reduced to  I have disabled the test for now.

Comment 6 Curtis Windatt

2011-11-22 11:05:02 EST

Even reducing the iterations to under 5 can cause the tests to fail.  No idea how I had the tests running on my local machine at all.  The tests will remain commented out for the foreseeable future.  I will run the test after making changes for target per project and if that ends the memory leak I will count that as a performance win.

Returning to fixed.