Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 334757

Summary:

[perfs] Performance regression in WorkflowTests#testBigWorkflowNoUI()Tag1

Product:

[Eclipse Project] Platform

Reporter:

Satyam Kandula <satyam.kandula>

Component:

Team

Assignee:

Platform Team Inbox <platform-team-inbox>

Status:

RESOLVED WORKSFORME

QA Contact:

Severity:

normal

Priority:

CC:

kim.moir, tomasz.zarna

Version:

3.7

Keywords:

performance, test

Target Milestone:

---

Hardware:

OS:

Windows XP

URL:

http://download.eclipse.org/eclipse/downloads/drops/I20110315-0800/performance/epwin2/Scenario511.html

Whiteboard:

Bug Depends on:

Bug Blocks:

313891

Attachments:

Description	Flags
Full log	none

Description Satyam Kandula

2011-01-19 06:20:57 EST

There is a 50-100% regression in the test org.eclipse.team.tests.ccvs.ui.benchmark.WorkflowTests#testBigWorkflowNoUI()Tag1. The regression seems to have got introduced between 11th and 13th January builds.

Comment 1 Tomasz Zarna

2011-01-20 06:10:20 EST

It looks bad indeed. I can see the regression in N20110115-2000 and I20110118-0800. What's interesting, on the results page for I20110118-0800[3] you can see that I20110114-1330[2], which had no contribution from the team, already had the regression. I20110111-0800[1] looks fine. I will keep an eye on the next perf tests results.

Workspace contribution:
[1] Bug 315694 and Bug 333812
[2] nothing
[3] Bug 294925 and Bug 332728

Comment 2 Satyam Kandula

2011-01-26 01:53:30 EST

There is also around 25% performance regression in org.eclipse.team.tests.ccvs.ui.benchmark.SyncTests#testSync100NoUI()ModifiedFiles on both RHEL and SLED machines. There is also 25% regression in SyncTests#testSync100NoUI()RemovedFiles on RHEL -- this test is good on SLED. 
This regression also seems to have got introduced between 11th and 13th Jan.

Comment 3 Tomasz Zarna

2011-01-28 07:12:16 EST

Examining perf results from http://fullmoon.ottawa.ibm.com/downloads/drops/I20110126-1800/performance/eplnx1/Scenario511.html I can see that WorkflowTests#testBigWorkflowNoUI()Tag1 results were pretty good for I20110124-1345 and I20110125-2012 but then it peaked again. Satyam, do you have any idea why did this happen?

Comment 4 Tomasz Zarna

2011-02-02 09:40:35 EST

Created attachment 188145 [details]
Full log

I've run the perf tests locally for I20101208-1300 (3.7M4) and I20110127-2034 (3.7M5). Here is an excerpt for the scenarios for which a regression was indicated on the server:

== ModifiedFiles (3.7M4 vs 3.7M5)
Elapsed Process:        6.18s vs 5.92s
CPU Time:                1.2s  vs 1.04s

== Tag1
Elapsed Process:        12.3s vs 15s
CPU Time:                490ms vs 425ms

Only Elapsed Process metric for Tag1 showed that the scenario ran longer (ca -21%?). Full log attached.

Comment 5 Tomasz Zarna

2011-02-10 11:34:56 EST

Last week the results for org.eclipse.team.tests.ccvs.ui.benchmark.WorkflowTests#testBigWorkflowNoUI()Tag1 skyrocketed to -314.7 %. Kim do you have any idea why this may be happening? Is it us or the server?

Comment 6 Tomasz Zarna

2011-03-22 06:52:45 EDT

The results are back to normal. They have been stable for the last couple of builds[1][2]. The only problem remaining is the fact that baseline hasn't been measured for the long time - filed bug 340639 for that.

[1] http://download.eclipse.org/eclipse/downloads/drops/I20110315-0800/performance/epwin2/Scenario511.html
[2] http://download.eclipse.org/eclipse/downloads/drops/I20110315-0800/performance/epwin3/Scenario511.html