Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 342884

Summary: [releng] Add performance tests to the build
Product: [ECD] Orion Reporter: Felipe Heidrich <eclipse.felipe>
Component: RelengAssignee: Project Inbox <orion.releng-inbox>
Status: RESOLVED WONTFIX QA Contact:
Severity: normal    
Priority: P3 CC: aniefer, mamacdon
Version: unspecified   
Target Milestone: ---   
Hardware: PC   
OS: Windows 7   
Whiteboard:
Attachments:
Description Flags
my ant script to run the tests
none
the XSL transform
none
result example none

Description Felipe Heidrich CLA 2011-04-14 15:25:11 EDT 
I have released some performance test for the editor:
http://git.eclipse.org/c/e4/org.eclipse.orion.client.git/commit/?id=4c7201a932031062044de2adb9a971aa5f60d1e0

They will run normally as the other tests as part of the orion build process.
But the tests in org.eclipse.orion.client\bundles\org.eclipse.orion.client.editor\web\js-tests\test-performance.js should be handled differently:

1) They do not need to run in parallel, is better than the only thing running on the machine at the time is that one test.
2) We need to save the time for each testcase and what type of variation on that time is normal. We will need to run than a few times for that.
3) Fail when the time differences is too big.
Comment 1 Felipe Heidrich CLA 2011-04-14 15:27:46 EDT 
Created attachment 193287 [details]
my ant script to run the tests

note this script runs the tests each browser at the time (instead of all browsers testing at the same time).
Comment 2 Felipe Heidrich CLA 2011-04-14 15:28:14 EDT 
Created attachment 193288 [details]
the XSL transform
Comment 3 Felipe Heidrich CLA 2011-04-14 15:29:39 EDT 
Created attachment 193289 [details]
result example

That is the output I get on my machine.
Comment 4 Andrew Niefer CLA 2011-04-15 13:42:52 EDT 
These tests have been automatically picked up by the existing js test framework.
See them for example at the bottom of 
http://download.eclipse.org/e4/orion/drops/N201104142000/Chrome-11.0-testResults.html

Browsers are currently run sequentially one after the other, we currently aren't able to say anything about what else is running at the same time as the build machine is shared with other projects.

We need to 
1) figure out how to save and track the times
2) Investigate isolating the tests more, perhaps by running them in hudson
Comment 5 Felipe Heidrich CLA 2011-04-18 09:54:02 EDT 
Note that I can change my code to conform with any design you come up with.
The only difference now between the the performance tests and regular tests are the file name (which has -performance.js in it) and the name of the test suite ("Performance").
Comment 6 Felipe Heidrich CLA 2011-04-18 09:56:33 EDT 
As for the UI (that can be changed by the XSL transformation), it would be cool if we could have all browser on the same page to make it easier to compare the result between two browser.
Comment 7 Andrew Niefer CLA 2011-04-18 14:45:53 EDT 
It looks like perhaps the best way to do this is to write a java performance test.
(http://dev.eclipse.org/viewcvs/viewvc.cgi/org.eclipse.test.performance/doc/Performance%20Tests%20HowTo.html?view=co)

Instead of being a "real" performance test, this would just read the TEST-*.Performance.xml results of the javascript tests and then log the data as if those were normal java performance tests.  This allows us to reuse the existing performance framework used by the SDK.
Comment 8 Felipe Heidrich CLA 2011-04-19 10:51:35 EDT 
Sounds good to me.
Comment 9 John Arthorne CLA 2015-05-05 15:50:01 EDT 
Closing as part of a mass clean up of inactive bugs. Please reopen if this problem still occurs or is relevant to you. For more details see:


https://dev.eclipse.org/mhonarc/lists/orion-dev/msg03444.html
Comment 10 John Arthorne CLA 2015-05-05 16:02:43 EDT 
Closing as part of a mass clean up of inactive bugs. Please reopen if this problem still occurs or is relevant to you. For more details see:


https://dev.eclipse.org/mhonarc/lists/orion-dev/msg03444.html