Community
Participate
Working Groups
I have released some performance test for the editor: http://git.eclipse.org/c/e4/org.eclipse.orion.client.git/commit/?id=4c7201a932031062044de2adb9a971aa5f60d1e0 They will run normally as the other tests as part of the orion build process. But the tests in org.eclipse.orion.client\bundles\org.eclipse.orion.client.editor\web\js-tests\test-performance.js should be handled differently: 1) They do not need to run in parallel, is better than the only thing running on the machine at the time is that one test. 2) We need to save the time for each testcase and what type of variation on that time is normal. We will need to run than a few times for that. 3) Fail when the time differences is too big.
Created attachment 193287 [details] my ant script to run the tests note this script runs the tests each browser at the time (instead of all browsers testing at the same time).
Created attachment 193288 [details] the XSL transform
Created attachment 193289 [details] result example That is the output I get on my machine.
These tests have been automatically picked up by the existing js test framework. See them for example at the bottom of http://download.eclipse.org/e4/orion/drops/N201104142000/Chrome-11.0-testResults.html Browsers are currently run sequentially one after the other, we currently aren't able to say anything about what else is running at the same time as the build machine is shared with other projects. We need to 1) figure out how to save and track the times 2) Investigate isolating the tests more, perhaps by running them in hudson
Note that I can change my code to conform with any design you come up with. The only difference now between the the performance tests and regular tests are the file name (which has -performance.js in it) and the name of the test suite ("Performance").
As for the UI (that can be changed by the XSL transformation), it would be cool if we could have all browser on the same page to make it easier to compare the result between two browser.
It looks like perhaps the best way to do this is to write a java performance test. (http://dev.eclipse.org/viewcvs/viewvc.cgi/org.eclipse.test.performance/doc/Performance%20Tests%20HowTo.html?view=co) Instead of being a "real" performance test, this would just read the TEST-*.Performance.xml results of the javascript tests and then log the data as if those were normal java performance tests. This allows us to reuse the existing performance framework used by the SDK.
Sounds good to me.
Closing as part of a mass clean up of inactive bugs. Please reopen if this problem still occurs or is relevant to you. For more details see: https://dev.eclipse.org/mhonarc/lists/orion-dev/msg03444.html
