Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 334285

Summary:	[server] Prepare performance tests for all EFS implementations in Orion
Product:	[ECD] Orion	Reporter:	Tomasz Zarna <tomasz.zarna>
Component:	Client	Assignee:	Tomasz Zarna <tomasz.zarna>
Status:	RESOLVED INVALID	QA Contact:
Severity:	normal
Priority:	P3	CC:	Szymon.Brandys
Version:	0.2	Keywords:	performance
Target Milestone:	---
Hardware:	PC
OS:	Windows XP
Whiteboard:
Bug Depends on:	336096
Bug Blocks:

Description Tomasz Zarna

2011-01-13 11:47:29 EST

I'm working on improving performance of git EFS implementation (bug 334094), because we already know it's slow comparing to local file system. But we don't have precise data on how much faster the later actually is. I hope that in the future there will be other EFS implementations working in Orion so it would be great if we could compare their performance as well.

As the first step, I would like to prepare a perf. test for common workflow and run it for local file system and git. The test should allow to add and compare other EFS implementation as they show up.

Comment 1 Tomasz Zarna

2011-01-18 09:46:19 EST

Here are the first results:

Scenario 'org.eclipse.orion.server.tests.filesystem.git.performance.GitPerformanceTest#commonOperations()' (average over 100 samples):
  Used Java Heap:            3,27M      (95% in [-4,5M, 11,03M])       Measurable effect: 15,65M (0,4 SDs) (required sample size for an effect of 5% of stdev: 6401)
  Working Set:             481,4K       (95% in [-354,11K, 1,29M])     Measurable effect: 1,64M (0,4 SDs) (required sample size for an effect of 5% of stdev: 6400)
  Committed:               175,52K      (95% in [-822,59K, 1,15M])     Measurable effect: 1,96M (0,4 SDs) (required sample size for an effect of 5% of stdev: 6401)
  Working Set Peak:        346,52K      (95% in [45,68K, 647,36K])     Measurable effect: 606,47K (0,4 SDs) (required sample size for an effect of 5% of stdev: 6400)
  Elapsed Process:         928ms        (95% in [904ms, 951ms])        Measurable effect: 47ms (0,4 SDs) (required sample size for an effect of 5% of mean: 106)
  Kernel time:             699ms        (95% in [688ms, 710ms])        Measurable effect: 21ms (0,4 SDs)
  Page Faults:               1,06K      (95% in [785, 1,33K])          Measurable effect: 551 (0,4 SDs) (required sample size for an effect of 5% of stdev: 6400)
  CPU Time:                850ms        (95% in [819ms, 882ms])        Measurable effect: 64ms (0,4 SDs) (required sample size for an effect of 5% of mean: 228)
  GDI Objects:               0          (95% in [0, 0])               

Scenario 'org.eclipse.orion.server.tests.filesystem.performance.LocalPerformanceTest#commonOperations()' (average over 100 samples):
  Used Java Heap:           28,05K      (95% in [25,56K, 30,53K])      Measurable effect: 5K (0,4 SDs) (required sample size for an effect of 5% of mean: 1274)
  Working Set:               8,76K      (95% in [55, 17,46K])          Measurable effect: 17,55K (0,4 SDs) (required sample size for an effect of 5% of stdev: 6400)
  Committed:                 7,96K      (95% in [-20,55K, 36,47K])     Measurable effect: 57,47K (0,4 SDs) (required sample size for an effect of 5% of stdev: 6401)
  Working Set Peak:          0          (95% in [0, 0])               
  Elapsed Process:           6ms        (95% in [4ms, 7ms])            Measurable effect: 3ms (0,4 SDs) (required sample size for an effect of 5% of stdev: 6401)
  Kernel time:               5ms        (95% in [3ms, 6ms])            Measurable effect: 2ms (0,4 SDs) (required sample size for an effect of 5% of stdev: 6400)
  Page Faults:               6          (95% in [2, 10])               Measurable effect: 7 (0,4 SDs) (required sample size for an effect of 5% of stdev: 6400)
  CPU Time:                  5ms        (95% in [4ms, 7ms])            Measurable effect: 3ms (0,4 SDs) (required sample size for an effect of 5% of stdev: 6400)
  GDI Objects:               0          (95% in [0, 0])  
  
The first scenario was for git filestore, the later - local file system. Both tests executed the same set of operations: create folder, create files, list files for folder, edit file etc. The results exceed my expectations, I didn't think git is sooo slow i.e. over 100x slower then local fs. There is a lot to improve here. Bug 334094 is about changing the current situation.

Comment 2 Tomasz Zarna

2011-01-19 12:30:39 EST

With the latest fix from bug 334094 the results for git went down to:

Scenario 'org.eclipse.orion.server.tests.filesystem.git.performance.GitPerformanceTest#commonOperations()' (average over 100 samples):
  Used Java Heap:            7,77M      (95% in [3,34M, 12,19M])       Measurable effect: 8,92M (0,4 SDs) (required sample size for an effect of 5% of stdev: 6400)
  Working Set:             272,96K      (95% in [-159,79K, 705,71K])   Measurable effect: 872,39K (0,4 SDs) (required sample size for an effect of 5% of stdev: 6400)
  Committed:               -12,88K      (95% in [-423,82K, 398,06K])   Measurable effect: 828,42K (0,4 SDs) (required sample size for an effect of 5% of stdev: 6400)
  Working Set Peak:        256,2K       (95% in [-7,71K, 520,11K])     Measurable effect: 532,02K (0,4 SDs) (required sample size for an effect of 5% of stdev: 6400)
  Elapsed Process:         900ms        (95% in [874ms, 926ms])        Measurable effect: 52ms (0,4 SDs) (required sample size for an effect of 5% of mean: 135)
  Kernel time:             670ms        (95% in [660ms, 680ms])        Measurable effect: 20ms (0,4 SDs)
  Page Faults:               1,04K      (95% in [843, 1,24K])          Measurable effect: 395 (0,4 SDs) (required sample size for an effect of 5% of mean: 5789)
  CPU Time:                824ms        (95% in [795ms, 854ms])        Measurable effect: 59ms (0,4 SDs) (required sample size for an effect of 5% of mean: 210)
  GDI Objects:               0          (95% in [0, 0])  
  
Not much, but it's the right direction ;)

Comment 3 Szymon Brandys

2011-03-30 09:58:52 EDT