Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.
Bug 259287 - extremely poor refreshLocalResourceImpl performance
Summary: extremely poor refreshLocalResourceImpl performance
Status: RESOLVED FIXED
Alias: None
Product: Subversive
Classification: Technology
Component: Core (show other bugs)
Version: 0.7   Edit
Hardware: PC Linux
: P3 normal (vote)
Target Milestone: ---   Edit
Assignee: Igor Burilo CLA
QA Contact:
URL:
Whiteboard:
Keywords:
: 270358 (view as bug list)
Depends on:
Blocks: 287559
  Show dependency tree
 
Reported: 2008-12-18 12:39 EST by Igor Fedorenko CLA
Modified: 2013-02-14 04:30 EST (History)
10 users (show)

See Also:


Attachments
Path for fix iterate (4.86 KB, patch)
2010-04-09 06:42 EDT, Alexey Mising name CLA
i.vinnykov: iplog+
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Igor Fedorenko CLA 2008-12-18 12:39:51 EST
I often have one of CPU cores running at 100% for several minutes after Refresh from local. Most of the time is spent in SVNRemoteStorage#refreshLocalResourceImpl according to Yourkit profiler. I had a quick look at this method and as far as I can tell it iterates over all known ILocalResources for each IResource to refresh. Kinda inefficient ;-) I can probably provide a patch, if you are interested.
Comment 1 Michael Powell CLA 2009-01-15 15:30:08 EST
Yes, please work on this as well as over all performance when refreshing. My dual core system frequently shoots to 50% and 75% CPU usage when refreshing. It gets quite annoying when this happens, because Eclipse becomes completely unresponsive.
Comment 2 Igor Burilo CLA 2009-02-27 03:47:27 EST
Igor, I would be appreciate if you provide a patch mentioned in description.

From Igor Fedorenko about slow update:
I believe slow refresh after update is caused by inefficient 
implementation of FileUtility#getPathNodes. It uses ArrayList#contains 
method which is slow for large collections. Simply switching to HashSet 
makes update much much faster (I am running patched version for several 
months now).
Comment 3 Jochen Stiepel CLA 2009-07-27 08:34:32 EDT
Igor Burilo you are requesting a patch from Igor Fedorenko in your last comment.

This patch is available here:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=266492
and the file here:
https://bugs.eclipse.org/bugs/attachment.cgi?id=126997

Please integrate this patch!

Currently we have a one SVN Repo with 10200 files in 1800 folder and an update using subclipse (Team Synchronise, Update All) on the complete project takes around 10 minutes.
A lot of this time is spend for the refresh operation of the eclipse workspace with a very high cpu usage.

I hope this patch will help, because 10 min. is way to long.
Comment 4 Alexey Mising name CLA 2010-04-09 06:42:10 EDT
Created attachment 164353 [details]
Path for fix iterate

I take some fixes of this. Please look at it, and apply, if it's good.
Comment 5 Igor Burilo CLA 2010-04-12 03:15:38 EDT
Thanks, Alexey. I'll review it.
Comment 6 Manuel Darveau CLA 2010-05-06 17:04:29 EDT
Is there a nightly build with this fix? I would be more than happy to test it.
Comment 7 Igor Burilo CLA 2010-05-07 03:18:05 EDT
(In reply to comment #6)
> Is there a nightly build with this fix? I would be more than happy to test it.
Manuel, it's not yet fixed.
Comment 8 Niklas Deutschmann CLA 2010-06-02 09:33:15 EDT
This may not be directly related, but I'm experiencing extremely poor performance after a share/commit when having a large number (>1000) of files in a single directory.

Sharing a project with 2000 small text files (having only the content "file [file number]") takes more than 30 minutes using Eclipse 3.5 (org.eclipse.team.svn_0.7.8.I20090506-1500 and the svnkit connector)

Look at org.polarion.team.svn.core.operation.local.RefreshResourcesOperation.runImpl(). For each file that is committed, the SVN metadata folder in the file's directory (.svn) is refreshed. This means that refresh may is in O(n^2) in the number of files, because the .svn\prop-base folder contains the base version of each file in the versioned directory.
Comment 9 Alexander Gurov CLA 2010-11-04 05:05:02 EDT
(In reply to comment #8)
> Look at
> org.polarion.team.svn.core.operation.local.RefreshResourcesOperation.runImpl().
> For each file that is committed, the SVN metadata folder in the file's
> directory (.svn) is refreshed. This means that refresh may is in O(n^2) in the
> number of files, because the .svn\prop-base folder contains the base version of
> each file in the versioned directory.

It is really not like that at all. This code reads SVN meta-information once per folder basis and not per file. So, if you have about 2000 files in few folders it will be readen few times (one time for each folder) but not 2000 times, i.e. there is no excessive reads.
Comment 10 Alexander Gurov CLA 2011-02-07 03:13:33 EST
*** Bug 270358 has been marked as a duplicate of this bug. ***
Comment 11 Alexander Gurov CLA 2011-06-17 13:06:00 EDT
Fixed.
Comment 12 Igor Vinnykov CLA 2012-05-15 11:27:03 EDT
Comment on attachment 164353 [details]
Path for fix iterate

Set iplog flag