Bugzilla – Full Text Bug Listing
| Summary: | Implement deferred in time fetch of affected paths for History View | ||
|---|---|---|---|
| Product: | [Technology] Subversive | Reporter: | nikolaus heger <nheger> |
| Component: | Core | Assignee: | Alexander Gurov <a.gurov> |
| Status: | CLOSED MOVED | QA Contact: | |
| Severity: | enhancement | ||
| Priority: | P3 | CC: | carsten.pfeiffer, nheger, yoda |
| Version: | 0.7 | ||
| Target Milestone: | --- | ||
| Hardware: | All | ||
| OS: | All | ||
| Whiteboard: | |||
Additional info: The same thing happens on svn compare. If I load history, then do a compare between two versions, again, I see a sustained 400KB/s download for about a minute before the compare appears. On a slow connection, it takes proportionally longer. Comparing the same file using svn diff -rn:m (filename) only takes a second. I have developed a theory: I think these commands are applied recursively, and access all branches which is what requires so much data to be downloaded. I found that if I add a new file to trunk and make a couple of revisions, then do history or compare, speed is reasonable. So we can conclude this happens only with multiple branches/tags - somehow all branches/tags are accessed. What may affect performance as well is that I don't have all branches checked out locally. I am going to test that next (getting all branches). The reason I don't have them is that each branch is over 600MB and we have many of them, so this is quite large. The issue seems to be a general SVN issue and also happens with subclipse. However, there is a workaround described here http://www.nabble.com/enhancement-to-show-history-of-genealogy-in-sublicpse-td16422529.html Could you please provide few more information: 1) what SVN version is installed on server side? 2) How many revisions you have for the element on which the problem is happened? 1) svn, version 1.4.6 (r28521) compiled Dec 22 2007, 08:52:58 Running on a Linux box 2) The element has 6 revisions. It's in trunk as well as 4 branches and 6 tags. Repository looks like this: / (root) /modules/moduleA/trunk/projectA/element.xml /modules/moduleA/branches/branch_1/projectA/element.xml ... /modules/moduleA/branches/tag_1/projectA/element.xml ... moduleA/trunk and each branch and tag contain approx. 36,000 files using ~830MB of disk space. The repository is just a week old so the vast majority of these files have very few revisions - the one I tried it on is one of the few. *** Bug 222632 has been marked as a duplicate of this bug. *** NOTE: Similar problem described in "linked" report also happened on Mac platform. <Copied from news group:> IMHO the problem is not related to tags or branches of existing resources. As you can see at Polarion public repository we have many tags and branches but never experianced with problem about you wrote: http://svn.polarion.org/repos/community/Subversive Unfortunately at this moment I can't say something reasonable about your situation except we have similar report from another Mac OS X user: 222632: Team Synchronization very slow https://bugs.eclipse.org/bugs/show_bug.cgi?id=222632 Marc experienced with the same problem: fetching history takes too much time. My testing shows that branches/tags have a lot to do with the issue. All evidence points to this. New observations: 1 - I discovered that history was a lot faster for new files (which prompted me to do the test in (2) below). 2 - I created a new file, text.txt, right next to the file that was taking so long. I made 2 revisions, and tested history view. It was reasonably fast at 6 seconds and I didn't notice any unusual network activity. It transferred a total of maybe 10KB. The difference between those two files seems to be only that the new one I created is not in any branches. 3 - I found that if I set the "history page size" to 1, it's very fast on the original file. I think that's because it will then only show the latest revision, and the latest version is newer than most branches and tags. To verify this, I kept the history page size at 1 and opened history view on a file that has never changed since the start of the repository. That means it has no revisions save the original, and it's in all branches/tags. Sure enough, a download of 10s of MB followed and history view was extremely slow - even with page size at 1. Is there a debug version of the code I could use? It seems like you can't recreate the problem but given how much data this downloads I think I could at least find the code that does this very quickly... it should be obvious. My colleague tried this on Windows and saw the same issue (in our repository, e.g. everything else being equal, so I don't think the Mac is to blame). I have new, and significant information on our repository. I have found a potential cause for the slowness. When I issue a "svn log --verbose (filename)" on one of the slow files, I get a list of "changed paths" which each revision. That is interesting but has a fatal problem with the initial revision. The file was there from the beginning, and our repository was set up from scratch with an existing code base, so the last rev looks like this: r9 | rob | 2008-04-06 12:54:48 +0700 (Sun, 06 Apr 2008) | 1 line Changed paths: A /modules/moduleA A /modules/moduleA/branches A /modules/moduleA/branches/branch1 ..... And it now goes on to list all 55,000 files that were checked in on the initial checkin, each one on it's own line. That's over 6MB of data, just for the log on a single file. All operations that use a verbose svn log would be affected by this. It compresses fairly well but even compressed a file with the same content (copied and pasted) is still close to 400KB, more than I want to download to get the history. The link I mentioned above mentions that the authors implemented a client side log caching algorithm. That's very simple to do since the svn log is always moving forward. Therefore, if you locally cached the log up to rev X, all you would need to do to get the new log up to revision Y is do a svn log from rev X to rev Y, then append that partial log to the cached log. Huge amounts of bandwidth would be saved, and the system would be way faster. So this is a feature request. In the meantime, I am going to check into how we can restructure our repository such that this doesn't happen quite so badly. HTTP compression would probably help a lot. Supposedly, TortoiseSVN on Windows does this as well. Thank you for your investigation. I hope HTTP compression should help in this case. In any case described behaviour is not acceptable for Synchronize or Annotate views. So, I will check if affected paths are grabbed by corresponding code parts or not. TortoiseSVN implements log caching - I see many ways in which this might be useful for any SVN client. See here: http://tortoisesvn.tigris.org/tsvn_1.5_releasenotes.html I see one little problem with log messages caching: SVN allows to override revision author and message using "svn propset --revprop" command (if corresponding hooks are enabled on the server side). So, in that case log messages cache will go to out-of-date state. The other problem with the log cache would be that it could grow uncontrollably. E.g. worst case for our repository would be 6MB x 55,000 or over 300GB for the logs alone. We have now resolved the issue. I want to point you to our current solution (server side) and a client side solution in subclipse: ==== Client side solution === Using Subclipse, I have the option to "Fetch affected paths on demand". That makes the initial history fast as affected paths are not downloaded. I also have the option of hiding the affected paths altogether in the history view. Subversive also offers the option to hide affected paths in the history view (didn't see that before). However, removing the view didn't stop subversive from downloading the paths anyway. So the solution for subversive is to offer a "fetch affected paths on demand" option. ==== Server side solution ==== While subversion doesn't allow deleting of history, you can do it indirectly by doing the following: - Export the current directory at revision X. Let X be 100 to "chop off" the first 100 entries in the history. - Delete the original repository / replace with new one - Import what was exported before - Clients have to delete their local versions and re-check out the entire repository so the version numbers are correct. We are pretty confident that will work. It will remove the extremely large paths from the history and so get around the problem. Affected paths should be fetched only in case if corresponding view part is visible. Otherwise paths should be fetched in background after affected paths view becomes visible. This issue has been migrated to https://gitlab.eclipse.org/eclipse/subversive/subversive/-/issues/17. |
Steps to reproduce: - Set up svn repository ising svn:// protocol - Access any file in Eclipse, select Show History Expected: History shows in a short amount of time Actual: History takes a very long time to show up, several minutes. Also, 10s of MB are downloaded in the process. Regression: native svn vlient "svn log (filename)" is equivalent to this operation and only takes a second and a few KB to complete Version: 0.7.0.v20080404 SVN Client: org.eclipse.team.svn.connector.javahl 2.0.0.v20080404 svn:1.4.6 (r28521) jni:0.9.0 JVM Properties: {java.vendor=Apple Inc., org.osgi.supports.framework.extension=true, sun.management.compiler=HotSpot Client Compiler, osgi.framework.beginningstartlevel=1, os.name=Mac OS X, osgi.ws=carbon, java.vm.specification.vendor=Sun Microsystems Inc., java.runtime.version=1.5.0_13-b05-237, org.apache.commons.logging.simplelog.log.httpclient.wire.header=off, osgi.instance.area=file:/Users/nik/work/prophet/workspaces/sandbox/modules/investools/, user.name=nik, awt.nativeDoubleBuffering=true, org.osgi.framework.language=en, user.language=en, org.osgi.framework.processor=i386, osgi.syspath=/Applications/eclipse/plugins, sun.boot.library.path=/System/Library/Frameworks/JavaVM.framework/Versions/1.5.0/Libraries, osgi.manifest.cache=/Applications/eclipse/configuration/org.eclipse.osgi/manifests, osgi.compatibility.bootdelegation=true, eof=eof, java.version=1.5.0_13, org.osgi.framework.os.name=MacOSX, user.timezone=Asia/Bangkok, sun.arch.data.model=32, java.endorsed.dirs=/System/Library/Frameworks/JavaVM.framework/Versions/1.5.0/Home/lib/endorsed, sun.cpu.isalist=, sun.jnu.encoding=MacRoman, file.encoding.pkg=sun.io, org.osgi.framework.vendor=Eclipse, file.separator=/, java.specification.name=Java Platform API Specification, java.class.version=49.0, user.country=US, org.eclipse.equinox.launcher.splash.location=/Applications/eclipse/plugins/org.eclipse.platform_3.3.3.r33x_r20080129/splash.bmp, java.home=/System/Library/Frameworks/JavaVM.framework/Versions/1.5.0/Home, osgi.os=macosx, eclipse.commands=-os macosx -ws carbon -arch x86 -showsplash -launcher /Applications/eclipse/Eclipse investools.app/Contents/MacOS/eclipse -name Eclipse --launcher.library /Applications/eclipse/Eclipse investools.app/Contents/MacOS/../../../plugins/org.eclipse.equinox.launcher.carbon.macosx_1.0.3.R33x_v20080118/eclipse_1023.so -startup /Applications/eclipse/Eclipse investools.app/Contents/MacOS/../../../plugins/org.eclipse.equinox.launcher_1.0.1.R33x_v20080118.jar -keyring /Users/nik/.eclipse_keyring -consoleLog -showlocation -data /Users/nik/work/prophet/workspaces/sandbox/modules/investools -clean -vm /System/Library/Frameworks/JavaVM.framework , java.vm.info=mixed mode, osgi.splashLocation=/Applications/eclipse/plugins/org.eclipse.platform_3.3.3.r33x_r20080129/splash.bmp, os.version=10.5.2, osgi.arch=x86, path.separator=:, java.vm.version=1.5.0_13-119, org.osgi.supports.framework.fragment=true, osgi.framework.shape=jar, osgi.instance.area.default=file:/Users/nik/Documents/workspace/, java.awt.printerjob=apple.awt.CPrinterJob, sun.io.unicode.encoding=UnicodeLittle, org.osgi.framework.version=1.4.0, awt.toolkit=apple.awt.CToolkit, osgi.install.area=file:/Applications/eclipse/, osgi.framework=file:/Applications/eclipse/plugins/org.eclipse.osgi_3.3.2.R33x_v20080105.jar, user.home=/Users/nik, osgi.bundlestore=/Applications/eclipse/configuration/org.eclipse.osgi/bundles, osgi.splashPath=platform:/base/plugins/org.eclipse.platform, eclipse.consoleLog=true, osgi.nl=en_US, java.specification.vendor=Sun Microsystems Inc., java.library.path=.:/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java, java.vendor.url=http://www.apple.com/, org.osgi.framework.os.version=10.5.2, eclipse.startTime=1208485181907, java.vm.vendor=Apple Inc., gopherProxySet=false, java.runtime.name=Java(TM) 2 Runtime Environment, Standard Edition, java.class.path=/Applications/eclipse/Eclipse investools.app/Contents/MacOS/../../../plugins/org.eclipse.equinox.launcher_1.0.1.R33x_v20080118.jar, osgi.requiredJavaVersion=1.5, org.apache.commons.logging.simplelog.log.org.apache.commons.httpclient=off, eclipse.vm=/System/Library/Frameworks/JavaVM.framework, java.vm.specification.name=Java Virtual Machine Specification, java.vm.specification.version=1.0, sun.cpu.endian=little, org.eclipse.swt.internal.carbon.smallFonts=, sun.os.patch.level=unknown, org.apache.commons.logging.simplelog.defaultlog=off, java.io.tmpdir=/tmp, java.vendor.url.bug=http://bugreport.apple.com/, eclipse.product=org.eclipse.platform.ide, os.arch=i386, java.awt.graphicsenv=apple.awt.CGraphicsEnvironment, java.ext.dirs=/Library/Java/Extensions:/System/Library/Java/Extensions:/System/Library/Frameworks/JavaVM.framework/Versions/1.5.0/Home/lib/ext, mrj.version=1040.1.5.0_13-237, user.dir=/Applications/eclipse/Eclipse investools.app/Contents/MacOS, org.osgi.supports.framework.requirebundle=true, osgi.clean=true, line.separator= , java.vm.name=Java HotSpot(TM) Client VM, org.apache.commons.logging.Log=org.apache.commons.logging.impl.SimpleLog, eclipse.ee.install.verify=false, file.encoding=MacRoman, osgi.framework.version=3.3.2.R33x_v20080105, eclipse.buildId=M20071023-1652, eclipse.vmargs=-Xdock:icon=../Resources/Eclipse.icns -XstartOnFirstThread -Xms40m -Xmx512m -XX:MaxPermSize=256m -Dosgi.requiredJavaVersion=1.5 -Dorg.eclipse.swt.internal.carbon.smallFonts -Djava.class.path=/Applications/eclipse/Eclipse investools.app/Contents/MacOS/../../../plugins/org.eclipse.equinox.launcher_1.0.1.R33x_v20080118.jar , java.specification.version=1.5, org.osgi.framework.executionenvironment=OSGi/Minimum-1.0,OSGi/Minimum-1.1,JRE-1.1,J2SE-1.2,J2SE-1.3,J2SE-1.4,J2SE-1.5, osgi.logfile=/Users/nik/work/prophet/workspaces/sandbox/modules/investools/.metadata/.log, osgi.configuration.area=file:/Applications/eclipse/configuration/}