Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 346282

Summary:	SVN checkout is slow
Product:	z_Archived	Reporter:	Nicolas Bros <nicolas.bros>
Component:	Buckminster	Assignee:	buckminster.core-inbox <buckminster.core-inbox>
Status:	RESOLVED INVALID	QA Contact:
Severity:	normal
Priority:	P3	CC:	thomas
Version:	unspecified
Target Milestone:	---
Hardware:	PC
OS:	Linux
Whiteboard:

Description Nicolas Bros

2011-05-18 11:25:08 EDT

I am building MoDisco using Buckminster on hudson.eclipse.org.
There, it imports plug-ins using "file:///svnroot", so it should be fast since there is no network transfer involved.

But the import takes a long time : 2746s for the last integration build (this is including the materialization of the target platform, but still).

Whereas when trying to do a checkout of the whole MoDisco svn manually, it terminates in two minutes.

Comment 1 Nicolas Bros

2011-05-18 11:35:36 EDT

When doing the same build with the nightly job, which checks out the projects from svn first using Hudson, and then calls Buckminster to do the import with the rmap referring to the local projects (readerType="local"), the import is much faster : 930s instead of 2746s.

Comment 2 Nicolas Bros

2011-05-25 04:05:58 EDT

Here is a concrete example. I see in the build log (with debug level logging):

!ENTRY org.eclipse.buckminster.core 0 293 2011-05-25 03:48:31.470
!MESSAGE Checking out file:///svnroot/modeling/org.eclipse.mdt.modisco/plugins/trunk/org.eclipse.gmt.modisco.kdm.browser.extension using revision HEAD
Checking out file:///svnroot/modeling/org.eclipse.mdt.modisco/plugins/trunk/org.eclipse.gmt.modisco.kdm.browser.extension using revision HEAD
!ENTRY org.eclipse.buckminster.core 0 293 2011-05-25 03:49:03.225
...

So, it means it took 32 seconds to checkout this single project (1.6 MiB, 634 files) locally.

But if I try it from the command line on build.eclipse.org, I see it takes less than 3 seconds:

nbros@build ~/test
$ time svn checkout -q file:///svnroot/modeling/org.eclipse.mdt.modisco/plugins/trunk/org.eclipse.gmt.modisco.kdm.browser.extension

real    0m2.792s
user    0m0.140s
sys     0m0.364s

So, the question I'm asking is why is Buckminster taking 11 times longer to checkout the same project?
Is that the subclipse or subversive connector used by Buckminster that is responsible for this?
In that case, I think I would be better off checking out the projects first, and then using the "local" Buckminster reader.

Comment 3 Thomas Hallgren

2011-05-25 05:02:54 EDT

(In reply to comment #2)
> So, the question I'm asking is why is Buckminster taking 11 times longer to
> checkout the same project?

Buckminster doesn't just "check out". It runs in two phases, resolution and materialization. During the first phase a discovery mechanism will fetch selected files from the remote sources and build the cspec based model of the transitive scope. This takes some time. In the second phase, everything is materialized (i.e checked out from SCM's or copied from p2 repositories). This is per design and should account for some of the time spent.

I have no idea why the checkout of one single project takes five times longer. The reason is probably that you're using different clients (native SVN versus Subversive or Subclipse). I suggest you try this checkout without Buckminster in your Eclipse IDE with Subversive or Subclipse installed. If you still can see a significant difference in performance, the perhaps it's worth bringing it up with one of those projects.

Having Hudson check out everything and then use a local reader is generally a lot faster when you know exactly what to check out beforehand. Especially if everything is under the same root. We already do this for our own 3.7 build. All discovery is then performed using a local reader.

I'm closing this bug since I don't think Buckminster is to blame.

Comment 4 Nicolas Bros

2011-05-25 05:38:06 EDT

Thank you for this explanation. I have followed your advice, and the import step went from taking ~40 minutes to ~3 minutes by using Hudson's checked-out sources!