Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 367551

Summary: Scalability improvement in BuilderParticipant
Product: [Modeling] TMF Reporter: Vladimir Piskarev <vpiskarov>
Component: XtextAssignee: Project Inbox <tmf.xtext-inbox>
Status: CLOSED FIXED QA Contact:
Severity: enhancement    
Priority: P3 CC: mckinley1411, sebastian.zarnekow, sven.efftinge
Version: 2.3.0Flags: sebastian.zarnekow: juno+
Target Milestone: M5   
Hardware: PC   
OS: Windows XP   
Whiteboard:
Attachments:
Description Flags
Proposed patch
none
Updated patch sven.efftinge: iplog+

Description Vladimir Piskarev CLA 2011-12-26 08:11:46 EST
Build Identifier: 2.2.1

Profiling spotted a scalability bottleneck in BuilderParticipant, mainly due to its use of DerivedResourceMarkers#findDerivedResources (which is called for each delta and is really slow when the number of derived resources is an order of thousands). Please see the patch attached which aims to provide near-linear scalability and, together with other remedies (but being a major contributor), has allowed to reduce (clean-) build time from 36 minutes to about 4 minutes for a project containing 5000 Xtext-based 'entities' (with a Java file generated for each 'entity').

Reproducible: Always
Comment 1 Vladimir Piskarev CLA 2011-12-26 08:14:26 EST
Created attachment 208803 [details]
Proposed patch
Comment 2 Vladimir Piskarev CLA 2012-01-10 05:55:22 EST
Please note that there is a bug in the patch: we should check for marker existence in DerivedResourceMarkers#findDerivedResources(List<IMarker> generatorMarkers, String source) method, i.e.

    if (marker.exists() && (source == null || source.equals(marker.getAttribute(...
Comment 3 Sebastian Zarnekow CLA 2012-01-10 05:58:32 EST
Could you please update the patch. Thanks!
Comment 4 Vladimir Piskarev CLA 2012-01-10 09:02:55 EST
Created attachment 209259 [details]
Updated patch

Sure, here it is.
Comment 5 Sven Efftinge CLA 2012-01-23 09:05:39 EST
pushed to master. I did some changes please verify if it's still what you were up to.
Comment 6 McKinley CLA 2012-01-23 13:33:16 EST
Will this fix speed up to the code generation of individual Xtend files? I have found the code gen to be longer than a 2 or 3 seconds in some cases depending on the class and what Eclipse is doing. It can be noticeable.
Comment 7 Vladimir Piskarev CLA 2012-01-24 02:38:46 EST
(In reply to comment #5)
> please verify if it's still what you were up to

Sven, I believe calling derivedResourceMarkers.findDerivedResourceMarkers (BuilderParticipant.java, line 135) instead of the proposed derivedResourceMarkers.findGeneratorMarkers changes semantics somewhat. The latter additionally filters the markers by generator id. Perhaps it's #findDerivedResourceMarkers(IResource file, String generatorId) that was meant to be used?

Also the methods #findGeneratorMarkers(IContainer container), #findGeneratorMarkers(IContainer container, String generator) and #findDerivedResources(List<IMarker> generatorMarkers, String source) introduced by the patch seem irrelevant (and are never called indeed) after your changes.

By the way, bug 367564 is even easier to fix and might provide speed up (although less dramatic) especially for Xtend2 (its ClasspathBasedChecks validator can be really slow on large projects without the fix) ;-)
Comment 8 Vladimir Piskarev CLA 2012-01-24 03:38:23 EST
(In reply to comment #6)
> Will this fix speed up to the code generation of individual Xtend files?

First and foremost, the fix is about scalability and is very noticeable on clean builds of large projects (thousands of files). I think it may be much less noticeable on incremental builds of individual files (unless it's a really large project perhaps), but honestly has no objective data about it.
Comment 9 Sven Efftinge CLA 2012-01-24 04:15:16 EST
Thank you for verifying. I changed the call to also pass the generator id.
Comment 10 McKinley CLA 2012-01-25 01:30:17 EST
I tested with the latest integration build of Eclipse 3.8 and the Xtend nightly after

http://git.eclipse.org/c/tmf/org.eclipse.xtext.git/commit/?id=c96be95f8573fabf2c11a340a7a403782028a2f5

The change or something in 3.8 (or not having m2eclipse installed :) made the incremental build of Xtend files feel closer to plain JDT for classes of similar method size and count.
Comment 11 Vladimir Piskarev CLA 2012-01-25 02:32:42 EST
(In reply to comment #10)
> I tested with the latest integration build of Eclipse 3.8 and the Xtend nightly
> after
> 
> http://git.eclipse.org/c/tmf/org.eclipse.xtext.git/commit/?id=c96be95f8573fabf2c11a340a7a403782028a2f5

Thanks for sharing your results. If possible it would be interesting to try on a previous nightly build of Xtext (without m2eclipse :) to know if it's the fix that provided speed up or something else.
Comment 12 McKinley CLA 2012-02-01 05:34:44 EST
I have not taken the time to test against Indigo. I see a more pressing performance problem. I have a medium-sized class and hover over variables, members, extension method, etc. Pauses the entire editor for several seconds. Here is a screen cast where Visualvm only exaggerates the pauses slightly.

http://vimeo.com/36008963

Thanks
Comment 13 Karsten Thoms CLA 2017-09-19 16:55:51 EDT
Closing all bugs that were set to RESOLVED before Neon.0
Comment 14 Karsten Thoms CLA 2017-09-19 17:06:44 EDT
Closing all bugs that were set to RESOLVED before Neon.0