Bugzilla – Full Text Bug Listing
| Summary: | C/C++ indexer too slow | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Tools] CDT | Reporter: | Ed Burnette <ed.burnette> | ||||||||||||||
| Component: | cdt-parser | Assignee: | Bogdan Gheorghe <gheorghe> | ||||||||||||||
| Status: | RESOLVED FIXED | QA Contact: | |||||||||||||||
| Severity: | major | ||||||||||||||||
| Priority: | P2 | CC: | jave27, Jeff.Polzin, Mikhail.Voronin, mlists, nikolaymetchev, oleg.krasilnikov, sreinstein, weisen123+bugzilla | ||||||||||||||
| Version: | 2.0 | ||||||||||||||||
| Target Milestone: | 3.1 | ||||||||||||||||
| Hardware: | PC | ||||||||||||||||
| OS: | Windows XP | ||||||||||||||||
| Whiteboard: | |||||||||||||||||
| Attachments: |
|
||||||||||||||||
|
Description
Ed Burnette
Would it be possible for you to zip up and attach your workspace? I just downloaded CDT RC2 and it seems to be faster. After reinstalling CDT and restarting Eclipse I closed and reopened my project, which triggered an index rebuild which took 7 minutes and 10 seconds. The project is proprietary closed source so I can't zip it up for you, sorry. I added some more (hopefully the last of the) external header file directories and the it took 17 minutes and 30 seconds to rebuild the index with RC2. According to task manager it seems to be more I/O bound now, with only about 40% CPU utilization. The external header files are on a fast network drive. But a file monitor tool shows that the same headers are being searched, opened, and read over and over during the indexing process. We are planning a number of performance improvements for the next point release which will improve the overall speed of the indexing operation by caching common include files and improving I/O performance when creating the index. I think it is too slow anyhow,even with basic includes and a main file,eclipse will exhaust all CPU usage and lose response.I have to kill it to continue working with my PC. I would also add that it takes too much time to update the indexer data. I have a small sized project (~150 files). If I change something in one of them, it takes the indexer about 30sec-1min to update. With slow, it also ocuppies too much memory. In my system, the memory occupied is up to 600M, and increase continuely. My System: CPU: P4 2.4G Memory: 1024M OS: Windows 2000 Eclipse: 3.1.1 CDT: 3.0.1 Compiler: MinGW 3.1 C/C++ project only includes one main.cpp file, the C/C++ indexer occupies 99% CPU Usage and about 500M memory, and takes more than 5 minutes every time I change the source. Created attachment 32196 [details]
Job Manager delay elimination
Currently, JobManager.run() (see code below) waits 50 ms after _every_ background job finished.
May be it's reasonable in some cases (allow concurrent processes to run), but during indexation
we are spending this time after each separate file processed !
When I have commented this delay (as shown below), indexing process has finished much faster.
Saved time depends of project, but it is significant in any case:
~25% on "C" Midninght Commander source.
~45-50% on "C++" Mozilla source
It seems to be we can improve indexing performance either by removing this delay at all,
or by decreasing waiting time as possible.
Created attachment 32197 [details]
Test to check indexing performance
Simple indexing performance test to org.eclipse.core.tests.
This test can be used, in particular, to check a gain of
previous modification.
Created attachment 32198 [details]
Test to check indexing performance
Simple indexing performance test to org.eclipse.core.tests.
This test can be used, in particular, to check a gain of
previous modification.
Comment on attachment 32197 [details]
Test to check indexing performance
Zip file was added below as separate attach. Current patch is not required now.
Created attachment 32830 [details]
Test case for indexing performance.
The test case requires sample source files to index.
This sample data will be attached as separate zip file.
Created attachment 32831 [details]
Sample data for CDT index parformance test
File should be placed to org.eclipse.cdt.core.tests\resources\zips\
Created attachment 32832 [details]
Sample data (part 2) for CDT index performance test
File should be placed to org.eclipse.cdt.core.tests\resources\zips\
Also experiencing this issue on Ubuntu Breezy AMD64 with an AMD64 3400+ processor and 1GB RAM. Working on a project (Pingus) with about 1000 total files if you include the game engine (ClanLib). It takes *way* too long to reindex, and it does it just about every time I open the project. Slows the entire system down to a crawl. Specs: * Java JDK 1.5.0 AMD64 * Eclipse 3.1.1 * CDT 3.0.1 * Ubuntu Breezy with gcc 4.0.2 (if that matters) Let me know if you need any files. I've eventually had to go back to setting CDT to using "No Indexer", so it's usuable now. Thanks! This bug has been submitted long time ago and targeted to "Future" I propose to change target to CDT 3.0.2 and add all patches (or, at least, 1st one) to this release. Code patch itself is local, it affects 1 method only. Test case with sample data does not affect CDT execution flow. So it seems like it can be accepted in 3.0.2. Ed, you've raised this against the 2.0 stream, when it probably meant 3.0 stream. Do you want to change the version number to something slightly higher so that it gets on their radar? I'm guessing that unless it has 3.2 on it, it's not going to get addressed any time soon ... I think we're supposed to leave the version number alone so it represents when the problem was first noticed. BTW I'm not currently using CDT because of this problem and a few others (e.g., dealing with thousands of include files, partially instantiated playpens, and our wierd custom build system, etc.) so somebody else will have to say if the problem still exists in current versions, sorry. Since proposed fix is applicable to CDT 3.1, I have changed target. I agree that current problem is too global (applicable both to 2.0, 3.0, 3.1 CDT and so on). But, new its incarnations can be tracked by bugs. I propose to close this item after patches will be applied. It will be done today. Code patch is applied. Tests will be delivered soon. Test patch is applied today. Sample data also added. Pardon my ignorance but does the resolution of this bug solve bug 105803 and bug 81769? In particular as Bug 105803 points out it takes 7 hours to index the mozilla source. Has that figure changed much? Nikolay, you are right, mentioned bugs are related to current one. Moreover, bug 105803 is may be the better place for such fixes. But I was not aware of it when I've created patch in current bug. It's not nice but it cannot be redone. What about the problem resolution. I can say: "partially". There were some separate fixes related to indexer performance. Most of them are to be included to CDT 3.1. So we will see their cumulative effect. May be it will run twice as faster. May be by 50%, or even 20% only. But I hope that result be visible anyway. From the other hand, even triple or quadruple speed-up will not satisfy users completely: "My project indexation runs 2 hours ! It is terrible !" It's true even we have had 7 hours before. So, bug #105803 can be left open forever: we'll always have anything to speed-up in indexer. But current bug have no any reason to be still opened. May be it would be better to mark such bugs as duplicated. Moreover, PDOM implementation will made Full Indexer problems less actual. |