Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 368160

Summary:	Improve treatment of dependent projects and external SDKs
Product:	[Tools] CDT	Reporter:	Markus Schorn <mschorn.eclipse>
Component:	cdt-indexer	Assignee:	Markus Schorn <mschorn.eclipse>
Status:	RESOLVED FIXED	QA Contact:	Markus Schorn <mschorn.eclipse>
Severity:	normal
Priority:	P3	CC:	cdtdoug, eclipse.sprigogin, eostroukhov, jens.elmenthaler, malaperle, mober.at+eclipse, yevshif, zeratul976
Version:	8.0
Target Milestone:	8.1.0
Hardware:	PC
OS:	Windows 7
Whiteboard:
Bug Depends on:
Bug Blocks:	351659

Description Markus Schorn

2012-01-09 08:58:33 EST

Current approach:
=================
When indexing a project the indexes of the dependency projects (as specified via project references) are reused. An external SDK is treated equally to a dependency project.

This approach has a few problems:
(1) The indexer has to perform lookups in a composite index that combines all dependency projects. I don't have experience with such a setup myself, however the layer is not very efficient. Bug 351659 comment 7 gives some indication that a sufficient number of project references kills the perfomance of the indexer.
(2) It is tricky to understand which configuration is used to index a header file that can be found in a dependency project. Call the main project 'A' and its dependent project 'B'. Normally, the header is indexed in the context of 'B' and sources from A in the context of A. However, if there is no source file in B, that also includes the header, then depending on the user's preference settings the header is not indexed in the context of B. In this case it will be indexed in A with the configuration taken from there. 
(3) The approach is not deterministic. Depending on the order in which files are added to the projects and when include statements are added a header file files may be indexed in the dependent project or taken directly from the dependency project.
(4) The approach requires us to index projects before their dependencies. We break this rule when indexing files from the editor with high priority, which is another source for non-deterministic indexing behavior. Also the sorting makes it difficult to parallelize the indexing accross multiple projects.


Alternate suggestion:
=====================
I suggest to change the above behavior to the following:
  * Index all projects independently
  * For the purpose of content-assist, navigation and add-include use the 
    composite index, that includes the dependent projects.
  * For other tools (include browser, call-hierarchy, ...) use the entire index,
    just as it is done today.

This approach will duplicate header information in every dependent prjoject that uses the header. (This is similar to how information about system headers is duplicated in each project that uses the system header). While this increases the size of the indexes, the duplication has the advantage that the heades can be indexed differently depending on the project they are used in.

Comment 1 Sergey Prigogin

2012-01-09 13:06:40 EST

Makes perfect sense to me.

Comment 2 Doug Schaefer

2012-01-10 10:32:24 EST

If I remember correctly, that's how the PDOM indexer worked originally, no?

Either way, +1 from me.

Comment 3 Markus Schorn

2012-01-11 01:37:42 EST

Sergey and Doug, thanks for your feedback.

(In reply to comment #2)
> If I remember correctly, that's how the PDOM indexer worked originally, no?
> Either way, +1 from me.
It did prior to CDT 4.0, when we did not have the ability to combine PDOMs at all. Andrew F.'s work on external SDKs and composite indexes gave us more flexibility. The decision to use this flexibility for indexing dependent projects was probably not the best one.

Comment 4 Nathan Ridge

2012-01-12 18:21:09 EST

(In reply to comment #0)
> Current approach:
> =================
> When indexing a project the indexes of the dependency projects (as specified
> via project references) are reused. An external SDK is treated equally to a
> dependency project.

What qualifies as an external SDK? For example, if I use the Boost libraries from several projects, but I don't have a project for the Boost libraries themselves, are they considered an external SDK?

Comment 5 Markus Schorn

2012-01-24 03:52:37 EST

(In reply to comment #4)
> What qualifies as an external SDK? For example, if I use the Boost libraries
> from several projects, but I don't have a project for the Boost libraries
> themselves, are they considered an external SDK?
No, the header files that are (possibly indirectly) included from a file in your project will be indexed although they are external to the workspace. 
However, you could create a pre-built index for the boost library, which needs to be registered via an CDT extension point, before it can be used. It then provides index information for workspace external files, even if they are not included from a source in a project.

Comment 6 Markus Schorn

2012-01-24 04:06:09 EST

Sergey, I think all of the contributed fragments (PDOM and non-PDOM) should get the same treatment. If there needs to be a difference in where a fragment wants to participate, I want to make this part of the extension point. With that you could register any fragment to for instance participates in call-hierarchy, but not in content assist. Because I am lazy I want to limit the possibilities to the tools where we actually need to make a difference. The tools that need to be able to work with contributed fragments are:

Search, Call Hierarchy, Type Hierarchy, Include Browser, Content Assist, Navigation, Add Import. (Did I miss a tool?)

If you do want your extension fragments not to be used for any of the tools above, please tell me so. I'll create the possibility to exempt them via an attribute in the extension point.

Comment 7 Sergey Prigogin

2012-01-24 13:08:38 EST

(In reply to comment #6)
The proposal makes a lot of sense. It would be nice to cover all the tools that you listed. I can make changes to all remaining tools after you change one or two.

Comment 8 Markus Schorn

2012-01-25 05:37:25 EST

(In reply to comment #7)
> (In reply to comment #6)
> The proposal makes a lot of sense. It would be nice to cover all the tools that
> you listed. I can make changes to all remaining tools after you change one or
> two.

Actually it does not make much difference whether to allow the customization for one of more tools, I'll do them all.

Comment 9 Sergey Prigogin

2012-01-25 10:53:13 EST

(In reply to comment #8)
Refactoring should be added to the list of tools.

Comment 10 Markus Schorn

2012-01-27 05:13:27 EST

I have committed my changes.

You may want to revise my change to CIndex.findNames(...):
Because the indexer can supersede an SDK when a header is included I have readded the behavior that IIndex.findNames(...) skips names in SDKs when a name in pdom of the indexer has been found. This is not a perfect solution (names should also be skipped when just the file is present in a pdom of the indexer, however I did not come up with an efficient way of detecting that). Also, you may not want this for your fragments, in which case I offer to make it configurable.

In case this work does not match your needs, go ahead and change it.

(In reply to comment #9)
> Refactoring should be added to the list of tools.
I did not touch refactoring, you can add it if it is necessary.

Comment 11 CDT Genie

2012-02-29 13:22:48 EST

*** cdt git genie on behalf of Markus Schorn ***

    Bug 368160: Dependent projects and SDKs.

[*] http://git.eclipse.org/c/cdt/org.eclipse.cdt.git/commit/?id=fa55d3662d571d16f7c82de3574168d6c0971f04

Comment 12 Markus Schorn

2012-03-21 10:12:04 EDT

My changes work as they are, if necessary additional options can be added.