Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 319186

Summary: NPE in CPPSemantics.reduceToViable because of type marshalling error
Product: [Tools] CDT Reporter: Josh Kelley <joshkel>
Component: cdt-indexerAssignee: Markus Schorn <mschorn.eclipse>
Status: RESOLVED FIXED QA Contact: Markus Schorn <mschorn.eclipse>
Severity: normal    
Priority: P3 CC: yevshif
Version: 7.0   
Target Milestone: 7.0.1   
Hardware: PC   
OS: Windows 7   
Whiteboard:
Attachments:
Description Flags
Log file showing errors
none
Second log file showing different errors
none
Source files necessary to create errors in a default Eclipse project
none
testcase + fix mschorn.eclipse: iplog-

Description Josh Kelley CLA 2010-07-07 15:54:22 EDT
Build Identifier: 20100617-1415

Since upgrading to Helios, many of my files aren't being properly indexed.  I checked the .log file and found unmarshalling errors and NullPointerExceptions in CPPSemantics.reduceToViable.

I'm using Eclipse CDT to edit CodeGear C++Builder source files.  C++Builder uses several vendor extensions to C++, and these indexer problems seem to be more common in C++Builder source files, so it may be that one of these extensions is triggering the bug, but I've used previous versions of Eclipse CDT without any problems.

Strangely, if I comment out the #include's for CodeGear's header files, save my source file, let the indexer index it, then uncomment the #include's, everything appears to work.

Reproducible: Always

Steps to Reproduce:
1. Open a C++Builder project in Eclipse Helios?
2. Note that all method implementations can't resolve the class that they're methods of.
Comment 1 Josh Kelley CLA 2010-07-07 15:55:24 EDT
Created attachment 173705 [details]
Log file showing errors
Comment 2 Markus Schorn CLA 2010-07-08 07:34:51 EDT
I have reviewed the code and cannot find the error. I need to reproduce the problem to fix the issue. To find the construct that triggers the issue please do the following:
1. Put your project into the state where the error occurs.
2. Choose a file for which the error was reported (e.g. GasMix.cpp in the 
   attached log-file)
3. Whenever you edit the file and save it, a similar error should be logged
   again.
4. By commenting away function definitions and function declarations in the
   file, try to figure out which one triggers the bug.

As an alternative you can run the preprocessor on GasMix.cpp and attach the result, I might be able to reproduce the issue from there. (gcc -E -P <your include path and defines> GasMix.cpp)
Comment 3 Josh Kelley CLA 2010-07-08 12:41:38 EDT
I confirmed that I can open the same projects in Galileo with no errors.

I also noticed while working on this that Eclipse's memory usage can get VERY high - Galileo could happily index my entire project with a max heap space of 512MB, while Helios can throw out of heap errors with the max heap space set to 1024MB or higher.  I'm not sure if this is related or not.

I think that I've pinpointed the error (or at least one cause of the error).  From what I can tell, code similar to the following triggers the error:

  VersionLabel->Caption = String("Version ") + software_version;

String is a class provided by C++Builder (actually, a typedef for C++Builder's UnicodeString class).  software_version is a const char *.  String provides an operator+ that takes a const String&, and it provides a conversion constructor from const char*.  So the problematic line of code is implicitly invoking the conversion constructor and then calling operator+(const String&).

The implicit call to this conversion constructor seems to be what's causing the error.  I was also able to get an error with the following code:

void DoSomething(const String& s)
{
}

void Test()
{
  DoSomething("Hi");
}

I get different errors depending on whether I'm using String or UnicodeString (even though String is just a typedef for UnicodeString) and depending on whether I'm using the DoSomething test or the operator+ test.  I'll attach an updated log file showing both sets of errors.

I've tried making up my own sample class with a conversion constructor and using it in place of String, but I can't duplicate the error.  I assume there's something unique about UnicodeString's definition that's causing the problem, but I looked at C++Builder's definition of UnicodeString but don't see anything odd.

If I preprocess the file to include all header files, the error goes away.
(I had to munge the file a bit after preprocessing to get Eclipse to be happy with it; before munging, the file as 440,000 lines, due to all of the inclusions, so maybe I accidentally changed something...)

If I include <ustring.h> (which provides UnicodeString) but no other headers in my file, I get the error.

If I don't include any header files and instead copy the entire contents of <ustring.h> directly into my file, I don't get the error.

If I copy and paste only the UnicodeString definition from ustring.h and remove all includes (even the #include that provides the String typedef) but don't update my code to use UnicodeString instead of String, then Eclipse still recognizes String as a typedef for UnicodeString and still generates an error.  

C++Builder provides a reserved word __fastcall, which Eclipse doesn't recognize.  To work around this, I had added an __fastcall to Eclipse's C++ symbols with no value.  Strangely, if I remove __fastcall from Eclipse's C++ symbols, the error no longer occurs.  (But I suspect that the error may no longer occur because Eclipse is no longer able to parse some definitions or declarations involving __fastcall, so that's probably not a good solution.)

Any suggestions on where to proceed from here?
Comment 4 Josh Kelley CLA 2010-07-08 12:42:09 EDT
Created attachment 173787 [details]
Second log file showing different errors
Comment 5 Markus Schorn CLA 2010-07-09 03:44:40 EDT
(In reply to comment #3)
> ...
> If I include <ustring.h> (which provides UnicodeString) but no other 
> headers in my file, I get the error.
> 
> If I don't include any header files and instead copy the entire contents of
> <ustring.h> directly into my file, I don't get the error.
> 
> If I copy and paste only the UnicodeString definition from ustring.h and 
> remove all includes (even the #include that provides the String typedef) 
> but don't update my code to use UnicodeString instead of String, then 
> Eclipse still recognizes String as a typedef for UnicodeString and still 
> generates an error.  
> ...

That looks promising. What I need to reproduce the issue is the preprocessed version of the minimal example that is failing (even if the preprocessed version does not fail). It may also be sufficient to look into the example source plus the header 'ustring.h'. If you do not want to attach these files you can mail them directly to me.

The root problem is, that something is stored in the index in a way such that it cannot be used (the unmarshalling error). The error goes away when you paste the content of the header into the source because then there is no need to look up the information in the index.
Comment 6 Josh Kelley CLA 2010-07-12 10:56:43 EDT
Created attachment 174037 [details]
Source files necessary to create errors in a default Eclipse project
Comment 7 Josh Kelley CLA 2010-07-12 10:57:01 EDT
I attached two source files that generate the errors.  Thank you.
Comment 8 Markus Schorn CLA 2010-07-19 04:33:09 EDT
(In reply to comment #7)
> I attached two source files that generate the errors.  Thank you.

Thanks, I can reproduce the issue.
Comment 9 Markus Schorn CLA 2010-07-19 08:54:22 EDT
Created attachment 174611 [details]
testcase + fix
Comment 10 Markus Schorn CLA 2010-07-19 08:57:13 EDT
The fix requires a new incompatible index version. --> CDT 7.0.1 will rebuild the index.
Comment 11 Markus Schorn CLA 2010-07-19 08:57:40 EDT
Fixed in 7.0.1 and 8.0 > 20100719.
Comment 12 CDT Genie CLA 2010-07-19 09:23:07 EDT
*** cdt cvs genie on behalf of mschorn ***
Bug 319186: Marshalling char16_t and char32_t.

[*] IndexCPPBindingResolutionBugs.java 1.44 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt-core/org.eclipse.cdt.core.tests/parser/org/eclipse/cdt/internal/index/tests/IndexCPPBindingResolutionBugs.java?root=Tools_Project&r1=1.43&r2=1.44

[*] CBasicType.java 1.22 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt-core/org.eclipse.cdt.core/parser/org/eclipse/cdt/internal/core/dom/parser/c/CBasicType.java?root=Tools_Project&r1=1.21&r2=1.22

[*] CPPBasicType.java 1.23 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt-core/org.eclipse.cdt.core/parser/org/eclipse/cdt/internal/core/dom/parser/cpp/CPPBasicType.java?root=Tools_Project&r1=1.22&r2=1.23

[*] PDOM.java 1.152 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt-core/org.eclipse.cdt.core/parser/org/eclipse/cdt/internal/core/pdom/PDOM.java?root=Tools_Project&r1=1.151&r2=1.152

[*] CBasicType.java 1.21.2.1 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt-core/org.eclipse.cdt.core/parser/org/eclipse/cdt/internal/core/dom/parser/c/CBasicType.java?root=Tools_Project&r1=1.21&r2=1.21.2.1

[*] CPPBasicType.java 1.22.2.1 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt-core/org.eclipse.cdt.core/parser/org/eclipse/cdt/internal/core/dom/parser/cpp/CPPBasicType.java?root=Tools_Project&r1=1.22&r2=1.22.2.1

[*] PDOM.java 1.150.2.1 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt-core/org.eclipse.cdt.core/parser/org/eclipse/cdt/internal/core/pdom/PDOM.java?root=Tools_Project&r1=1.150&r2=1.150.2.1

[*] IndexCPPBindingResolutionBugs.java 1.41.2.3 http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.cdt-core/org.eclipse.cdt.core.tests/parser/org/eclipse/cdt/internal/index/tests/IndexCPPBindingResolutionBugs.java?root=Tools_Project&r1=1.41.2.2&r2=1.41.2.3