Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 335142

Summary: [Doc] *.doc.*.jar bundles contain Lucene 1.9.1 instead of 2.9.1 search indexes
Product: [Eclipse Project] Platform Reporter: Holger Voormann <eclipse>
Component: RelengAssignee: Platform-Releng-Inbox <platform-releng-inbox>
Status: CLOSED FIXED QA Contact:
Severity: major    
Priority: P3 CC: cgold, daniel_megert, john.arthorne, Mike_Wilson
Version: 3.7   
Target Milestone: 3.7 M5   
Hardware: PC   
OS: Windows XP   
Whiteboard:

Description Holger Voormann CLA 2011-01-23 21:46:29 EST
In
- org.eclipse.jdt.doc.isv
- org.eclipse.jdt.doc.user
- org.eclipse.pde.doc.user 
- org.eclipse.platform.doc.isv
- org.eclipse.platform.doc.user 
the search index in the "index/" subfolder is compiled with Lucene 1.9.1 (see index/indexed_dependencies) instead of Lucene 2.9.1, which has been shipped since Eclipse 3.7 M3 (see bug 248986). Because the version of Lucene which with these indexes are created (1.9.1) does not match the runtime version (2.9.1), the prebuilt indexes are ignored: the search index will be created from scratch instead of merging the prebuilt indexes.

Due to the speed-up by Lucene 2.9.1 on my laptop the first query takes about 7 seconds without (or with wrong) prebuilt indexes to create the search index from scratch – not much more than the 3 or 4 seconds with the correct 2.9.1 indexes. Either the indexes should be compiled with Lucene 2.9.1 or removed (saves 2.6 MB but increases first query wait time from 4 to 7 seconds).
Comment 1 Dani Megert CLA 2011-01-24 04:28:26 EST
John, McQ, I guess we should do this.
Comment 2 John Arthorne CLA 2011-01-24 09:16:41 EST
(In reply to comment #1)
> John, McQ, I guess we should do this.

Yes. I think Chris G mentioned this in a call recently. Does anybody know how these indexes are being created? I would have thought this would happen "for free" after switching to building with the new lucene (the index must get created in each build or they would be incorrect).
Comment 3 Chris Goldthorpe CLA 2011-01-24 12:37:50 EST
I will also take a look at this.
Comment 4 Chris Goldthorpe CLA 2011-01-24 14:46:37 EST
It appears that the problem existed during M4 when we were building using a 1.9.1 version of Lucene. There is a trace option - org.eclipse.help/debug/search that turns on tracing when building the search indexes.

This is the output from Eclipse SDK 3.6.1

Search index for org.eclipse.jdt.doc.isv is prebuilt with path "index"
Search index for org.eclipse.jdt.doc.user is prebuilt with path "index"
Search index for org.eclipse.pde.doc.user is prebuilt with path "index"
Search index for org.eclipse.platform.doc.isv is prebuilt with path "index"
Search index for org.eclipse.platform.doc.user is prebuilt with path "index"
Building search index-  new docs: 2035, preindexed: 2020, remaining: 15
Milliseconds to update search index =  = 2937

This was the output from 3.7M4, note that it finds 0 preindexed docs and takes 9766 milliseconds

Search index for org.eclipse.jdt.doc.isv is prebuilt with path "index"
Search index for org.eclipse.jdt.doc.user is prebuilt with path "index"
Search index for org.eclipse.pde.doc.user is prebuilt with path "index"
Search index for org.eclipse.platform.doc.isv is prebuilt with path "index"
Search index for org.eclipse.platform.doc.user is prebuilt with path "index"
Building search index-  new docs: 2041, preindexed: 0, remaining: 2041
Milliseconds to update search index =  = 9766

And this is the output from I20110118-0800

Search index for org.eclipse.jdt.doc.isv is prebuilt with path "index"
Search index for org.eclipse.jdt.doc.user is prebuilt with path "index"
Search index for org.eclipse.pde.doc.user is prebuilt with path "index"
Search index for org.eclipse.platform.doc.isv is prebuilt with path "index"
Search index for org.eclipse.platform.doc.user is prebuilt with path "index"
Building search index-  new docs: 2043, preindexed: 1254, remaining: 789
Milliseconds to update search index =  = 5891

It appears that the problem is starting to go away as the doc bundles are built using 3.6M4, however platform.doc.user and jdt.doc.user  have not been rebuilt since M4 so the prebuilt indexes are not loading correctly.

John - can you run the release tool on all of the doc projects? ( I don't have commit rights to the jdt docs ).
Comment 5 John Arthorne CLA 2011-01-24 16:02:04 EST
(In reply to comment #4)
> John - can you run the release tool on all of the doc projects? ( I don't have
> commit rights to the jdt docs ).

It looks like they have all been tagged today.
Comment 6 Chris Goldthorpe CLA 2011-01-25 12:42:47 EST
I just tested using I20110124-1800 and the prebuilt indexes are now all working correctly. 

Building search index-  new docs: 2043, preindexed: 2028, remaining: 15

Resolving as FIXED.
Comment 7 Holger Voormann CLA 2011-01-28 01:55:22 EST
Verified in I20110127-2034-win32. Thanks.