Bugzilla – Full Text Bug Listing
| Summary: | JDT and PDE User pre-built index is missing in recent N-builds | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | [Eclipse Project] Platform | Reporter: | Dani Megert <daniel_megert> | ||||||
| Component: | Releng | Assignee: | Platform-Releng-Inbox <platform-releng-inbox> | ||||||
| Status: | RESOLVED FIXED | QA Contact: | |||||||
| Severity: | normal | ||||||||
| Priority: | P3 | CC: | david_williams, markus.kell.r | ||||||
| Version: | 4.4 | ||||||||
| Target Milestone: | 4.4 M3 | ||||||||
| Hardware: | All | ||||||||
| OS: | All | ||||||||
| Whiteboard: | |||||||||
| Attachments: |
|
||||||||
|
Description
Dani Megert
How confident are we in those tests ... or, put another way, how can I tell definitively if index is present in a downloaded SDK? And, it's not that I have good news that its merely the tests are bad. I don't "see" any index in tonight's N-build. I downloaded http://www.eclipse.org/downloads/download.php?file=/eclipse/downloads/drops4/N20130930-2000/eclipse-SDK-N20130930-2000-linux-gtk-x86_64.tar.gz&mirror_id=1 And I don't see an "index page" for any of the "books" in the help ... as determined by bringing up help, and clicking on the "Index" tab in left navigator. (See attachment). If there's not errors in build log (and I have not looked) I wonder if things are getting "packaged" to the wrong place? Created attachment 236001 [details]
help dialog with PDE and Index selected.
Looking into the bundle org.eclipse.pde.doc.user I see files such as index/_0.frq index/segments.gen index/_0.fdx index/_0.tis index/_0.fdt index/indexed_contributions index/_0.tii index/_0.fnm index/_0.nrm index/indexed_dependencies index/segments_1 index/indexed_docs index/_0.prx The look "reasonable" (little that I know what they should look like) but one likely error is in index/indexed_contributions its content is = = = = = #This is a generated file; do not edit. #Mon Sep 30 21:06:57 EDT 2013 org.eclipse.pde.doc.user=org.eclipse.pde.doc.user\n3.10.0.qualifier = = = = = The fact that it ends with ".qualifier" looks pretty suspicious. And ... I think this confirms what the unit tests are saying These two bundles contain no "index" files at all: org.eclipse.jdt.doc.user_3.10.0.N20130930-2000.jar org.eclipse.platform.doc.user_4.4.0.N20130930-2000.jar And all three of these have the ".qualifier" in index/indexed_contributions org.eclipse.jdt.doc.isv_3.10.0.N20130930-2000.jar org.eclipse.platform.doc.isv_4.4.0.N20130930-2000.jar org.eclipse.platform.doc.user_4.4.0.N20130930-2000.jar But ... red herring ... I checked the I-build from M2, specifically org.eclipse.pde.doc.user_3.10.0.v20130918-1843.jar And its index/indexed_contributions also says merely "qualifier". org.eclipse.pde.doc.user=org.eclipse.pde.doc.user\n3.10.0.qualifier The mystery continues. To cross reference, see bug 417578 where we are tracking some "undo" stuff related to original attempts to fix bug 415008. Created attachment 236002 [details]
3 buildDoc_log.xml from 9/30 N build
Just wanted to capture the buildDoc_log.xml from an N-build. Apparently we only capture them for 3 doc bundles?
org.eclipse.jdt.doc.isv
org.eclipse.pde.doc.user
org.eclipse.platform.doc.isv
Don't recall why that was, but doesn't "match" exactly those that are failing to generate indexes or anything.
Nothing obvious jumped out from logs, to me, but ... I've not read them real closely.
Not sure how we ended up so inconsistent with the "app arguments" for building doc ... but, ironically, the two that do not have any index files, are the two that name "build.index" as an explicit ant target. I'm assuming we should strive for more consistency in what we call? Or, are there really different "functional" purposes for each? org.eclipse.jdt.doc.isv pom.xml 118: <appArgLine>-consolelog -debug -application org.eclipse.ant.core.antRunner -buildfile buildDoc.xml -Dbasedir.properties=cbi_basedirs.properties -logger org.apache.tools.ant.XmlLogger -verbose -logfile buildDoc_log.xml</appArgLine> org.eclipse.jdt.doc.user pom.xml 47: <appArgLine>-application org.eclipse.ant.core.antRunner -buildfile build.xml build.index</appArgLine> org.eclipse.pde.doc.user pom.xml 115: <appArgLine>-consolelog -debug -application org.eclipse.ant.core.antRunner -buildfile buildDoc.xml -Dbasedir.properties=cbi_basedirs.properties -logger org.apache.tools.ant.XmlLogger -verbose -logfile buildDoc_log.xml</appArgLine> org.eclipse.platform.doc.isv pom.xml 263: <appArgLine>-consolelog -debug -application org.eclipse.ant.core.antRunner -buildfile buildDoc.xml -Dbasedir.properties=cbi_basedirs.properties -logger org.apache.tools.ant.XmlLogger -verbose -logfile buildDoc_log.xml</appArgLine> org.eclipse.platform.doc.user pom.xml 47: <appArgLine>-application org.eclipse.ant.core.antRunner -buildfile customBuildCallbacks.xml build.index</appArgLine> I've added more (consistent) logging to those antRunner calls where we did not add it previously. http://git.eclipse.org/c/platform/eclipse.platform.common.git/commit/?id=fa52512e792541d10b05c63a8bf4215f8b939d05 I've also added more (consistent) prereqs to the version of "eclipse" we install for ant runner. http://git.eclipse.org/c/platform/eclipse.platform.common.git/commit/?id=3b0cf9e57c5ab91d54e4f9184804e53c53c12eb9 We may change this again, in future, once fate of bug 418392 is clarified. Reading to your comment, it seems you already answered all the questions you previously posted (like e.g. how can I see that the index is really missing, or, whether the tests are trustworthy). If not, please ask the open questions again. I agree that we want the doc generation to be consistent over all the 5 bundles. While the pom, build scripts and manifest currently differ between the doc bundles, it does not explain why the index is there in I-build works but not in the N-builds. (In reply to Dani Megert from comment #10) > While the pom, build scripts and manifest currently differ between the doc > bundles, it does not explain why the index is there in I-build works but not > in the N-builds. No question about that. But, thought there might be some "indeterminate behavior" introduced (such as a "change in build order"?). The improved logging might help spot the trouble more easily. In any case, while I do know now what the junit tests are looking for, I am still puzzled why even those with indexes were not showing up in the UI (for that one N-build I looked at). Is that to be expected? That they all have to be there for any to show up? Or, is that a separate issue? Or a hint that something deeper is wrong with indexing (for N-builds ... haven't looked at today's I-build yet)? The buildDoc_log.xml are available in I20131001-0800, so let's wait for the next N-build that should include the missing logs (for the first time, see comment 8). (In reply to David Williams from comment #11) > I am > still puzzled why even those with indexes were not showing up in the UI (for > that one N-build I looked at). David, I guess you have a wrong expectation for what these help indexes are. They are not an "Index" topic that should show up somewhere in the help TOC. The help indexes are Lucene indexes that are used to speed up full-text searches in help documents. The only effect you see in the UI is that the very first search in the infocenter only takes 1 s to collect the indexes from all installed doc bundles. In N-builds, the very first search takes several seconds longer until the indexes are built (depending on your hard disk speed). Note that we only pre-build an index for "en", so if your machine doesn't default to an English locale, then you have to launch Eclipse with "-nl en" to make use of the pre-built index. (In reply to Markus Keller from comment #12) > > David, I guess you have a wrong expectation for what these help indexes are. > They are not an "Index" topic that should show up somewhere in the help TOC. > > The help indexes are Lucene indexes that are used to speed up full-text > searches in help documents. The only effect you see in the UI is that the > very first search in the infocenter only takes 1 s to collect the indexes > from all installed doc bundles. In N-builds, the very first search takes > several seconds longer until the indexes are built (depending on your hard > disk speed). Note that we only pre-build an index for "en", so if your > machine doesn't default to an English locale, then you have to launch > Eclipse with "-nl en" to make use of the pre-built index. Thanks! (that helps a lot). I've just now finished an "N-build" on my local test machine, and all the doc bundles have indices in them. So, if production build tonight does not, first thing I'll do is study the "build repo" to see what is not getting "cleaned up" or "reset" from one build to another. In the worst case, we may have to remove something and let it re-clone ... unless it is yet some other even more obscure, subtle bug specific to eclipse.org infrastructure (which, would be surprising, sort of :). From downloading and visually inspecting tonight's N-build, it appears all indices are back. It is conceivable that specifying "runtime.compatibility" and "compatibility.state" in the "running Eclipse" might have helped these be produced correctly. But, as said earlier, that does not really explain why there would be a difference between I-builds and N-build. But, I did look at the build order (pasted below) and the ones that were missing indices are the first two built. <guessing ahead> While each module has its own "running eclipse" (in 'work' directory), there is part of each of those that points back to a common "localRepo" (via config.ini) ... so ... might be some effect of order ... and only "logical" thing that'd make sense for I-builds is that by chance it was correct when those bundles were built, and then "swapped in" by comparator ... but dates of I-build doc bundles (also below) don't really support that. = = = = N20130930-2000 (no indexes) 208 [INFO] org.eclipse.platform.doc.isv ...................... SUCCESS [2:47.868s] 209 [INFO] org.eclipse.jdt.doc.isv ........................... SUCCESS [40.379s] 210 [INFO] org.eclipse.jdt.doc.user .......................... SUCCESS [4.133s] 215 [INFO] org.eclipse.platform.doc.user ..................... SUCCESS [4.048s] 272 [INFO] org.eclipse.pde.doc.user .......................... SUCCESS [19.638s] N20131001-2000 (has indexes) 208 [INFO] org.eclipse.platform.doc.isv ...................... SUCCESS [2:50.098s] 209 [INFO] org.eclipse.jdt.doc.isv ........................... SUCCESS [54.937s] 210 [INFO] org.eclipse.jdt.doc.user .......................... SUCCESS [8.521s] 215 [INFO] org.eclipse.platform.doc.user ..................... SUCCESS [6.786s] 272 [INFO] org.eclipse.pde.doc.user .......................... SUCCESS [20.372s] versions from this morning's (10/1) I-build org.eclipse.platform.doc.isv_4.4.0.v20131001-0519.jar org.eclipse.jdt.doc.isv_3.10.0.v20130920-0916.jar org.eclipse.jdt.doc.user_3.10.0.v20130930-0447.jar org.eclipse.platform.doc.user_4.4.0.v20130806-1450.jar org.eclipse.pde.doc.user_3.10.0.v20130930-1512.jar I think http://git.eclipse.org/c/platform/eclipse.platform.common.git/commit/?id=3b0cf9e57c5ab91d54e4f9184804e53c53c12eb9 did the trick. The reason why it "worked" in the I-builds is because there were no changes in the those bundles recently and hence the old version which still had the index got used. The comparator reported the missing files (full log and doc log [1]) but it did not end up in the unanticipated log [2] and hence was not noticed. [1] http://download.eclipse.org/eclipse/downloads/drops4/I20130924-1130/buildlogs/comparatorlogs/buildtimeComparatorDocBundle.log.txt [2] http://download.eclipse.org/eclipse/downloads/drops4/I20130924-1130/buildlogs/comparatorlogs/buildtimeComparatorUnanticipated.log.txt Filed bug 418488 to fix that. |