Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 364980

Summary: Platform doc ships different binary content with identical version
Product: [Eclipse Project] Platform Reporter: Martin Oberhuber <mober.at+eclipse>
Component: RelengAssignee: Kim Moir <kim.moir>
Status: VERIFIED FIXED QA Contact:
Severity: normal    
Priority: P3 CC: daniel_megert, deepakazad, john.arthorne, kim.moir, markus.kell.r, pwebster
Version: 3.7.1   
Target Milestone: 3.7.2   
Hardware: All   
OS: All   
Whiteboard:
Bug Depends on: 364979    
Bug Blocks:    

Description Martin Oberhuber CLA 2011-11-28 12:40:07 EST
Build ID: Eclipse 3.7.1

Comparing the contents of Eclipse 3.7.1 against Eclipse 3.7.0 it turns out
that the

   org.eclipse.platform.doc.user_3.7.0.v20110601-0800.jar

bundle has different binary content although it has the identical version.
This is a problem which undermines the trust in incremental software delivery.

Investigation shows that this is 
(a) partially due to Lucene and the Pre-generated help index (bug 364979),
(b) partially due to the generated "build.xml" file being embedded.

While (a) may need some investigation, I'd guess that (b) should just be fixed; and, I'm wondering why the differing binary content has not been caught by the comparator, which could have just provisioned the original 3.7.0 bundle into 3.7.1 rather than provisioning a re-generated bundle with the same version (which is undesirable).
Comment 1 John Arthorne CLA 2011-11-28 13:21:28 EST
I believe the comparator ignores the doc bundles. Since they always have auto-generated content in the build the comparator would always indicate they are changed. Maybe in this case we should always prefer the old version already in the repository over the newly built version.
Comment 2 Kim Moir CLA 2011-11-28 14:21:44 EST
The older version is always preferred. Not sure how this happened.  My suspicion is that I cleaned the repo of old builds and a build with the new bundle was included.  Although, this shouldn't have happened because I always include the last build in the repo, I don't entirely clean the repo to avoid this exact problem.  In any case, I'll increment the version of the platform.doc.user bundle in the 3.7.2 stream.  

One approach to avoid this problem in the future would be to increment the version of this bundle every build.  I do this with the branding bundles by concatenating the buildId to the build.properties.  However, not sure if this is the right approach given that we don't want users to gratuitously download the doc bundles especially given their size.
Comment 3 John Arthorne CLA 2011-11-28 14:29:43 EST
platform ISV doc in particular is 35MB. We certainly don't want that version incremented every build.
Comment 4 Martin Oberhuber CLA 2011-11-28 14:36:57 EST
I think the correct fix would be in Lucene, making the doc / index builds fully reproducable (as per bug 364979), then the comparator could be re-enabled. I have also brought this up with Lucene on ASF Jira, let's see how they react.

For now, Kim's approach preferring the repo seems to be good.

On the ISV docs, I suppose these should be re-generated when any public Javadoc changes? Is this automated today or do we just manually touch the qualifier at some point or rebuild unconditionally?
Comment 5 John Arthorne CLA 2011-12-02 09:23:54 EST
*** Bug 365414 has been marked as a duplicate of this bug. ***
Comment 6 John Arthorne CLA 2011-12-02 09:29:13 EST
(In reply to comment #4)
> On the ISV docs, I suppose these should be re-generated when any public Javadoc
> changes? Is this automated today or do we just manually touch the qualifier at
> some point or rebuild unconditionally?

I think it's a case of this being a theoretical problem that doesn't come up in practice very often for releases (if ever). Usually doc is the last thing we are working on at the end of a release after code is frozen, so typically it is getting modified and tagged right at the end of a release cycle. So I imagine there are sometimes I-builds with stale javadoc, but in releases it wouldn't come up. Perhaps in maintenance release this could be an issue, although changes to API doc in maintenance releases is quite oncommon. Rather than tagging it automatically for every single build, I would opt for something like a release checklist where we make sure it is tagged at the end of the release cycle.

The generated help index case is more interesting to me. Ideally the index is a function of the input documentation, so it would change if and only if the input text changed. For the build.xml this sounds like something that just shouldn't be in our bin.includes.
Comment 7 Dani Megert CLA 2011-12-08 03:33:36 EST
This bug is really an edge case that got produced only because we switched to Git between 3.7 and 3.7.1 and then just used the same tag for all doc bundles in the same repo to simplify the build input. Using a new tag on the unchanged 'org.eclipse.platform.doc.user' caused it to be re-built with a different content. This won't happen in the future since we now use a script to auto-tag only the changed bundles.

>For the build.xml this sounds like something that just
>shouldn't be in our bin.includes.
+1.

We could increase the version number to cover this case but I'd rather leave it as is: the version (excluding the qualifier) stands for the content/input and not the produced bits. The fact that the produced bits are different does not imply that the version must be increased for the unchanged content.
Comment 8 Dani Megert CLA 2011-12-10 01:54:31 EST
> On the ISV docs, I suppose these should be re-generated when any public Javadoc
> changes? Is this automated today
No. I filed bug 366287 to investigate an automated solution.
Comment 9 Kim Moir CLA 2011-12-13 16:51:32 EST
Verified fixed in M20111212-1000