Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 327176

Summary: several features fail comparator tests
Product: [WebTools] WTP Source Editing Reporter: David Williams <david_williams>
Component: wst.sseAssignee: David Williams <david_williams>
Status: RESOLVED FIXED QA Contact: Nitin Dahyabhai <thatnitind>
Severity: blocker    
Priority: P3 CC: keith.chong.ca
Version: unspecified   
Target Milestone: ---   
Hardware: PC   
OS: Windows 7   
Whiteboard:

Description David Williams CLA 2010-10-07 01:28:34 EDT
These test results are not published yet, (still running) but peeking on server I can see the xml_ui and xml_sdk features need to be retagged. Full comparator messages are below, but they basically say the xml_ui and xml_sdk features have exact same version numbers in M2 and this weeks's I-build, but different contents. That of course is a big no-no and we need to focus on not delivering (even declaring) builds that have this violation. If people install based on features, they won't update their xml_ui one, even though there is new contents. This general issue is caused by bug 208143 and is further described in bug 322601. Essentially, the theory goes, a small change in a bundle's version get's lost (truncated) in the feature's suffix. Hence, simply retagging the features to today's date will make them distinct from M2 versions. Once we confirm the releng test is valid ... and the contents really have changed from M2 :) Unfortunately, there's no easy way to tell (from these results) which bundle inside the feature changed since M2 ... but, hopefully SSE team will know right off if any changed, thus confirming the releng test. 

I've marked as blocker simply because technically we shouldn't let such an identical-but-changed feature out into the wild ... and we should respin to pick up a fix. 







 13 Message 2
 14 canonical: org.eclipse.update.feature,org.eclipse.wst.xml_sdk.feature,3.3.0.v201007311522-7A78-8DXJQUlJHRDD2LBB_qiiymz
 15 Difference found for canonical: org.eclipse.update.feature,org.eclipse.wst.xml_sdk.feature,3.3.0.v201007311522-7A78-8DXJQUlJHRDD2LBB_qiiymz between file:/home/data/httpd/download.eclipse.org/webtools/downloads/d    rops/R3.3.0/S-3.3.0M2-20100923155521/repository/ and file:/shared/webtools/projects/wtp-R3.3.0-I/workdir/I-3.3.0-20101007023510/buildrepository/wst-sdk
 16 The entry "Feature: org.eclipse.wst.xml_ui.feature 3.3.0.v201007311522-7H7DFYzDxumThWc9oigOk5b6p2Mb" is not present in both features.
 17 The entry "Feature: org.eclipse.wst.xml_ui.feature.source 3.3.0.v201007311522-7H7DFYzDxumThWc9oigOk5b6p2Mb" is not present in both features.
 18
 19 Message 3
 20 canonical: org.eclipse.update.feature,org.eclipse.wst.ws_ui.feature,3.3.0.v201008130125-7I7AFbIEtEoUAjkBlK9msVeS4ulj
 21 Difference found for canonical: org.eclipse.update.feature,org.eclipse.wst.ws_ui.feature,3.3.0.v201008130125-7I7AFbIEtEoUAjkBlK9msVeS4ulj between file:/home/data/httpd/download.eclipse.org/webtools/downloads/dro    ps/R3.3.0/S-3.3.0M2-20100923155521/repository/ and file:/shared/webtools/projects/wtp-R3.3.0-I/workdir/I-3.3.0-20101007023510/buildrepository/jst-sdk
 22 The entry "Feature: org.eclipse.wst.xml_ui.feature 3.3.0.v201007311522-7H7DFYzDxumThWc9oigOk5b6p2Mb" is not present in both features.
Comment 1 David Williams CLA 2010-10-07 01:33:24 EDT
Oh, rereading message, I see it involves wst.ws_ui.feature also .. because of the xml_ui feature. Fixing xml_ui might fix them all (since then would probably change the suffix) ... or, they could all be retagged to play it safe.
Comment 2 David Williams CLA 2010-10-07 01:46:41 EDT
Here's the analysis and recommendations I have so far. 

I looked at M2 page to infer tagged version of maps in M2, namely *20100923155521. 

I compared the HEAD version of source editing maps with that version and see the only ones changed (related to xml_ui feature) are actually in xml_core feature: 
xml.core and sse.core bundles. 

So, I think what has happened is that the xml_core feature has changed its suffix (since didn't show up in comparator test), but just changed a small amount, and that amount is not sufficient to "drive" a change up to the xml_ui feature. Hence xml_ui feature has same version, but they will have a slightly different version of xml_core feature in it.
Comment 3 David Williams CLA 2010-10-07 02:13:32 EDT
Oh ... and did I mention one significant aspect of our new build procedure ... the way it works, is if the p2 finds identical versions, it always uses the old one and ignores the new one in creating our final repo and zip files. So, good news is, I guess we don't have to always consider blocking .... we are not really delivering the bad one ... we just are not delivering a new one like we planned (which, that would also often be blocking, but not always). 

I have confirmed the analysis above, but see I didn't go far enough.  
Indeed, xml_core feature is different in the two xml_ui features as expected, 

   <includes
         id="org.eclipse.wst.xml_core.feature"
         version="3.3.0.v201007311522-7C7OFchF7RZHQHIAQ3MuPd"/>

   <includes
         id="org.eclipse.wst.xml_core.feature"
         version="3.3.0.v201007311522-7C7OFchF7RZHQHIAN-NmPT"/


And, here's the surprise, this different was enough to make the xml_ui features different versions: 

3.3.0.v201007311522-7H7DFYzDxumThWc9oigOk5b6p2Mb
3.3.0.v201007311522-7H7DFYzDxumThWc9oigOk5b6lz0N

The "loss" doesn't come until the xml_sdk feature, and there the version ids really are exacty the same, even though they have different xml_ui features in them: 

org.eclipse.wst.xml_sdk.feature_3.3.0.v201007311522-7A78-8DXJQUlJHRDD2LBB_qiiymz
org.eclipse.wst.xml_sdk.feature_3.3.0.v201007311522-7A78-8DXJQUlJHRDD2LBB_qiiymz


Don't you just love losey hash functions :) 

I'm assuming something similar is happening in wst.ws case,
Comment 4 David Williams CLA 2010-10-07 02:16:41 EDT
So, here's what I suggest ... that we re-tag the xml_core feature with "today's date" to make it very distinct (since that is where the changes really are, it seems kind of logical to retag them). and then rebuild and see if that change is large enough to then cause the hashing/suffix algorithms to work better in xml_ui and xml_sdk and wst.ws. My guess it it will be sufficient. 

Let's see ... where can we find a source editing committer this late at night :)
Comment 5 David Williams CLA 2010-10-07 02:38:28 EDT
I've tagged the xml_core feature file, even though no change in it, per se ... had to remember to uncheck the "tag only if changed" checkbox. 

I restarted the build so this change can be ready for Thursday's smoke test.
Comment 6 David Williams CLA 2010-10-07 09:07:15 EDT
My fix to xml_core did fix the xml_sdk problem, but, not wst.ws_ui. In fact, a couple of others showed up on the respin ... not sure if that's because xml_core was fixed, or if other changes were released last minute. See full list of the three below. 

Part of the problem might be that xml is one of our overly nested features so a small change there would tend to be reflected (or not) in many other features that "include" it. We may want to make it a high priority to "flatten" (see bug 297398).

Rather than go through and do a detailed analysis of each new failure, I'd prefer to go ahead and "touch" each feature that failed to have its suffix incremented properly by the suffix-hash heuristic, just to minimize the chances for yet another respin. 

In fact, I have "touched" the three features below. I will let current test run finish, but then rebuild before asking for a smoke tests. 

= = = = 

canonical: org.eclipse.update.feature,org.eclipse.jst.web_sdk.feature,3.3.0.v201008101217-7B7A-99ZJTVcioRVwaTYBi5dUz0v
Difference found for canonical: org.eclipse.update.feature,org.eclipse.jst.web_sdk.feature,3.3.0.v201008101217-7B7A-99ZJTVcioRVwaTYBi5dUz0v between file:/shared/webtools/committers/wtp-R3.3.0-I/20101007023510/I-3.3.0-20101007023510/repository/ and file:/shared/webtools/projects/wtp-R3.3.0-I/workdir/I-3.3.0-20101007063954/buildrepository/jst-sdk
The entry "Feature: org.eclipse.jst.web_ui.feature 3.3.0.v201008101217-7F7AFMTC25Tne2z0okhpVuy3vkJk" is not present in both features.
The entry "Feature: org.eclipse.jst.web_ui.feature.source 3.3.0.v201008101217-7F7AFMTC25Tne2z0okhpVuy3vkJk" is not present in both features.


canonical: org.eclipse.update.feature,org.eclipse.wst.web_core.feature,3.3.0.v201008062100-7E7EFKtAJtn6xjw1vNVG6NXvS8Lr
Difference found for canonical: org.eclipse.update.feature,org.eclipse.wst.web_core.feature,3.3.0.v201008062100-7E7EFKtAJtn6xjw1vNVG6NXvS8Lr between file:/shared/webtools/committers/wtp-R3.3.0-I/20101007023510/I-3.3.0-20101007023510/repository/ and file:/shared/webtools/projects/wtp-R3.3.0-I/workdir/I-3.3.0-20101007063954/buildrepository/wst-sdk
The entry "Feature: org.eclipse.wst.xml_core.feature 3.3.0.v201010070622-7C7OFchF7RZHQHIAQ3MuPd" is not present in both features.


canonical: org.eclipse.update.feature,org.eclipse.wst.ws_ui.feature,3.3.0.v201008130125-7I7AFbIEtEoUAjkBlK9msVeS4ulj
Difference found for canonical: org.eclipse.update.feature,org.eclipse.wst.ws_ui.feature,3.3.0.v201008130125-7I7AFbIEtEoUAjkBlK9msVeS4ulj between file:/home/data/httpd/download.eclipse.org/webtools/downloads/drops/R3.3.0/S-3.3.0M2-20100923155521/repository/ and file:/shared/webtools/projects/wtp-R3.3.0-I/workdir/I-3.3.0-20101007063954/buildrepository/jst-sdk
The entry "Feature: org.eclipse.wst.xml_ui.feature 3.3.0.v201007311522-7H7DFYzDxumThWc9oigOk6T6v2Mb" is not present in both features.
Comment 7 David Williams CLA 2010-10-07 12:53:15 EDT
I was slightly incorrect in previous comment. Only the last one, wst.ws_ui.feature was a true problem and needed to be touched. 

The other two cases were actually showing "violations" with previous I-builds (which had never been promoted) and not the reference M2 build. 

I suspect it'll be nearly impossible to be pure each and every build, and should probably change scripts to stick to using only declared builds as "references". 

I say this because the rebuild flagged more things as same version, different contents, but each case was just against a previous I-build. I suspect it'll be hard to get "perfection" without tagging every feature ... but ... maybe there's some common pattern .. such as, if we touch one part of a "set" (such as "core") then touch all parts, UI and SDK ... and anything else that _includes_ them?