Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 325181

Summary: We should use "comparator" to detect changed bundles
Product: [WebTools] WTP Releng Reporter: David Williams <david_williams>
Component: relengAssignee: David Williams <david_williams>
Status: RESOLVED FIXED QA Contact: David Williams <david_williams>
Severity: enhancement    
Priority: P3 Keywords: plan
Version: 3.10   
Target Milestone: 3.10.0   
Hardware: PC   
OS: Windows 7   
See Also: https://bugs.eclipse.org/bugs/show_bug.cgi?id=322601
https://bugs.eclipse.org/bugs/show_bug.cgi?id=325158
https://bugs.eclipse.org/bugs/show_bug.cgi?id=325311
https://bugs.eclipse.org/bugs/show_bug.cgi?id=326018
Whiteboard:
Bug Depends on:    
Bug Blocks: 322601    
Attachments:
Description Flags
example comparator output none

Description David Williams CLA 2010-09-13 19:15:17 EDT 
There are some tools/techniques that can be used during a build (or mirror operation) to detect if a bundle has changed, but its version (even qualifier) has not changed).  We should add this functionality to our builds. 

For more information, see, for example, 
http://aniefer.blogspot.com/2009/08/versioning-p2-slides-from-eclipsecon.html

I'll attach a sample log I've obtained in some early testing. 

We'll have to add some "releng tests" to "fail" if unexpected errors show up in a log (and/or, improve "comparator" not to fail/log certain types of errors, such differences in "about.mappings". 

One complication of using this process, is that there can be cases where what is "in the latest build" ends up not being what is delivered. For example, if someone uses a final static constant from some other pre-req package, this will be "inlined" into the bundle. If that prereq bundles happens to change the value of that constant, then it will not be required for the consumer to increment their version (or, at least, retag with current date/time) so "new constant" version will appear newer, and be picked up, instead of having exact same version as old constant version, so p2 and installers will not know it was supposed to get a new version.
Comment 1 David Williams CLA 2010-09-13 19:20:44 EDT 
>> then it will not be required for the consumer to increment their version

should have been it "will be required".
Comment 2 David Williams CLA 2010-09-13 19:23:35 EDT 
In short, the thing to remember is, we version our source (and that sets bundle version) ... but other things can effect the finally built byte codes. 

The advantage, of course, is that we need to detect these situations anyway, at some point, or else things will mysteriously go wrong with some updates or installs. 

so, this new process should help us find it earlier, before release.
Comment 3 David Williams CLA 2010-09-13 20:12:34 EDT 
Created attachment 178782 [details]
example comparator output

Most of these "errors" are about the "about.mappings" file. It is normal for that to change from build to build, and infact would be good to not "update" a bundle just based on the date it was built alone. At first, I was thinking it might be good to get ride of these about mappings files ... we don't especially use them ... but conversely, it might actually be good to this this record or "build date" inside the jar, especially if we don't update/ship new ones, if build date was only thing that changed. (for bugs related to these "poor" messages, see bug 325067 and bug 325134).

So, minus the about.mappings changes, there's only 4 "issues": 

Difference found for packed: osgi.bundle,org.eclipse.wst.xsl.sdk.documentation,1.0.100.v201006012005 between file:/home/data/httpd/download.eclipse.org/webtools/downloads/drops/R3.2.1/R-3.2.1-20100730021206/repository/ and file:/home/data/httpd/download.eclipse.org/webtools/downloads/drops/R3.2.2/M-3.2.2-20100910064233/repository/
	Binary file build.xml: sizes differ by 15 bytes.

Difference found for canonical: osgi.bundle,org.eclipse.wst.xsl.sdk.documentation,1.0.100.v201006012005 between file:/home/data/httpd/download.eclipse.org/webtools/downloads/drops/R3.2.1/R-3.2.1-20100730021206/repository/ and file:/home/data/httpd/download.eclipse.org/webtools/downloads/drops/R3.2.2/M-3.2.2-20100910064233/repository/
	Binary file build.xml: sizes differ by 15 bytes.

Difference found for canonical: osgi.bundle,org.eclipse.wst.jsdt.core.tests.model,1.0.301.v201007080225 between file:/home/data/httpd/download.eclipse.org/webtools/downloads/drops/R3.2.1/R-3.2.1-20100730021206/repository/ and file:/home/data/httpd/download.eclipse.org/webtools/downloads/drops/R3.2.2/M-3.2.2-20100910064233/repository/
	IOException comparing /tmp/sourceartifactworkspace_JavaSearch_corrupt_jar6259798808892428264.jar and /tmp/sourceartifactworkspace_JavaSearch_corrupt_jar3276306043517706990.jar 
	error in opening zip file

Difference found for packed: osgi.bundle,org.eclipse.wst.jsdt.core.tests.model,1.0.301.v201007080225 between file:/home/data/httpd/download.eclipse.org/webtools/downloads/drops/R3.2.1/R-3.2.1-20100730021206/repository/ and file:/home/data/httpd/download.eclipse.org/webtools/downloads/drops/R3.2.2/M-3.2.2-20100910064233/repository/
	IOException comparing /tmp/sourceartifactworkspace_JavaSearch_corrupt_jar7150382796895120839.jar and /tmp/sourceartifactworkspace_JavaSearch_corrupt_jar5255069612613771745.jar 
	error in opening zip file

two of those, are obviously test cases, where jars/zips can not be opened. 

The other two, might deserve some attention. I don't think a "build.xml" file should normally be "in" a delivered bundle anyway, but doubt its actually hurting anything.
Comment 4 David Williams CLA 2010-10-13 14:18:47 EDT 
I've implemented this new process in our scripts. Probably needs a little fine tuning so will leave this bug open a bit longer. 

Note I have written up a general description in on of our WTP Build wiki pages: 

http://wiki.eclipse.org/WTP/Build/Accounting_for_history_during_builds

(which also needs a little more fine tuning :)
Comment 5 David Williams CLA 2011-05-17 14:32:59 EDT 
fixed long ago