Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 441889

Summary:	refactor "production test scripts"
Product:	[Eclipse Project] Platform	Reporter:	David Williams <david_williams>
Component:	Releng	Assignee:	David Williams <david_williams>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	P3
Version:	4.5
Target Milestone:	4.4.1
Hardware:	PC
OS:	Linux
Whiteboard:
Bug Depends on:
Bug Blocks:	374441, 437069

Description David Williams

2014-08-15 14:50:26 EDT

As I've looked into enabling running our performance tests on Hudson, I was repeatedly "blocked" by problems in our "production scripts" which run the unit tests on multiple platforms, multiple Hudson's, etc. and as I fixed things and looked deeper realized what bad shape they are in simply due to years of "hacks and patches" BEFORE moving to eclipse.org hardware, Hudson, Git, etc., which caused many more "hacks and patches", rapidly made, "just to get things working". 

And, the good news is, with some refactoring and clean-up, they can be made much simpler, more "reusable", more reliable, with potential for more capability.

So, while we could have gotten the performance tests running with even more "hacks and patches" we would be making the problems even worse and be even more tightly tied to running in only one environment that others would have a hard time duplicating. 

This last part, about others being able to run the tests, is essential for the general effort of "getting the performance tests" running again, since as I mentioned several years ago, with the early prototypes, some of the performance tests simply do not run any longer (or, hang, or something) so other committers and community must have the ability to run them locally, to figure out if they can or should be fixed, or just remove the "performance test" marker. 

My plan is to start introducing the changes to "production tests scripts" soon ... which, does have the potential to (temporarily) break our regular unit tests due to me either moving the changes to Git, or (more likely) subtle differences between Linux, windows, and macs.  

And longer term, some of these changes will "trickle down" to the "packaged" scripts that are part of eclipse-Automated-Tests-*.zip, which in turn will take some changes to the documentation for running them, at 
https://wiki.eclipse.org/Platform-releng/Automated_Testing
So I will "take my time" making those changes, and be sure there is time for community review and comment. Bug 437069 was already open, for other reasons, so I will track the "packaged version" refactoring there.

Comment 1 David Williams

2014-08-21 01:50:14 EDT

First installment of this change: 

http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/commit/?id=b18c279a3983b14d85b82e7cdc3fc37ee954c928

To make use of this will also take a substantial change in our "Hudson jobs". (And, by substantial, I mean most of what is there removed, to be replaced by a the complex stuff there now removed, and replaced with just a few commands (and, a couple more variables ... and even those can be removed in future. 

When current nightlies finish, I'll make those changes and then before doing another nightly, try a few "retests" (being sure to save away current results in a safe place, in case of total failure). 

Full disclosure: This is well tested on Linux, and some of the refactoring should minimize the differences between platforms (letting Ant and Hudson handle it, instead of us) but, there will likely need to be some tweaks done for Mac and Windows. 

Once basics confirmed, I will also put in R4_4_maintenance branch. (It is basically a complete replacement of everything under .../production/testscripts, i.e. not a lot of subtle differences between streams ... by design ... just as it is now.

Comment 2 David Williams

2014-08-27 09:34:17 EDT

Yikes, just realized I haven't "sync'd" this up to maintenance stream yet, so plan to do that before SR1. 

Then other refinements will be rolled into bug 437069.

Comment 3 David Williams

2014-08-27 11:20:51 EDT

The maintenance tests did initially fail (since I'd forgotten to "sync up" the new code (which interfaces with Hudson, etc.) but after doing that, they seem to be progressing normally. 

[I will need a small tweak after today to put back in the part that pulls the testing interface based on 'hash' of what produced the build, but since did it too late for that today, had to simply use the tip of R4_4_maintenance stream. (and, again, this is really only effected by what's in aggregator/production/testscripts ... not the "whole project" or anything.)]