Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 342087

Summary: [JUnit] Compare results of different JUnit executions
Product: [Eclipse Project] JDT Reporter: Nitin Dahyabhai <thatnitind>
Component: UIAssignee: JDT-UI-Inbox <jdt-ui-inbox>
Status: ASSIGNED --- QA Contact:
Severity: enhancement    
Priority: P3 CC: daniel_megert, markus.kell.r, remy.suen
Version: 3.7Keywords: helpwanted
Target Milestone: ---   
Hardware: All   
OS: All   
Whiteboard:

Description Nitin Dahyabhai CLA 2011-04-06 18:38:52 EDT 
In WTP Source Editing we spend time, when we find it, bolstering our compatibility with the W3C DOM unit test suite (bug 214439).  Currently we fail nearly 100 of the 499 tests, and I find it difficult making forward progress as it is easy to make changes that (hypothetically) let us pass 5 tests while causing 4 tests that *used* to pass to now fail.  The overall numbers are better but it's basically a regression at that point, and one that's hard to figure out unless there's a way for me to compare the results of the suite executions before and after my changes (not comparing the results in a single execution as is offered for expected/actual).

Ideally when I bring up the History of runs, there'd be a new button to compare the exactly two runs I've selected so I can know both the net pass/fail change as well as easily spot tests that previously passed but now fail, and vice-versa.
Comment 1 Dani Megert CLA 2011-04-07 02:24:41 EDT 
Nitin, maybe a workaround for you might be to export the results to XML (via JUnit view's history drop down menu) and then use text / XML compare?
Comment 2 Markus Keller CLA 2011-04-07 08:22:17 EDT 
Good idea, I'll keep this in mind for 3.8. This would also be nice item for a new contributor to work on.

Apart from a Compare... button in the history dialog, we should also add Import.../Export... buttons (bug 185056) to make it easier to compare external test runs (e.g. from a build server).