Bugzilla – Full Text Bug Listing
| Summary: | [JUnit] Eclipse JUnit Runner does not work well with stateful runners | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Eclipse Project] JDT | Reporter: | Mike Goodwin <mkpgoodwin> | ||||
| Component: | UI | Assignee: | JDT-UI-Inbox <jdt-ui-inbox> | ||||
| Status: | CLOSED WONTFIX | QA Contact: | |||||
| Severity: | normal | ||||||
| Priority: | P4 | CC: | markus.kell.r | ||||
| Version: | 3.4 | ||||||
| Target Milestone: | --- | ||||||
| Hardware: | All | ||||||
| OS: | All | ||||||
| Whiteboard: | stalebug | ||||||
| Attachments: |
|
||||||
org.junit.internal.runners.TestClassMethodsRunner is an internal JUnit 4.3.1 class, which is not even available any more in JUnit 4.4 and 4.5. You should probably refactor your tests to use @Parameterized. In general, tests should be self-contained. Re-running individual tests should behave the same as running the individual tests alone. The Eclipse JUnit support just adds some UI to make JUnit easier to use, but we won't do more than what would also work if you ran JUnit from the command line. Sorry for reusing internal code. Actually I had not noticed, but it is was convenient to do so. In anycase it is not related to the issue. If what you say is true then there is virtually no point in custom runners, except for a) running test classes differently (e.g. the parameterized runner) b) gathering test classes differently From the design of JUnit4 this is clearly not the intended limit of its extensability. That the filtering mechanism has been done in this general way is because it does not want to make assumptions about tests. I believe it provides the cleanest mechanism for reexcuting tests. It seems to me that it is most powerful and helpful to think of tests as logical entities anyway, a test is an abstraction. A graphical runner should not make any assumptions about the tests being passed into it. I would not be surpised if it simplified the graphical runners code to just have one test to reconstruct and a filter to apply each time, rather than trying discover what file a test belongs too. What you are saying about the commandline I think is not a fair comparison as the command line does not offer the chance to reexecute selected tests. I do think that JUnit4 is an attempt, not to move away from the dogmatism of pure unit testing, but to leave that in the core and to see what happens with extensions around it. Graphical runners for instance do not need to be tied to this dogma, and indeed are no longer in the core. Having one versatile runner for all different types of tests is certainly better than having one for unit testing, one for integrated testing ... etc, given the similarities in the outputs. Can I ask, what would be the disadvantages of doing it the way I suggest? Theoretically, if I were to make the change, would it be possible to get it integrated into a future version, or even a point version of eclipse? I don't want to be a pain, but I thought I would reopen this bug to see if I can get my comments addressed. I think the runner is missing a trick here and that as of JUnit 4 the test output is intended to be more general and not related to how the test was created. OK, I think I understand. RemoteTestRunner#rerunTest(RerunRequest) should load the original test class again, pass it on to JUnit4TestClassReference(Class<?>), which should call Request.aClass(clazz) on the original class and then call Request#filterWith(..) on the original class request to restrict the actual execution to the test or suite to run. However, we must be careful that we only do this if the original request's Runner implements Filterable (otherwise, the whole original test would be run again). Yes, that sounds about right to me. It is unfortunate that by design the filtering is left to the implementation and can be done incorrectly. Thanks for the consideration. This bug hasn't had any activity in quite some time. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're closing this bug. If you have further information on the current state of the bug, please add it and reopen this bug. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. -- The automated Eclipse Genie. |
Created attachment 110635 [details] Project with custom test runners that exhibit the problem After running tests you can run a part of the test suite that has just executed, by selecting run from the context menu of the nodes in the tree. The problem is that if one tries to configure all tests globabally with a suite that uses a runner that constructs all subtests passing in an argument for configuration, re-executing a testcase does not re-execute with the passed in arguments. Rather the class the test method is on is recreated and the default runner for that class is used. The issue is that it should, I believe, reconstruct all the tests, as they were run initially. Then when executing it filters down to just the ones that you want to rerun. Instead it is just guessing how to reconstruct the tests, which only works in the simple case. A project is attached which demonstrates this. Execute 'src/s/suite.java' as a junit test case. This will execute a super suite that executes two classes, which each have two methods. Values are printed (an int which is passed in as configuration). If you rerun the method then a different value is printed - which is the default value the TestClassRunner uses.