Community
Participate
Working Groups
The Eclipse JUnit 4 runner correctly treats @Ignore'd tests as being skipped, marks them with a special decoration, and does not count them in the totals. Tests that fail with an AssumptionViolationException should have exactly the same behavior, but they do not have a special decoration, they show up in the totals, and they look like "passed" tests. The JUnit docs for the org.junit.Assume class states "The default JUnit runner treats tests with failing assumptions as ignored. Custom runners may behave differently" [1] . It makes sense for Eclipse's runner to follow suit. [1] http://www.junit.org/apidocs/org/junit/Assume.html -- Configuration Details -- Product: Eclipse SDK 3.7.1.v201109091335 (org.eclipse.sdk.ide) Installed Features: org.eclipse.jdt 3.7.1.r371_v20110810-0800-7z8gFcoFMLfTabvKsR5Qm9rBGEBK
(In reply to comment #0) > "Custom runners may behave differently" Technically the Eclipse behavior is not wrong :-) However, I agree that Eclipse's runner should follow suit.
Created attachment 211836 [details] test case showing the problem (Java JUnit test source code) Test case showing the problem. # Keep the now += ... line commented and ## all is green (since assumption is false) - yet should be reported as ignored ## compile error flagged by Eclipse also passes :/ # Uncomment the now +=... line and ## Compile error comes to life ## The default failing test fails as expected Actually only the default test that fails as it's not yet implemented is enough to replicate the main issue. The other is a curious addition that compile error in test can pass. That may not be necessarily Eclipse test runner's problem. Eclipse data: Eclipse Java EE IDE for Web Developers. Version: Indigo Service Release 1 Build id: 20110916-0149
This is not an Eclipse problem. When you run a test class with JUnitCore.main(..), then a test whose assumption doesn't hold is also considered as successful, see the test case below (Run As > Java Application). If you want this to be changed, please file a bug against JUnit. Comment 2 is about an assumption in an @Before method. Again, we behave the same as the JUnit text runner. package testing; import static org.junit.Assert.fail; import static org.junit.Assume.assumeTrue; import org.junit.Ignore; import org.junit.Test; import org.junit.runner.JUnitCore; public class AssumptionTests { @Test public void testOK() throws Exception { // pass } // @Test // public void testFail() throws Exception { // fail(); // } @Test @Ignore public void testIgnored() throws Exception { fail(); } @Test public void testAssume() throws Exception { assumeTrue(false); fail(); } public static void main(String[] args) { JUnitCore.main(AssumptionTests.class.getName()); } }
bq. This is not an Eclipse problem. I disagree. This may be a problem stemming from JUnit's infrastructure but arguing that Eclipse is simulating a text runner is like saying Porches should run like hay wagons just because they have four wheels in common. Assumption-ignored tests convey an important information back to the user -- that a test was partially executed but had to stop along the way because of a failed constraint. It is this information that is important, not what JUnit does (for clarity: I think the design of run listeners in JUnit is flawed but it's another discussion). Speaking from practice, they are very important in projects that use them. For an example see Apache Lucene where randomized testing is used so tests are not identical from run to run -- assumptions are the only way of ignoring invalid parameter/ component configurations. If nothing else, maven, IntelliJ, JUnit4 runner -- they all do report assumption ignored tests.
(In reply to comment #3) > This is not an Eclipse problem. Markus, I'd like you to take a moment to reconsider your resolution of this bug. The title may be misleading but the underlying problem seems real. I've looked at the JUnit code and it does seem to notify listeners of assumption failures, but the eclipse listener doesn't do anything with it. Please take a look at JUnit4TestListener.java, which I assume is the listener that handles the junit run. I expect that if it had handled testAssumptionFailure(Failure) Example: @Override public void testAssumptionFailure(Failure failure) { testIgnored(Description plan); } I appreciate your time and look forward to your response. Of course, I could be totally wrong, I'm not entirely familiar with the eclipse source... but it does look to me like a eclipse issue. see: JUnit4TestListener at http://dev.eclipse.org/viewcvs/viewvc.cgi/org.eclipse.jdt.junit4.runtime/src/org/eclipse/jdt/internal/junit4/runner/JUnit4TestListener.java method runLeaf of Junit ParentRunner: https://github.com/KentBeck/junit/blob/master/src/main/java/org/junit/runners/ParentRunner.java -thank you
Just a comment -- ideally, ignored and assumption-ignored would be two different outcomes (because they are different outcomes!), marked with a different decorator. Even marking a test as ignored would be helpful and an improvement though.
As per prior comments, the resolution appears, to me, a trifle 'hasty' and would like it to be reconsidered. Maven's runner handles this as per 'preference' (failed assumption = skip) so Eclipse should also be capable of doing so, development effort notwithstanding ;) In many patterns the distinction (success-v-skip) is important - I, along with others, utilise 'assumeNotException(e)' in test cases alongside 'NotImplementedException' (or synonym) in incomplete development code (or even in the test cases themselves). Eclipse says all my tests completed succesfully - not true. Maven says it skipped those hitting incomplete dev code - ie. testing did not fail (some may have, some may have succeed) but testing was not complete, exactly the desired effect. TIA SteveG
RunListener#testAssumptionFailure(..) has only been added after Eclipse's JUnit4TestListener had been implemented: https://github.com/KentBeck/junit/commit/ca6d91f41d566b21b643b013e843732e1315652f But the JUnit 4 guys neither specified this with an @since tag nor did they update their own listeners. We could indeed handle assumption failures specially.
Created attachment 225868 [details] Screenshot with UI changes
Created attachment 225869 [details] Patch
This patch adds support for assumption failures to the Eclipse JUnit runner. For the UI changes see the attached screenshot. It shows the AssumptionViolationException stack trace and an own counter for the assumption failures (much like the ignored counter). An icon is still missing, a variant of the ignored icon might be ok. Technically assumption failures are treated like other failures, so that also the AssumptionViolationException stack can be transport to the UI without big changes. The remote protocol is enhanced in the same way as the @Ignored test prefix, so that it should be compatible even with older clients.
+1 from me.
Upvoted. Markus, Compilation error and green tests? Really? And you say it's OK, NOT misleading? It's not an attack, I'm just really thrown off-balance if you will confirm you do actually mean this. Let's say that JUnit does NOT differentiate between "failed test due to compile error" and "ignored tests due to assumption not met". That is a bug in their code. I'd advise against adopting this in Eclipse, as this will only mean problems later on, when this will be fixed in JUnit. I'd say, make Eclipse handle this well and open a bug in JUnit, with your test case showing the problem.
This is not a bug in JUnit, it's how it's been for ages and it's the only way of propagating assumption failures from within your code. If you're not using assumptions you'll never see assumption-ignored tests. If you do, you're familiar with their internal structure and a stack trace shouldn't be surprising (and won't be since you want to know which assumption and where fired).
(In reply to comment #14) > This is not a bug in JUnit, it's how it's been for ages and it's the only > way of propagating assumption failures from within your code. If you're not > using assumptions you'll never see assumption-ignored tests. If you do, > you're familiar with their internal structure and a stack trace shouldn't be > surprising (and won't be since you want to know which assumption and where > fired). First your comment Dawid that makes me wanna disagree (about this being "the only way". :-) I've replied via email, explaining myself, as this is a side-topic to most important issue here: Eclipse should NOT report "assumption not met" as "test has passed, all is green". Neither should JUnit, for that matter. So, even IF JUnit does report it so, this is clearly an error, which I sure hope Eclipse will not strengthen, nor spread nor perpetuate. Such test has NOT passed. It did NOT even run, therefore marking it as "error" or "success" is an error in itself. Which, in a nutshell, is what you said Dawid, in your former post here. About this being two very different outcomes.
Assumption ignored test actually _did_ run. At least partially (including rules and hooks). So it's not a black-and-white situation. But I agree that they should be marked distinctively different from success/failure/error/ignored status -- it's yet another outcome of a test (partially excecuted, but ignored). Dawid
That's what is proposed by the current patch: have a different icon for such tests, maybe similar to the ignored status, but still different. We could also rename the "Failure Trace" label so that it's not suggesting a real failure. @Tomasz: maybe you have concrete ideas how this could be visualized.
(In reply to comment #17) > That's what is proposed by the current patch: have a different icon for such > tests, maybe similar to the ignored status, but still different. We could > also rename the "Failure Trace" label so that it's not suggesting a real > failure. > > @Tomasz: maybe you have concrete ideas how this could be visualized. @Christian, neat one. I'll be ok with this, if I may offer: 1) assumptionFailure -> assumptionNotMet - on a number of cases I've seen assumptions used as "run tests only if assumption is met, as there's NO POINT in doing so otherwise" - so, not a failure but rather ignored tests - just as you say when you speak about Failure Trace label renaming - I fully support that. 2) icon - would it be a problem if it would be same as ignored, but instead of just having the sheet slashed with / we would have it marked with "A" letter? that would be similar enough, yet different enough (I hope I'm clear enough here! ;D) With these two I'll be more than just OK. 3) (cosmetics), I've sent you an email. Thanks!
(In reply to comment #10) > Created attachment 225869 [details] [diff] > Patch Thanks for the good quality patch, Christian! I've used most of it, except for the serialization, due to two issues: 1. When you launched multiple tests (or the same test multiple times), then the JUnit view didn't show the assumption failure traces any more when you switched back to an earlier test run. 2. Other test frameworks like TestNG already used a different notation with a <skipped> element, and Ant 1.9.0 adopted the same: https://issues.apache.org/bugzilla/show_bug.cgi?id=43969 We try to stay as close as possible to the Ant JUnit format, so I've also switched to <skipped>. Ant 1.9.0 currently doesn't record the assumption failure trace, but since I agree it makes sense to show it in the JUnit view, I just added the failure trace as CDATA in the <skipped> element. Regarding the icons, I tried it with a grayed or lighter version of the 'ignored' icon, but it didn't look good (and a minor change in tone is also not good for accessibility). In the end, I used the old 'ignored' icon for assumption failures and added an 'empty test' icon for 'ignored' tests. And in the counter panel, the (<x> ignored, <y> assumption failures) message used too much space. I changed this to a common (<x> skipped) count and kept the full text in the tooltip. And I fixed the bogus "Integer.toBinaryString(assumptionFailureCount)"... Released with http://git.eclipse.org/c/jdt/eclipse.jdt.ui.git/commit/?id=f884bb73f0186fc8e55c458c1d817fa2cc34b4f4 (Christian's patch) and http://git.eclipse.org/c/jdt/eclipse.jdt.ui.git/commit/?id=050dcb5f7e736575a72442d32d95e5dc688bf508 (my changes + icons).
.
Verified in I20130430-2000.
The icons for skipped tests have been adapted a bit, see bug 406976.
