Bugzilla – Full Text Bug Listing
| Summary: | Unstable UI tests | ||
|---|---|---|---|
| Product: | [Technology] EGit | Reporter: | Robin Stocker <robin> |
| Component: | UI | Assignee: | Project Inbox <egit.ui-inbox> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | major | ||
| Priority: | P3 | CC: | daniel_megert, matthias.sohn |
| Version: | unspecified | ||
| Target Milestone: | 3.2 | ||
| Hardware: | All | ||
| OS: | All | ||
| Whiteboard: | |||
|
Description
Robin Stocker
A not yet quite successful start: http://egit.eclipse.org/r/#change,3419 It's a pain when getting failed tests on unrelated changes. We should either fix the test or disable it. Another try: https://git.eclipse.org/r/8384 The latest failure was from testPushToOriginRemoteNode. My current theory why that fails: Push itself is done, but the dialog takes its time to be displayed because RefUpdateElement does some rev walking, see getCommits called from getChildren. RefUpdateElement is used in PushResultTable in the result dialog. I verified that this is called before the dialog is shown, so this could be the remaining problem. Also, the machine running these seems to be extremely slow or overused :/. Any ideas how to solve this? Increasing the timeout? (In reply to comment #4) > The latest failure was from testPushToOriginRemoteNode. My current theory > why that fails: > > Push itself is done, but the dialog takes its time to be displayed because > RefUpdateElement does some rev walking, see getCommits called from > getChildren. RefUpdateElement is used in PushResultTable in the result > dialog. > > I verified that this is called before the dialog is shown, so this could be > the remaining problem. > > Also, the machine running these seems to be extremely slow or overused :/. > > Any ideas how to solve this? Increasing the timeout? I didn't look at the code, so I don't know to which "timeout" you refer. I suggest to simply wait on a specific condition (e.g. dialog correctly displays the items) while still driving the event loop. Here's how we do this in our tests: http://git.eclipse.org/c/jdt/eclipse.jdt.ui.git/tree/org.eclipse.jdt.ui.tests/test%20plugin/org/eclipse/jdt/testplugin/util/DisplayHelper.java (In reply to comment #5) > I didn't look at the code, so I don't know to which "timeout" you refer. I > suggest to simply wait on a specific condition (e.g. dialog correctly > displays the items) while still driving the event loop. Here's how we do > this in our tests: > http://git.eclipse.org/c/jdt/eclipse.jdt.ui.git/tree/org.eclipse.jdt.ui. > tests/test%20plugin/org/eclipse/jdt/testplugin/util/DisplayHelper.java The SWTBot timeout, which is 10 seconds currently. SWTBot does this already, but it seems the machine is too slow to finish before the timeout. I pushed a new change which increases the timeout to 20 seconds, maybe that helps: https://git.eclipse.org/r/8692 Since the Hudson move, some UI tests have been even more unstable, which is not good. Here's a change series which should improve it (with these changes, I had the first run without failures on my machine): https://git.eclipse.org/r/17565 https://git.eclipse.org/r/17566 https://git.eclipse.org/r/17567 https://git.eclipse.org/r/17568 https://git.eclipse.org/r/17569 https://git.eclipse.org/r/17570 https://git.eclipse.org/r/17571 https://git.eclipse.org/r/17572 https://git.eclipse.org/r/17573 https://git.eclipse.org/r/17574 The latest initiative made this problem far less severe. There are still some failures from time to time, but they are the minority now. I think we can close this and improve individual tests when needed. (In reply to Robin Stocker from comment #8) > The latest initiative made this problem far less severe. There are still > some failures from time to time, but they are the minority now. I think we > can close this and improve individual tests when needed. I agree, UI tests are pretty stable thanks to the effort you spent on improving them, thanks again for this work :-) Should we change the configuration of the egit.gerrit build job verifying new changes to fail the build when a UI test fails ? At the moment UI tests failures are reported as UNSTABLE, we could consider to instead FAIL these builds in order to ensure that the stability of the UI tests doesn't degrade over time. (In reply to Matthias Sohn from comment #9) > (In reply to Robin Stocker from comment #8) > > The latest initiative made this problem far less severe. There are still > > some failures from time to time, but they are the minority now. I think we > > can close this and improve individual tests when needed. > > I agree, UI tests are pretty stable thanks to the effort you spent on > improving them, thanks again for this work :-) No problem :). I also submitted this change right now, which seems to improve stability a bit more: https://git.eclipse.org/r/#/c/18646/ > Should we change the configuration of the egit.gerrit build job verifying > new changes to fail the build when a UI test fails ? At the moment UI tests > failures are reported as UNSTABLE, we could consider to instead FAIL these > builds in order to ensure that the stability of the UI tests doesn't degrade > over time. Yes, I think we can try that now. |