Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 495810

Summary:	Rebalance SWTBot tests between the two sub-suites based on running time and stability
Product:	[Modeling] Sirius	Reporter:	Pierre-Charles David <pierre-charles.david>
Component:	Core	Assignee:	Pierre-Charles David <pierre-charles.david>
Status:	CLOSED FIXED	QA Contact:
Severity:	enhancement
Priority:	P3	CC:	steve.monnier
Version:	unspecified	Keywords:	triaged
Target Milestone:	5.0.0M7
Hardware:	All
OS:	All
See Also:	https://git.eclipse.org/r/75130 https://git.eclipse.org/r/75131 https://git.eclipse.org/c/sirius/org.eclipse.sirius.git/commit/?id=81e6704269be18786e4ae5849c73704aa4a681eb https://git.eclipse.org/c/sirius/org.eclipse.sirius.git/commit/?id=61f4bbb5a3fbeee06fc410cfe02bc26e5093b1f1 https://git.eclipse.org/r/76335 https://git.eclipse.org/c/sirius/org.eclipse.sirius.git/commit/?id=4a24ca12b1d86d1cd38949f659c45d3acba015e9
Whiteboard:

Description Pierre-Charles David

2016-06-09 11:47:14 EDT

In order to get faster feedback, we have split our longest test suite (the SWTbot one) in two parts, that can be run in parallel (given enough slots on a CI server). The split was done purely based on running time, in order to have roughly equivalent times for both parts.

In practice, part 1 takes significantly longer than part 2: around 1h35min vs 1h10min.

An additional observation is part 1 is much more frequently "green" (with no failure) than part 2.

The goal of this ticket is to move the less stable tests from part 1 into part 2, both to even out the suite's running time, and to help segregate the more stable tests (in part 1) from the more unreliable ones (in part 2).

It's a low priority task, which will be done iteratively as a background task:
* note which tests in part 1 fail during the nightly sirius.tests-master job;
* move them to part 2 as long as part 1 takes longer than part 2 (this will even out their times);
* if part 2 starts to take longer than part 1 to run (because we moved to many tests there), move some of the most stable tests from part 2 into part 1 to keep the balance.
* repeat until part 1 only contains reliable tests and is consistently green.

In the end, the part in which a test is should give us some information about its expected reliability, and help us focus our efforts on improving this reliability.

Comment 1 Pierre-Charles David

2016-06-09 11:49:06 EDT

Some first candidates that are currently in SWTBot-Part1 but have failed at least once for no good reason recently:

CenteredEdgesRepairTest
ChildrenPositionStabilityAfterParentResizeTest
CustomizationPropertySectionsTests
HideRevealDiagramElementsLabelsTest
HideRevealTableLinesTest
LabelFontModificationsTest
LinkWithEditorFeatureWithModelExplorerViewTest
MetamodelPropertyTabTests
NavigationDescriptionFromTableAndTreeTest
OpenMultipleRepresentationsTest
RefreshWithCustomizedStyleFromAppearanceTabTests
RefreshWithCustomizedStyleTests
ResizeKindEditorTest
STD006
STD008
ViewpointSpecificationProjectCreationTest

Comment 2 Eclipse Genie

2016-06-12 04:01:58 EDT

New Gerrit change created: https://git.eclipse.org/r/75130

Comment 3 Eclipse Genie

2016-06-12 08:45:05 EDT

New Gerrit change created: https://git.eclipse.org/r/75131

Comment 4 Eclipse Genie

2016-06-12 12:18:02 EDT

Gerrit change https://git.eclipse.org/r/75130 was merged to [master].
Commit: http://git.eclipse.org/c/sirius/org.eclipse.sirius.git/commit/?id=81e6704269be18786e4ae5849c73704aa4a681eb

Comment 5 Eclipse Genie

2016-06-12 12:18:04 EDT

Gerrit change https://git.eclipse.org/r/75131 was merged to [master].
Commit: http://git.eclipse.org/c/sirius/org.eclipse.sirius.git/commit/?id=61f4bbb5a3fbeee06fc410cfe02bc26e5093b1f1

Comment 6 Pierre-Charles David

2016-06-13 03:06:35 EDT

Other candidates identified by recent failures:
- STD010
- PinnedElementsTest
- InvalidMetamodelRessourceTest

Comment 7 Pierre-Charles David

2016-06-14 03:23:14 EDT

More candidates:
- org.eclipse.sirius.tests.swtbot.EditorSavingTest
- org.eclipse.sirius.tests.swtbot.table.HideRevealTableColumnsTest

Comment 8 Pierre-Charles David

2016-06-14 04:30:59 EDT

I've created https://hudson.eclipse.org/sirius/job/sirius.tests-swtbot/ which only runs the SWTBot suites, and on a single target platform (neon), to iterate more quickly on this.

Comment 9 Pierre-Charles David

2016-06-28 09:37:52 EDT

Current status:

Among all the candidates liste on comments above, all of these have not been moved yet:

EditorSavingTest
HideRevealTableColumnsTest
HideRevealTableLinesTest
InvalidMetamodelRessourceTest
LabelFontModificationsTest
LinkWithEditorFeatureWithModelExplorerViewTest
NavigationDescriptionFromTableAndTreeTest
OpenMultipleRepresentationsTest
PinnedElementsTest
RefreshWithCustomizedStyleFromAppearanceTabTests
RefreshWithCustomizedStyleTests
ResizeKindEditorTest
STD006
STD008
STD010
ViewpointSpecificationProjectCreationTest


During the last 6 builds run on https://hudson.eclipse.org/sirius/job/sirius.tests-swtbot/, the actual failures in SWTBot-Part 1 were:
- LinkWithEditorFeatureWithModelExplorerViewTest.testLinkWithEditorFeatureWithModelExplorerView
- NavigationDescriptionFromTableAndTreeTest.testOpenDescriptionAvailableFromEditionTablesWithPrecondition
- NavigationDescriptionFromTableAndTreeTest.testOpenDescriptionAvailableFromTrees
- RefreshAfterViewCreationTest.testEdgeCreationOnBorderedNodeOnContainerInContainerCreationWithAutoRefresh
- ViewpointSpecificationProjectCreationTest.test_ViewpointViewpoint_Specification_Project_Creation

Only RefreshAfterViewCreationTest was not already listed as a candidate. These 4 classes will be moved next.

Comment 10 Eclipse Genie

2016-06-30 09:54:06 EDT

New Gerrit change created: https://git.eclipse.org/r/76335

Comment 11 Eclipse Genie

2016-07-01 08:05:49 EDT

Gerrit change https://git.eclipse.org/r/76335 was merged to [master].
Commit: http://git.eclipse.org/c/sirius/org.eclipse.sirius.git/commit/?id=4a24ca12b1d86d1cd38949f659c45d3acba015e9

Comment 12 Pierre-Charles David

2016-09-07 10:21:27 EDT

Some errors still occur for no good reason on the part-1 suite, but overall it's much better than before. Moving to 5.0 for the remaining work.

Comment 13 Pierre-Charles David

2017-04-05 10:55:37 EDT

While still not perfect, we'll consider this done for now.

Comment 14 Pierre-Charles David

2017-06-29 03:32:31 EDT

Available in Sirius 5.0.0, see https://wiki.eclipse.org/Sirius/5.0.0 for details.