Attachment 105532 Details for Bug 211777 – Updated Part 2

Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Updated Part 2

TPTP_Testing_Strategy_Part_2.html (text/html), 28.28 KB, created by Alan Haggarty

on 2008-06-20 14:30:43 EDT

(hide)

Description:

Filename:

MIME Type:

Creator: Alan Haggarty

Created: 2008-06-20 14:30:43 EDT

Size: 28.28 KB

patch

obsolete

><!DOCTYPE html PUBLIC "-//w3c//dtd html 4.0 transitional//en">
><html><head><title>TPTP Testing Strategy Part 2 - Test Conventions and Best Practices</title>
>
><meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
><link rel="stylesheet" type="text/css" href="http://www.eclipse.org/tptp/eclipse_style.css">
><link rel="stylesheet" type="text/css" href="http://www.eclipse.org/tptp/home/documents/process/TPTP_Testing_Strategy_files/strategy_doc_style.css">
><script type="text/javascript">
>		function expandOrCollapse()
>		{
>			element = document.getElementById("invisible");
>			if (element != null)
>			{
>				element.id = "dynamicAppearance";
>				document.getElementById("changeHistoryLink").lastChild.nodeValue = "Hide the Change History";
>				return;
>			}
>
>			element = document.getElementById("dynamicAppearance");
>			if (element != null)
>			{
>				element.id = "invisible";
>				document.getElementById("changeHistoryLink").lastChild.nodeValue = "Show the Change History";
>			}
>		}
>  </script></head><body>
><div id="page_header">Eclipse Test and Performance Tools Platform
>Project</div>
><div id="page_sub_header">TPTP Testing Strategy Part 2 - Test Conventions and Best Practices</div>
><div id="section_header">TPTP Testing Strategy Part 2 - Test Conventions and Best Practices</div>
>
><div class="content">
><table id="main_content">
>	<tbody>
>		<tr>
>			<td><b>Title:</b></td>
>			<td>TPTP Testing Strategy</td>
>		</tr>
>		<tr>
>			<td><b>Author:</b></td>
>
>			<td>Alan Haggarty (haggarty@ca.ibm.com)</td>
>		</tr>
>	</tbody>
></table>
><a href="javascript:expandOrCollapse()"><b><span id="changeHistoryLink">Show the Change History</span></b></a>
><table id="invisible" border="1">
>	<tbody>
>		<tr>
>			<th align="left" bgcolor="#dfdfdf" width="10%">Name</th>
>
>			<th align="left" bgcolor="#dfdfdf" width="20%">Date</th>
>			<th align="left" bgcolor="#dfdfdf" width="30%">Modified Sections
>			</th>
>		</tr>
>		<tr>
>			<td>Alan Haggarty (haggarty@ca.ibm.com)</td>
>			<td>June 12, 2008</td>
>			<td>Initial Creation</td>
>
>		</tr>
>	</tbody>
></table>  	
><br>
><div id="section_hlight">Table of Contents</div>
><div id="main_content">
><a href="#1.0">1.0 TPTP Test Project Conventions</a><br>
><a href="#1.1">1.1 Naming Conventions</a><br>
><a href="#1.2">1.2 Test Project Detailed Structure</a><br>
><a href="#2.0">2.0 Test Creation Best Practices</a><br>
><a href="#2.1">2.1 Creating Manual Test Suites</a><br>
><a href="#2.2">2.2 Creating JUnit and JUnit Plug-in Test Suites</a><br>
><a href="#2.3">2.3 Creating AGR Test Suites</a><br>
><a href="#3.0">3.0 Test Execution Best Practices</a><br>
><a href="#3.1">3.1 Smoke Testing</a><br>
><a href="#4.0">4.0 TPTP Test Project Exceptions</a><br>
><a href="#5.0">5.0 The Common Test Infrastructure</a><br>
><a href="#5.1">5.1 Reference Platform</a><br>
><a href="#5.2">5.2 Adding Tests</a><br>
><a href="#5.3">5.3 Installing and Executing</a><br>
></div>
><br>
><div id="section_hlight"><a name="1.0">1.0 TPTP Test Project Conventions</a></div>
><div id="main_content">
></div>
><br>
><div id="section_hlight"><a name="1.1">1.1 Naming Conventions</a></div>
><div id="main_content">
>	<ol>
>	 <li>All names for testing resources and folder names MUST adhere to the following rules:
>	 	<ul>
>      <li>Upper or lower case letter (for example, A-Z or a-z).</li>
>      <li>Digits (for example, 0 - 9).</li>
>      <li>Underscore (for example, _).</li>
>    </ul>
>  	</li>
>  	<li>Names must NOT contain any white space characters (for example, tab, space, and new line).</li>
>  	<li>Every name should begin with an upper or lower case letter (for example, A-Z or a-z). 
>  		All other characters in the name should be upper or lower case letters, digits, or the underscore.</li>
>  	<li>Java keywords (for example, import, if, etc.) cannot be used for names of JUnit test resources.</li>
>  	<li>The name of the Deployment file (deployment files are often used for remote launch of test suites) may contain '.' characters. 
>  		The name should be meaningful to the project and the host that it is associated with. 
>  		For example, a local deployment file for the org.eclipse.hyades.use.cases folder can be org.eclipse.hyades.use.cases.localhost.deploy</li>
>  	<li>The name and directory structure containing the generated execution history file MUST be the same as it will be stored in CVS. 
>  		If the generated execution history file is either moved or renamed, the linkage with the associated test suite will break thereby 
>  		corrupting generated reports. In the event of an existing execution history in the same directory, 
>  		TPTP Testing Tools will append a millisecond time stamp to the name of the new execution history file.</li>
>  	<li>Test suite names must start with the bugzilla component name associated with the plug-in or component under test.<br>
>  		If the test suite contains several functions, you can either use the bugzilla component name as the name of the test suite or 
>  		use the bugzilla component as a prefix.</li>
>  	<li>Multiple JRE or platform test suite names must end with the name, vendor, and version of the JRE or platform under test. 
>  	These test suites should reference the test suite(s) and/or test case(s) for the plug-in or component to avoid redundancy.</li>
>	</ol>
></div>
><br>
><div id="section_hlight"><a name="1.2">1.2 Test Project Detailed Structure</a></div>
><div id="main_content">
>	<table border="1" width="100%">
>	<tbody>
>		<tr>
>			<th align="left">Name</th>
>			<th align="left">Purpose</th>
>		</tr>
>		<tr>
>			<td>deployment</td>
>			<td>Directory containing all deployment configurations for executing the test suites in the test project.</td>
>		</tr>
>		<tr>
>			<td>gui</td>
>			<td>Directory containing all AGR test suites for the test project.</td>
>		</tr>
>		<tr>
>			<td>gui/AllTests.testsuite</td>
>			<td>AGR test suite to execute all lower level AGR test
>			suites for the test project.  Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP1/AllTP1GUITests.testsuite (see Table 2).</td>
>		</tr>
>		<tr>
>			<td>gui/AllSmokeTests.testsuite</td>
>			<td>AGR smoke test suite to execute all lower level AGR smoke test
>			suites for the test project.  Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP2/AllTP2GUITests.testsuite (see Table 2).</td>
>		</tr>
>		<tr>
>			<td>gui_results</td>
>			<td>Directory containing all execution histories for automated
>			GUI test suites for the test project.</td>
>		</tr>
>		<tr>
>			<td>junit_plugin</td>
>			<td>Directory containing all JUnit Plug-in test suites for the test project.</td>
>		</tr>
>		<tr>
>			<td>junit_plugin/AllTests.testsuite</td>
>			<td>JUnit Plug-in test suite to execute all lower level JUnit Plug-in test
>			suites for the test project.  Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP1/AllTP1JUnitPluginTests.testsuite (see Table 2).</td>
>		</tr>
>		<tr>
>			<td>junit_plugin/AllSmokeTests.testsuite</td>
>			<td>JUnit Plug-in smoke test suite to execute all lower level JUnit Plug-in smoke test
>			suites for the test project.  Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP2/AllTP2JUnitPluginTests.testsuite (see Table 2).</td>
>		</tr>
>		<tr>
>			<td>junit_plugin_results</td>
>			<td>Directory containing all execution histories for JUnit Plug-in test
>			suites for the test project.</td>
>		</tr>
>		<tr>
>			<td>junit</td>
>			<td>Directory containing all JUnit test suites for the test project.</td>
>		</tr>
>		<tr>
>			<td>junit/AllTests.testsuite</td>
>			<td>JUnit test suite to execute all lower level JUnit test
>			suites for the test project.  Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP1/AllTP1JUnitTests.testsuite (see Table 2).</td>
>		</tr>
>		<tr>
>			<td>junit/AllSmokeTests.testsuite</td>
>			<td>JUnit smoke test suite to execute all lower level JUnit smoke test
>			suites for the test project.  Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP2/AllTP2JUnitTests.testsuite (see Table 2).</td>
>		</tr>
>		<tr>
>			<td>junit_results</td>
>			<td>Directory containing all execution histories for JUnit test
>			suites for the test project.</td>
>		</tr>
>		<tr>
>			<td>manual</td>
>			<td>Directory containing all manual test suites for the test project.</td>
>		</tr>
>		<tr>
>			<td>manual/AllTests.testsuite</td>
>			<td>Manual test suite to execute all lower level manual test
>			suites for the test project.  Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP1/AllTP1ManualTests.testsuite (see Table 2).</td>
>		</tr>
>		<tr>
>			<td>manual/AllSmokeTests.testsuite</td>
>			<td>Manual smoke test suite to execute all lower level manual smoke test
>			suites for the test project.  Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP2/AllTP2ManualTests.testsuite (see Table 2).</td>
>		</tr>
>		<tr>
>			<td>manual_results</td>
>			<td>Directory containing all execution histories for manual test
>			suites for the test project.</td>
>		</tr>
>		<tr>
>			<td>resources</td>
>			<td>Directory containing any resources required to execute the
>			test suites for the test project.</td>
>		</tr>
>		<tr>
>			<td>src</td>
>			<td>Directory containing generated JUnit Java source code for
>			the JUnit and JUnit Plug-in test suites and Java source code corresponding to
>			verification hooks of AGR test suites for the test project.</td>
>		</tr>
>	</tbody>
></table>
><b>Table 1: Required structure for contents of a test project.</b>
><p>
>Each of the subfolders contains a test project named org.eclipse.hyades.use.cases for testing functionality 
>that crosses plug-in or component boundaries.
></p>
>
><p>
>The <a href="http://dev.eclipse.org/viewcvs/index.cgi/test-results/platform/org.eclipse.hyades.tests/?root=TPTP_Project">test-results/platform/org.eclipse.hyades.tests</a> 
>test project contains all root-level test suites for TPTP.  Table 2 details the required structure for contents of the test-results/platform/org.eclipse.hyades.tests test project.
></p>
>
><table border="1" width="100%">
>	<tbody>
>		<tr>
>			<th align="left">Name</th>
>			<th align="left">Purpose</th>
>		</tr>
>		<tr>
>			<td>deployment</td>
>
>			<td>Directory containing all deployment configurations for executing the root-level test suites in TPTP.</td>
>		</tr>
>		<tr>
>			<td>enablement</td>
>			<td>Directory containing all the TPTP enablement tests.</td>
>		</tr>
>		<tr>
>
>			<td>TVT</td>
>			<td>Directory containing all the TPTP Translation Verification Tests (TVT).</td>
>		</tr>
>		<tr>
>			<td>src</td>
>			<td>Directory containing generated JUnit Java source code for
>			the root-level JUnit and JUnit Plug-in test suites in TPTP and Java source code corresponding to
>			verification hooks of root-level AGR test suites in TPTP.</td>
>	  </tr>
>
>		<tr>
>			<td>TP1</td>
>			<td>Directory containing all root-level test suites to execute all lower level test suites 
>				during test pass 1 (TP1).</td>
>		</tr>
>		<tr>
>			<td>TP1/AllTP1GUITests.testsuite</td>
>			<td>Root-level AGR test suite to execute all lower level AGR test suites during test pass 1</td>
>		</tr>	
>		<tr>
>			<td>TP1/AllTP1JUnitPluginTests.testsuite</td>
>			<td>Root-level JUnit Plug-in test suite to execute all lower level JUnit Plug-in test suites during test pass 1</td>
>		</tr>		
>		<tr>
>			<td>TP1/AllTP1JUnitTests.testsuite</td>
>			<td>Root-level JUnit test suite to execute all lower level JUnit test suites during test pass 1</td>
>		</tr>	
>		<tr>
>			<td>TP1/AllTP1ManualTests.testsuite</td>
>			<td>Root-level manual test suite to execute all lower level manual test suites during test pass 1</td>
>		</tr>			
>		
>		<tr>
>			<td>TP2</td>
>			<td>Directory containing all root-level test suites to execute all lower level test suites 
>				during test pass 2 (TP2).</td>
>		</tr>
>		<tr>
>			<td>TP2/AllTP2GUITests.testsuite</td>
>			<td>Root-level AGR test suite to execute all lower level AGR test suites during test pass 2</td>
>		</tr>	
>		<tr>
>			<td>TP2/AllTP2JUnitPluginTests.testsuite</td>
>			<td>Root-level JUnit Plug-in test suite to execute all lower level JUnit Plug-in test suites 
>				during test pass 2</td>
>		</tr>		
>		<tr>
>			<td>TP2/AllTP2JUnitTests.testsuite</td>
>			<td>Root-level JUnit test suite to execute all lower level JUnit test suites during test pass 2</td>
>		</tr>	
>		<tr>
>			<td>TP2/AllTP2ManualTests.testsuite</td>
>			<td>Root-level manual test suite to execute all lower level manual test suites during test pass 2</td>
>		</tr>		
>		
>		<tr>
>			<td>BVT</td>
>			<td>Directory containing all root-level test suites to execute all lower level test suites 
>				during build verification test (BVT).</td>
>		</tr>
>		<tr>
>			<td>BVT/AllBVTGUITests.testsuite</td>
>			<td>Root-level AGR test suite to execute all lower level AGR test suites during build verification test</td>
>		</tr>	
>		<tr>
>			<td>BVT/AllBVTJUnitPluginTests.testsuite</td>
>			<td>Root-level JUnit Plug-in test suite to execute all lower level JUnit Plug-in test suites 
>				during build verification tests</td>
>		</tr>		
>		<tr>
>			<td>BVT/AllBVTJUnitTests.testsuite</td>
>			<td>Root-level JUnit test suite to execute all lower level JUnit test suites during build verification tests</td>
>		</tr>
>			
>	</tbody>
></table>
><b>Table 2: Required structure for contents of the test-results/platform/org.eclipse.hyades.tests test project.</b>
>
><p>The execution history files (*.execution) generated when executing test suites are checked-in to the CVS branch for the 
>release under test by the committer for the associated plug-in or component.  For example, the execution history files generated from 
>testing the current full release would be checked in to HEAD.  Execution history files are checked in to the *_results folder for the type of 
>test suite.  For example, execution history files generated from executing a manual test suite would be checked in the manual_results folder.  To 
>delineate between test executions on multiple platforms, each *_results folder contains the following subfolders for each supported platform:
>
></p>
>
><ul>
>	<li>AIX</li>
>	<li>AS_400</li>
>	<li>HP_UX</li>
>	<li>Linux_390</li>
>	<li>Linux_x86</li>
>
>	<li>OS_390</li>
>	<li>Solaris</li>
>	<li>Windows</li>
>	<li>Linux PPC64</li>
></ul>
>
><p>
>Supported platform subfolders may be optionally appended with the platform version and release:
></p>
>
><ul>
>	<li>AIX_5_2_0</li>
>	<li>AS_400_5_2</li>
>	<li>HP_UX_11</li>
>	<li>Linux_390_SuSE_ Enterprise_7_0</li>
>	<li>Linux_x86_RedHat_Enterprise_2_1</li>
>
>	<li>Linux_x86_SLES_10</li>
>	<li>OS_390_1_6</li>
>	<li>Solaris_9</li>
>	<li>Windows_2000</li>
>	<li>Windows_XP</li>
>	<li>Windows_Vista</li>
>
>	<li>Linux_PPC64<br>
>	</li>
></ul>
>
><p>Test execution reports are generated and checked in to CVS by the
>TPTP Project Lead or PMC representative at predetermined milestones in a 
>test pass for public web access.  Test execution reports are available
>at: <a href="http://www.eclipse.org/tptp/">www.eclipse.org/tptp</a> &gt;&gt; Development Plans &gt;&gt; Test Pass Reports
></p>
></div>
><br>
><div id="section_hlight"><a name="2.0">2.0 Test Creation Best Practices</a></div>
><div id="main_content">
>	<p>
>	Based on the previous experiences of test resource creators and testers, the following general recommendations will assist 
>	testers on creating test cases. Section 2.1 - 2.3 go into more details about best practices for creating manual, JUnit, JUnit Plug-in, 
>	and AGR test suites.
></p>
><ol>   
>	<li>A test case should always have a specific purpose for testing a scenario. Having many test cases in a test suite does not necessarily ensure better quality. Each test case must be well thought out and have a specific purpose. Simply testing the basic functionality of a tool is often not enough to ensure high quality standards. Boundary cases are usually considered to be interesting test cases. As an example, consider the case of testing a function that removes entries from a table. The following test cases should be considered:
>		<ul>
>     <li>Removing the first entry (a boundary case)</li>
>     <li>Removing the second entry (tests basic functionality)</li>
>     <li>Removing the last entry (a boundary case)</li>
>    </ul>
>   </li>
>  <li>Avoid repeating the same set of steps in multiple test cases. If two test cases require the user to import a log file, 
>  	then create three test cases. The first test case will only serve as setting the environment for successive test cases and the 
>  	second and third test case will test the specific functionality in mind.</li>
>  <li>This point provides a basic summary of the first two: apply the high cohesion principle to test cases. 
>  	Cohesion means assigning specific responsibilities to test cases. A test case requiring a user to import a log file, 
>  	profile an application, and use the Probekit has little common in its responsibilities and thus does not follow the high cohesion principle.</li>
>  <li>Avoid unnecessarily complicating test cases that are designed to test functionality and not performance. 
>  	For example, a test case that uses a class with 3 methods to test the Java Profiler is likely as good as a test case 
>  	that uses a class with 10 methods.</li>
>  <li>Comment test cases thoroughly in the description section of the editor including manual steps for what the test case is performing.</li>
>  <li>If any resource (such as a datapool) needs to be modified before the test suite is executed, 
>   	then include that as part of the description of the test suite. 
>  Any other important information that the tester should be aware of needs to be included in the description of the test suite.</li>
>  <li>If an existing manual test case is being automated, then add '(DEPRECATED - Use <automated test suite name>)' 
>  to the beginning of the description of the manual test case.</li>
>  </ol>
></div>
><br>
><div id="section_hlight"><a name="2.1">2.1 Creating Manual Test Suites</a></div>
><div id="main_content">
>	<p>
>	Create small and terse manual test cases that cover well-defined partitions of functionality. 
>	Smaller sized test cases provide easier units of work for one tester to execute, 
>	more realistic reporting statistics and concise steps to reproduce defects.
></p>
><ol>
>	<li>Create manual test suites with sufficient test cases for one tester to complete in less than an hour. 
>	Smaller sized test suites provide easier units of work dividing test suites between testers while decreasing the risk of lost execution data.</li>
>	<li>In the event of manual test suites being executed by multiple testers, 
>		have the tester(s) tabulate the execution histories on paper and the tester with largest assigned test cases will record 
>		all the execution histories in the TPTP Manual Test Client.</li>
>	<li>When structuring the internal organization as the associated test suite folder (for example, manual), 
>		create a summary test suite for the subfolder (for example,/manual/AllTests.testsuite). 
>		The summary test suite (for example,/manual/AllTests.testsuite) will reference all test suites within the current folder. 
>		This summary test suites is registered with the root-level manual test suite 
>		(for example, test-results/platform/org.eclipse.hyades.tests/AllManualTests.testsuite), 
>		thereby reducing work when creating and removing plug-in or component test suites.</li>
>	<li>Although the TPTP Manual Test Client permits recording more than one execution verdict per test case, 
>		it is recommended to only record one execution verdict. 
>		Otherwise, TPTP Testing Tools arbitrates the execution verdicts based on the following precedence rules (decreasing order) 
>		which may cause unintended execution histories:
>		<ul>
>			<li>Error</li>
>			<li>Fail</li>
>			<li>Inconclusive</li>
>			<li>Pass</li>
>		</ul>			
>		</li>
>	</ol>
></div>
><br>
><div id="section_hlight"><a name="2.2">2.2 Creating JUnit and JUnit Plug-in Test Suites</a></div>
><div id="main_content">
>	<ol>
>		<li>Create small and terse unit test cases that cover individual methods or sequence of methods.
>		Smaller sized test cases provide more realistic reporting statistics and concise steps to reproduce defects.</li>
>		<li>When structuring the internal organization as the associated test suite folders (for example, junit and junit_plugin), 
>			create summary test suites for each subfolder (for example,<i>/junit/AllTests.testsuite</i> and <i>/junit_plugin/AllTests.testsuite</i>). 
>			The summary test suites (for example,<i>/junit/AllTests.testsuite and /junit_plugin/AllTests.testsuite</i>) 
>			will reference all test suites within the current folder. 
>			These summary test suites are registered with the root-level JUnit and JUnit Plug-in test suites 
>			(for example, <i>test-results/platform/org.eclipse.hyades.tests/AllJUnitTests.testsuite</i> and <i>test-results/platform/org.eclipse.hyades.tests/AllJUnitPluginTests.testsuite</i>), 
>		thereby reducing work when creating and removing plug-in or component test suites.</li>
>	</ol>
></div>
><br>
><div id="section_hlight"><a name="2.3">2.3 Creating AGR Test Suites</a></div>
><div id="main_content">
>	<p>
>	The user is expected to have reviewed the available 
>	<a href=http://www.eclipse.org/tptp/test/documents/gettingstarted/Automated_GUI_Recorder_Getting_Started.html#3.0>AGR User Guide</a> before reading the recommendations below. 
>	The user guide is available on the document section linked from TPTP's web site.
></p>
><ol>
>	<li>Ensure that a recording session is short. Avoid having a generated macro for a test case that is longer than 100 lines.</li>
>	<li>As mentioned previously make sure that the set of test cases included in a test suite don't perform redundant operations. 
>		For example, if two test cases require launching a process, then launch the process once and use it for each of the test cases that follow. 
>		At the same time the test cases shouldn't be too dependent on the environment setup by previous test cases. 
>		A test case should easily be executable in quick mode without having to run many test cases that precede it. 
>		It becomes difficult to debug a test case when it is dependent on more than one test case that precedes it.</li>
>	<li>Avoid using absolute paths in a test case. Absolute paths should be replaced with relative paths using one of the available static variables 
>		(for example, %testsuiteProjectLocation%). Other modifiable fields that are dependent on the environment that the test suite is launched in 
>		(for example, host name) should be linked to a datapool variable.</li>
>	<li>Use a logger to make it convenient to debug verification hooks. 
>	See <i>test-results/platform/org.eclipse.hyades.use.cases/src/org.eclipse.hyades.use.cases.auto.common/Logger.java</i></li>
>	<li>Comment the test case thoroughly in the description section of the test case. Include all manual steps that the test case is performing.</li>
>	<li>If any resource (such as a datapool) needs to be modified before the test suite is executed, 
>		then include that as part of the description of the test suite. 
>		Any other important information that the tester should be aware of needs to be included in the description of the test suite.</li>
>	<li>If a manual test case is automated, then add '(DEPRECATED - Use &lt;automated test suite name&gt;)' 
>		to the beginning of the description of the manual test case.</li>
>	<li>Avoid position-based recording as much as possible. If position-based recording must be used for a component, 
>		then try to include all such test cases of the component in a separate test suite. 
>		Clearly note the resolution that the tester should use as part of the description of the test suite.</li>
>	<li>Verification hooks should verify the value of controls rather than their logical representation. 
>		For example, if a tree item represents object A, then verify the existence of the tree item by testing that its text field 
>		is the same as the name of object A as opposed to just simply testing for the existence of object A.</li>
></ol>
><p>
>There is a recommended structure for the internal content of the 'gui' folder that committers are encouraged to use when automating existing manual test suites:
></p>
><ol>
>	<li>The folders that are a direct child of the 'gui' folder should correspond exactly to the folders that are a direct child of the 'manual' folder. 
>		When structuring the internal organization as the associated test suite folder (for example, gui), 
>		create a summary test suite for the subfolder (for example,<i>/gui/AllTests.testsuite</i>). 
>		The summary test suite (for example,<i>/gui/AllTests.testsuite</i>) will reference all test suites within the current folder. 
>		This summary test suites is registered with the root-level gui test suite 
>		(for example, <i>test-results/platform/org.eclipse.hyades.tests/AllGUITests.testsuite</i>), 
>		thereby reducing work when creating and removing plug-in or component test suites.</li>
>	<li>The name of the automated gui test suite should correspond exactly to the name of the manual test suite that is being automated.</li>
>	<li>A resource directory should be used to contain the resources required for the verification hooks of a test suite. 
>		The name of the directory should be &lt;test-suite-name&gt;.Resources 
>			(For example the resource directory for the Monitor.UI.LogSets test suite is Monitor.UI.LogSets.Resources. 
>			This directory may contain any number of log files or any other files that the test suite depends on). 
>			If a datapool is being associated with a test suite, then include the datapool as part of this directory 
>			(see the <a href=http://www.eclipse.org/tptp/test/documents/gettingstarted/Automated_GUI_Recorder_Getting_Started.html#3.0>AGR User Guide</a> for more details).</li>
>	<li>Use the directory 'Common.Resources' to store resources that are shared amongst multiple test suites</li>
></ol>
></div>
><br>
><div id="section_hlight"><a name="3.0">3.0 Test Execution Best Practices</a></div>
><div id="main_content">
></div>
><br>
><div id="section_hlight"><a name="3.1">3.1 Smoke Testing</a></div>
><div id="main_content">
>	<p>
>	Smoke testing, or patch/maintenance release testing, refers to partial regression test for testing the majority of the functionality in less time than a full test pass.
>	</p>
>	<p>
>	Smoke test suites should cover:
>		<ul>
>			<li>All major use cases.</li>
>			<li>All major supported platforms.</li>
>			<li>All major supported JREs.</li>
>			<li>All automated test suites (for example, JUnit, JUnit Plug-in, and AGR test suites).</li>
>		</ul>
>	</p>
>	<p>
>	Smoke test suites should reference the test suite(s) and/or test case(s) for the plug-in or component to avoid redundancy.
>	</p>
></div>
><br>
><div id="section_hlight"><a name="4.0">4.0 TPTP Test Project Exceptions</a></div>
><div id="main_content">
>	<p>
>		Some TPTP tests are not manual, AGR or TPTP JUnit/JUnit Plug-in tests as described above.
>	</p>
>	<p>
>		<b>JVMPI Profiler Test Suites</b>
>	</p>
>	<p>
>	  The JVMTI Profiler tests (Platform.Agents.JVMPI.*) are run like regular JUnit tests but require manual configuration of the local host and installation and configuration of a test server on   the remote host. 
>	  This is documented in <i>test-results/platform/org.eclipse.tptp.ac.testautomation/automation-files/notes/ACTestAutomation.doc.	</i>
>	</p>
></div>
><br>
><div id="section_hlight"><a name="5.0">5.0 The Common Test Infrastructure</a></div>
><div id="main_content">
>	<p>
>		The common test infrastructure is used for build verification test.
>	</p>
></div>
><br>
><div id="section_hlight"><a name="5.1">5.1 Reference Platform</a></div>
><div id="main_content">
>	<p>
>	TPTP will use reference platforms. New code will be expected to be tested against the reference platform before being committed.
>	</p>
>	<p>
>	The build verification test will run on the reference platform.
>	</p>
>	<p>
>	The current reference platform for TPTP is: IBM Java 1.5 (latest SR) and Windows XP/x86.
>	</p>
></div>
><br>
><div id="section_hlight"><a name="5.2">5.2 Adding Tests</a></div>
><div id="main_content">
>	<p>
>	Tests can be added to the BVT testsuites which are:<br>
>	org.eclipse.hyades.tests\BVT\AllBVTGUITests.testsuite<br>
>	org.eclipse.hyades.tests\BVT\AllBVTJUnitPluginTests.testsuite<br>
>	org.eclipse.hyades.tests\BVT\AllBVTJUnitTests.testsuite<br>
></p>
></div>
><br>
><div id="section_hlight"><a name="5.3">5.3 Installing and Executing</a></div>
><div id="main_content">
>	<p>
>	All tests in the root level BVT testsuites should be run automatically every build.<br>
>	Developers are expected to test locally before committing changes to CVS.
></p>
><p>
>	For setup and execution please see the BVT Readme at <a href=http://dev.eclipse.org/viewcvs/index.cgi/platform/org.eclipse.tptp.platform.releng.tools/org/eclipse/tptp/platform/releng/tools/testautomation/readme.html?revision=1.1&root=TPTP_Project>
>	org.eclipse.tptp.platform.releng.tools/testautomation/readme.html</a>
></p>
><p>
>	We do not need to check-in the execution results to CVS due to disk
>space limitations and polluting of test pass results. Each developer can rerun the automated tests to reproduce a failure.
></p>
></div>
></body>
></html>

<!DOCTYPE html PUBLIC "-//w3c//dtd html 4.0 transitional//en">
<html><head><title>TPTP Testing Strategy Part 2 - Test Conventions and Best Practices</title>

<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<link rel="stylesheet" type="text/css" href="http://www.eclipse.org/tptp/eclipse_style.css">
<link rel="stylesheet" type="text/css" href="http://www.eclipse.org/tptp/home/documents/process/TPTP_Testing_Strategy_files/strategy_doc_style.css">
<script type="text/javascript">
		function expandOrCollapse()
		{
			element = document.getElementById("invisible");
			if (element != null)
			{
				element.id = "dynamicAppearance";
				document.getElementById("changeHistoryLink").lastChild.nodeValue = "Hide the Change History";
				return;
			}

			element = document.getElementById("dynamicAppearance");
			if (element != null)
			{
				element.id = "invisible";
				document.getElementById("changeHistoryLink").lastChild.nodeValue = "Show the Change History";
			}
		}
  </script></head><body>
<div id="page_header">Eclipse Test and Performance Tools Platform
Project</div>
<div id="page_sub_header">TPTP Testing Strategy Part 2 - Test Conventions and Best Practices</div>
<div id="section_header">TPTP Testing Strategy Part 2 - Test Conventions and Best Practices</div>

<div class="content"><!-- Main content goes here -->
<table id="main_content">
	<tbody>
		<tr>
			<td><b>Title:</b></td>
			<td>TPTP Testing Strategy</td>
		</tr>
		<tr>
			<td><b>Author:</b></td>

			<td>Alan Haggarty (haggarty@ca.ibm.com)</td>
		</tr>
	</tbody>
</table>
<a href="javascript:expandOrCollapse()"><b><span id="changeHistoryLink">Show the Change History</span></b></a>
<table id="invisible" border="1">
	<tbody>
		<tr>
			<th align="left" bgcolor="#dfdfdf" width="10%">Name</th>

			<th align="left" bgcolor="#dfdfdf" width="20%">Date</th>
			<th align="left" bgcolor="#dfdfdf" width="30%">Modified Sections
			</th>
		</tr>
		<tr>
			<td>Alan Haggarty (haggarty@ca.ibm.com)</td>
			<td>June 12, 2008</td>
			<td>Initial Creation</td>

		</tr>
	</tbody>
</table>  	
<br>
<div id="section_hlight">Table of Contents</div>
<div id="main_content">
<a href="#1.0">1.0 TPTP Test Project Conventions</a><br>
<a href="#1.1">1.1 Naming Conventions</a><br>
<a href="#1.2">1.2 Test Project Detailed Structure</a><br>
<a href="#2.0">2.0 Test Creation Best Practices</a><br>
<a href="#2.1">2.1 Creating Manual Test Suites</a><br>
<a href="#2.2">2.2 Creating JUnit and JUnit Plug-in Test Suites</a><br>
<a href="#2.3">2.3 Creating AGR Test Suites</a><br>
<a href="#3.0">3.0 Test Execution Best Practices</a><br>
<a href="#3.1">3.1 Smoke Testing</a><br>
<a href="#4.0">4.0 TPTP Test Project Exceptions</a><br>
<a href="#5.0">5.0 The Common Test Infrastructure</a><br>
<a href="#5.1">5.1 Reference Platform</a><br>
<a href="#5.2">5.2 Adding Tests</a><br>
<a href="#5.3">5.3 Installing and Executing</a><br>
</div>
<br>
<div id="section_hlight"><a name="1.0">1.0 TPTP Test Project Conventions</a></div>
<div id="main_content">
</div>
<br>
<div id="section_hlight"><a name="1.1">1.1 Naming Conventions</a></div>
<div id="main_content">
	<ol>
	 <li>All names for testing resources and folder names MUST adhere to the following rules:
	 	<ul>
      <li>Upper or lower case letter (for example, A-Z or a-z).</li>
      <li>Digits (for example, 0 - 9).</li>
      <li>Underscore (for example, _).</li>
    </ul>
  	</li>
  	<li>Names must NOT contain any white space characters (for example, tab, space, and new line).</li>
  	<li>Every name should begin with an upper or lower case letter (for example, A-Z or a-z). 
  		All other characters in the name should be upper or lower case letters, digits, or the underscore.</li>
  	<li>Java keywords (for example, import, if, etc.) cannot be used for names of JUnit test resources.</li>
  	<li>The name of the Deployment file (deployment files are often used for remote launch of test suites) may contain '.' characters. 
  		The name should be meaningful to the project and the host that it is associated with. 
  		For example, a local deployment file for the org.eclipse.hyades.use.cases folder can be org.eclipse.hyades.use.cases.localhost.deploy</li>
  	<li>The name and directory structure containing the generated execution history file MUST be the same as it will be stored in CVS. 
  		If the generated execution history file is either moved or renamed, the linkage with the associated test suite will break thereby 
  		corrupting generated reports. In the event of an existing execution history in the same directory, 
  		TPTP Testing Tools will append a millisecond time stamp to the name of the new execution history file.</li>
  	<li>Test suite names must start with the bugzilla component name associated with the plug-in or component under test.<br>
  		If the test suite contains several functions, you can either use the bugzilla component name as the name of the test suite or 
  		use the bugzilla component as a prefix.</li>
  	<li>Multiple JRE or platform test suite names must end with the name, vendor, and version of the JRE or platform under test. 
  	These test suites should reference the test suite(s) and/or test case(s) for the plug-in or component to avoid redundancy.</li>
	</ol>
</div>
<br>
<div id="section_hlight"><a name="1.2">1.2 Test Project Detailed Structure</a></div>
<div id="main_content">
	<table border="1" width="100%">
	<tbody>
		<tr>
			<th align="left">Name</th>
			<th align="left">Purpose</th>
		</tr>
		<tr>
			<td>deployment</td>
			<td>Directory containing all deployment configurations for executing the test suites in the test project.</td>
		</tr>
		<tr>
			<td>gui</td>
			<td>Directory containing all AGR test suites for the test project.</td>
		</tr>
		<tr>
			<td>gui/AllTests.testsuite</td>
			<td>AGR test suite to execute all lower level AGR test
			suites for the test project.  Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP1/AllTP1GUITests.testsuite (see Table 2).</td>
		</tr>
		<tr>
			<td>gui/AllSmokeTests.testsuite</td>
			<td>AGR smoke test suite to execute all lower level AGR smoke test
			suites for the test project.  Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP2/AllTP2GUITests.testsuite (see Table 2).</td>
		</tr>
		<tr>
			<td>gui_results</td>
			<td>Directory containing all execution histories for automated
			GUI test suites for the test project.</td>
		</tr>
		<tr>
			<td>junit_plugin</td>
			<td>Directory containing all JUnit Plug-in test suites for the test project.</td>
		</tr>
		<tr>
			<td>junit_plugin/AllTests.testsuite</td>
			<td>JUnit Plug-in test suite to execute all lower level JUnit Plug-in test
			suites for the test project.  Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP1/AllTP1JUnitPluginTests.testsuite (see Table 2).</td>
		</tr>
		<tr>
			<td>junit_plugin/AllSmokeTests.testsuite</td>
			<td>JUnit Plug-in smoke test suite to execute all lower level JUnit Plug-in smoke test
			suites for the test project.  Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP2/AllTP2JUnitPluginTests.testsuite (see Table 2).</td>
		</tr>
		<tr>
			<td>junit_plugin_results</td>
			<td>Directory containing all execution histories for JUnit Plug-in test
			suites for the test project.</td>
		</tr>
		<tr>
			<td>junit</td>
			<td>Directory containing all JUnit test suites for the test project.</td>
		</tr>
		<tr>
			<td>junit/AllTests.testsuite</td>
			<td>JUnit test suite to execute all lower level JUnit test
			suites for the test project.  Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP1/AllTP1JUnitTests.testsuite (see Table 2).</td>
		</tr>
		<tr>
			<td>junit/AllSmokeTests.testsuite</td>
			<td>JUnit smoke test suite to execute all lower level JUnit smoke test
			suites for the test project.  Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP2/AllTP2JUnitTests.testsuite (see Table 2).</td>
		</tr>
		<tr>
			<td>junit_results</td>
			<td>Directory containing all execution histories for JUnit test
			suites for the test project.</td>
		</tr>
		<tr>
			<td>manual</td>
			<td>Directory containing all manual test suites for the test project.</td>
		</tr>
		<tr>
			<td>manual/AllTests.testsuite</td>
			<td>Manual test suite to execute all lower level manual test
			suites for the test project.  Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP1/AllTP1ManualTests.testsuite (see Table 2).</td>
		</tr>
		<tr>
			<td>manual/AllSmokeTests.testsuite</td>
			<td>Manual smoke test suite to execute all lower level manual smoke test
			suites for the test project.  Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP2/AllTP2ManualTests.testsuite (see Table 2).</td>
		</tr>
		<tr>
			<td>manual_results</td>
			<td>Directory containing all execution histories for manual test
			suites for the test project.</td>
		</tr>
		<tr>
			<td>resources</td>
			<td>Directory containing any resources required to execute the
			test suites for the test project.</td>
		</tr>
		<tr>
			<td>src</td>
			<td>Directory containing generated JUnit Java source code for
			the JUnit and JUnit Plug-in test suites and Java source code corresponding to
			verification hooks of AGR test suites for the test project.</td>
		</tr>
	</tbody>
</table>
<b>Table 1: Required structure for contents of a test project.</b>
<p>
Each of the subfolders contains a test project named org.eclipse.hyades.use.cases for testing functionality 
that crosses plug-in or component boundaries.
</p>

<p>
The <a href="http://dev.eclipse.org/viewcvs/index.cgi/test-results/platform/org.eclipse.hyades.tests/?root=TPTP_Project">test-results/platform/org.eclipse.hyades.tests</a> 
test project contains all root-level test suites for TPTP.  Table 2 details the required structure for contents of the test-results/platform/org.eclipse.hyades.tests test project.
</p>

<table border="1" width="100%">
	<tbody>
		<tr>
			<th align="left">Name</th>
			<th align="left">Purpose</th>
		</tr>
		<tr>
			<td>deployment</td>

			<td>Directory containing all deployment configurations for executing the root-level test suites in TPTP.</td>
		</tr>
		<tr>
			<td>enablement</td>
			<td>Directory containing all the TPTP enablement tests.</td>
		</tr>
		<tr>

			<td>TVT</td>
			<td>Directory containing all the TPTP Translation Verification Tests (TVT).</td>
		</tr>
		<tr>
			<td>src</td>
			<td>Directory containing generated JUnit Java source code for
			the root-level JUnit and JUnit Plug-in test suites in TPTP and Java source code corresponding to
			verification hooks of root-level AGR test suites in TPTP.</td>
	  </tr>

		<tr>
			<td>TP1</td>
			<td>Directory containing all root-level test suites to execute all lower level test suites 
				during test pass 1 (TP1).</td>
		</tr>
		<tr>
			<td>TP1/AllTP1GUITests.testsuite</td>
			<td>Root-level AGR test suite to execute all lower level AGR test suites during test pass 1</td>
		</tr>	
		<tr>
			<td>TP1/AllTP1JUnitPluginTests.testsuite</td>
			<td>Root-level JUnit Plug-in test suite to execute all lower level JUnit Plug-in test suites during test pass 1</td>
		</tr>		
		<tr>
			<td>TP1/AllTP1JUnitTests.testsuite</td>
			<td>Root-level JUnit test suite to execute all lower level JUnit test suites during test pass 1</td>
		</tr>	
		<tr>
			<td>TP1/AllTP1ManualTests.testsuite</td>
			<td>Root-level manual test suite to execute all lower level manual test suites during test pass 1</td>
		</tr>			
		
		<tr>
			<td>TP2</td>
			<td>Directory containing all root-level test suites to execute all lower level test suites 
				during test pass 2 (TP2).</td>
		</tr>
		<tr>
			<td>TP2/AllTP2GUITests.testsuite</td>
			<td>Root-level AGR test suite to execute all lower level AGR test suites during test pass 2</td>
		</tr>	
		<tr>
			<td>TP2/AllTP2JUnitPluginTests.testsuite</td>
			<td>Root-level JUnit Plug-in test suite to execute all lower level JUnit Plug-in test suites 
				during test pass 2</td>
		</tr>		
		<tr>
			<td>TP2/AllTP2JUnitTests.testsuite</td>
			<td>Root-level JUnit test suite to execute all lower level JUnit test suites during test pass 2</td>
		</tr>	
		<tr>
			<td>TP2/AllTP2ManualTests.testsuite</td>
			<td>Root-level manual test suite to execute all lower level manual test suites during test pass 2</td>
		</tr>		
		
		<tr>
			<td>BVT</td>
			<td>Directory containing all root-level test suites to execute all lower level test suites 
				during build verification test (BVT).</td>
		</tr>
		<tr>
			<td>BVT/AllBVTGUITests.testsuite</td>
			<td>Root-level AGR test suite to execute all lower level AGR test suites during build verification test</td>
		</tr>	
		<tr>
			<td>BVT/AllBVTJUnitPluginTests.testsuite</td>
			<td>Root-level JUnit Plug-in test suite to execute all lower level JUnit Plug-in test suites 
				during build verification tests</td>
		</tr>		
		<tr>
			<td>BVT/AllBVTJUnitTests.testsuite</td>
			<td>Root-level JUnit test suite to execute all lower level JUnit test suites during build verification tests</td>
		</tr>
			
	</tbody>
</table>
<b>Table 2: Required structure for contents of the test-results/platform/org.eclipse.hyades.tests test project.</b>

<p>The execution history files (*.execution) generated when executing test suites are checked-in to the CVS branch for the 
release under test by the committer for the associated plug-in or component.  For example, the execution history files generated from 
testing the current full release would be checked in to HEAD.  Execution history files are checked in to the *_results folder for the type of 
test suite.  For example, execution history files generated from executing a manual test suite would be checked in the manual_results folder.  To 
delineate between test executions on multiple platforms, each *_results folder contains the following subfolders for each supported platform:

</p>

<ul>
	<li>AIX</li>
	<li>AS_400</li>
	<li>HP_UX</li>
	<li>Linux_390</li>
	<li>Linux_x86</li>

	<li>OS_390</li>
	<li>Solaris</li>
	<li>Windows</li>
	<li>Linux PPC64</li>
</ul>

<p>
Supported platform subfolders may be optionally appended with the platform version and release:
</p>

<ul>
	<li>AIX_5_2_0</li>
	<li>AS_400_5_2</li>
	<li>HP_UX_11</li>
	<li>Linux_390_SuSE_ Enterprise_7_0</li>
	<li>Linux_x86_RedHat_Enterprise_2_1</li>

	<li>Linux_x86_SLES_10</li>
	<li>OS_390_1_6</li>
	<li>Solaris_9</li>
	<li>Windows_2000</li>
	<li>Windows_XP</li>
	<li>Windows_Vista</li>

	<li>Linux_PPC64<br>
	</li>
</ul>

<p>Test execution reports are generated and checked in to CVS by the
TPTP Project Lead or PMC representative at predetermined milestones in a 
test pass for public web access.  Test execution reports are available
at: <a href="http://www.eclipse.org/tptp/">www.eclipse.org/tptp</a> &gt;&gt; Development Plans &gt;&gt; Test Pass Reports
</p>
</div>
<br>
<div id="section_hlight"><a name="2.0">2.0 Test Creation Best Practices</a></div>
<div id="main_content">
	<p>
	Based on the previous experiences of test resource creators and testers, the following general recommendations will assist 
	testers on creating test cases. Section 2.1 - 2.3 go into more details about best practices for creating manual, JUnit, JUnit Plug-in, 
	and AGR test suites.
</p>
<ol>   
	<li>A test case should always have a specific purpose for testing a scenario. Having many test cases in a test suite does not necessarily ensure better quality. Each test case must be well thought out and have a specific purpose. Simply testing the basic functionality of a tool is often not enough to ensure high quality standards. Boundary cases are usually considered to be interesting test cases. As an example, consider the case of testing a function that removes entries from a table. The following test cases should be considered:
		<ul>
     <li>Removing the first entry (a boundary case)</li>
     <li>Removing the second entry (tests basic functionality)</li>
     <li>Removing the last entry (a boundary case)</li>
    </ul>
   </li>
  <li>Avoid repeating the same set of steps in multiple test cases. If two test cases require the user to import a log file, 
  	then create three test cases. The first test case will only serve as setting the environment for successive test cases and the 
  	second and third test case will test the specific functionality in mind.</li>
  <li>This point provides a basic summary of the first two: apply the high cohesion principle to test cases. 
  	Cohesion means assigning specific responsibilities to test cases. A test case requiring a user to import a log file, 
  	profile an application, and use the Probekit has little common in its responsibilities and thus does not follow the high cohesion principle.</li>
  <li>Avoid unnecessarily complicating test cases that are designed to test functionality and not performance. 
  	For example, a test case that uses a class with 3 methods to test the Java Profiler is likely as good as a test case 
  	that uses a class with 10 methods.</li>
  <li>Comment test cases thoroughly in the description section of the editor including manual steps for what the test case is performing.</li>
  <li>If any resource (such as a datapool) needs to be modified before the test suite is executed, 
   	then include that as part of the description of the test suite. 
  Any other important information that the tester should be aware of needs to be included in the description of the test suite.</li>
  <li>If an existing manual test case is being automated, then add '(DEPRECATED - Use <automated test suite name>)' 
  to the beginning of the description of the manual test case.</li>
  </ol>
</div>
<br>
<div id="section_hlight"><a name="2.1">2.1 Creating Manual Test Suites</a></div>
<div id="main_content">
	<p>
	Create small and terse manual test cases that cover well-defined partitions of functionality. 
	Smaller sized test cases provide easier units of work for one tester to execute, 
	more realistic reporting statistics and concise steps to reproduce defects.
</p>
<ol>
	<li>Create manual test suites with sufficient test cases for one tester to complete in less than an hour. 
	Smaller sized test suites provide easier units of work dividing test suites between testers while decreasing the risk of lost execution data.</li>
	<li>In the event of manual test suites being executed by multiple testers, 
		have the tester(s) tabulate the execution histories on paper and the tester with largest assigned test cases will record 
		all the execution histories in the TPTP Manual Test Client.</li>
	<li>When structuring the internal organization as the associated test suite folder (for example, manual), 
		create a summary test suite for the subfolder (for example,/manual/AllTests.testsuite). 
		The summary test suite (for example,/manual/AllTests.testsuite) will reference all test suites within the current folder. 
		This summary test suites is registered with the root-level manual test suite 
		(for example, test-results/platform/org.eclipse.hyades.tests/AllManualTests.testsuite), 
		thereby reducing work when creating and removing plug-in or component test suites.</li>
	<li>Although the TPTP Manual Test Client permits recording more than one execution verdict per test case, 
		it is recommended to only record one execution verdict. 
		Otherwise, TPTP Testing Tools arbitrates the execution verdicts based on the following precedence rules (decreasing order) 
		which may cause unintended execution histories:
		<ul>
			<li>Error</li>
			<li>Fail</li>
			<li>Inconclusive</li>
			<li>Pass</li>
		</ul>			
		</li>
	</ol>
</div>
<br>
<div id="section_hlight"><a name="2.2">2.2 Creating JUnit and JUnit Plug-in Test Suites</a></div>
<div id="main_content">
	<ol>
		<li>Create small and terse unit test cases that cover individual methods or sequence of methods.
		Smaller sized test cases provide more realistic reporting statistics and concise steps to reproduce defects.</li>
		<li>When structuring the internal organization as the associated test suite folders (for example, junit and junit_plugin), 
			create summary test suites for each subfolder (for example,<i>/junit/AllTests.testsuite</i> and <i>/junit_plugin/AllTests.testsuite</i>). 
			The summary test suites (for example,<i>/junit/AllTests.testsuite and /junit_plugin/AllTests.testsuite</i>) 
			will reference all test suites within the current folder. 
			These summary test suites are registered with the root-level JUnit and JUnit Plug-in test suites 
			(for example, <i>test-results/platform/org.eclipse.hyades.tests/AllJUnitTests.testsuite</i> and <i>test-results/platform/org.eclipse.hyades.tests/AllJUnitPluginTests.testsuite</i>), 
		thereby reducing work when creating and removing plug-in or component test suites.</li>
	</ol>
</div>
<br>
<div id="section_hlight"><a name="2.3">2.3 Creating AGR Test Suites</a></div>
<div id="main_content">
	<p>
	The user is expected to have reviewed the available 
	<a href=http://www.eclipse.org/tptp/test/documents/gettingstarted/Automated_GUI_Recorder_Getting_Started.html#3.0>AGR User Guide</a> before reading the recommendations below. 
	The user guide is available on the document section linked from TPTP's web site.
</p>
<ol>
	<li>Ensure that a recording session is short. Avoid having a generated macro for a test case that is longer than 100 lines.</li>
	<li>As mentioned previously make sure that the set of test cases included in a test suite don't perform redundant operations. 
		For example, if two test cases require launching a process, then launch the process once and use it for each of the test cases that follow. 
		At the same time the test cases shouldn't be too dependent on the environment setup by previous test cases. 
		A test case should easily be executable in quick mode without having to run many test cases that precede it. 
		It becomes difficult to debug a test case when it is dependent on more than one test case that precedes it.</li>
	<li>Avoid using absolute paths in a test case. Absolute paths should be replaced with relative paths using one of the available static variables 
		(for example, %testsuiteProjectLocation%). Other modifiable fields that are dependent on the environment that the test suite is launched in 
		(for example, host name) should be linked to a datapool variable.</li>
	<li>Use a logger to make it convenient to debug verification hooks. 
	See <i>test-results/platform/org.eclipse.hyades.use.cases/src/org.eclipse.hyades.use.cases.auto.common/Logger.java</i></li>
	<li>Comment the test case thoroughly in the description section of the test case. Include all manual steps that the test case is performing.</li>
	<li>If any resource (such as a datapool) needs to be modified before the test suite is executed, 
		then include that as part of the description of the test suite. 
		Any other important information that the tester should be aware of needs to be included in the description of the test suite.</li>
	<li>If a manual test case is automated, then add '(DEPRECATED - Use &lt;automated test suite name&gt;)' 
		to the beginning of the description of the manual test case.</li>
	<li>Avoid position-based recording as much as possible. If position-based recording must be used for a component, 
		then try to include all such test cases of the component in a separate test suite. 
		Clearly note the resolution that the tester should use as part of the description of the test suite.</li>
	<li>Verification hooks should verify the value of controls rather than their logical representation. 
		For example, if a tree item represents object A, then verify the existence of the tree item by testing that its text field 
		is the same as the name of object A as opposed to just simply testing for the existence of object A.</li>
</ol>
<p>
There is a recommended structure for the internal content of the 'gui' folder that committers are encouraged to use when automating existing manual test suites:
</p>
<ol>
	<li>The folders that are a direct child of the 'gui' folder should correspond exactly to the folders that are a direct child of the 'manual' folder. 
		When structuring the internal organization as the associated test suite folder (for example, gui), 
		create a summary test suite for the subfolder (for example,<i>/gui/AllTests.testsuite</i>). 
		The summary test suite (for example,<i>/gui/AllTests.testsuite</i>) will reference all test suites within the current folder. 
		This summary test suites is registered with the root-level gui test suite 
		(for example, <i>test-results/platform/org.eclipse.hyades.tests/AllGUITests.testsuite</i>), 
		thereby reducing work when creating and removing plug-in or component test suites.</li>
	<li>The name of the automated gui test suite should correspond exactly to the name of the manual test suite that is being automated.</li>
	<li>A resource directory should be used to contain the resources required for the verification hooks of a test suite. 
		The name of the directory should be &lt;test-suite-name&gt;.Resources 
			(For example the resource directory for the Monitor.UI.LogSets test suite is Monitor.UI.LogSets.Resources. 
			This directory may contain any number of log files or any other files that the test suite depends on). 
			If a datapool is being associated with a test suite, then include the datapool as part of this directory 
			(see the <a href=http://www.eclipse.org/tptp/test/documents/gettingstarted/Automated_GUI_Recorder_Getting_Started.html#3.0>AGR User Guide</a> for more details).</li>
	<li>Use the directory 'Common.Resources' to store resources that are shared amongst multiple test suites</li>
</ol>
</div>
<br>
<div id="section_hlight"><a name="3.0">3.0 Test Execution Best Practices</a></div>
<div id="main_content">
</div>
<br>
<div id="section_hlight"><a name="3.1">3.1 Smoke Testing</a></div>
<div id="main_content">
	<p>
	Smoke testing, or patch/maintenance release testing, refers to partial regression test for testing the majority of the functionality in less time than a full test pass.
	</p>
	<p>
	Smoke test suites should cover:
		<ul>
			<li>All major use cases.</li>
			<li>All major supported platforms.</li>
			<li>All major supported JREs.</li>
			<li>All automated test suites (for example, JUnit, JUnit Plug-in, and AGR test suites).</li>
		</ul>
	</p>
	<p>
	Smoke test suites should reference the test suite(s) and/or test case(s) for the plug-in or component to avoid redundancy.
	</p>
</div>
<br>
<div id="section_hlight"><a name="4.0">4.0 TPTP Test Project Exceptions</a></div>
<div id="main_content">
	<p>
		Some TPTP tests are not manual, AGR or TPTP JUnit/JUnit Plug-in tests as described above.
	</p>
	<p>
		<b>JVMPI Profiler Test Suites</b>
	</p>
	<p>
	  The JVMTI Profiler tests (Platform.Agents.JVMPI.*) are run like regular JUnit tests but require manual configuration of the local host and installation and configuration of a test server on   the remote host. 
	  This is documented in <i>test-results/platform/org.eclipse.tptp.ac.testautomation/automation-files/notes/ACTestAutomation.doc.	</i>
	</p>
</div>
<br>
<div id="section_hlight"><a name="5.0">5.0 The Common Test Infrastructure</a></div>
<div id="main_content">
	<p>
		The common test infrastructure is used for build verification test.
	</p>
</div>
<br>
<div id="section_hlight"><a name="5.1">5.1 Reference Platform</a></div>
<div id="main_content">
	<p>
	TPTP will use reference platforms. New code will be expected to be tested against the reference platform before being committed.
	</p>
	<p>
	The build verification test will run on the reference platform.
	</p>
	<p>
	The current reference platform for TPTP is: IBM Java 1.5 (latest SR) and Windows XP/x86.
	</p>
</div>
<br>
<div id="section_hlight"><a name="5.2">5.2 Adding Tests</a></div>
<div id="main_content">
	<p>
	Tests can be added to the BVT testsuites which are:<br>
	org.eclipse.hyades.tests\BVT\AllBVTGUITests.testsuite<br>
	org.eclipse.hyades.tests\BVT\AllBVTJUnitPluginTests.testsuite<br>
	org.eclipse.hyades.tests\BVT\AllBVTJUnitTests.testsuite<br>
</p>
</div>
<br>
<div id="section_hlight"><a name="5.3">5.3 Installing and Executing</a></div>
<div id="main_content">
	<p>
	All tests in the root level BVT testsuites should be run automatically every build.<br>
	Developers are expected to test locally before committing changes to CVS.
</p>
<p>
	For setup and execution please see the BVT Readme at <a href=http://dev.eclipse.org/viewcvs/index.cgi/platform/org.eclipse.tptp.platform.releng.tools/org/eclipse/tptp/platform/releng/tools/testautomation/readme.html?revision=1.1&root=TPTP_Project>
	org.eclipse.tptp.platform.releng.tools/testautomation/readme.html</a>
</p>
<p>
	We do not need to check-in the execution results to CVS due to disk
space limitations and polluting of test pass results. Each developer can rerun the automated tests to reproduce a failure.
</p>
</div>
</body>
</html>

Actions: View

Attachments on bug 211777: 102509 | 102510 | 104770 | 104771 | 105529 | 105532