View All Attachments for Bug 211777

Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Obsolete attachments are hidden. To view all attachments (including obsolete) click here.

Attachment #105529
Updated part 1	text/html	2008-06-20 14:24:02 EDT	15.59 KB	no flags	Details

<!DOCTYPE html PUBLIC "-//w3c//dtd html 4.0 transitional//en">
<html><head><title>TPTP Testing Strategy Part 1 - Creating and Executing TPTP Tests</title>



<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<link rel="stylesheet" type="text/css" href="http://www.eclipse.org/tptp/eclipse_style.css">
<link rel="stylesheet" type="text/css" href="http://www.eclipse.org/tptp/home/documents/process/TPTP_Testing_Strategy_files/strategy_doc_style.css">
<script type="text/javascript">
		function expandOrCollapse()
		{
			element = document.getElementById("invisible");
			if (element != null)
			{
				element.id = "dynamicAppearance";
				document.getElementById("changeHistoryLink").lastChild.nodeValue = "Hide the Change History";
				return;
			}

			element = document.getElementById("dynamicAppearance");
			if (element != null)
			{
				element.id = "invisible";
				document.getElementById("changeHistoryLink").lastChild.nodeValue = "Show the Change History";
			}
		}
  </script></head><body>
<div id="page_header">Eclipse Test and Performance Tools Platform
Project</div>
<div id="page_sub_header">TPTP Testing Strategy Part 1 - Creating and Executing TPTP Tests</div>
<div id="section_header">TPTP Testing Strategy Part 1 - Creating and Executing TPTP Tests</div>

<div class="content"><!-- Main content goes here -->
<table id="main_content">
	<tbody>
		<tr>
			<td><b>Title:</b></td>
			<td>TPTP Testing Strategy</td>
		</tr>
		<tr>
			<td><b>Author:</b></td>

			<td>Alan Haggarty (haggarty@ca.ibm.com)</td>
		</tr>
	</tbody>
</table>
<a href="javascript:expandOrCollapse()"><b><span id="changeHistoryLink">Show the Change History</span></b></a>
<table id="invisible" border="1">
	<tbody>
		<tr>
			<th align="left" bgcolor="#dfdfdf" width="10%">Name</th>

			<th align="left" bgcolor="#dfdfdf" width="20%">Date</th>
			<th align="left" bgcolor="#dfdfdf" width="30%">Modified Sections
			</th>
		</tr>
		<tr>
			<td>Alan Haggarty (haggarty@ca.ibm.com)</td>
			<td>June 4, 2008</td>
			<td>Initial Creation</td>

		</tr>
	</tbody>
</table>  	
<br>
<div id="section_hlight">Table of Contents</div>
<div id="main_content">
<a href="#1.0">1.0 Preparation</a><br>
<a href="#1.1">1.1 Install the TPTP Testing Tools</a><br>
<a href="#1.2">1.2 Extracting the Test Resources</a><br>
<a href="#1.3">1.3 Structure of the Test Resources</a><br>
<a href="#2.0">2.0 Creating New TPTP Tests</a><br>
<a href="#3.0">3.0 Executing Tests</a><br>
<a href="#3.1">3.1 Creating Test Deployments</a><br>
<a href="#3.2">3.2 Executing Manual Test Suites</a><br>
<a href="#3.3">3.3 Executing JUnit and JUnit Plug-in Test Suites</a><br>
<a href="#3.4">3.4 Executing AGR Test Suites</a><br>
<a href="#3.5">3.5 Viewing Test Logs</a><br>
<a href="#3.6">3.6 Submitting Execution Histories</a><br>
<a href="#4.0">4.0 Help</a><br>
</div>
<br>
<div id="section_hlight"><a name="1.0">1.0 Preparation</a></div>
<div id="main_content">
</div>
<br>
<div id="section_hlight"><a name="1.1">1.1 Install the TPTP Testing Tools</a></div>
<div id="main_content">
	<ol>
		<li>Open the <a href=http://www.eclipse.org/tptp/home/downloads>TPTP download site.</a></li>
	  <li>Select the release of TPTP to be tested and download and unzip the TPTP plugins as well as the requirements listed 
	  	in the TPTP Plugins for Eclipse section.</li>
	  <li>Download and install the Manual Test Tools ("As Is Components" section).</li>
	  <li>Download and install the BIRT Test Reports ("As Is Components" section).</li>
	  <li>Download and install Automated GUI Recording (As Is Components section).
	         <br><i>Note: Remember to restart eclipse with the clean option after installing the AGR</i></li>
	  <li>If remote execution is required download and unzip the Agent Controller (Agent Controller section).</li>
	  <li>The Installation Guide and Release Notes are located in the Documentation section.</li>
	</ol>
</div>
<br>
<div id="section_hlight"><a name="1.2">1.2 Extracting the Test Resources</a></div>
<div id="main_content">
	<p>Test suites, cases, deployment configurations and execution histories are stored in the TPTP CVS repository 
		(dev.eclipse.org:/cvsroot/tptp  or www.eclipse.org/tptp  &gt;&gt; CVS Repository), under the test-results  module.</p>
	<p>The test-results module contains subfolders for each of the four subprojects - monitoring, platform, test and trace
		 - in the TPTP project. Each subfolder contains the test projects for that subproject.</p>
	<ol>
		 <li>Check out the individual test projects under test-results/monitoring/, test-results/platform/, test-results/test/, or test-results/trace/ to the local workspace.
						<br><b>For Example: </b><a href=http://dev.eclipse.org/viewcvs/index.cgi/test-results/platform/org.eclipse.hyades.use.cases/?root=TPTP_Project>test-results/platform/org.eclipse.hyades.use.cases</a></li>

   	<li>To ensure that the local workspace contains the latest test projects, update at the beginning of each test pass and report generation.</li>
	</ol>
</div>
<br>
<div id="section_hlight"><a name="1.3">1.3 Structure of the Test Resources</a></div>
<div id="main_content">
	<p>Test projects must follow a defined structure.
	</p>
	<p>
	Please see Part 2 for the required structure.<br>
	<a href=TPTP_Testing_Strategy_Part_2.html#1.2>Test Project Detailed Structure</a><br>
	<a href=TPTP_Testing_Strategy_Part_2.html#2.0>Test Creation Best Practices</a><br>
</p>
</div>
<br>
<div id="section_hlight"><a name="2.0">2.0 Creating New TPTP Tests</a></div>
<div id="main_content">
	<p>To create new TPTP Manual, JUnit, JUnit Plug-in and AGR test suites and cases please follow the testing documentation
		for each type of test. The location of the documentation can be found in the <a href="#4.0">Help section</a> of this document.
	</p>
	<p>The <a href=http://www.eclipse.org/tptp/home/documents/process/TPTP_Manual_Test_Case_Generator.html>TPTP Manual Test Case Generator</a>
		 may be used to generate the structure of manual test case descriptions based on the standardized structured manual test case format.
	</p> 
	<p>Please see <a href=TPTP_Testing_Strategy_Part_2.html#2.0>Test Creation Best Practices</a> in Part 2 for further information on 
	creating effective TPTP test cases.
	</p>
</div>
<br>
<div id="section_hlight"><a name="3.0">3.0 Executing Tests</a></div>
<div id="main_content">
</div>
<br>
<div id="section_hlight"><a name="3.1">3.1 Creating Test Deployments</a></div>
<div id="main_content">
	<p>
	Creating Test Deployments: Detailed steps for creating test deployments can be found in Eclipse Help &gt;&gt; Help Contents &gt;&gt; Testing Applications &gt;&gt; Creating a Test Deployment
	</p>
	<p>Create a Test Launch Configuration</p>
	<ol>
   <li>Create a new launch configuration by selecting the entry Test from the list of configurations in Run &gt;&gt; Run Configurations</li>
   <li>Use the Select test to run pane to navigate to and select the test suite you wish to run.</li>
   <li>Select the deployment for running the test suite. By default the test will be deployed locally. Manual tests are always run in the local deployment.</li>
   <li>In the Test Logs tab override the default log configuration to match the desired log directory (for example plugin_results). 
   	Please refer to the <a href=TPTP_Testing_Strategy_Part_2.html#1.1>Naming Conventions</a> section of Part 2 for the names of result directories.</li>
  </ol>
</div>
<br>
<div id="section_hlight"><a name="3.2">3.2 Executing Manual Test Suites</a></div>
<div id="main_content">
	<ol>
	   <li>Open the Test perspective.</li>
   <li>In the Test Navigator view, select the manual test suite to be executed.</li>
   <li>Launch the selected test suite by creating a Test configuration in the Run dialog
   	<ul>
		<li>Under the Test tab select the test suite</li>
		<li>Choose the results folder associated with the selected test suite and deployment platform<br>
	<b>Note:</b> The name and directory structure containing the generated execution history file 
	<b>MUST</b> be the same as it will be in CVS in order for generated reports to work correctly.</li>
		<li>Click Run</li>
		</ul>
		</li>
   <li>Launch a candidate workbench on the testing platform</li>
   <li>Execute each test case as specified in the TPTP Manual Test View on the local host.</li>
   <li>Record the result of the test case execution by capturing the status of the test case execution using the following explanation for TPTP Testing Tools Status options:
	<table border="1" width="100%">
		<tbody>
			<tr>
				<th align="left">TPTP Testing Tools Status</th>
				<th align="left">Explanation</th>
			</tr>
			<tr>
				<td>Error</td>
				<td>Do not use.</td>
			</tr>
			<tr>
				<td>Fail</td>
				<td>The test case could be executed and failed due to a
				blocking defect. <br>
				</td>
			</tr>
			<tr>
				<td>Inconclusive</td>
				<td>The test case was not applicable OR the test case could be executed and failed due to a non-blocking defect.
			A non-blocking defect is defined as a nice-to-have fix that does not degrade the core functionality being tested by the test case. 
			</td>
			</tr>
			<tr>
				<td>Pass</td>
				<td>The test case could be executed and passed.</td>
			</tr>
		</tbody>
		
	</table>
	For verdicts of Fail or Inconclusive please provide a defect number and explanation in the test log.</li>
</ol>
</div>
<br>
<div id="section_hlight"><a name="3.3">3.3 Executing JUnit and JUnit Plug-in Test Suites</a></div>
<div id="main_content">
	<ol>
		<li>Open the Test perspective.</li>
   	<li>In the Test Navigator view, select the JUnit test suite to be executed.</li>
   	<li>Launch the selected test suite by creating a Test configuration in the Run dialog
   		<ul>
				<li>Under the Test tab select the test suite</li>
				<li>Select the deployment.<br>
				To run a test remotely you will need to create and configure a new test deployment<br>
				For JUnit plug-in tests you also may want to specify deployment on a different workbench location.</li>
				<li>Choose the results folder associated with the selected test suite and deployment platform<br>
				<b>Note: </b>The name and directory structure containing the generated execution history file MUST be the same as it will be in CVS in order for generated reports to work correctly.</li>
				<li>Click Run</li>
			</ul>
		</li>
   	<li>An execution history will be generated according to the following explanation of TPTP Testing Tools Status options:
	<table border="1" width="100%">
		<tbody>
			<tr>
				<th align="left">TPTP Testing Tools Status</th>
				<th align="left">Explanation</th>
			</tr>
			<tr>
				<td>Error</td>
				<td>The test case was or could not be executed due to a run
				time error.</td>
			</tr>
			<tr>
				<td>Fail</td>
				<td>The test case could be executed and failed due to a defect.</td>
			</tr>
			<tr>
				<td>Pass</td>
				<td>The test case could be executed and passed.</td>
			</tr>
		</tbody>
	</table>
	The Fail and Error verdict events in the test log will contain a stack trace that can jump to source.</li>
</ol>
</div>
<br>
<div id="section_hlight"><a name="3.4">3.4 Executing AGR Test Suites</a></div>
<div id="main_content">
	<ol>
		<li>Ensure that the Agent Controller is installed and started.</li>
   <li>Open the Test perspective.</li>
   <li>In the Test Navigator view, select the Automated GUI Test to be executed.</li>
   <li>Execute the test suite by creating a launch configuration item.<br>
		<b>Important:</b> Using the 'Quick Run' mode will not generate an execution history file. Make sure that a proper launch configuration is used.</li>
   <li>The runner will generate an execution history according to the following explanation of TPTP Testing Tools Status options:
   <table border="1" width="100%">
		<tbody>
			<tr>
				<th align="left">TPTP Testing Tools Status</th>
				<th align="left">Explanation</th>
			</tr>
			<tr>
				<td>Error</td>
				<td>The test case was or could not be executed due to a run
				time error.</td>
			</tr>
			<tr>
				<td>Fail</td>
				<td>The test case could be executed and failed due to a defect.</td>
			</tr>
			<tr>
				<td>Pass</td>
				<td>The test case could be executed and passed.</td>
			</tr>
		</tbody>
	</table>	
	The Fail and Error verdict events in the test log will contain a stack trace that can jump to source.
   	</li>
	</ol>
	
	
</div>
<br>
<div id="section_hlight"><a name="3.5">3.5 Viewing Test Logs</a></div>
<div id="main_content">
	<p>
		To open a test log file, use the Open File window, or double-click in the Test Navigator the file with the extension, .execution.
	</p>
	<ol>
   <li>From the File menu, click Open File.</li>
   <li>Browse to and select the test log file to open. (Test log files use the extension, .execution.)</li>
   <li>Click OK.</li>
	</ol>
</div>
<br>
<div id="section_hlight"><a name="3.6">3.6 Submitting Execution Histories</a></div>
<div id="main_content">
	<ol>
		<li>Execution histories should be checked-in to the CVS branch for the release under test.  
			They are checked into a results folder under the associated plugin for the type of test executed 
			(manual_results, junit_results, junit_plugin_results, gui_results). To delineate between test executions on multiple platforms, 
			each results folder contains subfolders for the supported platforms.
			<br>
			<b>For Example: </b>The results of executing the manual test 
			<i>org.eclipse.hyades.use.cases/manual/Profiling_and_Logging/Platform.Communication.Control_Channel_Remote_Profiling.Windows_IA32.testsuite</i> 
			should be generated to and checked into <i>org.eclipse.hyades.use.cases/manual_results/Profiling_and_Logging/Windows_XP</i>
		</li>
   <li>Testers who are not committers should email the execution history files to the committer for the associated component.
			<br>
			Include the following in your email:
			<ul>
				<li>Name of tester if not same as sender</li>
				<li>Deployment platform(s) including name, version and release.</li>
				<li>Execution specific information including name, version and release.</li>
				<li>TPTP driver build id and test pass identifier.</li>
			</ul>
		</li>
	</ol>
</div>
<br>
<div id="section_hlight"><a name="4.0">4.0 Help</a></div>
<div id="main_content">
	<p>
	<b>General Help</b><br><br>
	TPTP General documentation can be found in the Eclipse Help under Help &gt;&gt; Help Contents &gt;&gt; Testing Applications<br>
	<a href=http://www.eclipse.org/tptp/test/documents/gettingstarted/Automated_GUI_Recorder_Getting_Started.html>
		Getting Started with TPTP's Automated GUI Recorder</a><br>
	<a href=http://www.eclipse.org/tptp/test/documents/gettingstarted/Manual_Test_Tools_Getting_Started.html>
		Getting Started with TPTP's Manual Test Tools</a><br>
	<a href=http://www.eclipse.org/tptp/test/documents/gettingstarted/BIRT_Test_Reports_Getting_Started.html>
	Getting Started with TPTP's Business Intelligence and Reporting Tools (BIRT) Test Reports</a><br>
</p>
<p>
	<b>Creating Tests</b><br><br>
	Manual test cases and suites: Eclipse Help &gt;&gt; Help Contents &gt;&gt;  Testing Applications &gt;&gt; Testing Manually<br>
	TPTP JUnit: Eclipse Help &gt;&gt; Help Contents &gt;&gt;  Testing Applications &gt;&gt; Testing with JUnit<br>
	TPTP JUnit Plug-in: Eclipse Help &gt;&gt; Help Contents &gt;&gt; Testing Applications &gt;&gt; Testing plug-ins with JUnit<br>
	AGR: See the <a href=http://www.eclipse.org/tptp/test/documents/gettingstarted/Automated_GUI_Recorder_Getting_Started.html#3.0>
		Documentation section of the AGR Getting Started Guide</a><br>
</p>
	<p>
	<b>More Resources</b><br><br>
	<a href=http://www.eclipse.org/tptp/>Eclipse Test & Performance Tools Platform Project Web Site</a><br>
	<a href=http://www.eclipse.org/tptp/home/documents/extenders.php>Documentation for TPTP extenders</a><br>
	<a href=http://www.eclipse.org/tptp/home/project_info/general/mailnews.php>TPTP Mailing Lists and Newsgroups</a>
	</p>
</div>
<br>
</body>
</html>

Attachment #105532
Updated Part 2	text/html	2008-06-20 14:30:43 EDT	28.28 KB	no flags	Details

<!DOCTYPE html PUBLIC "-//w3c//dtd html 4.0 transitional//en">
<html><head><title>TPTP Testing Strategy Part 2 - Test Conventions and Best Practices</title>

<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<link rel="stylesheet" type="text/css" href="http://www.eclipse.org/tptp/eclipse_style.css">
<link rel="stylesheet" type="text/css" href="http://www.eclipse.org/tptp/home/documents/process/TPTP_Testing_Strategy_files/strategy_doc_style.css">
<script type="text/javascript">
		function expandOrCollapse()
		{
			element = document.getElementById("invisible");
			if (element != null)
			{
				element.id = "dynamicAppearance";
				document.getElementById("changeHistoryLink").lastChild.nodeValue = "Hide the Change History";
				return;
			}

			element = document.getElementById("dynamicAppearance");
			if (element != null)
			{
				element.id = "invisible";
				document.getElementById("changeHistoryLink").lastChild.nodeValue = "Show the Change History";
			}
		}
  </script></head><body>
<div id="page_header">Eclipse Test and Performance Tools Platform
Project</div>
<div id="page_sub_header">TPTP Testing Strategy Part 2 - Test Conventions and Best Practices</div>
<div id="section_header">TPTP Testing Strategy Part 2 - Test Conventions and Best Practices</div>

<div class="content"><!-- Main content goes here -->
<table id="main_content">
	<tbody>
		<tr>
			<td><b>Title:</b></td>
			<td>TPTP Testing Strategy</td>
		</tr>
		<tr>
			<td><b>Author:</b></td>

			<td>Alan Haggarty (haggarty@ca.ibm.com)</td>
		</tr>
	</tbody>
</table>
<a href="javascript:expandOrCollapse()"><b><span id="changeHistoryLink">Show the Change History</span></b></a>
<table id="invisible" border="1">
	<tbody>
		<tr>
			<th align="left" bgcolor="#dfdfdf" width="10%">Name</th>

			<th align="left" bgcolor="#dfdfdf" width="20%">Date</th>
			<th align="left" bgcolor="#dfdfdf" width="30%">Modified Sections
			</th>
		</tr>
		<tr>
			<td>Alan Haggarty (haggarty@ca.ibm.com)</td>
			<td>June 12, 2008</td>
			<td>Initial Creation</td>

		</tr>
	</tbody>
</table>  	
<br>
<div id="section_hlight">Table of Contents</div>
<div id="main_content">
<a href="#1.0">1.0 TPTP Test Project Conventions</a><br>
<a href="#1.1">1.1 Naming Conventions</a><br>
<a href="#1.2">1.2 Test Project Detailed Structure</a><br>
<a href="#2.0">2.0 Test Creation Best Practices</a><br>
<a href="#2.1">2.1 Creating Manual Test Suites</a><br>
<a href="#2.2">2.2 Creating JUnit and JUnit Plug-in Test Suites</a><br>
<a href="#2.3">2.3 Creating AGR Test Suites</a><br>
<a href="#3.0">3.0 Test Execution Best Practices</a><br>
<a href="#3.1">3.1 Smoke Testing</a><br>
<a href="#4.0">4.0 TPTP Test Project Exceptions</a><br>
<a href="#5.0">5.0 The Common Test Infrastructure</a><br>
<a href="#5.1">5.1 Reference Platform</a><br>
<a href="#5.2">5.2 Adding Tests</a><br>
<a href="#5.3">5.3 Installing and Executing</a><br>
</div>
<br>
<div id="section_hlight"><a name="1.0">1.0 TPTP Test Project Conventions</a></div>
<div id="main_content">
</div>
<br>
<div id="section_hlight"><a name="1.1">1.1 Naming Conventions</a></div>
<div id="main_content">
	<ol>
	 <li>All names for testing resources and folder names MUST adhere to the following rules:
	 	<ul>
      <li>Upper or lower case letter (for example, A-Z or a-z).</li>
      <li>Digits (for example, 0 - 9).</li>
      <li>Underscore (for example, _).</li>
    </ul>
  	</li>
  	<li>Names must NOT contain any white space characters (for example, tab, space, and new line).</li>
  	<li>Every name should begin with an upper or lower case letter (for example, A-Z or a-z). 
  		All other characters in the name should be upper or lower case letters, digits, or the underscore.</li>
  	<li>Java keywords (for example, import, if, etc.) cannot be used for names of JUnit test resources.</li>
  	<li>The name of the Deployment file (deployment files are often used for remote launch of test suites) may contain '.' characters. 
  		The name should be meaningful to the project and the host that it is associated with. 
  		For example, a local deployment file for the org.eclipse.hyades.use.cases folder can be org.eclipse.hyades.use.cases.localhost.deploy</li>
  	<li>The name and directory structure containing the generated execution history file MUST be the same as it will be stored in CVS. 
  		If the generated execution history file is either moved or renamed, the linkage with the associated test suite will break thereby 
  		corrupting generated reports. In the event of an existing execution history in the same directory, 
  		TPTP Testing Tools will append a millisecond time stamp to the name of the new execution history file.</li>
  	<li>Test suite names must start with the bugzilla component name associated with the plug-in or component under test.<br>
  		If the test suite contains several functions, you can either use the bugzilla component name as the name of the test suite or 
  		use the bugzilla component as a prefix.</li>
  	<li>Multiple JRE or platform test suite names must end with the name, vendor, and version of the JRE or platform under test. 
  	These test suites should reference the test suite(s) and/or test case(s) for the plug-in or component to avoid redundancy.</li>
	</ol>
</div>
<br>
<div id="section_hlight"><a name="1.2">1.2 Test Project Detailed Structure</a></div>
<div id="main_content">
	<table border="1" width="100%">
	<tbody>
		<tr>
			<th align="left">Name</th>
			<th align="left">Purpose</th>
		</tr>
		<tr>
			<td>deployment</td>
			<td>Directory containing all deployment configurations for executing the test suites in the test project.</td>
		</tr>
		<tr>
			<td>gui</td>
			<td>Directory containing all AGR test suites for the test project.</td>
		</tr>
		<tr>
			<td>gui/AllTests.testsuite</td>
			<td>AGR test suite to execute all lower level AGR test
			suites for the test project.  Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP1/AllTP1GUITests.testsuite (see Table 2).</td>
		</tr>
		<tr>
			<td>gui/AllSmokeTests.testsuite</td>
			<td>AGR smoke test suite to execute all lower level AGR smoke test
			suites for the test project.  Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP2/AllTP2GUITests.testsuite (see Table 2).</td>
		</tr>
		<tr>
			<td>gui_results</td>
			<td>Directory containing all execution histories for automated
			GUI test suites for the test project.</td>
		</tr>
		<tr>
			<td>junit_plugin</td>
			<td>Directory containing all JUnit Plug-in test suites for the test project.</td>
		</tr>
		<tr>
			<td>junit_plugin/AllTests.testsuite</td>
			<td>JUnit Plug-in test suite to execute all lower level JUnit Plug-in test
			suites for the test project.  Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP1/AllTP1JUnitPluginTests.testsuite (see Table 2).</td>
		</tr>
		<tr>
			<td>junit_plugin/AllSmokeTests.testsuite</td>
			<td>JUnit Plug-in smoke test suite to execute all lower level JUnit Plug-in smoke test
			suites for the test project.  Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP2/AllTP2JUnitPluginTests.testsuite (see Table 2).</td>
		</tr>
		<tr>
			<td>junit_plugin_results</td>
			<td>Directory containing all execution histories for JUnit Plug-in test
			suites for the test project.</td>
		</tr>
		<tr>
			<td>junit</td>
			<td>Directory containing all JUnit test suites for the test project.</td>
		</tr>
		<tr>
			<td>junit/AllTests.testsuite</td>
			<td>JUnit test suite to execute all lower level JUnit test
			suites for the test project.  Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP1/AllTP1JUnitTests.testsuite (see Table 2).</td>
		</tr>
		<tr>
			<td>junit/AllSmokeTests.testsuite</td>
			<td>JUnit smoke test suite to execute all lower level JUnit smoke test
			suites for the test project.  Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP2/AllTP2JUnitTests.testsuite (see Table 2).</td>
		</tr>
		<tr>
			<td>junit_results</td>
			<td>Directory containing all execution histories for JUnit test
			suites for the test project.</td>
		</tr>
		<tr>
			<td>manual</td>
			<td>Directory containing all manual test suites for the test project.</td>
		</tr>
		<tr>
			<td>manual/AllTests.testsuite</td>
			<td>Manual test suite to execute all lower level manual test
			suites for the test project.  Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP1/AllTP1ManualTests.testsuite (see Table 2).</td>
		</tr>
		<tr>
			<td>manual/AllSmokeTests.testsuite</td>
			<td>Manual smoke test suite to execute all lower level manual smoke test
			suites for the test project.  Must be referenced by test-results/platform/org.eclipse.hyades.tests/TP2/AllTP2ManualTests.testsuite (see Table 2).</td>
		</tr>
		<tr>
			<td>manual_results</td>
			<td>Directory containing all execution histories for manual test
			suites for the test project.</td>
		</tr>
		<tr>
			<td>resources</td>
			<td>Directory containing any resources required to execute the
			test suites for the test project.</td>
		</tr>
		<tr>
			<td>src</td>
			<td>Directory containing generated JUnit Java source code for
			the JUnit and JUnit Plug-in test suites and Java source code corresponding to
			verification hooks of AGR test suites for the test project.</td>
		</tr>
	</tbody>
</table>
<b>Table 1: Required structure for contents of a test project.</b>
<p>
Each of the subfolders contains a test project named org.eclipse.hyades.use.cases for testing functionality 
that crosses plug-in or component boundaries.
</p>

<p>
The <a href="http://dev.eclipse.org/viewcvs/index.cgi/test-results/platform/org.eclipse.hyades.tests/?root=TPTP_Project">test-results/platform/org.eclipse.hyades.tests</a> 
test project contains all root-level test suites for TPTP.  Table 2 details the required structure for contents of the test-results/platform/org.eclipse.hyades.tests test project.
</p>

<table border="1" width="100%">
	<tbody>
		<tr>
			<th align="left">Name</th>
			<th align="left">Purpose</th>
		</tr>
		<tr>
			<td>deployment</td>

			<td>Directory containing all deployment configurations for executing the root-level test suites in TPTP.</td>
		</tr>
		<tr>
			<td>enablement</td>
			<td>Directory containing all the TPTP enablement tests.</td>
		</tr>
		<tr>

			<td>TVT</td>
			<td>Directory containing all the TPTP Translation Verification Tests (TVT).</td>
		</tr>
		<tr>
			<td>src</td>
			<td>Directory containing generated JUnit Java source code for
			the root-level JUnit and JUnit Plug-in test suites in TPTP and Java source code corresponding to
			verification hooks of root-level AGR test suites in TPTP.</td>
	  </tr>

		<tr>
			<td>TP1</td>
			<td>Directory containing all root-level test suites to execute all lower level test suites 
				during test pass 1 (TP1).</td>
		</tr>
		<tr>
			<td>TP1/AllTP1GUITests.testsuite</td>
			<td>Root-level AGR test suite to execute all lower level AGR test suites during test pass 1</td>
		</tr>	
		<tr>
			<td>TP1/AllTP1JUnitPluginTests.testsuite</td>
			<td>Root-level JUnit Plug-in test suite to execute all lower level JUnit Plug-in test suites during test pass 1</td>
		</tr>		
		<tr>
			<td>TP1/AllTP1JUnitTests.testsuite</td>
			<td>Root-level JUnit test suite to execute all lower level JUnit test suites during test pass 1</td>
		</tr>	
		<tr>
			<td>TP1/AllTP1ManualTests.testsuite</td>
			<td>Root-level manual test suite to execute all lower level manual test suites during test pass 1</td>
		</tr>			
		
		<tr>
			<td>TP2</td>
			<td>Directory containing all root-level test suites to execute all lower level test suites 
				during test pass 2 (TP2).</td>
		</tr>
		<tr>
			<td>TP2/AllTP2GUITests.testsuite</td>
			<td>Root-level AGR test suite to execute all lower level AGR test suites during test pass 2</td>
		</tr>	
		<tr>
			<td>TP2/AllTP2JUnitPluginTests.testsuite</td>
			<td>Root-level JUnit Plug-in test suite to execute all lower level JUnit Plug-in test suites 
				during test pass 2</td>
		</tr>		
		<tr>
			<td>TP2/AllTP2JUnitTests.testsuite</td>
			<td>Root-level JUnit test suite to execute all lower level JUnit test suites during test pass 2</td>
		</tr>	
		<tr>
			<td>TP2/AllTP2ManualTests.testsuite</td>
			<td>Root-level manual test suite to execute all lower level manual test suites during test pass 2</td>
		</tr>		
		
		<tr>
			<td>BVT</td>
			<td>Directory containing all root-level test suites to execute all lower level test suites 
				during build verification test (BVT).</td>
		</tr>
		<tr>
			<td>BVT/AllBVTGUITests.testsuite</td>
			<td>Root-level AGR test suite to execute all lower level AGR test suites during build verification test</td>
		</tr>	
		<tr>
			<td>BVT/AllBVTJUnitPluginTests.testsuite</td>
			<td>Root-level JUnit Plug-in test suite to execute all lower level JUnit Plug-in test suites 
				during build verification tests</td>
		</tr>		
		<tr>
			<td>BVT/AllBVTJUnitTests.testsuite</td>
			<td>Root-level JUnit test suite to execute all lower level JUnit test suites during build verification tests</td>
		</tr>
			
	</tbody>
</table>
<b>Table 2: Required structure for contents of the test-results/platform/org.eclipse.hyades.tests test project.</b>

<p>The execution history files (*.execution) generated when executing test suites are checked-in to the CVS branch for the 
release under test by the committer for the associated plug-in or component.  For example, the execution history files generated from 
testing the current full release would be checked in to HEAD.  Execution history files are checked in to the *_results folder for the type of 
test suite.  For example, execution history files generated from executing a manual test suite would be checked in the manual_results folder.  To 
delineate between test executions on multiple platforms, each *_results folder contains the following subfolders for each supported platform:

</p>

<ul>
	<li>AIX</li>
	<li>AS_400</li>
	<li>HP_UX</li>
	<li>Linux_390</li>
	<li>Linux_x86</li>

	<li>OS_390</li>
	<li>Solaris</li>
	<li>Windows</li>
	<li>Linux PPC64</li>
</ul>

<p>
Supported platform subfolders may be optionally appended with the platform version and release:
</p>

<ul>
	<li>AIX_5_2_0</li>
	<li>AS_400_5_2</li>
	<li>HP_UX_11</li>
	<li>Linux_390_SuSE_ Enterprise_7_0</li>
	<li>Linux_x86_RedHat_Enterprise_2_1</li>

	<li>Linux_x86_SLES_10</li>
	<li>OS_390_1_6</li>
	<li>Solaris_9</li>
	<li>Windows_2000</li>
	<li>Windows_XP</li>
	<li>Windows_Vista</li>

	<li>Linux_PPC64<br>
	</li>
</ul>

<p>Test execution reports are generated and checked in to CVS by the
TPTP Project Lead or PMC representative at predetermined milestones in a 
test pass for public web access.  Test execution reports are available
at: <a href="http://www.eclipse.org/tptp/">www.eclipse.org/tptp</a> &gt;&gt; Development Plans &gt;&gt; Test Pass Reports
</p>
</div>
<br>
<div id="section_hlight"><a name="2.0">2.0 Test Creation Best Practices</a></div>
<div id="main_content">
	<p>
	Based on the previous experiences of test resource creators and testers, the following general recommendations will assist 
	testers on creating test cases. Section 2.1 - 2.3 go into more details about best practices for creating manual, JUnit, JUnit Plug-in, 
	and AGR test suites.
</p>
<ol>   
	<li>A test case should always have a specific purpose for testing a scenario. Having many test cases in a test suite does not necessarily ensure better quality. Each test case must be well thought out and have a specific purpose. Simply testing the basic functionality of a tool is often not enough to ensure high quality standards. Boundary cases are usually considered to be interesting test cases. As an example, consider the case of testing a function that removes entries from a table. The following test cases should be considered:
		<ul>
     <li>Removing the first entry (a boundary case)</li>
     <li>Removing the second entry (tests basic functionality)</li>
     <li>Removing the last entry (a boundary case)</li>
    </ul>
   </li>
  <li>Avoid repeating the same set of steps in multiple test cases. If two test cases require the user to import a log file, 
  	then create three test cases. The first test case will only serve as setting the environment for successive test cases and the 
  	second and third test case will test the specific functionality in mind.</li>
  <li>This point provides a basic summary of the first two: apply the high cohesion principle to test cases. 
  	Cohesion means assigning specific responsibilities to test cases. A test case requiring a user to import a log file, 
  	profile an application, and use the Probekit has little common in its responsibilities and thus does not follow the high cohesion principle.</li>
  <li>Avoid unnecessarily complicating test cases that are designed to test functionality and not performance. 
  	For example, a test case that uses a class with 3 methods to test the Java Profiler is likely as good as a test case 
  	that uses a class with 10 methods.</li>
  <li>Comment test cases thoroughly in the description section of the editor including manual steps for what the test case is performing.</li>
  <li>If any resource (such as a datapool) needs to be modified before the test suite is executed, 
   	then include that as part of the description of the test suite. 
  Any other important information that the tester should be aware of needs to be included in the description of the test suite.</li>
  <li>If an existing manual test case is being automated, then add '(DEPRECATED - Use <automated test suite name>)' 
  to the beginning of the description of the manual test case.</li>
  </ol>
</div>
<br>
<div id="section_hlight"><a name="2.1">2.1 Creating Manual Test Suites</a></div>
<div id="main_content">
	<p>
	Create small and terse manual test cases that cover well-defined partitions of functionality. 
	Smaller sized test cases provide easier units of work for one tester to execute, 
	more realistic reporting statistics and concise steps to reproduce defects.
</p>
<ol>
	<li>Create manual test suites with sufficient test cases for one tester to complete in less than an hour. 
	Smaller sized test suites provide easier units of work dividing test suites between testers while decreasing the risk of lost execution data.</li>
	<li>In the event of manual test suites being executed by multiple testers, 
		have the tester(s) tabulate the execution histories on paper and the tester with largest assigned test cases will record 
		all the execution histories in the TPTP Manual Test Client.</li>
	<li>When structuring the internal organization as the associated test suite folder (for example, manual), 
		create a summary test suite for the subfolder (for example,/manual/AllTests.testsuite). 
		The summary test suite (for example,/manual/AllTests.testsuite) will reference all test suites within the current folder. 
		This summary test suites is registered with the root-level manual test suite 
		(for example, test-results/platform/org.eclipse.hyades.tests/AllManualTests.testsuite), 
		thereby reducing work when creating and removing plug-in or component test suites.</li>
	<li>Although the TPTP Manual Test Client permits recording more than one execution verdict per test case, 
		it is recommended to only record one execution verdict. 
		Otherwise, TPTP Testing Tools arbitrates the execution verdicts based on the following precedence rules (decreasing order) 
		which may cause unintended execution histories:
		<ul>
			<li>Error</li>
			<li>Fail</li>
			<li>Inconclusive</li>
			<li>Pass</li>
		</ul>			
		</li>
	</ol>
</div>
<br>
<div id="section_hlight"><a name="2.2">2.2 Creating JUnit and JUnit Plug-in Test Suites</a></div>
<div id="main_content">
	<ol>
		<li>Create small and terse unit test cases that cover individual methods or sequence of methods.
		Smaller sized test cases provide more realistic reporting statistics and concise steps to reproduce defects.</li>
		<li>When structuring the internal organization as the associated test suite folders (for example, junit and junit_plugin), 
			create summary test suites for each subfolder (for example,<i>/junit/AllTests.testsuite</i> and <i>/junit_plugin/AllTests.testsuite</i>). 
			The summary test suites (for example,<i>/junit/AllTests.testsuite and /junit_plugin/AllTests.testsuite</i>) 
			will reference all test suites within the current folder. 
			These summary test suites are registered with the root-level JUnit and JUnit Plug-in test suites 
			(for example, <i>test-results/platform/org.eclipse.hyades.tests/AllJUnitTests.testsuite</i> and <i>test-results/platform/org.eclipse.hyades.tests/AllJUnitPluginTests.testsuite</i>), 
		thereby reducing work when creating and removing plug-in or component test suites.</li>
	</ol>
</div>
<br>
<div id="section_hlight"><a name="2.3">2.3 Creating AGR Test Suites</a></div>
<div id="main_content">
	<p>
	The user is expected to have reviewed the available 
	<a href=http://www.eclipse.org/tptp/test/documents/gettingstarted/Automated_GUI_Recorder_Getting_Started.html#3.0>AGR User Guide</a> before reading the recommendations below. 
	The user guide is available on the document section linked from TPTP's web site.
</p>
<ol>
	<li>Ensure that a recording session is short. Avoid having a generated macro for a test case that is longer than 100 lines.</li>
	<li>As mentioned previously make sure that the set of test cases included in a test suite don't perform redundant operations. 
		For example, if two test cases require launching a process, then launch the process once and use it for each of the test cases that follow. 
		At the same time the test cases shouldn't be too dependent on the environment setup by previous test cases. 
		A test case should easily be executable in quick mode without having to run many test cases that precede it. 
		It becomes difficult to debug a test case when it is dependent on more than one test case that precedes it.</li>
	<li>Avoid using absolute paths in a test case. Absolute paths should be replaced with relative paths using one of the available static variables 
		(for example, %testsuiteProjectLocation%). Other modifiable fields that are dependent on the environment that the test suite is launched in 
		(for example, host name) should be linked to a datapool variable.</li>
	<li>Use a logger to make it convenient to debug verification hooks. 
	See <i>test-results/platform/org.eclipse.hyades.use.cases/src/org.eclipse.hyades.use.cases.auto.common/Logger.java</i></li>
	<li>Comment the test case thoroughly in the description section of the test case. Include all manual steps that the test case is performing.</li>
	<li>If any resource (such as a datapool) needs to be modified before the test suite is executed, 
		then include that as part of the description of the test suite. 
		Any other important information that the tester should be aware of needs to be included in the description of the test suite.</li>
	<li>If a manual test case is automated, then add '(DEPRECATED - Use &lt;automated test suite name&gt;)' 
		to the beginning of the description of the manual test case.</li>
	<li>Avoid position-based recording as much as possible. If position-based recording must be used for a component, 
		then try to include all such test cases of the component in a separate test suite. 
		Clearly note the resolution that the tester should use as part of the description of the test suite.</li>
	<li>Verification hooks should verify the value of controls rather than their logical representation. 
		For example, if a tree item represents object A, then verify the existence of the tree item by testing that its text field 
		is the same as the name of object A as opposed to just simply testing for the existence of object A.</li>
</ol>
<p>
There is a recommended structure for the internal content of the 'gui' folder that committers are encouraged to use when automating existing manual test suites:
</p>
<ol>
	<li>The folders that are a direct child of the 'gui' folder should correspond exactly to the folders that are a direct child of the 'manual' folder. 
		When structuring the internal organization as the associated test suite folder (for example, gui), 
		create a summary test suite for the subfolder (for example,<i>/gui/AllTests.testsuite</i>). 
		The summary test suite (for example,<i>/gui/AllTests.testsuite</i>) will reference all test suites within the current folder. 
		This summary test suites is registered with the root-level gui test suite 
		(for example, <i>test-results/platform/org.eclipse.hyades.tests/AllGUITests.testsuite</i>), 
		thereby reducing work when creating and removing plug-in or component test suites.</li>
	<li>The name of the automated gui test suite should correspond exactly to the name of the manual test suite that is being automated.</li>
	<li>A resource directory should be used to contain the resources required for the verification hooks of a test suite. 
		The name of the directory should be &lt;test-suite-name&gt;.Resources 
			(For example the resource directory for the Monitor.UI.LogSets test suite is Monitor.UI.LogSets.Resources. 
			This directory may contain any number of log files or any other files that the test suite depends on). 
			If a datapool is being associated with a test suite, then include the datapool as part of this directory 
			(see the <a href=http://www.eclipse.org/tptp/test/documents/gettingstarted/Automated_GUI_Recorder_Getting_Started.html#3.0>AGR User Guide</a> for more details).</li>
	<li>Use the directory 'Common.Resources' to store resources that are shared amongst multiple test suites</li>
</ol>
</div>
<br>
<div id="section_hlight"><a name="3.0">3.0 Test Execution Best Practices</a></div>
<div id="main_content">
</div>
<br>
<div id="section_hlight"><a name="3.1">3.1 Smoke Testing</a></div>
<div id="main_content">
	<p>
	Smoke testing, or patch/maintenance release testing, refers to partial regression test for testing the majority of the functionality in less time than a full test pass.
	</p>
	<p>
	Smoke test suites should cover:
		<ul>
			<li>All major use cases.</li>
			<li>All major supported platforms.</li>
			<li>All major supported JREs.</li>
			<li>All automated test suites (for example, JUnit, JUnit Plug-in, and AGR test suites).</li>
		</ul>
	</p>
	<p>
	Smoke test suites should reference the test suite(s) and/or test case(s) for the plug-in or component to avoid redundancy.
	</p>
</div>
<br>
<div id="section_hlight"><a name="4.0">4.0 TPTP Test Project Exceptions</a></div>
<div id="main_content">
	<p>
		Some TPTP tests are not manual, AGR or TPTP JUnit/JUnit Plug-in tests as described above.
	</p>
	<p>
		<b>JVMPI Profiler Test Suites</b>
	</p>
	<p>
	  The JVMTI Profiler tests (Platform.Agents.JVMPI.*) are run like regular JUnit tests but require manual configuration of the local host and installation and configuration of a test server on   the remote host. 
	  This is documented in <i>test-results/platform/org.eclipse.tptp.ac.testautomation/automation-files/notes/ACTestAutomation.doc.	</i>
	</p>
</div>
<br>
<div id="section_hlight"><a name="5.0">5.0 The Common Test Infrastructure</a></div>
<div id="main_content">
	<p>
		The common test infrastructure is used for build verification test.
	</p>
</div>
<br>
<div id="section_hlight"><a name="5.1">5.1 Reference Platform</a></div>
<div id="main_content">
	<p>
	TPTP will use reference platforms. New code will be expected to be tested against the reference platform before being committed.
	</p>
	<p>
	The build verification test will run on the reference platform.
	</p>
	<p>
	The current reference platform for TPTP is: IBM Java 1.5 (latest SR) and Windows XP/x86.
	</p>
</div>
<br>
<div id="section_hlight"><a name="5.2">5.2 Adding Tests</a></div>
<div id="main_content">
	<p>
	Tests can be added to the BVT testsuites which are:<br>
	org.eclipse.hyades.tests\BVT\AllBVTGUITests.testsuite<br>
	org.eclipse.hyades.tests\BVT\AllBVTJUnitPluginTests.testsuite<br>
	org.eclipse.hyades.tests\BVT\AllBVTJUnitTests.testsuite<br>
</p>
</div>
<br>
<div id="section_hlight"><a name="5.3">5.3 Installing and Executing</a></div>
<div id="main_content">
	<p>
	All tests in the root level BVT testsuites should be run automatically every build.<br>
	Developers are expected to test locally before committing changes to CVS.
</p>
<p>
	For setup and execution please see the BVT Readme at <a href=http://dev.eclipse.org/viewcvs/index.cgi/platform/org.eclipse.tptp.platform.releng.tools/org/eclipse/tptp/platform/releng/tools/testautomation/readme.html?revision=1.1&root=TPTP_Project>
	org.eclipse.tptp.platform.releng.tools/testautomation/readme.html</a>
</p>
<p>
	We do not need to check-in the execution results to CVS due to disk
space limitations and polluting of test pass results. Each developer can rerun the automated tests to reproduce a failure.
</p>
</div>
</body>
</html>