Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 87285

Summary:	[Test Automation] Automated Test Execution Harness Tests
Product:	z_Archived	Reporter:	Scott E. Schneider <scott.schneider>
Component:	TPTP	Assignee:	Paul Slauenwhite <paulslau>
Status:	CLOSED WONTFIX	QA Contact:
Severity:	normal
Priority:	P2	CC:	jkubasta, jptoomey, kdsiefke, paulslau, toddmm
Version:	unspecified
Target Milestone:	---
Hardware:	All
OS:	Windows XP
Whiteboard:	closed471
Bug Depends on:
Bug Blocks:	171827

Description Scott E. Schneider

2005-03-07 12:02:16 EST

Some of the more tricky test execution defects only appear after repeated use 
of the test execution harness by exhaustive downstream products.  One 
improvement to the TPTP test execution harness would include automated test 
execution runs (for example 100+ back-to-back, some overlapping) to fully 
exercise the capacity and scaling of the execution harness.  Some defects 
appear after socket errors and these socket errors might appear due to the 
current network conditions and not the particular test executing.  The ability 
for the test execution harness to recover (reset the sessions that are cached) 
and continue executing tests will be seen after these semi-long running 
automated tests of test execution can be run.

This item consists of designing and implementing the tests (probably JUnit 
running the test execution harness in standalone mode in some cases and 
perhaps PDE JUnit to exercise other cases (since more items are cached in this 
case) and then running these tests regularly (should be easy since they are 
automated).

Comment 1 Scott E. Schneider

2005-04-25 13:08:22 EDT

This should take advantage the new headless test execution via ant/batch 
support put into 3.3/4.0.

Comment 2 Scott E. Schneider

2005-06-17 14:26:43 EDT

Try to have something prototyped and in use by 7/16/2005 -- for 
performance/memory efficiency testing and functional testing.

Comment 3 Scott E. Schneider

2005-08-01 00:45:29 EDT

[ sizing = 5 days ]

Comment 4 Scott E. Schneider

2005-08-21 19:11:05 EDT

Depends on features that do not get delivered until 4.1 i2 -- not possible until
after i2 (or adopted during i2).

Comment 5 Todd Merriweather

2006-04-06 15:35:50 EDT

Deferring to i3

Comment 6 DuWayne Morris

2006-05-12 13:48:50 EDT

This is a testing enhancement that will be worked on during 4.2i4.  Officially setting target to 4.2.1 since this is not a stop-ship defect.

Comment 7 DuWayne Morris

2006-07-21 13:02:20 EDT

Changing milestone to 4.3 since 4.2.1 has been changed to only major approved
defect fixes.

Comment 8 DuWayne Morris

2006-10-23 11:12:01 EDT

Changing target to 4.4, since 4.3 is closed except for blocking or critical defects.

Comment 9 Paul Slauenwhite

2007-01-16 11:44:57 EST

[Test Automation] Automate manual test suites using the Automated GUI Recorder.

This defect is for the sustaining work required in TPTP for automating the manual test suites using the Automated GUI Recorder for the following test projects:

test-results\test\org.eclipse.hyades.execution.harness.tests

Comment 10 Paul Slauenwhite

2007-03-19 11:04:26 EDT

Please target to i3 since this Test Automation defect will be fixed during the remaining test passes.

Comment 11 DuWayne Morris

2007-03-19 14:29:02 EDT

Moving target to 4.4i3 per Paul's note and meeting minutes.

Comment 12 Paul Slauenwhite

2007-06-25 08:07:45 EDT

4.4 is complete.  Re-targeting to 4.4.0.1.

Comment 13 Paul Slauenwhite

2007-06-25 12:16:22 EDT

This defect is not stop-ship for 4.4.0.1; decreasing priority.

Comment 14 Paul Slauenwhite

2007-06-27 10:17:21 EDT

Retargeting to 4.4.1 to allow 4.4.0.1 Test Project resources to be allocating throughout the other sub-project to complete the patch release.

Comment 15 Paul Slauenwhite

2007-08-22 12:08:52 EDT

Deferring to 4.5 to smoke test 4.4.1 in preparation for 4.4.1 TP1.

Comment 16 Paul Slauenwhite

2007-11-07 06:32:53 EST

Bulk update of the target field to reflect the change in iteration numbers.

Comment 17 Paul Slauenwhite

2008-01-07 09:25:47 EST

Deferring to I6.  

We already cover multiple test execution launches with our existing test bucket, but we will need to review these tests to determine if they satisfy this defect.

Comment 18 Paul Slauenwhite

2008-07-02 14:36:20 EDT

Part of the work for this defect should be add some stress tests (possible automated) for URL and JUnit tests.  For JUnit, we do run repeated tests but not more than 10 and no parallel execution.

Comment 19 Paul Slauenwhite

2008-07-22 07:44:20 EDT

Reducing priority of test automation defects.

Comment 20 Paul Slauenwhite

2009-06-09 11:10:03 EDT

Over the past two releases (4.5.x and 4.6.x), numerous manual tests have been automated.  Currently, there are insufficient resources to automate the remaining manual tests.

Comment 21 Kathy Chan

2010-11-18 18:55:34 EST

As of TPTP 4.6.0, TPTP is in maintenance mode and focusing on improving quality by resolving relevant enhancements/defects and increasing test coverage through test creation, automation, Build Verification Tests (BVTs), and expanded run-time execution. As part of the TPTP Bugzilla housecleaning process (see http://wiki.eclipse.org/Bugzilla_Housecleaning_Processes), this enhancement/defect is verified/closed by the Project Lead since this enhancement/defect has been resolved and unverified for more than 1 year and considered to be fixed. If this enhancement/defect is still unresolved and reproducible in the latest TPTP release (http://www.eclipse.org/tptp/home/downloads/), please re-open.