Community
Participate
Working Groups
The Test project currently lacks the ability to effectively test across various platforms, operating systems, JVMs and various other configurations and effectively report on the results. As a workaround, TPTP's test process involves making a copy of each test for every OS on which the test is run (so we can report on which OSs have and have not been covered in a test pass.) We are now extending this system to further cover the JVM used in test execution. Done properly, this will create a crossproduct of JVMs and OSs, unnecessarily duplicating tests, some of which may require ongoing maintenance (even if they only reference test cases in other test suites.) The common test runner should send back configuration information about the system under test, and that configuration information should be persisted in the test log model. This data would then be available for reporting, and reports could be written to show Platform, OS, JVM and configuration coverage without requiring copies of tests. The easiest way to do this would be to add a contained CFGLocation on the TPFExecutionResult, and then specify the configuration information on the CFGLocation (which already supports this type of data), but this will require some design work to validate. This enhancement will require work on the test model, the common runner, and will require one or more new XML fragments and corresponding loader(s).
I think we should strongly consider this enhancement for TPTP 4.5.
I can't set the version field to 4.5, because the values on go up to 4.4. (See http://www.eclipse.org/tptp/home/documents/process/development/bugzilla.html )
This is am important feature. Certain contextual information is important for understanding the execution of a test case. For example: -Defect(s) (e.g. one or more defect number for failed manual test cases) -Driver e.g. product build date or driver number) -Tester(s) (e.g. one or more testers) -Platform(s) (e.g. one or more execution platforms) -Run-time (e.g. JRE, dependency levels, etc.)
Correcting priority since not a 4.5 candidate enhancement (see http://www.eclipse.org/tptp/home/documents/process/development/bugzilla.html).
As of TPTP 4.6.0, TPTP is in maintenance mode and focusing on improving quality by resolving relevant defects and increasing test coverage through test creation, automation, Build Verification Tests (BVTs), and expanded run-time execution. As such, TPTP is not delivering enhancements. As part of the TPTP Bugzilla housecleaning process (see http://wiki.eclipse.org/Bugzilla_Housecleaning_Processes), this enhancement is resolved as WONTFIX. For this enhancement to be considered, please re-open with an attached patch including the Description Document (see http://www.eclipse.org/tptp/home/documents/process/development/description_documents.html), code (see http://www.eclipse.org/tptp/home/documents/resources/TPTPDevGuide.htm), and test cases (see http://www.eclipse.org/tptp/home/documents/process/TPTP_Testing_Strategy.html).
As of TPTP 4.6.0, TPTP is in maintenance mode and focusing on improving quality by resolving relevant enhancements/defects and increasing test coverage through test creation, automation, Build Verification Tests (BVTs), and expanded run-time execution. As part of the TPTP Bugzilla housecleaning process (see http://wiki.eclipse.org/Bugzilla_Housecleaning_Processes), this enhancement/defect is verified/closed by the Project Lead since this enhancement/defect has been resolved and unverified for more than 1 year and considered to be fixed. If this enhancement/defect is still unresolved and reproducible in the latest TPTP release (http://www.eclipse.org/tptp/home/downloads/), please re-open.
