Community
Participate
Working Groups
Testsuites can reference other testsuites in two different ways: Type 1: A testsuite references another testsuite. This means that when you run the testsuite it should run the referenced testsuites Type 2: A testsuite references test cases in another testsuite. This means that you run the testsuite but not run the referenced testsuite. Instead the testsuite execution history should contain results for the the test cases. This is much like an "inheritance" model. The way the referenced testsuite are modeled is the same. As a result the reporting infrastructure can not determine if the testsuite is referenced in a way similar to type 1 or type 2. In the case the references are type 1 both the testsuite and referenced test suite should show up in the report. However, in the case the references are type 2 the referenced test suite should not show up. There needs to be a way for the model to distinguish between both types of references.
This does need to be addressed, but is an enhancement request that could be handled in 4.4. It iwll require some investigation to understand how much if this is facade behaviour vs model intent.
This is an important piece of work to enable accurate test results reporting, including in our own project.
I think the issue here is that you are misinterpreting the intent of the referencedTestSuites collection in the test model. This collection is a simple reference collection to easily track all other test suites that are referenced by a given test suite's behavior without needing to navigate the entire behavioral model. It's not supposed to track the reason for the reference, but that information is certainly still in the model. If you want to understand the distinction between a referenced test suite that is invoked in its entirety and a referenced test suite that has only some of its test cases invoked, you can easily do that by examining the invocations in the behavior model. Do a query to retrieve all the BVRExecutionOccurrence objects in the model and look at the otherBehavior attribute of each of those objects. For each of those otherBehavior attributes, if it is contained by a TPFTestSuite, then we're invoking a test suite. If it is contained by a TPFTestCase, then we're invoking a test case. I'd be happy to help get the reports running correctly, though I suggest we close this enhancement as invalid.
(In reply to comment #3) I would agree since the report generators need to be modified to not traverse the referenced suites (https://bugs.eclipse.org/bugs/show_bug.cgi?id=187242).
Invalidating this bug as per comments.
As of TPTP 4.6.0, TPTP is in maintenance mode and focusing on improving quality by resolving relevant defects and increasing test coverage through test creation, automation, Build Verification Tests (BVTs), and expanded run-time execution. As such, TPTP is not delivering enhancements. As part of the TPTP Bugzilla housecleaning process (see http://wiki.eclipse.org/Bugzilla_Housecleaning_Processes), this enhancement is resolved as WONTFIX. For this enhancement to be considered, please re-open with an attached patch including the Description Document (see http://www.eclipse.org/tptp/home/documents/process/development/description_documents.html), code (see http://www.eclipse.org/tptp/home/documents/resources/TPTPDevGuide.htm), and test cases (see http://www.eclipse.org/tptp/home/documents/process/TPTP_Testing_Strategy.html).
As of TPTP 4.6.0, TPTP is in maintenance mode and focusing on improving quality by resolving relevant enhancements/defects and increasing test coverage through test creation, automation, Build Verification Tests (BVTs), and expanded run-time execution. As part of the TPTP Bugzilla housecleaning process (see http://wiki.eclipse.org/Bugzilla_Housecleaning_Processes), this enhancement/defect is verified/closed by the Project Lead since this originator of this enhancement/defect has an inactive Bugzilla account and considered to be fixed. If this enhancement/defect is still unresolved and reproducible in the latest TPTP release (http://www.eclipse.org/tptp/home/downloads/), please re-open.
