Bugzilla – Full Text Bug Listing
| Summary: | Build an automated test framework that executes and reports results of tests with every development build | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | z_Archived | Reporter: | amehrega | ||||||
| Component: | TPTP | Assignee: | Navid Mehregani <nmehrega> | ||||||
| Status: | CLOSED WONTFIX | QA Contact: | |||||||
| Severity: | enhancement | ||||||||
| Priority: | P2 | CC: | bjiang, hkyleung, jptoomey, nmehrega, paulslau, ruthdaly, scott.schneider, slavescu, srinivas.p.doddapaneni | ||||||
| Version: | unspecified | Keywords: | helpwanted | ||||||
| Target Milestone: | --- | ||||||||
| Hardware: | PC | ||||||||
| OS: | Windows XP | ||||||||
| URL: | http://www.eclipse.org/tptp/groups/Architecture/documents/features/hf_112928.html | ||||||||
| Whiteboard: | closed471 | ||||||||
| Bug Depends on: | 111544, 144958 | ||||||||
| Bug Blocks: | 171827 | ||||||||
| Attachments: |
|
||||||||
|
Description
amehrega
Created attachment 28402 [details]
Design document
Here's the design document. Criticisms are welcome!
Adding Scott to the CC list. Excellent! As soon as TPTP 4.1 calms down I'd like to review this in more detail and contribute if I can -- thanks. This is great. Scott and I are submitting an abstract to EclipseCon2006 for a tutorial that will cover this and more, so we'd both be interested in helping get this up as a testbed for our eclipsecon tutorial. I am all for moving forward in this area. I am concerned about doing this headless as in the current strcuture vs running detached, whihc is to me a much more important way to run. *** Bug 93911 has been marked as a duplicate of this bug. *** I didn't want to put additional dependencies on this enhancement -- I would like Ali to add these as dependencies if deemed by him, but here are related enhancements in TPTP 4.2 (test interpretation, test reporting, monitoring and profiling services). bugzilla_114531 bugzilla_114534 bugzilla_111542 bugzilla_111544 Thanks for your comments Scott. Of the ones that you've listed, I declared this bug as a dependency of 111544. The rest are nice to have. A bit of comment on each: 111542 - test interpretation automatable service: This will be useful to have. I wouldn't put too much of an affort in providing a lot of details in this service. I think many people will only be interested in aggregated information (i.e. percentage passed, failed, inconclusive, etc...) 114534 - monitoring automatable service: This will be nice to have for larger projects that are expected to always be in production (e.g. DBMS - performing queries and determining number of disk operations, memory access, and etc...). 114531 - profiling automatable service: Very useful to have if it only provides aggregate information and displays the difference between the previous and current build. Sort of what Eclipse does: http://fullmoon.torolab.ibm.com/downloads/drops/R-3.1.1-200509290840/performance/performance.php Comment on attachment 28402 [details]
Design document
See posted description document
Posted updated description document: - Made the workload estimate more realistic http://www.eclipse.org/tptp/groups/Architecture/documents/features/hf_112928.html This framework should be more general than just targeting the build environment, as the "requirement summary" suggests. This feature proposal should be positioned as a generic framework for executing and reporting the results of automated test cases. The build environment can leverage this framework to run some build verification tests (BVT) which are designed to test the basic functionality of the product, and used as a indicator of the quality of the build. TPTP has a large and growing number of function verification test cases (FVT). From build's point of view, it is an overkill to run all TPTP FVT test cases in every development build. The way I envision the test teams to use this framework is to have a user interface to display all avialable test suites avialable, and the tester can select which of the test cases to run, provide the build ID of the driver to be tested, press a button, and the test cases are run and reports generated. The framework is an enabling technology. As for the deployment of the framework in the TPTP project (such as build and test) and the resulting change of the execution of TPTP test strategy, should be further discussed with teams involved. To integrate the framework in the build, I need to know the interface of the framework, and its assumption of operating system, system resources, etc. There is also work involved in creating BVT test cases for some components. The workload estimate for build and infrastructure of 1PD is an underestimate. There was an enhancement for running BVT in the build (bug 93911), but was marked as a duplicated of this bug. I intend to reopen the enhancement 93911 to track the deployment of this framework in the build, and mark enhancement 112928 a dependency of 93911. proposed to be planned for 4.2 but no resources are available (helpwanted keyword added) Target set to 4.2 Is this containable in I3. If set the target milestone accordingly, other defer (by sertting version to 'future' and target milestone '--'). Not containable in 4.2. Setting version to future and target milestone to --. Changing the component to TPTP.Testing Created attachment 46136 [details]
Design Document
Re-assigning to Navid. He's taken over this enhancement. Mass update of P1 enhancements and defects targetted to future to P2. As of TPTP 4.6.0, TPTP is in maintenance mode and focusing on improving quality by resolving relevant defects and increasing test coverage through test creation, automation, Build Verification Tests (BVTs), and expanded run-time execution. As such, TPTP is not delivering enhancements. As part of the TPTP Bugzilla housecleaning process (see http://wiki.eclipse.org/Bugzilla_Housecleaning_Processes), this enhancement is resolved as WONTFIX. For this enhancement to be considered, please re-open with an attached patch including the Description Document (see http://www.eclipse.org/tptp/home/documents/process/development/description_documents.html), code (see http://www.eclipse.org/tptp/home/documents/resources/TPTPDevGuide.htm), and test cases (see http://www.eclipse.org/tptp/home/documents/process/TPTP_Testing_Strategy.html). As of TPTP 4.6.0, TPTP is in maintenance mode and focusing on improving quality by resolving relevant defects and increasing test coverage through test creation, automation, Build Verification Tests (BVTs), and expanded run-time execution. As such, TPTP is not delivering enhancements. As part of the TPTP Bugzilla housecleaning process (see http://wiki.eclipse.org/Bugzilla_Housecleaning_Processes), this enhancement is resolved as WONTFIX. For this enhancement to be considered, please re-open with an attached patch including the Description Document (see http://www.eclipse.org/tptp/home/documents/process/development/description_documents.html), code (see http://www.eclipse.org/tptp/home/documents/resources/TPTPDevGuide.htm), and test cases (see http://www.eclipse.org/tptp/home/documents/process/TPTP_Testing_Strategy.html). As of TPTP 4.6.0, TPTP is in maintenance mode and focusing on improving quality by resolving relevant enhancements/defects and increasing test coverage through test creation, automation, Build Verification Tests (BVTs), and expanded run-time execution. As part of the TPTP Bugzilla housecleaning process (see http://wiki.eclipse.org/Bugzilla_Housecleaning_Processes), this enhancement/defect is verified/closed by the Project Lead since this enhancement/defect has been resolved and unverified for more than 1 year and considered to be fixed. If this enhancement/defect is still unresolved and reproducible in the latest TPTP release (http://www.eclipse.org/tptp/home/downloads/), please re-open. |