Bugzilla – Full Text Bug Listing
| Summary: | Provide TPTP Testing Process (part 1 and part 2). | ||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | z_Archived | Reporter: | Paul Slauenwhite <paulslau> | ||||||||||||||
| Component: | TPTP | Assignee: | Alan Haggarty <haggarty> | ||||||||||||||
| Status: | CLOSED FIXED | QA Contact: | |||||||||||||||
| Severity: | normal | ||||||||||||||||
| Priority: | P1 | CC: | jcayne, jkubasta | ||||||||||||||
| Version: | unspecified | Keywords: | plan | ||||||||||||||
| Target Milestone: | --- | Flags: | paulslau:
review+
jcayne: review+ paulslau: review+ |
||||||||||||||
| Hardware: | All | ||||||||||||||||
| OS: | All | ||||||||||||||||
| Whiteboard: | |||||||||||||||||
| Bug Depends on: | 211752 | ||||||||||||||||
| Bug Blocks: | |||||||||||||||||
| Attachments: |
|
||||||||||||||||
|
Description
Paul Slauenwhite
Alan, will this be completed in i8? I'll provide a new version this week. Created attachment 102509 [details]
Updated Part 1 document
Created attachment 102510 [details]
Updated Part 2 document
I have attached updated documents for both parts. This is about as close as I can get without participation from people with more intimate association with the specific tests.
For example Joel and Paul likely need to help fill out the bvt section of part 2 with the current state.
I believe I made most of the updates for the last review of part 1. Part 2 review comments wanted to move some sections to part 1 but I have decided to keep them separate to reduce the size and scope of part 1.
Links within the documents and to each other are not created, as well the final formatting should match the existing tptp web pages.
Paul Slauenwhite, can you please review? Joel, could you review these documents from a BVT perspective? Reviewed with comments: General: -Formatting (as you have said), including missing/external/TOC links, TPTP web document format (see http://www.eclipse.org/tptp/home/documents/process/TPTP_Testing_Strategy.html), and section content (start each major section with a paragraph and not a bullet list - see 1.2 in Part 1). Also, 2.3.2 is Part 1 is indented. -Remove the screen capture (image001.jpg). -Change any AGR documentation references to http://www.eclipse.org/tptp/test/documents/gettingstarted/Automated_GUI_Recorder_Getting_Started.html. -Change any manual test tools documentation references to http://www.eclipse.org/tptp/test/documents/gettingstarted/Manual_Test_Tools_Getting_Started.html. -Change any Business Intelligence and Reporting Tools (BIRT) test reports documentation references to http://www.eclipse.org/tptp/test/documents/gettingstarted/BIRT_Test_Reports_Getting_Started.html. -We should have the next new co-op try these instructions to validate their usefulness and check for missing content. Part 1: -We need to add the new structure in CVS, which now to three sets of root level test suites in the TP1, TP2, BVT directories. -Change 'Executing and Creating TPTP tests' to 'Creating and Executing TPTP tests'. -1.1: Add manual test tools and BIRT test reports. -1.3: Move to Part 2 and reference from section 1.3. -2.0: Just reference the help section from the first paragraph. The help Seton should contain the TPTP help and all of the help for all of As-Is test components. -4.2: Move to before section 3.1. -4.3 - 4.5: Move to a section section between section 3.0 and 4.0. -The 'Submitting Execution Histories' section is missing a section number. Part 2: -1.1: Determine what is beneficial from https://bugs.eclipse.org/bugs/show_bug.cgi?id=104409#c0 and copy that content to #7. -5.0: Provide a link to org.eclipse.tptp.platform.releng.tools/testautomation/readme.html. -5.0: Add "We do not need to check-in the execution results to CVS due to disk space limitations and polluting of our test pass results. Each developer can rerun the automated tests to reproduce a failure." -5.0: Need more content here about what is expected from developers for BVT (e.g. all tests in the root level BVT test suites are ran every build and it is expected that developers will test locally before checking into CVS). -Joel needs to provide a section on the BVT architecture for others to use if they want to replicate the same infrastructure. Created attachment 104770 [details]
Part 1
Part 1 updated after Paul's review.
Created attachment 104771 [details]
Part 2
Part 2 updated after Paul's review
I think I got the formatting and links updated ok. The stylesheets are full reference. References to Patr 2 are local (assumes the two docs are in the same directory. A couple of things still need addressing: > Part 1: > -We need to add the new structure in CVS, which now to three sets of root level > test suites in the TP1, TP2, BVT directories. Please give me more specific detail on this. Are the existing file names in test-results/platform/org.eclipse.hyades.tests correct anymore or replaced by these? > -4.3 - 4.5: Move to a section section between section 3.0 and 4.0. Seems like creating a new section for nothing. After the other changes there is nothing else in between. > Part 2: > -1.1: Determine what is beneficial from > https://bugs.eclipse.org/bugs/show_bug.cgi?id=104409#c0 and copy that content > to #7. I only added one comment about using the component as prefix if necessary. Most of the other stuff in the log was to do with renaming, which is done now. > -Joel needs to provide a section on the BVT architecture for others to use if > they want to replicate the same infrastructure. Also, as I have no experience with th BVT any other details about it should be explicitly provided by Joel or Paul. We could update this dynamically as the BVT becomes more mainstream. (In reply to comment #10) > > Part 1: > > -We need to add the new structure in CVS, which now to three sets of root level > > test suites in the TP1, TP2, BVT directories. > > Please give me more specific detail on this. Are the existing file names in > test-results/platform/org.eclipse.hyades.tests correct anymore or replaced by > these? test-results/platform/org.eclipse.hyades.tests/TP1/AllTP1<JUnit/JUnitPlugin/Manual/GUI>Tests.testsuite: The root level test suite for the test type to be ran during Test Pass 1 (TP1). test-results/platform/org.eclipse.hyades.tests/TP2/AllTP2<JUnit/JUnitPlugin/Manual/GUI>Tests.testsuite: The root level test suite for the test type to be ran during Test Pass 2 (TP2). test-results/platform/org.eclipse.hyades.tests/BVT/AllBVT<JUnit/JUnitPlugin/GUI>Tests.testsuite: The root level test suite for the test type to be ran during the Build Verification Test (BVT). test-results/platform/org.eclipse.hyades.tests/TVT/<release>/*: The Translation Verification Tests (TVT). > > -4.3 - 4.5: Move to a section section between section 3.0 and 4.0. > > Seems like creating a new section for nothing. After the other changes there is > nothing else in between. My point here is that the section on submitting execution histories (4.3) should be directly after the sections on executing the test suites (e.g. 3.5). > > Part 2: > > -1.1: Determine what is beneficial from > > https://bugs.eclipse.org/bugs/show_bug.cgi?id=104409#c0 and copy that content > > to #7. > > I only added one comment about using the component as prefix if necessary. Most > of the other stuff in the log was to do with renaming, which is done now. The naming is covered by 1.1 - #7. > > -Joel needs to provide a section on the BVT architecture for others to use if > > they want to replicate the same infrastructure. > > Also, as I have no experience with th BVT any other details about it should be > explicitly provided by Joel or Paul. We could update this dynamically as the > BVT becomes more mainstream. Sounds fine. Covered by https://bugs.eclipse.org/bugs/show_bug.cgi?id=211752. Review with comments: General: -Looking really good! -Shouldn't you be the initial author of these two documents? -Lets get these documents into CVS (HEAD) and evolve them over time. We should have the next new co-op try these instructions to validate their usefulness and check for missing content. Part 1: -Move 4.2 and 4.3 to 3.5 and 3.6. -Add a link to the Best Practices section in Part 2 to section 2.0. -If everything in section 4.1 is covered in Part 2, remove that section (actually all of 4.0 given the first comment). Reviewed with focus on The Common Test Infrastructure section. It may be preferable to create AllTests test suites for each sub-project, for example org.eclipse.hyades.tests\BVT\Platform\AllBVTJUnitTests.testsuite, to match the structure of the Test Project Conventions. Created attachment 105529 [details]
Updated part 1
Created attachment 105532 [details] Updated Part 2 Attached are updates to Part 1 and 2 addressing Paul's most recent review comments. These should now be complete to be checked in. Changes made: - Updated structure tables in part 2 to refer to TP1, TP2 and BVT directories. - Removed section 4 in part 1. 4.1 contents added to part 2 and 4.2,4,3 added to section 3 of part 1 - Added a link to the Best Practices section in Part 2 to section 2.0. BVT comments did not require immediate changes. If we change the structure it is appropriate to update these documents at that time. I will add a comment to https://bugs.eclipse.org/bugs/show_bug.cgi?id=211752 instructing its fixer to update this document with the results of new BVT documentation. Final drafts committed to Head, posted to WIKI (http://wiki.eclipse.org/4.5_Test_Automation_Initiative#Testing_Process), and added to website: http://www.eclipse.org/tptp/home/documents/committers.php (In reply to comment #16) > Final drafts committed to Head, posted to WIKI > (http://wiki.eclipse.org/4.5_Test_Automation_Initiative#Testing_Process), and > added to website: http://www.eclipse.org/tptp/home/documents/committers.php > Updated the old TPTP Test Strategy (http://www.eclipse.org/tptp/home/documents/process/TPTP_Testing_Strategy.html) to only reference the new Testing Strategy documents (Part 1 and Part 2). Added the new Testing Strategy documents (Part 1 and Part 2) to http://www.eclipse.org/tptp/home/documents/committers.php. Closing. |