Community
Participate
Working Groups
Add BVT results to TPTP test reports. Our BVT results should reported in our TPTP Test reports so testers do not have to run the test suites in the BVT buckets during the test pass.
Is there any preference for where in the cvs should we put the results? Joel has suggested something like "test-results/platform/org.eclipse.hyades.use.cases/bvt_results". We just have to remember to get the test pass report to pick up the results from the new directory.
(In reply to comment #1) > Is there any preference for where in the cvs should we put the results? > Joel has suggested something like > "test-results/platform/org.eclipse.hyades.use.cases/bvt_results". We just have > to remember to get the test pass report to pick up the results from the new > directory. > Yes. Firstly, they need to be generated in the same directory that they will reside in CVS. Secondly, they need to be in the directory that contains the test suites, that is org.eclipse.hyades.tests. I would create your BVT_results directory and under that, directories for target OSes (Windows_XP in this case). Since we have execution results for several JREs, I have created a AllBVT<test type>Tests_<JRE vendor>_JRE_<JRE version>.testsuite for each JRE (referening the AllBVT<test type>Tests.testsuite) so the results for each JRE will be reported. Please start executing these JRE-specific test suites. Lastly, the TPTP Test Strategy (http://www.eclipse.org/tptp/home/documents/process/TPTP_Testing_Strategy.html) and the BVT document needs to be updated with this information.
Steps required: 1) Check the BVT test results in to /test-results/platform/org.eclipse.hyades.tests/BVT_results. 2) Add /test-results/platform/org.eclipse.hyades.tests/BVT/AllBVT<JUnit/JUnitPlugin/GUI>Tests_<JRE vendor>_JRE_<JRE version>.testsuite to the TP1 and TP2 automatically generated reports. 3) Update the TPTP Test Strategy (http://www.eclipse.org/tptp/home/documents/process/TPTP_Testing_Strategy.html) and the BVT document with this information.
A question about how this should be checked in: since the executions don't have time stamps attached to their names when created (ie. it isn't named AllBVTJUnitTests_IBM_JRE_150_1184101222031.execution) I can't check them straight into the results folders. So our choices are that I rename them before submitting with some form of date tag or that I overwrite the previous results with the new ones (this has the down side of only having the latest results and having to look in the history to get older ones).
Having multiple results would create a lot of files in the directory. Overwriting a single file with the build time stamp in the comment could be an option.
(In reply to comment #5) > Having multiple results would create a lot of files in the directory. > Overwriting a single file with the build time stamp in the comment could be an > option. > I tend to agree. If developers need to access an execution result, they can use the CVS history. Having the build ID in the comment field would be a requirement to associated an execution result with a build.
All Test executions are now being checked into the CVS
Added BVT JUnit test results to TP1 and TP2.
(In reply to comment #3) > 3) Update the TPTP Test Strategy > (http://www.eclipse.org/tptp/home/documents/process/TPTP_Testing_Strategy.html) > and the BVT document with this information. The point here is that we need to make testers aware that the BVTs are running some of there test pass tests, whcih do not need to be rerun. For example: http://wiki.eclipse.org/TPTP_Test_Tools_Project_Tests#JUnit We should make the following changes to http://www.eclipse.org/tptp/home/documents/process/test/automation/process/TPTP_Testing_Strategy_Part_2.html#5.0: -Rename it to include Build Verification Test (BVT). -Include a new section that describes that the BVT results are checked in to CVS (and where), added to the test pass reports, and where to find the BVT reports and log (e.g. see http://wiki.eclipse.org/TPTP_Test_Tools_Project_Tests#JUnit). -Remove for section 5.3: "We do not need to check-in the execution results to CVS due to disk space limitations and polluting of test pass results. Each developer can rerun the automated tests to reproduce a failure."
Created attachment 134782 [details] Patch for Testing Strategy Paul, Can you please review. Thanks!
(In reply to comment #10) > Created an attachment (id=134782) [details] > Patch for Testing Strategy > > Paul, > > Can you please review. > > Thanks! > Adding myself to the review flag.
Reviewed and approved with the following comments: -Add Windows XP/x86 to: The build verification tests are run against IBM Java 1.5 and 1.6, and Sun Java 1.5 and 1.6 (latest SR). -Change the following: org.eclipse.hyades.tests\BVT\AllBVTGUITests_<JRE Vendor>_JRE_<version>.execution<br> org.eclipse.hyades.tests\BVT\AllBVTJUnitPluginTests_<JRE Vendor>_JRE_<version>.execution<br> org.eclipse.hyades.tests\BVT\AllBVTJUnitTests_<JRE Vendor>_JRE_<version>.execution<br> to (adding _results): org.eclipse.hyades.tests\BVT_results\AllBVTGUITests_<JRE Vendor>_JRE_<version>.execution<br> org.eclipse.hyades.tests\BVT_results\AllBVTJUnitPluginTests_<JRE Vendor>_JRE_<version>.execution<br> org.eclipse.hyades.tests\BVT_results\AllBVTJUnitTests_<JRE Vendor>_JRE_<version>.execution<br> -Add a sentence to section 5.3 saying that tests included the BVTs should be executed during the test pass since they are include in the test pass reports.
Created attachment 135218 [details] Patch for Testing Strategy Updated with Paul's comments.
Patch checked in for website update.
Note, this is blocked by https://bugs.eclipse.org/bugs/show_bug.cgi?id=276121.
Dependency on 210289 no longer required (see https://bugs.eclipse.org/bugs/show_bug.cgi?id=210289#c5). Josh, can you please refactor the BVT Ant file to use the results property to save the BVT test results in \org.eclipse.hyades.tests\BVT_results (see /org.eclipse.hyades.test.core.services.tests/resources/Ant/build.xml for an example).
Results are now being generated under org.eclipse.hyades.tests\BVT_results and are being checked in on every run. Is there anything left for this bug or can we close it now?
(In reply to comment #17) > Results are now being generated under org.eclipse.hyades.tests\BVT_results and > are being checked in on every run. Is there anything left for this bug or can > we close it now? > No, we are good.
Verified in http://www.eclipse.org/tptp/test/reports/4.6_i3_TP2/index.html. Closing.
