Community
Participate
Working Groups
Build updates for the TPTP BVTs. There are several updates required of the Release Engineering Team for the TPTP BVTs to improve our testing effectiveness: Required (in order): 1) Change the BVTs to start after every build completes. 2) Only check out the test plug-ins and download the TPTP driver once for the four JVM runs. In addition, only extract the test plug-ins required to run the BVT (currently deleted). 3) Merge the four BVT reports into one BVT report containing the following: AllBVTJUnitTests_IBM_JRE_150 [TPTP JUnit Test] AllBVTJUnitTests_IBM_JRE_160 [TPTP JUnit Test] AllBVTJUnitTests_Sun_JRE_150 [TPTP JUnit Test] AllBVTJUnitTests_Sun_JRE_160 [TPTP JUnit Test] 4) Add a not run(question mark icon), pass (green checkmark icon), or fail (red x icon) indicators for each build. The icon would be a link to the BVT page for that build. We would add another column to the builds (e.g. http://www.eclipse.org/tptp/home/downloads/?ver=4.6.1) for BVT results where users/consumers can easily determine the quality (from a BVT perspective) of the driver. This also gets BVT failures front-and-center. The aggregated verdict can be captured from the test interrogation task executed on all four BVT execution results. Optional: 1) Add a button to the internal build page to start the BVTs for the last published build. There should be a check to see if a BVT is already running and a configuration setting to delete (default)/not delete the work-space/temp working directory on the deployed machine.
Kathy, can we get this into plan for TPTP 4.6.2 as part of our efforts to improve the quality of the project through continuous testing (BVTs)?
Joel, once you determine the set of BVT test plug-ins, can you post a Team Project Set file?
Created attachment 153458 [details] BVT Project Set File BVT Project Set File based on test projects checked out for the latest BVTs.
Created attachment 153538 [details] BVT Project Set File Update to remove Test API Recorder tests. These are the basis for (2) in comment #0 and reduces the checkout of test projects to only those being tested.
(In reply to comment #4) > Created an attachment (id=153538) [details] > BVT Project Set File > > Update to remove Test API Recorder tests. > > These are the basis for (2) in comment #0 and reduces the checkout of test > projects to only those being tested. Can you please add this as a subsection of http://www.eclipse.org/tptp/home/documents/process/test/automation/process/TPTP_Testing_Strategy_Part_2.html#5.0?
Created attachment 161320 [details] Patch to update Test Strategy Adds BVT Project Set File to Test Strategy page.
The following items were completed from the list in the bug description: 1) BVTs are scheduled to run daily and check for the latest builds. 2) Only the required test plug-ins are checked out during BVT (as listed in the posted psf file). The required downloads are only retrieved once. 4) The status for the BVT can be based on the status for the all-in-one downloads. Remaining items from the bug description: 1) Start BVTs once a build finish. 3) The reports can be merged into one report. 4) Add an icon to show the BVT status on the build page. There is no plan to do the remaining BVT updates.
Closing.
