Community
Participate
Working Groups
Setting the ANT tptp.test.results property requires adding a trailing / to the property in order for the result to be stored with a specific name in a folder. For example, the results should be stored in the folder junit_results/Windows_IA32 with the file name AllTests.execution. I tried the following to set a results property 1. Create the results property using <property name="tptp.test.results" value="junit_results/Windows_IA32/AllTests" />. The resulting file will be AllTests.execution located in the project's root directory. 2. Create the results property using <property name="tptp.test.results" value="junit_results/Windows_IA32/" />. This results in a WIN_IA32.execution file in the project/junit_results directory. 3. Create the results property with a trailing slash <property name="tptp.test.results" value="junit_results/Windows_IA32/AllTests/" />. The result is an AllTests.execution file in the project/junit_results/Windows_IA32 directory. Sample usage of the results property in the execution task: <tptp:execution resultsrefid="tptp.test.results"> <fileset dir="${project.dir}"> <include name="**/junit/*${testSuite}.testsuite" /> </fileset> </tptp:execution> Using the third method I was able to interrogate it and get the results, but it would also throw an exception trying to find a .execution file in project/junit_results/Windows_IA32/AllTests/. <tptp:interrogation verdictproperty="tptp.test.verdict"> <filelist refid="tptp.test.results"/> </tptp:interrogation> This throws the exception: [condition] org.eclipse.emf.ecore.resource.impl.ResourceSetImpl$1DiagnosticWrappedException: org.eclipse.core.internal.resources.ResourceException: Resource '/project/junit_results/Windows_IA32/AllTests/.execution' does not exist. [condition] at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.handleDemandLoadException(ResourceSetImpl.java:316) [condition] at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLoadHelper(ResourceSetImpl.java:275) [condition] at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.getResource(ResourceSetImpl.java:398) [condition] at org.eclipse.hyades.test.core.services.AbstractTestResultsInterrogationService.interrogate(AbstractTestResultsInterrogationService.java:197) [condition] at org.eclipse.hyades.test.core.services.AbstractTestResultsInterrogationService.execute(AbstractTestResultsInterrogationService.java:101) [condition] at org.eclipse.hyades.automation.server.AbstractService.execute(AbstractService.java:43) [condition] at org.eclipse.hyades.automation.server.ServiceProxy$1.run(ServiceProxy.java:141) [condition] at org.eclipse.hyades.automation.core.utils.ReusableThread.run(ReusableThread.java:131) [condition] at java.lang.Thread.run(Unknown Source) [condition] Caused by: org.eclipse.core.internal.resources.ResourceException: Resource '/project/junit_results/Windows_IA32/AllTests/.execution' does not exist. [condition] at org.eclipse.core.internal.resources.Resource.checkExists(Resource.java:310) [condition] at org.eclipse.core.internal.resources.Resource.checkAccessible(Resource.java:193) [condition] at org.eclipse.core.internal.resources.File.getContents(File.java:285) [condition] at org.eclipse.core.internal.resources.File.getContents(File.java:276) [condition] at org.eclipse.hyades.models.hierarchy.util.HierarchyURIConverterImpl$WorkbenchHelper.createPlatformResourceInputStream(HierarchyURIConverterImpl.java:290) [condition] at org.eclipse.hyades.models.hierarchy.util.HierarchyURIConverterImpl.createPlatformResourceInputStream(HierarchyURIConverterImpl.java:368) [condition] at org.eclipse.emf.ecore.resource.impl.URIConverterImpl.createInputStream(URIConverterImpl.java:538) [condition] at org.eclipse.emf.ecore.resource.impl.ResourceImpl.load(ResourceImpl.java:1152) [condition] at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLoad(ResourceSetImpl.java:256) [condition] at org.eclipse.emf.ecore.resource.impl.ResourceSetImpl.demandLoadHelper(ResourceSetImpl.java:271) [condition] ... 7 more
Duwayne, we need this defect for the Test Automation Initiative. Please provide a sizing and resolve early in I4. Thanks.
This defect is able to worked around. Retargetting.
Deferring to future as approved by the TPTP PMC (http://dev.eclipse.org/mhonarc/lists/tptp-pmc/msg04926.html).
Josh, this defect is blocking you adding the BVT test results to the test pass reports.
After testing the results property with the tptp:test and tptp:execution tasks, it is working as designed. However, the documentation requires refactoring to clarify the intent, requirements, and valid values for this property: -The directory containing the results must exist on the local file system. -Absolute (e.g. d:\myworkspace\myproject\myfolder\myresults\myresult[.execution]) or project-relative (e.g. myfolder\myresults\myresult[.execution]) path to the result file (with or without the .execution file extension). When the tptp:execution task is used and multiple test suites are executed, each test suite is saved to the same result file. If the overwrites property is false, new result files are created with the same result file name appended with the current time stamp (milliseconds) for each test suite. -Absolute (e.g. d:\myworkspace\myproject\myfolder\myresults\) or project-relative (e.g. myfolder\myresults\) path to the result file directory. We will use defect to refactor the documentation. In addition, the other properties (see org.eclipse.hyades.test.doc.user/ref/rsupported-test-execution-service-properties.htm) should be refactored to clarify the intent, requirements, and valid values (e.g. only absolute and project-relative paths) for these properties (see defect https://bugs.eclipse.org/bugs/show_bug.cgi?id=200586). The test cases (/org.eclipse.hyades.test.core.services.tests/resources/Ant/build.xml) also need to be refactored to test each of these requirements and valid values for this property. Josh, can you please refactor the BVT Ant file to use the results property to save the BVT test results in \org.eclipse.hyades.tests\BVT_results (see /org.eclipse.hyades.test.core.services.tests/resources/Ant/build.xml for an example).
(In reply to comment #5) > After testing the results property with the tptp:test and tptp:execution tasks, > it is working as designed. However, the documentation requires refactoring to > clarify the intent, requirements, and valid values for this property: > > -The directory containing the results must exist on the local file system. > > -Absolute (e.g. > d:\myworkspace\myproject\myfolder\myresults\myresult[.execution]) or > project-relative (e.g. myfolder\myresults\myresult[.execution]) path to the > result file (with or without the .execution file extension). When the > tptp:execution task is used and multiple test suites are executed, each test > suite is saved to the same result file. If the overwrites property is false, > new result files are created with the same result file name appended with the > current time stamp (milliseconds) for each test suite. > > -Absolute (e.g. d:\myworkspace\myproject\myfolder\myresults\) or > project-relative (e.g. myfolder\myresults\) path to the result file directory. > > We will use defect to refactor the documentation. In addition, the other > properties (see > org.eclipse.hyades.test.doc.user/ref/rsupported-test-execution-service-properties.htm) > should be refactored to clarify the intent, requirements, and valid values > (e.g. only absolute and project-relative paths) for these properties (see > defect https://bugs.eclipse.org/bugs/show_bug.cgi?id=200586). > > The test cases > (/org.eclipse.hyades.test.core.services.tests/resources/Ant/build.xml) also > need to be refactored to test each of these requirements and valid values for > this property. > > Josh, can you please refactor the BVT Ant file to use the results property to > save the BVT test results in \org.eclipse.hyades.tests\BVT_results (see > /org.eclipse.hyades.test.core.services.tests/resources/Ant/build.xml for an > example). > After testing the results property with the tptp:test and tptp:execution tasks using the TPTP-4.6.0-200906041936 build, I am getting mixed results. As such, we need to reevaluate the org.eclipse.hyades.test.core.services.AbstractTestExecutionService.beforeLaunch(String) method. Josh, please use the following for now, which appears to create the execution results in the correct directory but without a name (e.g. .execution): results="BVT_results\" overwrite="false" Also, please change the tptp.test.results used in resultsrefid/refid to test_results_reference_ID since tptp.test.results is a reserved property name.
Done and done, I'm assuming that your choice of slash direction for results "\" is environment dependent.
(In reply to comment #7) > Done and done, I'm assuming that your choice of slash direction for results "\" > is environment dependent. > No, since the paths are handled as URIs. However, we will need clarification in then documentation.
I got the results to generate with a named execution using results="BVT_results\${testSuite}_${targetJRE}", so that does still work it seems.
This is blocking the TPTP BVTs.
Defect 296286 will cover documentation updates to the results property. This defect will address the inconsistencies with the results property.
(In reply to comment #6) > (In reply to comment #5) > > After testing the results property with the tptp:test and tptp:execution tasks, > > it is working as designed. However, the documentation requires refactoring to > > clarify the intent, requirements, and valid values for this property: > > > > -The directory containing the results must exist on the local file system. > > > > -Absolute (e.g. > > d:\myworkspace\myproject\myfolder\myresults\myresult[.execution]) or > > project-relative (e.g. myfolder\myresults\myresult[.execution]) path to the > > result file (with or without the .execution file extension). When the > > tptp:execution task is used and multiple test suites are executed, each test > > suite is saved to the same result file. If the overwrites property is false, > > new result files are created with the same result file name appended with the > > current time stamp (milliseconds) for each test suite. > > > > -Absolute (e.g. d:\myworkspace\myproject\myfolder\myresults\) or > > project-relative (e.g. myfolder\myresults\) path to the result file directory. > > > > We will use defect to refactor the documentation. In addition, the other > > properties (see > > org.eclipse.hyades.test.doc.user/ref/rsupported-test-execution-service-properties.htm) > > should be refactored to clarify the intent, requirements, and valid values > > (e.g. only absolute and project-relative paths) for these properties (see > > defect https://bugs.eclipse.org/bugs/show_bug.cgi?id=200586). > > > > The test cases > > (/org.eclipse.hyades.test.core.services.tests/resources/Ant/build.xml) also > > need to be refactored to test each of these requirements and valid values for > > this property. > > > > Josh, can you please refactor the BVT Ant file to use the results property to > > save the BVT test results in \org.eclipse.hyades.tests\BVT_results (see > > /org.eclipse.hyades.test.core.services.tests/resources/Ant/build.xml for an > > example). > > > > After testing the results property with the tptp:test and tptp:execution tasks > using the TPTP-4.6.0-200906041936 build, I am getting mixed results. As such, > we need to reevaluate the > org.eclipse.hyades.test.core.services.AbstractTestExecutionService.beforeLaunch(String) > method. > > Josh, please use the following for now, which appears to create the execution > results in the correct directory but without a name (e.g. .execution): > > results="BVT_results\" > overwrite="false" > > Also, please change the tptp.test.results used in resultsrefid/refid to > test_results_reference_ID since tptp.test.results is a reserved property name. When the results property contains only one slash character (e.g. BVT_results/), the org.eclipse.hyades.test.core.services.AbstractTestExecutionService.beforeLaunch(String) method strips off the last slash character and incorrectly interprets as the execution result file name without the file extension (BVT_results/.execution).
The 'results' property have the following meaning (see org.eclipse.hyades.test.core.services.AbstractTestExecutionService.resultsSpecifier): Optional results property representing the workspace-relative (prefixed with the project name) or project-relative path to the execution histories. Possible path values include: Directory: Directory or path to a directory with a trailing path separator character representing the execution history location. Execution histories are named using the default execution history name (see ExecutionHistoryLaunchConfigurationFacade.getDefaultExecutionHistoryName(ILaunchConfiguration)) for all test executions. For example: [\]<directory>\ [\][<project name>]\<directory>\ File: File name or path to a file name with an optional .execution file extension (no trailing path separator character) representing the execution history location and execution history name for all test executions. For example: [\]<execution history name>[.execution] [\]<directory>\<execution history name>[.execution] [\][<project name>]\<directory>\<execution history name>[.execution] Notes: 1. The directory structure must exist on the local file system. 2. Path separator characters are platform-independent (slash ('/') or backslash ('\') characters are interchangeable). 3. The path may contain an optional leading path separator character. Note, the results property does NOT support absolute paths. The attached patch resolves this defect's symptom and provides a utility class (org.eclipse.hyades.test.core.services.internal.util.ServiceUtil) for resolving containers and file names from service properties. If the container/directory referenced by the results property does not exist on the local file system, an error is logged and test execution is halted. Added new JUnit Plug-in test suite (/org.eclipse.hyades.test.core.services.tests/junit_plugin/Test.Execution.ServiceUtilTest.testsuite) for org.eclipse.hyades.test.core.services.internal.util.ServiceUtil (and added to http://wiki.eclipse.org/TPTP_Test_Tools_Project_Tests) and updated /org.eclipse.hyades.test.core.services.tests/manual/Test.Execution.Services.testsuite to test various combinations of the results property. The attached patch and test cases checked in to CVS (HEAD).
The fix does not handle file names in the results property.
(In reply to comment #13) > The 'results' property have the following meaning (see > org.eclipse.hyades.test.core.services.AbstractTestExecutionService.resultsSpecifier): > > Optional results property representing the workspace-relative (prefixed with > the project name) or project-relative path to the execution histories. Possible > path values include: > > Directory: > Directory or path to a directory with a trailing path separator character > representing the execution history location. Execution histories are named > using the default execution history name (see > ExecutionHistoryLaunchConfigurationFacade.getDefaultExecutionHistoryName(ILaunchConfiguration)) > for all test executions. For example: > [\]<directory>\ > [\][<project name>]\<directory>\ > > File: > File name or path to a file name with an optional .execution file extension (no > trailing path separator character) representing the execution history location > and execution history name for all test executions. For example: > [\]<execution history name>[.execution] > [\]<directory>\<execution history name>[.execution] > [\][<project name>]\<directory>\<execution history name>[.execution] > > Notes: > > 1. The directory structure must exist on the local file system. > 2. Path separator characters are platform-independent (slash ('/') or backslash > ('\') characters are interchangeable). > 3. The path may contain an optional leading path separator character. > > Note, the results property does NOT support absolute paths. > Instead, see the variable JavaDoc comment for org.eclipse.hyades.test.core.services.AbstractTestExecutionService.resultsSpecifier.
Created attachment 155434 [details] Patch.
Updated the following test suites to include all possible values for the results property: /org.eclipse.hyades.test.core.services.tests/junit_plugin/Test.Execution.ServiceUtilTest.testsuite /org.eclipse.hyades.test.core.services.tests/manual/Test.Execution.Services.testsuite The attached patch checked in to CVS (HEAD).
(In reply to comment #13) > The attached patch resolves this defect's symptom and provides a utility class > (org.eclipse.hyades.test.core.services.internal.util.ServiceUtil) for resolving > containers and file names from service properties. If the container/directory > referenced by the results property does not exist on the local file system, an > error is logged and test execution is halted. Changed files include: /org.eclipse.hyades.test.core/src/org/eclipse/hyades/test/core/services/internal/util/ServiceUtil.java (1.1) /org.eclipse.hyades.test.core/src/org/eclipse/hyades/test/core/services/AbstractTestExecutionService.java (1.36)
Closing.
