Community
Participate
Working Groups
The tests for SML-IF import/export are currently very weak. We need to beef up the non-UI tests with a lot more scenarios. We also need to implement some automated UI tests using the TPTP facilities for that.
moving target to i8
The non-UI tests are good and pretty thorough, as import/export are tested as part of the repository plugin tests. However, there are no documented UI tests. Given that there is a large learning curve and maintenance cost with the GUI recorder in TPTP, the best approach, at least for now, looks to be manual tests. We need to create manual tests for the following: * Using the SML-IF editor * Using the SML-IF import UI * Using the SML-IF export UI * Using the SML-IF new wizard (File > New > Systems Management > SML-IF file) Once these are in place, QA can test these UI components as part of an iteration test cycle.
Srini, this is something you might want to attempt. It should be easy enough to create basic manual tests based on the documentation.
Also, there should be a manual test for invoking the SML validator from the Eclipse context menu, in addition to the ones previously identified.
Adding Hubert & Sheldon to the cc: list in case there are questions from CA while I'm on vacation. Jack, please encourage the new CA folks to use this bugzilla defect to submit questions.
I believe Naveen, Ramesh, and Monica will all be working on this. I assigned it to Naveen so that one of them would be able to mark this as fixed when it is complete.
Created attachment 114427 [details] Update to testsuite
Naveen, please in the future use bugzilla for communication and attaching patches, instead of email. I have attached the testsuite here you sent me, and will review it shortly.
This is a good start, Naveen. There should be some more rigorous testing added to some of these tests. A few comments: 1. Typically we create an AllTests.testsuite that serves as a collection of individual testsuites. See /org.eclipse.cosmos.rm.validation.tests/junits/AllTests.testsuite for an example, where that suite just points to all the individual testsuites. This allows you to run tests for certain areas if you do not want to run all tests. I would suggest you create TestSMLIFEditor.testsuite, TestValidator.testsuite, TestImport.testsuite, etc. and have those invoked from AllTests in the same way. 2. I renamed All Tests.testsuite to not have a space in the name, since spaces can cause some problems on some platforms. Please sync with cvs to pick up this change. 3. For the test "Validate SML-IF File from SML-IF Editor" you probably want to add a step to ensure that double clicking on a failed validation file causes the editor to switch to the Source tab and the cursor to move to the line of the error. 4. I would suggest you indicate specific files to use from the validation.tests plugin, so the tester will try both a valid and an invalid SML-IF file. 5. For the test "Validate SML-IF File from Eclipse Context Menu", you should include a step for validating an invalid file to ensure the problems view then contains the validation errors 6. For the test "Create a new SML-IF File" add a step to validate the new SML-IF file to ensure the new file is a valid SML-IF document. 7. For the test "Import SML-IF Files into repository using SML-IF Editor" add variations for specifying source files from the file system, source files from the workspace, and files that do not exist. Also add cases for both destination files that exist and do not exist to ensure you are prompted to replace the file if it exists. Use a specific file from validation.tests plugin and tell the tester exactly which files should be created. Tell the tester to open the metadata file and inspect it for correctness. 8. For the test "Using SML-IF Export from Eclipse Context Menu" add some variations similar to the previous test such as destination files that exist vs. don't exist. Also add a test to add a rule binding and ensure the rule binding is generated in the output file. Also add a test to ensure that when you import a file to a directory and then export that directory back to a file that all the SML resources are preserved and the metadata is preserved. When you have done these changes, please attach the .testsuite files to this defect, and also attach any results files you generated from trying these tests. Thanks!
Created attachment 114995 [details] Manual Tests Test Suite
Created attachment 114996 [details] Export Operation Test suite
Created attachment 114997 [details] Import Operation Test suite
Created attachment 114998 [details] New SML-IF Test Suite
Created attachment 114999 [details] SML-IF Validator Test Suite
Naveen, I apologize for the delay in looking at this. Here is my feedback: I like how you have broken things up. In "Using SML-IF Export from Eclipse Context Menu" I would like to see you name a specific file. It's good to be very specific in manual tests so the tester does not have to think about what to do. I will make these changes but want you to know what I did. In step 2, you should have them click on the parent folder to do the export so that you export multiple documents together into one SML-IF file. Also, I made it clear that the tester should try both workspace and file system destinations, to make sure both function properly. I also added steps for testing the 2nd and 3rd page of the export wizard. They are very terse, so perhaps you want to flesh those out a bit. For the New SMLIF test, I added a few more steps. There probably should be a separate test for the Editor itself, because we want to be sure that when you open the editor on an existing SML-IF file, it correctly parses the file and populates the first 2 tabs. I basically added a step to verify that when you close the editor and then double click on the file to reopen it, everything looks as before. I also added a couple steps to verify the recent fix to bug 222225. In general, when we fix a bug, we should add a regular or manual JUnit step to make sure the fix is not regressed in the future. Please pick up the changes and see what you think. I made changes to the validator test as well, but the majority of them were to the two listed above. I think I left the import test as-is. Checked into CVS HEAD and COSMOS 1.0 branch, content same in both. I wasn't sure if we had to modify both streams since I presume Srinivas is using the tests from HEAD stream.
closing to cleanup bugzilla
