Community
Participate
Working Groups
The test harness should supply the identity of the server being tested, individual tests should then run against this repository.
Bugzilla test repositories are now setup and the test suite has been refactored to allow tests to run against multiple configurations. The idea is the following: The BugzillaFixture class provide static field for accessing to instances of BugzillaFixture that provide access to the various test configurations. Test cases can obtain the currently active fixture by invoking BugzillaFixture.current() and should run tests against that particular repository configuration. The AllBugzillaTestSuite generates tests suites that run the same set of tests multiple times against different configurations by activating a different fixture (i.e. changing the result of BugzillaFixture.current()) each a suite is run. I have converted and moved a few tests to a new core package in the tests plug-in. The tests are currently failing since they expect certain bugs to be present in the repository which do not exist, yet. We are currently in the process of figuring out how much initial setup (custom fields, products etc.) we should have in the database schema to make this work. Once that has been decided and the repositories have been initialized the core tests should be made passing again and then the remaining tests need to be refactored. In the end we'll hopefully have a test suite that is more portable, easier to adopt for new versions of Bugzilla and less fragile when bugs change in the test repository. Thomas, can you post what you have determined so far in terms of requirements for the database setup?
Here are the failing tests and some reason why they are currently failing: *AllBugzillaHeadlessStandaloneTests*, *BugzillaTaskDataHandlerTest*, *BugzillaRepositoryConnectorStandaloneTest*, *RepositoryReportFactoryTest*, *BugzillaSearchTest*, *BugzillaRepositoryConnectorTest*, *EncodingTest*, *BugzillaDateTimeTests* tests do not need anything special set up besides the 3 users. However all 9 failing tests in that suite need some tweaking so they put whatever they want to test against into the repository first (and clean up afterwards) *TaskEditorTest* This one should not be a bugzilla test in the first place. *BugzillaSearchPageTest* Failing on it's own, without the repository playing a part *BugzillaRepository32Test* Needs custom fields: * cf_colors {multi select} with allowed values: Red, Green, Yellow, Blue * some more custom fields are needed by testCustomAttributesNewTask, but that seems redundant, I think using cf_colors in that test should be enough Still needs work to actually create the bugs in the state it needs them.
It seems in terms of db structure, not a lot is needed. All tests test against the on-default created TestProduct/TestComponent. I think we should create at least 2 versions and 2 milestones just to have them around. Besides that, a custom field called "cf_colors", being a multi-select combo with values {Blue, Green, Yellow, Red} is necessary. Finally three users should be set up: admin/tests/guest (and for guest the group-regex for the editbugs group needs to be removed). The custom fields are required for Bugzilla 3.2 and above. Since we figured that we cannot automate the DB setup, we should see that everything we need is in there on the first setup to limit manual tweaks. So Steffen, Rob, any other things we might want to test against any time soon? Things that come to mind are: other custom field types, other products/components (to test switching those). As soon as we have determined the initial set, I can set up clean DBs, make dumps for each bugzilla version and come up with a script that removes the existing DB and replaces it with a DB created from that dump. If we would need additional setup work done, all the dumps affected need to be updated.
Thanks for posting the list! Sounds like we should create a schema that supports the set of available types and options for custom fields: http://www.bugzilla.org/docs/tip/en/html/custom-fields.html . Other setup that we discussed: - 3 Products - 2 Milestones - 2 Versions - Custom Fields of all available types - 2 Flags - 2 Keywords - Users: Admin, Guest, User For now 3.0.8, 3.2.4 and 3.4.1 are most important and we can worry about the other versions later. Once the repository setup is complete the goal is to get the tests in the core package passing again.
Alright, here is the initial setup for the database as agreed on: * admin@mylyn.eclipse.org * tests@mylyn.eclipse.org * guest@mylyn.eclipse.org * 2 additional products * have 2 components in each of the products * handful of versions for each * handful of milestones for each * two flags fo bugs * two flags for attachments * one custom_field of each type ** with, if applicable, a handful of allowed values ** the multi select combo be called cf_color with {Red, Yellow, Green, Blue} * two keywords
Awesome. This looks right to me.
Created attachment 147260 [details] patch v1 Here are changes that make the headless pass for all but: *2.18.6:* in many ways, I would say: ignore this *3.2.3 & 3.2.2:* for some strange reason the configuration does not contain the milestones. The XML file does, and it looks exactly like the 3.2.4 XML file (which passes). usetargetmilestones was disabled for those repositories, but I enabled it. Not sure what is going on there....maybe someone else can try reproducing this. So besides the weird 3.2.2/3.2.3 issue this should be fine. I added some tweaks to the test fixture classes, and removed usage of TasksUiPlugin as well as BugzillaCorePlugin instances. For headless tests the connector should be retrieved from the BugzillaFixture directly (using .connector()). Also I needed to make on method public for testing and add a check to the initializeTaskData to make it work in a headless environment.
Created attachment 147261 [details] mylyn/context/zip
Created attachment 147347 [details] patch v2 this patch removes 2.18.6, 3.22 and 3.2.3 from the Fixture, and all Tests pass Since we have the latest 3.2.x still in the fixture, this should be fine.
Created attachment 147348 [details] mylyn/context/zip
Great! Thanks Thomas. I think that we should take a look at the cyclic dependency between BugzillaCorePlugin and BugzillaRepositoryConnector and consider cleaning that up instead of keeping instances in sync in tests. Let's look over the code sometime today.
Created attachment 147383 [details] patch v3
Thanks Thomas! I have committed the patch and opened bug 289663 to track refactoring the circular dependency between BugzillaCorePlugin and BugzillaRepositoryConnector. I made a few minor modifications to make tests in BugzillaRepositoryConnectorStandaloneTest slightly more readable: * Only factor things that are common to all tests, e.g. only create a task in setup() if it's used by all tests. That avoids slowing down tests and could also cause unrelated failures of tests. * Avoid if (foo.equals(bar)) ... else fail(); constructs. It's better to simply assert the condition assertEquals(foo, bar); ... * Avoid loops in tests. Loops make it difficult to figure out why a test failed if it does.
