Bugzilla – Full Text Bug Listing
| Summary: | Continuous testing | ||
|---|---|---|---|
| Product: | [Eclipse Project] JDT | Reporter: | David Saff <david> |
| Component: | UI | Assignee: | JDT-UI-Inbox <jdt-ui-inbox> |
| Status: | RESOLVED WONTFIX | QA Contact: | |
| Severity: | enhancement | ||
| Priority: | P3 | CC: | akiezun, carsten.behring, dcarew, dcorbin, eclipse, ed.burnette, erich_gamma, fg, gian.franco.bonini, gunnar, horst.naujoks, jbr, jpshack, mlists, pombredanne, preuss, remy.suen, Sam.Mesh, tlroche |
| Version: | 3.0 | Keywords: | helpwanted |
| Target Milestone: | --- | ||
| Hardware: | All | ||
| OS: | All | ||
| URL: | http://pag.csail.mit.edu/~saff/continuoustesting.html | ||
| Whiteboard: | |||
| Attachments: | |||
Actually, rather than an attachment, I have set the URL to point to the page from which the plug-in can be downloaded. David, please use attachments rather than URLs. It makes things easier from the legal point of view. you can invalidate attachments so updating is easy. erich, can you comment? I have not looked at this plugin but if it behaves similarly to the background TestRunner built in the _Contributing To Eclipse_ book, I'd agree that this functionality is useful enough to be worthly of inclusion in JDT. Hi David, as a first step I suggest to make the availablilty of this plug-in known to the eclipse community. To do so announce it in the eclipse.tools.jdt newsgroup and refer to this bug to collect feedback and votes. A bug report doesn't get enough visibility. Regarding the next steps to get this code into eclipse: * the formal steps are described in the eclipse charter http://www.eclipse.org/eclipse/eclipse-charter.html. * You should also check out the license http://www.eclipse.org/org/documents/Eclipse%20EPL%202003_11_10%20Final.pdf. This means your current license will have to be adjusted ("This is research software with open source intent." isn't sufficient). Adam's suggestion to attach the plug-in to the bug report addresses the licensing issue. Attachements are considered as licensed under the eclipse license. Once the license of the plug-in is fixed the committers can take a look at the source and provide more detailed feedback about what is involved to get this into Eclipse. We just went through a 3.0 replanning exercise and the JDT plate is already pretty full. To get a new feature that isn't in the plan into Eclipse will require a lot of external help... Created attachment 8147 [details]
Contains five plugins that together make up the continuous testing feature
Finally, here is the plug-in attached. My apologies for the delay. I haven't
yet changed the displayed license.
This plug-in is similar to intent to the one developed in Erich Gamma and Kent Beck's _Contributing to Eclipse_. Gamma and Beck's plug-in includes an editor for excluding tests from automatic execution. Ours includes asynchronous test execution through the build framework, detailed specification of the launch configuration and environment for automatic tests, test prioritization, and other enhancements. To all who have contributed useful comments to this plug-in, thank you. I have been unable to devote time to this for several weeks due to unexpected events. However, because the idea of continuous testing and this plug-in are important parts of my research, I plan to continue supporting, enhancing, and publicizing the plug-in for several years. So, if the committers agree that this is an effective and important extension to Eclipse, I am happy to provide the external help necessary to make it happen. i volunteer to be the eclipse-side buddy for this contribution. my take is that nearly everybody would want test prioritization, so that'd the first one to think about integrating. Erich, what's your opinion? we need to see how the plugins co-play with pde.junit. I've gotten several reports of incompatibility with M7--I'm hoping to roll out an M7-compatible version soon, but until then, the plug-in only works with M6. Thanks. Adam having you as the eclipse buddy is great - thanks for the offer. I looked into the code and the UI. One issue is that several things had to be duplicated to get the continous testing integrated (this isn't David's fault). The duplication shows up both in the UI (separate test runner views) and the code (duplicated classes). The top priority should therefore be duplication reduction. This will make the continous plug-in thinner and the existing JUnit plug-in better. In particular it makes sure that we can maintain this code in the future. Here the staging that I suggest: 1) generalize the JUnit UI so that more than one result can be shown in the result view (similar to the search results view). This requires generalizing the TestRunInfos structure (the model split isn't clean currently). 2) generalize the existing JUnit plug-in code so that the continous plug-in doesn't have to duplicate code (RemoteTestRunner etc). 3) push down features from the continous testing plug-in that are not specific to continous testing. For example, showing markers for failures isn't continous testing specific is generally useful. 4) integrate continous testing To make this happen requires some test coverage (which I confess is currently not happy). David/Adam how does this sound to you. I understand that this is non-trivial amount of work. David, can you provide CVS access (R/O) to the plugin source? Created attachment 8447 [details]
Version 1.0.1: works with M6 and M7
I have now attached v1.0.1, which works with both M6 and M7. Further bug
reports and feature requests are appreciated.
Two more points about 1.0.1: 1) it's also available from the update site (http://pag.csail.mit.edu/~saff/eclipse) 2) The feature id has changed (for the last time), which might cause problems for people attempting a simple upgrade. Erich, Your ordering of tasks makes sense to me. I'll be meeting Adam in person soon to talk in more detail, and that might generate more thoughts. It sounds, though, like shoring up test coverage is actually step 0. Where do you recommend we start? Created attachment 8560 [details]
Changes to 1.0.1. Priority algorihms can be contributed +selection of tests feature added
Some changes compared to 1.0.1:
- prioritiesing algorithms can be added as extension to the plugin
- the current priority algorithms are implemented in an extra plugin as
extension points
- tests can not only be ordered but optionaly filtered as well(should be maybe
splitt into two extension points)
I made same changes to version 1_0_1 and posted it as an new attachemend and I would like you to check it, and give some comments, if any. Created attachment 8568 [details]
same as before, but source included
The same as before, but I forgot to include the source...
Created attachment 9099 [details]
Patched version of continuous testing plugin
This is a new version of the plugin. It contains as a seperate plugin with a
new algorithm, which selects automaticly the tests to run, dependend on the
source files which changed since the last build. It does that by searching all
dependend files of all changed files (recursivly), and checks if a test to run
is under the files. Only then the tests gets selected for running.
Works with eclipse 3.0M7
Has problems if tests and code are in different projects.
All, For now, we have decided to develop this as a stand-alone plug-in, resolving several issues so that it might be considered for inclusion in Eclipse 3.1. A new version that works with 3.0 M9 and 3.0 RC1 (and perhaps not earlier versions) is now at update site http://pag.csail.mit.edu/~saff/eclipse. More information can be found at http://pag.csail.mit.edu/~saff/continuoustesting.html (we hope to improve the web page soon). This new version includes Carsten Behring's enhancements, but does not include his separate plugin with the dependency algorithm. There is also now a separate bugzilla repository just for the plug-in, at http://pag.csail.mit.edu/zilla. Please make further enhancement requests and bug reports there. Share and Enjoy. All, There are now two mailing lists devoted to this plug-in. To add yourself to the list where new versions will be announced, go to http://lists.csail.mit.edu/mailman/listinfo/continuous-testing-plugin-announce . To discuss using and improving the plug-in, go to http://lists.csail.mit.edu/mailman/listinfo/continuous-testing-plugin-discuss . Thanks! David, Can you recomment newsgroups for these mailing lists. Are they registered on <gmane.org>? PS. What are the following lists? <http://lists.csail.mit.edu/mailman/listinfo/continuous-esting-plugin-announce> <http://lists.csail.mit.edu/mailman/listinfo/continuous-esting-plugin-discuss> -- 10x, Sam Mesh Thanks, I was not aware of gmane. These are now registered as gmane.comp.java.eclipse.continuous.testing.announce and gmane.comp.java.eclipse.continuous.testing.discuss Please take further meta-discussion of the lists to the discuss list itself. The *.esting.* lists are typos that will soon disappear. All, after a lot of hard work, we've just released version 1.1 of the plug-in. I'd be interested in any feedback: http://people.csail.mit.edu/u/k/kcheva/public_html/continuoustesting/ In particular, the new release contains patched versions of the org.eclipse.jdt.junit and org.eclipse.jdt.junit.runtime plug-ins, addressing all 4 items from Erich's comment #11. Test coverage has not been improved, but that is the next priority. If this feature is still of interest, I'd really love feedback on whether the steps taken so far make sense. I'll attach a zip of the feature for copyright reasons. Thanks. I don't seem to be able to submit an attachment. When this is resolved, I'll attach. Sorry. Still unable to create an attachment. Anyone have any ideas? Is there anything happening about this issue in Eclipse 3.2 stream? Erich, should this issue be also assigned to Markus? David already did lot of work in the area and it would be very interesting to have this functionality in Eclipse. David's work on continuous testing and his many contributions to the eclipse junit support are indeed highly appreciated.
The optional plug-in has allowed us to get experience with the continous testing approach. What we have found is that developers enjoy test prioritization ("run failed tests first"), but continous testing itself didn't get the acceptance level you might have expected.
For this reason we have included support for "run failed tests first" in 3.1, but didn't go further in 3.2 and I still think this is the right thing to do. Therefore I suggest to close this request. I'm moving the bug back to the inbox so that Martin has the final word.
It seems to me that standalone plugins, managed by the community seems is the right way to go. We unfortunatly don't have the manpower here to evolve and maintain this. David, let me know if we can help with patches to jdt.junit so you can provide your plugins without any changes to an install. Set to LATER, helpwanted *** Bug 171123 has been marked as a duplicate of this bug. *** As of now 'LATER' and 'REMIND' resolutions are no longer supported. Please reopen this bug if it is still valid for you. |
Attached is a plug-in that implements continuous testing, as described below. A user study with students has shown that continuous testing has measurable productivity benefits in certain situations--I think it could be a very useful part of the standard Eclipse toolset. It currently is only tested against Eclipse 3.0 M6. Continuous testing builds on the automated developer support in Eclipse to make it even easier to keep your Java code well-tested, if you have a JUnit test suite. With continuous testing enabled, as you edit your code, Eclipse runs your tests quietly in the background, and notifies you if any of them fail or cause errors. It is most useful in situations where you would already have a test suite while you are changing code: when performing maintenance, refactoring, or using test-first development. Continuous testing builds on the following features of Eclipse: * JUnit integration: Test suites run under continuous testing give you the same information, in the same format, that you already get from Eclipse's JUnit integration. However, continuous testing also helps to automate the choice of when to run tests, which tests to run, and in which order. * Compile error notification: As soon as you write code that contains a compilation error, Eclipse highlights the error in the text and the margin, and creates a task in the Problems table. This makes it easy both to quickly find out if what you wrote was wrong, and to step through and fix problems in an orderly fashion. With continuous testing, you get the same kind of notification when you write or edit code that causes one of your tests to fail. However, test failures are different from compile errors in several ways: test failures can sometimes not easily be tracked to a single line of code, and test failures also can provide more information, such as a backtrace, than compile errors. Continuous testing was first developed by David Saff at MIT as part of his PhD research. Let me know if it sounds useful to have it as part of the base distribution, and what next steps would be required to begin getting it to the state where that could happen.