Community
Participate
Working Groups
The JAX-WS Tools Test suite (org.eclipse.jst.ws.cxf.tests.AllTestsSuite) is frequently not finishing in WTP 3.3.0 and 3.2.3 builds. 5 of the last 8 3.3.0 builds and 2 of the last 8 3.2.2 builds had the test suite DNF'ing. Likely heavy server load. Need to investigate.
Created attachment 182429 [details] I-3.3.0-20101104042929 DNF Console Log Test suite console log for http://build.eclipse.org/webtools/committers/wtp-R3.3.0-I/20101104042929/I-3.3.0-20101104042929/
Typo above that should read 3.2.3 builds not 3.2.2 builds.
Created attachment 182436 [details] I-3.3.0-20101103182633 DNF console log
Created attachment 182437 [details] I-3.3.0-20101103082012 DNF console log
Created attachment 182440 [details] I-3.3.0-20101103012050 green build console log
Attaching the console logs of the three 3.3.0 builds prior to the I-3.3.0-20101104042929 build for comparison.
Comparing the log files of the DNF's builds with the green builds and also with local output it does look like heavy server load that's causing the tests not to finish. There are no standout stack traces in the DNF's logs when compared to the others that would point to another reason. Also from looking at the total run time for the test suite over the last while i've noticed successful test runs taking from ~550+s to ~999s to finish. Danail can you think of anything we can do to improve the running times / performance of the JAX-WS DOM tests?
Hi Shane, My observations are that the tests which verify that DOM model is updated upon resource changes take most of the execution time. These tests are written in an integration manner in order to verify that resource change events are delivered and handled properly. At first thought I cannot think how to make them run faster and preserve their integration nature at the same time. What options are there? Is it possible to tweak the time a test suite is expected to finish? Or maybe we could split the tests in several test suites which manage to finish in the expected time? Regards, Danail
I'm not sure what or if there's a limit on the time a test suite has to complete. I think there may be a limit for the entire build test run as a whole. Keith, do you know of any recommended approach on how to tackle this?
On Monday the I-3.3.0-20101115152837 build ran successfully with all the test suites running. Run time 614.555 seconds. I removed the following from the testsuite to see how it would affect the build on subsequent runs and if the testsuite as a whole would still DNF in any of them with a lighter load. org.eclipse.jst.ws.jaxws.dom.integration.tests.dom.AllTestsSuite org.eclipse.jst.ws.jaxws.dom.ui.tests.AllTestsSuite org.eclipse.jst.ws.jaxws.utils.tests.AllTestsSuite That left the following: org.eclipse.jst.ws.jaxws.dom.runtime.tests.AllTestsSuite org.eclipse.jst.ws.jaxws.core.tests.JAXWSCoreTestSuite org.eclipse.jst.ws.jaxws.core.annotation.validation.tests.JAXWSAnnotationValidationTestSuite org.eclipse.jst.ws.jaxb.core.tests.JAXBCoreTestSuite The next 2 test runs (I-3.3.0-20101115213040 I-3.3.0-20101116052013) completed successfully. Run times 513.964 & 562.983 seconds. The testsuite however still DNF'd in the I-3.3.0-20101117050243 build earlier today. There were also three over 3 DNF's in that build. org.eclipse.jst.jsf.core.tests.AllTests org.eclipse.jst.jsf.validation.el.tests.AllTests_1_1 org.eclipse.jst.pagedesigner.tests.AllTests_Part1
Hi Shane, Keith, Today I ran org.eclipse.jst.ws.jaxws.dom.runtime.tests.AllTestsSuite on my computer and it seems that the test execution deadlocks. This could be a possible reason for tests DNF. Is it possible that we got a JVM thread dump when a test gets into DNF state? I am attaching the thread dump from my execution. I have not investigated the deadlock in detail. However, from a first look I would say that the "ThreadContext" and "Worker-6" threads both wait for org.eclipse.wst.common.componentcore.internal.builder.DependencyGraphImpl to gather pending updates. We should probably get the jst.j2ee guys involved since their classpath container tries to use this dependency graph in synchronous manner Regads, Danail
Created attachment 184853 [details] JVM Thread dump
CC'ing David on this to see if we can get a thread dump when the test suite DNF's on the build machine.
(In reply to comment #13) > CC'ing David on this to see if we can get a thread dump when the test suite > DNF's on the build machine. We can ... but it takes me "watching" the tests, see it occur then "manually" snagging a thread dump. Which I'm happy to do ... just need to remember to watch the pot boil.
Hi David, I think that it would be very useful to improve the JUnit run infrastructure to get the thread dump into e.g. workspace log file whenever it identifies that the test is not finishing instead of manually obtaining it. Maybe the test report could even provide a link to the dump so that developers can easily see whether the test deadlocked or execution is simply taking too much time Regards, Danail
Created attachment 186590 [details] 3 thread dumps Here's three more thread dump files, from a recent 3.3 I build where the cxf tests appear to hang ... they normally finish quickly, but tonight was watching the build, and saw the were taking over 45 minutes, so I took these dumps, about a minute apart, and then killed the process. HTH
Thanks for the dumps, David. Unfortunately this is not what I expected :( In the dumps you attached I cannot find the complete stack trace of all currently running threads. ALso there is quite some cryptic information I don't know what tool you use to obtain the dump but my personal favorite is AdaptJ StackTrace (http://www.adaptj.com/webstart/stacktrace/app/launch.jnlp)
I actually managed to wrap my mind around the new (for me) format of the thread dump and saw that the test execution deadlocks the very same way it does on my side. I managed to find two related bugs which deal with the deadlock in org.eclipse.wst.common.componentcore.internal.builder.DependencyGraphImpl - bug 334050 and bug 331719 Unfortunately there is no progress there... Regards, Danail
