Bugzilla – Full Text Bug Listing
| Summary: | Test failures in nightly build | ||
|---|---|---|---|
| Product: | [Eclipse Project] Equinox | Reporter: | DJ Houghton <dj.houghton> |
| Component: | p2 | Assignee: | P2 Inbox <equinox.p2-inbox> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | P3 | CC: | irbull, jens.glander, kim.moir, t-oberlies |
| Version: | 3.7 | ||
| Target Milestone: | 3.7 M7 | ||
| Hardware: | PC | ||
| OS: | Mac OS X - Carbon (unsup.) | ||
| Whiteboard: | |||
Ian noticed that we might not be using -consoleLog when calling out to run the verifier application so I've modified the test code to add that arg for tonight's build. At least this will help us catch the output to see what the real error is. DJ, you were asking about the files in /tmp. I reran the p2 tests and there were files created that were not deleted after the test run drwxr-xr-x 3 kmoir Relenggrp 4096 Mar 28 20:48 bundlepool4883220661034501347.repo drwxr-xr-x 3 kmoir Relenggrp 4096 Mar 28 20:48 bundlepool2890193534444668015.repo drwxr-xr-x 3 kmoir Relenggrp 4096 Mar 28 20:48 bundlepool1519252368379183158.repo drwxr-xr-x 3 kmoir Relenggrp 4096 Mar 28 20:48 bundlepool1903679667036072078.repo drwxr-xr-x 3 kmoir Relenggrp 4096 Mar 28 20:47 bundlepool4584562551904563757.repo drwxr-xr-x 3 kmoir Relenggrp 4096 Mar 28 20:47 bundlepool828043049646067906.repo drwxr-xr-x 3 kmoir Relenggrp 4096 Mar 28 20:47 bundlepool4561647090524021591.repo drwxr-xr-x 3 kmoir Relenggrp 4096 Mar 28 20:47 bundlepool390940905583631328.repo drwxr-xr-x 3 kmoir Relenggrp 4096 Mar 28 20:47 bundlepool3608355845539495549.repo drwxr-xr-x 3 kmoir Relenggrp 4096 Mar 28 20:47 bundlepool8679726613325941370.repo drwxr-xr-x 3 kmoir Relenggrp 4096 Mar 28 20:47 bundlepool924242912620840271.repo The tests are failing because the verifier application fails to load: java.lang.RuntimeException: Application "org.eclipse.equinox.p2.tests.verifier.application" could not be found in the registry. The applications available are: org.eclipse.ant.core.antRunner, org.eclipse.equinox.app.error, org.eclipse.equinox.p2.director.app.application, org.eclipse.equinox.p2.director, org.eclipse.equinox.p2.garbagecollector.application, org.eclipse.equinox.p2.metadata.generator.EclipseGenerator, org.eclipse.equinox.p2.publisher.InstallPublisher, org.eclipse.equinox.p2.publisher.EclipseGenerator, org.eclipse.equinox.p2.publisher.ProductPublisher, org.eclipse.equinox.p2.publisher.FeaturesAndBundlesPublisher, org.eclipse.equinox.p2.reconciler.application, org.eclipse.equinox.p2.repository.repo2runnable, org.eclipse.equinox.p2.repository.metadataverifier, org.eclipse.equinox.p2.artifact.repository.mirrorApplication, org.eclipse.equinox.p2.metadata.repository.mirrorApplication, org.eclipse.equinox.p2.updatesite.UpdateSitePublisher, org.eclipse.equinox.p2.publisher.UpdateSitePublisher, org.eclipse.equinox.p2.publisher.CategoryPublisher, org.eclipse.help.base.infocenterApplication, org.eclipse.help.base.helpApplication, org.eclipse.help.base.indexTool, org.eclipse.ui.ide.workbench, org.eclipse.update.core.standaloneUpdate, org.eclipse.update.core.siteOptimizer. I don't know enough about the test infrastructure, but from some digging I don't see any recent changes (other than we moved our features into our own map file)... maybe that caused some problems. I found the dump shown above in the following log file: http://download.eclipse.org/eclipse/downloads/drops/I20110329-0800/testresults/linux.gtk.x86_6.0/org.eclipse.equinox.p2.tests.AutomatedTests.txt Moving your bundles into new map files doesn't matter. The map files are all concatenated by the build. I reran the p2 tests and the verifier bundle is there. I ran the osgi console and all the p2 test bundles resolve. Is there a way to tell if an application id is available from the osgi console? The OSGi console command is "apps". Kim, feel free to ping me when you're online and I can look into this further. I logged into the build machine and ran N20110329-2000/eclipse-testing/test-eclipse/eclipse and the test verifier application is available from that Eclipse instance, but maybe that's not the right one. Yes, DJ, that was the right Eclipse. Not sure what is happening then. I've looked into it more and the test actually takes the platform zip and extracts it to a separate location and then takes the verifier bundle and puts it in that location as well and then kicks off an external call to this new Eclipse. I've added some debug lines so we can see where this temp folder is. I've also released code which checks a System property and won't delete the temp folders if it is set. So if we want to run a test build with: p2.tests.doNotClean=true we can check. I'm releasing code which will detect a return code of 13 when we call out (to run the reconciler or verifier or another app) and then parse the error message and read the log file path from it and then put the log file contents into the fail string of an assert(false). Crazy hack but hopefully will give us more information. As a side note, the chmod log messages were gone when we commented out the SharedInstallTests so that's something that we need to investigate. We released changes to the build last night so we wouldn't delete the tmp folder and I've logged in and grabbed a copy. Initial viewing notes: - there is a log in the configuration area saying what we knew: verifier application is not available - there is no verifier (or *tests*) bundle in dropins or plugins - there is no verifier (or *tests*) bundle in the bundles.info I'll investigate further. (In reply to comment #12) > We released changes to the build last night so we wouldn't delete the tmp > folder and I've logged in and grabbed a copy. Initial viewing notes: > - there is a log in the configuration area saying what we knew: verifier > application is not available Yep. > - there is no verifier (or *tests*) bundle in dropins or plugins No verifier in dropins, makes sense (we copy it to plugins). No verifier in plugins... strange... we copy it there. > - there is no verifier (or *tests*) bundle in the bundles.info If there really is no verifier in the plugins directory, then the problem is with our copy method. This seems odd since any problems will likely throw an IOException, and this will instantly fail the test case. The other option is that the verifier is there and we fail to load it. This is possible since we are using Eclipse 3.6.2 + the verifier from head (3.7), but our version ranges seem to allow this. > > I'll investigate further. Regarding comment #11 The plugins directory does have the verifier bundle. When the tests are run, the test bundles are installed as needed and the install is cleaned before the next test suite is run. Since another test suite was run after the p2 tests, the verifier bundles didn't exist on the filesystem. I just reran the p2 tests by themselves and the verifier bundle seems to be there and the application appears to be available. -bash-3.2$ ls -ld *verifier* -rw-r--r-- 1 kmoir Relenggrp 8537 Apr 6 15:57 org.eclipse.equinox.p2.tests.verifier_1.0.100.N20110405-2000.jar -bash-3.2$ pwd /buildtest/N20110405-2000/eclipse-testing/test-eclipse/eclipse/plugins -bash-3.2$ hostname ejlnx1 -bash-3.2$ pwd /buildtest/N20110405-2000/eclipse-testing/test-eclipse/eclipse -bash-3.2$ /buildtest/N20110404-2000/jdk6_17/jre/bin/java -jar plugins/org.eclipse.equinox.launcher_1.2.0.N20110405-2000.jar -noexit -console osgi> apps org.eclipse.equinox.p2.tests.verifier.application [launchable] org.eclipse.pde.junit.runtime.nonuithreadtestapplication [not launchable] org.eclipse.ui.ide.workbench [launchable] This is a special test where we extract the platform zip to /tmp/xxx (and copy the verifier bundle to this location as well) and then call out to Eclipse to run it. We released code last night which doesn't clean up the tmp folders and I've checked all of them and the verifier bundle isn't in any. What is worrisome is that the System.out statements that we added to tell us where we were copying the bundle and what we were extracting, don't appear in our capture of the console logs. These should be executed by the outer Eclipse so we should still see them in the logs. (In reply to comment #15) > What is worrisome is that the System.out statements that we added to tell us > where we were copying the bundle and what we were extracting, don't appear in > our capture of the console logs. These should be executed by the outer Eclipse > so we should still see them in the logs. I see some printouts in the following log: http://download.eclipse.org/eclipse/downloads/drops/N20110405-2000/testresults/linux.gtk.x86_6.0/org.eclipse.equinox.p2.tests.AutomatedTests.txt Extracting: /buildtest/N20110405-2000/eclipse-testing/eclipse-platform-N20110405-2000-linux-gtk.tar.gz to: /tmp/1302089435525-0.42582375213316725 Extracting: /buildtest/N20110405-2000/eclipse-testing/eclipse-platform-N20110405-2000-linux-gtk.tar.gz to: /tmp/1302089587631-0.7288600688523417 Extracting: /buildtest/N20110405-2000/eclipse-testing/eclipse-platform-N20110405-2000-linux-gtk.tar.gz to: /tmp/1302089702109-0.8862407463726315 Extracting: /buildtest/N20110405-2000/eclipse-testing/eclipse-platform-3.6.2-linux-gtk.tar.gz to: /tmp/1302090095610-0.004588181345203335 STDOUT: Installing org.eclipse.platform.ide 3.7.0.I20110405-0800. STDOUT: Uninstalling org.eclipse.platform.ide 3.6.2.M20110210-1200. STDOUT: Operation completed in 65501 ms. Copying :org.eclipse.equinox.p2.tests.verifier to: /tmp/1302090095610-0.004588181345203335 Ah good find. I was looking in the regular linux logs. Maybe we should open a bug that there are too many log files to check? ;-) (In reply to comment #17) > Ah good find. I was looking in the regular linux logs. Maybe we should open a > bug that there are too many log files to check? ;-) The test infrastructure is all new to me, so I just assumed this was some sort of new recruit initiation torture that I'm expected to figure out :). Argh. One of the errors is coming from when we try and install the 3.5 version of the verifier bundle. It isn't resolving because some of the imports have changed. (used to import provisional APIs which have since moved) Changing the code to install the regular version of the verifier in #from36To37 should fix that one. (cut and paste problem when I added the 3.6 -> 3.7 test... not sure why it just started failing now though?) I'm still not sure why #install37From36 is failing. My colleague Jens Glander had analysed the same issue, and - as far as I know - has found the bug in the test code where the verifier bundle is installed in the wrong location. This is the message he sent to the mailing list: http://dev.eclipse.org/mhonarc/lists/p2-dev/msg03884.html I'll also try to put in on CC for this issue in case you have more questions to him. Thanks for the pointer. After reviewing the note and looking closer at the code, it appears the Install tests are extracting the platform zip to /tmp/xxx and then using the director to install into a temp location which is a sub-dir of /eclipse (p2_sniff_folder on the test machine). We copy the verifier bundle to /eclipse/p2_sniff_folder but when we try and run the verifier, we are running the eclipse in /tmp/xxx. The method #runVerifierBundle doesn't use the "destination" arg. The tests passed last night. Closing. . Great work DJ! Any idea why this suddenly started failing? Did something change in a test case or on the server? Or were these tests never properly running? No idea why they started failing. Perhaps they failed before and we always assumed failures in the Install/Update tests were firewall/server related and just ignored them? Ah well. From now on we will pay closer attention since now we understand what's going on. :-) |
There are a couple of test failures in the nightly build. Specifically: junit.framework.AssertionFailedError: expected:<0> but was:<13> at org.eclipse.equinox.p2.tests.full.From36to37.from36To37(From36to37.java:47) junit.framework.AssertionFailedError: expected:<0> but was:<13> at org.eclipse.equinox.p2.tests.full.Install37from36.install37From36(Install37from36.java:50) Looking at the console, (Test Results -> Console Output Logs -> linux.gtk-6.0_consolelog.txt) there seem to be several hundred of these messages: [java] STDERR: chmod: /tmp/1301323532998-0.790961705585606/eclipse: new permissions are r-xrwxr-x, not r-xr-xr-x [java] STDERR: chmod: /tmp/1301323532998-0.790961705585606/eclipse/notice.html: new permissions are r--rw-r--, not r--r--r-- [java] STDERR: chmod: /tmp/1301323532998-0.790961705585606/eclipse/configuration: new permissions are r-xrwxr-x, not r-xr-xr-x [java] STDERR: chmod: /tmp/1301323532998-0.790961705585606/eclipse/configuration/config.ini: new permissions are r--rw-r--, not r--r--r-- And then at the end of the file we see this: [java] STDERR: An error has occurred. See the log file [java] STDERR: /tmp/1301323924372-0.7007638145603529/eclipse/configuration/1301324824840.log. [java] STDERR: Eclipse: [java] STDERR: An error has occurred. See the log file [java] STDERR: /tmp/1301324829858-0.6040562064171939/eclipse/configuration/1301324898043.log. Unfortunately those folders have been cleaned up but we will keep an eye on the nightly build tonight to see if the same error occurs and if we can get more info from these other logs.