Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 352215

Summary: Unavailable port 2401 on startup causes hang on shutdown [was: Web and Jetty builds failing on Hudson]
Product: [RT] Virgo Reporter: Chris Frost <eclipse>
Component: unknownAssignee: Borislav Kapukaranov <b.kapukaranov>
Status: CLOSED FIXED QA Contact:
Severity: normal    
Priority: P3 CC: b.kapukaranov, glyn.normington, l.kirchev
Version: 3.0.0.M06   
Target Milestone: 3.0.0.RC1   
Hardware: PC   
OS: Mac OS X - Carbon (unsup.)   
Whiteboard:
Attachments:
Description Flags
Sample log taken from Hudson web-server build
none
Changes to Activator none

Description Chris Frost CLA 2011-07-15 09:59:04 EDT
During a smoke-test run the server is started up, tests run, then shutdown. Then the warm start is tested. Finally it does everything again. The second run through fails as the server fails to start up. This is because something fails to shutdown properly, probably a port.
Comment 1 Chris Frost CLA 2011-07-15 10:09:10 EDT
Created attachment 199749 [details]
Sample log taken from Hudson web-server build

I suspect this may be the cause but I'm not sure as it doesn't cause the server to stop starting. I only find this error on the logs for the second startup, the first startup is fine. One could argue that if you have telnet or ssh configured and it fails to start then the whole server should fail to start.
Comment 2 Glyn Normington CLA 2011-07-19 04:56:50 EDT
Targeting 3.0.0.RC1 as this is a "must fix".
Comment 3 Glyn Normington CLA 2011-07-21 08:23:13 EDT
Assigning back to virgo-inbox since my experiments seem to show that this is likely to be a bug in the Equinox console supportability bundle. Lazar is looking into it.

To reproduce (at least on Ubuntu, but probably everywhere), run a separate Equinox instance with a telnet console on port 2401:

java -jar org.eclipse.osgi-3.7.0.v20110224.jar -console 2401

and the run the (Tomcat) web server top-level build with parameters:

ant clean clean-integration package smoke-test

The following stack trace appears on startup:

[2011-07-21 06:26:31.049] INFO  iLogServiceListener@1cad157f org.osgi.service.log.LogService                                   Bundle org.eclipse.equinox.console.supportability_1.0.0.N20110626-2000, Service 139, ServiceEvent REGISTERED 
[2011-07-21 06:26:31.057] INFO  iLogServiceListener@1cad157f org.osgi.service.log.LogService                                   Bundle org.eclipse.equinox.console.supportability_1.0.0.N20110626-2000, Service 140, ServiceEvent REGISTERED 
[2011-07-21 06:26:31.059] INFO  main                         System.out                                                        Cannot start telnet. Reason: Port 2401 already in use 
[2011-07-21 06:26:31.059] ERROR main                         System.err                                                        java.lang.Exception: Port 2401 already in use 
[2011-07-21 06:26:31.059] ERROR main                         System.err                                                        	at org.eclipse.equinox.console.telnet.TelnetCommand.telnet(TelnetCommand.java:152) 
[2011-07-21 06:26:31.060] ERROR main                         System.err                                                        	at org.eclipse.equinox.console.telnet.TelnetCommand.start(TelnetCommand.java:89) 
[2011-07-21 06:26:31.060] ERROR main                         System.err                                                        	at org.eclipse.equinox.console.command.adapter.Activator$ProcessorCustomizer.addingService(Activator.java:81) 
[2011-07-21 06:26:31.060] ERROR main                         System.err                                                        	at org.eclipse.equinox.console.command.adapter.Activator$ProcessorCustomizer.addingService(Activator.java:1) 
[2011-07-21 06:26:31.060] ERROR main                         System.err                                                        	at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:980) 
[2011-07-21 06:26:31.060] ERROR main                         System.err                                                        	at org.osgi.util.tracker.ServiceTracker$Tracked.customizerAdding(ServiceTracker.java:1) 
[2011-07-21 06:26:31.061] ERROR main                         System.err                                                        	at org.osgi.util.tracker.AbstractTracked.trackAdding(AbstractTracked.java:262) 
[2011-07-21 06:26:31.061] ERROR main                         System.err                                                        	at org.osgi.util.tracker.AbstractTracked.trackInitial(AbstractTracked.java:185) 
[2011-07-21 06:26:31.061] ERROR main                         System.err                                                        	at org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:348) 
[2011-07-21 06:26:31.061] ERROR main                         System.err                                                        	at org.osgi.util.tracker.ServiceTracker.open(ServiceTracker.java:283) 
[2011-07-21 06:26:31.061] ERROR main                         System.err                                                        	at org.eclipse.equinox.console.command.adapter.Activator.start(Activator.java:202) 
[2011-07-21 06:26:31.062] ERROR main                         System.err                                                        	at org.eclipse.osgi.framework.internal.core.BundleContextImpl$1.run(BundleContextImpl.java:711) 
[2011-07-21 06:26:31.062] ERROR main                         System.err                                                        	at java.security.AccessController.doPrivileged(Native Method) 
[2011-07-21 06:26:31.062] ERROR main                         System.err                                                        	at org.eclipse.osgi.framework.internal.core.BundleContextImpl.startActivator(BundleContextImpl.java:702) 
[2011-07-21 06:26:31.062] ERROR main                         System.err                                                        	at org.eclipse.osgi.framework.internal.core.BundleContextImpl.start(BundleContextImpl.java:683) 
[2011-07-21 06:26:31.063] ERROR main                         System.err                                                        	at org.eclipse.osgi.framework.internal.core.BundleHost.startWorker(BundleHost.java:381) 
[2011-07-21 06:26:31.063] ERROR main                         System.err                                                        	at org.eclipse.osgi.framework.internal.core.AbstractBundle.start(AbstractBundle.java:299) 
[2011-07-21 06:26:31.063] ERROR main                         System.err                                                        	at org.eclipse.osgi.framework.internal.core.AbstractBundle.start(AbstractBundle.java:291) 
[2011-07-21 06:26:31.063] ERROR main                         System.err                                                        	at org.eclipse.virgo.osgi.launcher.FrameworkBuilder.installAndStartBundles(FrameworkBuilder.java:187) 
[2011-07-21 06:26:31.064] ERROR main                         System.err                                                        	at org.eclipse.virgo.osgi.launcher.FrameworkBuilder.start(FrameworkBuilder.java:159) 
[2011-07-21 06:26:31.064] ERROR main                         System.err                                                        	at org.eclipse.virgo.osgi.launcher.Launcher.main(Launcher.java:54) 
[2011-07-21 06:26:31.072] ERROR iLogServiceListener@1cad157f org.osgi.service.log.LogService                                   Bundle org.eclipse.equinox.cm_1.0.300.v20101204, null java.lang.NullPointerException: null
	at org.eclipse.equinox.console.telnet.TelnetCommand$TelnetConfigurator.updated(TelnetCommand.java:206)
	at org.eclipse.equinox.internal.cm.ManagedServiceTracker$1.run(ManagedServiceTracker.java:181)
	at org.eclipse.equinox.internal.cm.SerializedTaskQueue$1.run(SerializedTaskQueue.java:36)

[2011-07-21 06:26:31.072] INFO  iLogServiceListener@1cad157f org.osgi.service.log.LogService                                   Bundle org.eclipse.equinox.console.supportability_1.0.0.N20110626-2000, Service 141, ServiceEvent REGISTERED 

(Apparently the NPE will be fixed by bug 352211.) 

However, Virgo continues and starts up successfully (although there is no event log message or console message saying that 2401 is not available - the System.out message is redirected to Virgo's trace log).

During shutdown, the following stack trace appears:

[2011-07-21 06:27:11.073] ERROR iLogServiceListener@1cad157f org.osgi.service.log.LogService                                   Bundle org.eclipse.equinox.console.supportability_1.0.0.N20110626-2000, FrameworkEvent ERROR org.osgi.framework.BundleException: Exception in org.eclipse.equinox.console.command.adapter.Activator.stop() of bundle org.eclipse.equinox.console.supportability.
	at org.eclipse.osgi.framework.internal.core.BundleContextImpl.stop(BundleContextImpl.java:791)
	at org.eclipse.osgi.framework.internal.core.BundleHost.stopWorker(BundleHost.java:510)
	at org.eclipse.osgi.framework.internal.core.AbstractBundle.suspend(AbstractBundle.java:565)
	at org.eclipse.osgi.framework.internal.core.Framework.suspendBundle(Framework.java:1160)
	at org.eclipse.osgi.framework.internal.core.StartLevelManager.suspendAllBundles(StartLevelManager.java:615)
	at org.eclipse.osgi.framework.internal.core.StartLevelManager.doSetStartLevel(StartLevelManager.java:261)
	at org.eclipse.osgi.framework.internal.core.StartLevelManager.shutdown(StartLevelManager.java:215)
	at org.eclipse.osgi.framework.internal.core.InternalSystemBundle.suspend(InternalSystemBundle.java:284)
	at org.eclipse.osgi.framework.internal.core.Framework.shutdown(Framework.java:690)
	at org.eclipse.osgi.framework.internal.core.Framework.close(Framework.java:597)
	at org.eclipse.osgi.framework.internal.core.InternalSystemBundle$1.run(InternalSystemBundle.java:261)
	at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.IllegalStateException: telnet is not running.
	at org.eclipse.equinox.console.telnet.TelnetCommand.telnet(TelnetCommand.java:159)
	at org.eclipse.equinox.console.command.adapter.Activator.stop(Activator.java:300)
	at org.eclipse.osgi.framework.internal.core.BundleContextImpl$2.run(BundleContextImpl.java:771)
	at java.security.AccessController.doPrivileged(Native Method)
	at org.eclipse.osgi.framework.internal.core.BundleContextImpl.stop(BundleContextImpl.java:764)
	... 11 common frames omitted

so it seems as if the console supportability bundle is trying to close a port it didn't manage to open. I'm not sure why this should cause a hang, but it seems to.
Comment 4 Lazar Kirchev CLA 2011-07-21 08:54:28 EDT
Fix submitted to the supportability bundle in Equinox incubator.
Comment 5 Glyn Normington CLA 2011-07-21 08:56:07 EDT
(In reply to comment #4)
> Fix submitted to the supportability bundle in Equinox incubator.

Please could you attach the patch here for posterity?
Comment 6 Lazar Kirchev CLA 2011-07-21 09:04:19 EDT
(In reply to comment #5)
> (In reply to comment #4)
> > Fix submitted to the supportability bundle in Equinox incubator.
> 
> Please could you attach the patch here for posterity?

Unfortunately I already committed the changes to the repository. I cannot make a diff against it.
Comment 7 Glyn Normington CLA 2011-07-21 09:17:39 EDT
Created attachment 200091 [details]
Changes to Activator
Comment 8 Glyn Normington CLA 2011-07-21 09:58:14 EDT
Verified that the fix removes the hang on a local Ubuntu VM. We will not know whether this helps the CI server until we have put the bundle in the EBR and integrated it into the VWS build.
Comment 9 Chris Frost CLA 2011-07-22 08:53:10 EDT
Hudson is back to blue. The fix to the telnet port shutdown worked.