Community
Participate
Working Groups
Build Identifier: 20100917-0705 After changing the default JMX port (for instance change from 9875 to 9877), if I start the server with ./startup.sh &, and then try to shut it down with ./shutdown.sh, then I get the error following error message : The Server could not be reached, it may already be stopped. I have to manually kill the process to stop the server. Reproducible: Always Steps to Reproduce: 1.Edit dmk.sh to change the default JMX_PORT 2.Sart the server 3.Try to stop the server with shutdown.sh
Hi Lorie Rather than editing the script to change the default, you need to pass the parameter: -jmxport 9877 to both startup and shutdown scripts. This is because the default applied in the shutdown script is not obtained from dmk.sh. If you don't want to specify a parameter, dmk.sh calls bin/setenv.sh script, if present, to set environment variables and you can set $JMX_OPTS to "-jmxport 9877" and $JMX_PORT to "9877". Hope that helps.
Hello Glyn, Thank for your help. I don't want to pass the JMX port with command line, so I try to add the setenv.sh. The only configuration I could get the server to start with is export JMX_PORT="9877" export JMX_OPTS="-Dcom.sun.management.jmxremote.port=9877" (in setenv.sh file). But if I do this, when I shutdown the server, I get the error : Erreur: L'accès à la lecture du fichier de mots de passe doit être limité: /usr/lib/jvm/java-6-sun-1.6.0.22/jre/lib/management/jmxremote.password (Translation : the password file access must be limited). I tried with export JMX_OPTS="-jmxport 9877" as you advised me, but when I do so, the server doesn't even start with the message : Unrecognized option: -jmxport Could not create the Java virtual machine. What am I missing? Thanx again for your help. Best Regards.
Re-opening for further investigation.
I tested on the Virgo 3.0 line in the hope that the fix to bug 331299 would help after creating a setenv.sh containing: export JMX_PORT="9877" export JMX_OPTS="-Dcom.sun.management.jmxremote.port=9877" but shutdown still complained: $ bin/shutdown.sh Error: Password file not found: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/lib/management/jmxremote.password I'll leave this open in the backlog for inclusion in a future sprint.
Hi Lorie, In setenv.sh specify only the JMX_PORT i.e. export JMX_PORT="9877" Please do not specify export JMX_OPTS="-Dcom.sun.management.jmxremote.port=9877" because we also specify this in dmk.sh (JMX_OPTS) and this duplication is the cause for the "Error: Password file not found" that you observe. I also prepared a patch for the original problem "The Server could not be reached, it may already be stopped." Can you test that it is working for your use case. Best Regards Violeta Georgieva
Created attachment 188889 [details] Patch
Created attachment 188978 [details] Patch
Tested, committed and pushed. The fix will be available in 3.0.0.M01. Lorie, I'm leaving the status as RESOLVED. Could you please verify that this fixes the issue, if it works as you expect, please would you change the status to CLOSED. Regards Violeta
Dear Violeta, Thank you for the fix. I removed the export of JMX_OPTS in the setenv.sh, and integrated the modifications included in your patch. It gets a little farer : the server is initiating shutdown, stopping my bundles, but it doesn't really stops. The virgo process is still runing. And if I start again shutdown.sh, I get the following exception : javax.management.InstanceNotFoundException: org.eclipse.virgo.kernel:type=Shutdown at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getClassLoaderFor(DefaultMBeanServerInterceptor.java:1438) at com.sun.jmx.mbeanserver.JmxMBeanServer.getClassLoaderFor(JmxMBeanServer.java:1276) at com.sun.jmx.remote.security.MBeanServerAccessController.getClassLoaderFor(MBeanServerAccessController.java:329) at javax.management.remote.rmi.RMIConnectionImpl$5.run(RMIConnectionImpl.java:1326) at java.security.AccessController.doPrivileged(Native Method) at javax.management.remote.rmi.RMIConnectionImpl.getClassLoaderFor(RMIConnectionImpl.java:1323) at javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:771) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:305) at sun.rmi.transport.Transport$1.run(Transport.java:159) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:155) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:255) at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142) at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown Source) at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:993) at org.eclipse.virgo.kernel.shutdown.ShutdownClient.doShutdown(ShutdownClient.java:71) at org.eclipse.virgo.kernel.shutdown.ShutdownClient.performShutdown(ShutdownClient.java:50) at org.eclipse.virgo.kernel.shutdown.ShutdownClient.main(ShutdownClient.java:43) Any idea of where this new problem come from? Regards, Lorie
Looks like the shutdown sequence got stuck somewhere. Maybe try jps/jstack to see the stacks?
Created attachment 189117 [details] jstack result after shutdown Here is the result of jstack when I shut down the server with shutdown.sh Hope this will help you investigate.
(In reply to comment #11) > Created attachment 189117 [details] > jstack result after shutdown > > Here is the result of jstack when I shut down the server with shutdown.sh > Hope this will help you investigate. There are some signs of being held up. For instance the socket read in thread RMI TCP Connection(3)-192.168.2.46. The tricky thing is there isn't a Virgo stack frame in sight, so I think we need to turn to logging for more insight. For starters, please could you attach serviceability/logs/log.log for the hung instance after starting it up with -clean to minimise the start of the log.
Created attachment 189159 [details] log.log Attached the log file after a startup -clean, and shutdown
Nothing glaringly obvious in the log. I wonder if you would mind doing the same and attaching a log for the successful shutdown case when you haven't changed the port. Then we can play "spot the difference". Now that this bug is resolved, you should probably create a fresh bug for this problem, but that's not urgent.
I have tried the same thing on the other instance of Virgo in which I didn't changed the JMX port. I have the exact same issue when shutting the server down. It seems to be due to the plan I put in the pickup because as soon as I remove it, it stops properly. This is no longer related to the JMX bug, so I think this is not the best way to discuss about it. But do you have any clue to help me investigate this? BTW, I change the satus to closed, as the JMX issue is successfully fixed by the patch.
Probably best to take this up on the forum and see if there is anything "unusual" about your plan.
