357240 – Jetty's 100% cpu utilization (due to NIO select? )

Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 357240 - Jetty's 100% cpu utilization (due to NIO select? )

Summary: Jetty's 100% cpu utilization (due to NIO select? )

Status:	RESOLVED FIXED

Alias:	None

Product:	Jetty
Classification:	RT
Component:	server (show other bugs)
Version:	7.4.1
Hardware:	PC Windows 7

Importance:	P3 normal (vote)
Target Milestone:	7.5.x
Assignee:	Greg Wilkins
QA Contact:

URL:
Whiteboard:
Keywords:

Depends on:
Blocks:

Reported:	2011-09-09 10:34 EDT by Ryan Daley
Modified:	2014-05-18 06:15 EDT (History)
CC List:	3 users (show)

See Also:

Attachments
heap/thread dump for 100% cpu issue (41.41 KB, text/plain) 2011-09-09 12:06 EDT, Ryan Daley	no flags	Details
screenshot of cpu analysis (96.70 KB, image/png) 2011-09-16 16:49 EDT, Ryan Daley	no flags	Details
select CPU hotspot (41.78 KB, image/png) 2011-09-16 16:50 EDT, Ryan Daley	no flags	Details
select CPU backtrace (35.48 KB, image/png) 2011-09-16 16:51 EDT, Ryan Daley	no flags	Details
View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.

Description Ryan Daley

2011-09-09 10:34:00 EDT

Team Webtide,

We’re having the jetty 100% cpu utilization issue included below is what a Visual VM profiling shows as the methods consuming the cpu and it appears related to the NIO selector invocation issue. This seems like the issue Greg mentioned in his Feb 8 email to jetty-dev@eclipse.org see . Was there a patch for this problem or a later version we should be running that resolves this one, presently running 7.4.3 and seeing it under both Win XP and 7.

25% our servlet’s run method
17% our UDP traffic listener’s run method
17% o.e.j.io.nio.SelectorManager$SelectorSet.doSelect
15% java.util.TimerThread.run()  used to throttle waiting
09% o.e.j.client.HttpClient$1.run()


One time Visual VM noticed mid execution the SelectorManager no longer active and reported it was possibly unloaded and though Jetty became unresponsive it was running but the 100% cpu utilization went away.


I enabled the start and end dumps. What logging do I need to enable and/or in what files do I need to look for these to be captured in? I'm seeing it output to the console but it is tough to read mixed in with the application’s own console messages. When I have determined how to do so I will supplement this issue by attaching those logs at that time (if it is possible to attach supplements post submission that is).

HTH let me know what else I can provide, I'll look into the logging more. Can you please offer any additional direction that would be relevant in handling this issue. What has helped/resolved it in the past? What are things to check? etc.

Thanks,

Ryan

PS As a future log attachment will indicate we are running the Bayeux server now but the cpu problem was there before we went to using cometd/Bayeux.

Comment 1 Jesse McConnell

2011-09-09 10:41:17 EDT

if you can replicate on 7.5.1 which was released yesterday then we absolutely want to hear about this :)

generally what works best is a mess of thread dumps that help capture the busy loop, sometimes up to 100 are needed but if you could take say 10 and then attach a tarball of them all that would be great

cheers
jesse

Comment 2 Ryan Daley

2011-09-09 12:03:23 EDT

I am attaching heap dump (with thread dumps) from an XP run (issue platform non-discriminatory winxp/win7/linux) for starters. The busy threads appear to be these threads.

DestroyJavaVM
Thread-30
Thread-29
Thread-27
Thread-25
qtp49359-28 Acceptor1 SelectChannelConnector@0.0.0.0:8080 STARTING
qtp49359-21 Acceptor1 SelectChannelConnector@0.0.0.0:8080 STARTED
qtp49359-20 Acceptor1 SelectChannelConnector@0.0.0.0:8080 STARTED
RMI TCP accept - 0 (Visual VM shows this thread name in table twice 
                    two unique identically named threads? )

Comment 3 Ryan Daley

2011-09-09 12:06:18 EDT

Created attachment 203073 [details]
heap/thread dump for 100% cpu issue

see my comment 2 under bug comments

Comment 4 Ryan Daley

2011-09-09 12:09:28 EDT

Thanks Jesse will have to try 7.5.1 and advise.

Comment 5 Ryan Daley

2011-09-09 14:50:47 EDT

I should add as I mull over Greg's post on http://wiki.eclipse.org/Jetty/Feature/JVM_NIO_Bug that this is happening under the lastest early release of Java 7 as well as previous java versions. 

Also the attached heap dump summary was for the 17M .hprof binary file. You guys want the binary?

Comment 6 Ryan Daley

2011-09-16 16:48:12 EDT

Confirmed this is an issue still for us using 7.5.1. Here are a couple more thread dumps. Note the Acceptor0 thread BLOCKED if this is a possible problem indicator.

VisualVM shows the NIO hotspots see attached graphics which I assume are the blocking threads. Can Jetty be configured to run using BIO? Will tradeoff throughput for CPU load until this gets resolved but need to get CPU performance resolved immediately. Not a ton of concurrent users are expected and HTTP traffic while constantly sending updates are small per transmission  if that helps in determining if adjustments to threadpool, etc. are advisable. 

Also while the issue still seems to be around oejs.AbstractConnector$Acceptor and oejs.nio.... I'm not seeing any of the Jetty JVM NIO bug workaround messages coming out to the console in windows or linux.

Thanks!

Ryan

2011-09-16 14:37:46
Full thread dump Java HotSpot(TM) Client VM (21.0-b17 mixed mode, sharing):

"RMI TCP Connection(7)-155.34.190.20" daemon prio=6 tid=0x0514c800 nid=0x1158 runnable [0x07ccf000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:150)
        at java.net.SocketInputStream.read(SocketInputStream.java:121)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
        - locked <0x2a363e88> (a java.io.BufferedInputStream)
        at java.io.FilterInputStream.read(FilterInputStream.java:83)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:670)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:604)
        at java.lang.Thread.run(Thread.java:722)

"*** Profiler Agent Special Execution Thread 6" daemon prio=6 tid=0x04ace000 nid=0x190c in Object.wait() [0x07a7f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x29f7ab30> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:503)
        at org.netbeans.lib.profiler.server.ProfilerServer$SeparateCmdExecutionThread.run(ProfilerServer.java:289)
        - locked <0x29f7ab30> (a java.lang.Object)

"*** JFluid Monitor thread ***" daemon prio=10 tid=0x04acdc00 nid=0x118c waiting on condition [0x074ff000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.netbeans.lib.profiler.server.Monitors$SurvGenAndThreadsMonitor.run(Monitors.java:186)

"*** Profiler Agent Communication Thread" daemon prio=10 tid=0x04acd400 nid=0x14ec runnable [0x07d7f000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.read(SocketInputStream.java:150)
        at java.net.SocketInputStream.read(SocketInputStream.java:121)
        at java.net.SocketInputStream.read(SocketInputStream.java:203)
        at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2266)
        at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2446)
        at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2516)
        at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2588)
        at java.io.ObjectInputStream.read(ObjectInputStream.java:838)
        at org.netbeans.lib.profiler.wireprotocol.WireIO.receiveCommandOrResponse(WireIO.java:88)
        at org.netbeans.lib.profiler.server.ProfilerServer.listenToClient(ProfilerServer.java:1703)
        at org.netbeans.lib.profiler.server.ProfilerServer.run(ProfilerServer.java:684)

"JMX server connection timeout 47" daemon prio=6 tid=0x04acc800 nid=0x1740 in Object.wait() [0x07acf000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x29f7ac78> (a [I)
        at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:168)
        - locked <0x29f7ac78> (a [I)
        at java.lang.Thread.run(Thread.java:722)

"RMI Scheduler(0)" daemon prio=6 tid=0x04acc000 nid=0x1a7c waiting on condition [0x07a0f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x29f7ad00> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

"RMI TCP Accept-0" daemon prio=6 tid=0x04acb400 nid=0x410 runnable [0x077cf000]
   java.lang.Thread.State: RUNNABLE
        at java.net.DualStackPlainSocketImpl.accept0(Native Method)
        at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:121)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:396)
        at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:183)
        - locked <0x29f7d218> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:522)
        at java.net.ServerSocket.accept(ServerSocket.java:490)
        at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:387)
        at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
        at java.lang.Thread.run(Thread.java:722)

"qtp15202739-43" prio=6 tid=0x04acb000 nid=0x19cc waiting on condition [0x0754f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x297b0fb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
        at java.lang.Thread.run(Thread.java:722)

"DestroyJavaVM" prio=6 tid=0x0514bc00 nid=0x14d8 waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"BayeuxServer@23789217" daemon prio=6 tid=0x0514c400 nid=0x1bd8 in Object.wait() [0x073ff000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x29c362b0> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:552)
        - locked <0x29c362b0> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

"Thread-" prio=6 tid=0x0514b800 nid=0x1684 runnable [0x0733f000]
   java.lang.Thread.State: RUNNABLE
        at java.util.Vector.size(Vector.java:311)
        - locked <0x2997e950> (a java.util.Vector)
        at edu.mit.ll.sstbcore.webapps.RSCControl.find(RSCControl.java:381)
        at edu.mit.ll.sstbcore.webapps.RSCControl.isSystemFault(RSCControl.java:389)
        at edu.mit.ll.sstbcore.webapps.RadarManager.run(RadarManager.java:616)
        at java.lang.Thread.run(Thread.java:722)

"Thread-RemoteSentinelDisplay's UDPListener_1" prio=6 tid=0x0514b000 nid=0x16e8 runnable [0x0700f000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:295)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:277)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:158)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        - locked <0x29976198> (a sun.nio.ch.Util$2)
        - locked <0x299761a8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x29976120> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
        at edu.mit.ll.sstbcore.utilities.net.UDPListener.run(UDPListener.java:144)
        at java.lang.Thread.run(Thread.java:722)

"Thread-RemoteSentinel" prio=6 tid=0x0514a800 nid=0x19dc in Object.wait() [0x0714f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x29976328> (a edu.mit.ll.sstbcore.webapps.RemoteSentinel)
        at edu.mit.ll.sstbcore.webapps.SentinelWebServlet.run(SentinelWebServlet.java:374)
        - locked <0x29976328> (a edu.mit.ll.sstbcore.webapps.RemoteSentinel)
        at java.lang.Thread.run(Thread.java:722)

"Thread-localhost's_UDPListener's UDPListener_2" prio=6 tid=0x0514a400 nid=0x62c runnable [0x06b1f000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:295)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:277)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:158)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        - locked <0x29979560> (a sun.nio.ch.Util$2)
        - locked <0x29979570> (a java.util.Collections$UnmodifiableSet)
        - locked <0x299794e8> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
        at edu.mit.ll.sstbcore.utilities.net.UDPListener.run(UDPListener.java:144)
        at java.lang.Thread.run(Thread.java:722)

"Thread-FDLServlet" prio=6 tid=0x05149c00 nid=0x1ab0 in Object.wait() [0x06d8f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x299247a0> (a edu.mit.ll.sstbcore.webapps.FDLServlet)
        at edu.mit.ll.sstbcore.webapps.SentinelWebServlet.run(SentinelWebServlet.java:374)
        - locked <0x299247a0> (a edu.mit.ll.sstbcore.webapps.FDLServlet)
        at java.lang.Thread.run(Thread.java:722)

"Thread-localhost's_UDPListener's UDPListener_1" prio=6 tid=0x05149800 nid=0x1a70 runnable [0x06f1f000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:295)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:277)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:158)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        - locked <0x29927958> (a sun.nio.ch.Util$2)
        - locked <0x29927968> (a java.util.Collections$UnmodifiableSet)
        - locked <0x299278e0> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
        at edu.mit.ll.sstbcore.utilities.net.UDPListener.run(UDPListener.java:144)
        at java.lang.Thread.run(Thread.java:722)

"Thread-RRDLServlet" prio=6 tid=0x05149000 nid=0x1a5c in Object.wait() [0x06eaf000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x29927ae8> (a edu.mit.ll.sstbcore.webapps.RRDLServlet)
        at edu.mit.ll.sstbcore.webapps.SentinelWebServlet.run(SentinelWebServlet.java:374)
        - locked <0x29927ae8> (a edu.mit.ll.sstbcore.webapps.RRDLServlet)
        at java.lang.Thread.run(Thread.java:722)

"Thread-RRDLServlet's UDPListener_1" prio=6 tid=0x05148c00 nid=0x1b50 runnable [0x06e4f000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:295)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:277)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:158)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        - locked <0x2992acb8> (a sun.nio.ch.Util$2)
        - locked <0x2992acc8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x2992ac40> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
        at edu.mit.ll.sstbcore.utilities.net.UDPListener.run(UDPListener.java:144)
        at java.lang.Thread.run(Thread.java:722)

"HttpClient-29" daemon prio=6 tid=0x05148400 nid=0x1398 waiting on condition [0x06a4f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x298dcea8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
        at java.lang.Thread.run(Thread.java:722)

"HttpClient-28" daemon prio=6 tid=0x05147c00 nid=0x15e4 waiting on condition [0x0684f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x298dcea8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
        at java.lang.Thread.run(Thread.java:722)

"HttpClient-27" daemon prio=6 tid=0x05147800 nid=0x1b08 waiting on condition [0x064ff000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x298dcea8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
        at java.lang.Thread.run(Thread.java:722)

"HttpClient-26" daemon prio=6 tid=0x05147000 nid=0x196c waiting on condition [0x0661f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x298dcea8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
        at java.lang.Thread.run(Thread.java:722)

"HttpClient-25" daemon prio=6 tid=0x05146c00 nid=0x1b70 waiting on condition [0x067ff000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x298dcea8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
        at java.lang.Thread.run(Thread.java:722)

"HttpClient-24" daemon prio=6 tid=0x05146400 nid=0x16d8 waiting on condition [0x05e7f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x298dcea8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
        at java.lang.Thread.run(Thread.java:722)

"HttpClient-23" daemon prio=6 tid=0x05146000 nid=0x64c waiting on condition [0x0666f000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at org.eclipse.jetty.client.HttpClient$1.run(HttpClient.java:456)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
        at java.lang.Thread.run(Thread.java:722)

"HttpClient-22" daemon prio=6 tid=0x05145800 nid=0x116c runnable [0x0617f000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:295)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:277)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:158)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        - locked <0x298e0318> (a sun.nio.ch.Util$2)
        - locked <0x298e0328> (a java.util.Collections$UnmodifiableSet)
        - locked <0x298e02a0> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:487)
        at org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:225)
        at org.eclipse.jetty.client.SelectConnector.run(SelectConnector.java:127)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
        at java.lang.Thread.run(Thread.java:722)

"HashSessionScavenger-1" daemon prio=6 tid=0x05145000 nid=0x1488 in Object.wait() [0x05e2f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x2986a9c0> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:552)
        - locked <0x2986a9c0> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

"HashSessionScavenger-0" daemon prio=6 tid=0x050f3800 nid=0x1ab8 in Object.wait() [0x05f2f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x297d9940> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:552)
        - locked <0x297d9940> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

"qtp15202739-18 Selector0 SelectChannelConnector@0.0.0.0:8080 STARTED" prio=6 tid=0x050bb000 nid=0x1758 runnable [0x061cf000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:295)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:277)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:158)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        - locked <0x29c7ba50> (a sun.nio.ch.Util$2)
        - locked <0x29c7ba60> (a java.util.Collections$UnmodifiableSet)
        - locked <0x29c7b9d8> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:487)
        at org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:225)
        at org.eclipse.jetty.server.nio.SelectChannelConnector$1.run(SelectChannelConnector.java:268)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
        at java.lang.Thread.run(Thread.java:722)

"qtp15202739-17" prio=6 tid=0x050bac00 nid=0xbc8 waiting on condition [0x05f8f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x297b0fb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
        at java.lang.Thread.run(Thread.java:722)

"qtp15202739-16 Acceptor1 SelectChannelConnector@0.0.0.0:8080 STARTING" prio=6 tid=0x050ba400 nid=0x13f4 runnable [0x05ecf000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
        - locked <0x29c7bb88> (a java.lang.Object)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:92)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:780)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
        at java.lang.Thread.run(Thread.java:722)

"qtp15202739-15" prio=6 tid=0x050b6000 nid=0x194c waiting on condition [0x05c1f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x297b0fb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
        at java.lang.Thread.run(Thread.java:722)

"qtp15202739-14 Acceptor0 SelectChannelConnector@0.0.0.0:8080 STARTING" prio=6 tid=0x050b2800 nid=0x15ac waiting for monitor entry [0x05cff000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
        - waiting to lock <0x29c7bb88> (a java.lang.Object)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:92)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:780)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
        at java.lang.Thread.run(Thread.java:722)

"qtp15202739-13" prio=6 tid=0x050b2000 nid=0x1bc4 waiting on condition [0x05baf000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x297b0fb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
        at java.lang.Thread.run(Thread.java:722)

"qtp15202739-12 Selector1 SelectChannelConnector@0.0.0.0:8080 STARTED" prio=6 tid=0x050b1c00 nid=0x19e4 runnable [0x05a9f000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:295)
        at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:277)
        at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:158)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
        - locked <0x29c7eca8> (a sun.nio.ch.Util$2)
        - locked <0x29c7ecb8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x29c7ec30> (a sun.nio.ch.WindowsSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:487)
        at org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:225)
        at org.eclipse.jetty.server.nio.SelectChannelConnector$1.run(SelectChannelConnector.java:268)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
        at java.lang.Thread.run(Thread.java:722)

"qtp15202739-11" prio=6 tid=0x0508fc00 nid=0x185c waiting on condition [0x049bf000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x297b0fb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
        at java.lang.Thread.run(Thread.java:722)

"qtp15202739-10" prio=6 tid=0x0508f400 nid=0xed8 waiting on condition [0x0599f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x297b0fb8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
        at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
        at java.lang.Thread.run(Thread.java:722)

"Service Thread" daemon prio=6 tid=0x0232a800 nid=0x7c0 runnable [0x00000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread0" daemon prio=10 tid=0x02325c00 nid=0x1b0c waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Attach Listener" daemon prio=10 tid=0x02324c00 nid=0x11b0 runnable [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x02321c00 nid=0x1870 waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=8 tid=0x02312c00 nid=0x1b80 in Object.wait() [0x045af000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x296f27b8> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
        - locked <0x296f27b8> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)

"Reference Handler" daemon prio=10 tid=0x02311400 nid=0x14c0 in Object.wait() [0x0467f000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x296f2840> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:503)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
        - locked <0x296f2840> (a java.lang.ref.Reference$Lock)

"VM Thread" prio=10 tid=0x0230f400 nid=0x17cc runnable

"VM Periodic Task Thread" prio=10 tid=0x0233c400 nid=0x169c waiting on condition

JNI global references: 12173

Heap
 def new generation   total 8064K, used 2798K [0x24180000, 0x24a40000, 0x296d0000)
  eden space 7168K,  36% used [0x24180000, 0x24413d18, 0x24880000)
  from space 896K,  22% used [0x24880000, 0x248b2ad0, 0x24960000)
  to   space 896K,   0% used [0x24960000, 0x24960000, 0x24a40000)
 tenured generation   total 17664K, used 12946K [0x296d0000, 0x2a810000, 0x34180000)
   the space 17664K,  73% used [0x296d0000, 0x2a374920, 0x2a374a00, 0x2a810000)
 compacting perm gen  total 16896K, used 16744K [0x34180000, 0x35200000, 0x38180000)
   the space 16896K,  99% used [0x34180000, 0x351da1c0, 0x351da200, 0x35200000)
    ro space 10240K,  45% used [0x38180000, 0x38601888, 0x38601a00, 0x38b80000)
    rw space 12288K,  54% used [0x38b80000, 0x392004f8, 0x39200600, 0x39780000)
|
|
|
|
|
|
|
|
|
|
|
| Dump 2:
|
|
|
|
|
|
|
|
|
|
|
|

2011-09-16 15:51:55
Full thread dump Java HotSpot(TM) Client VM (21.0-b17 mixed mode, sharing):

"RMI TCP Connection(5)-155.34.190.20" daemon prio=6 tid=0x052fd800 nid=0x1c64 runnable [0x07acf000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:150)
	at java.net.SocketInputStream.read(SocketInputStream.java:121)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
	- locked <0x29cef890> (a java.io.BufferedInputStream)
	at java.io.FilterInputStream.read(FilterInputStream.java:83)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- <0x29ce3888> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"RMI TCP Connection(4)-155.34.190.20" daemon prio=6 tid=0x052fd000 nid=0x1ce0 runnable [0x0734f000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.read(SocketInputStream.java:150)
	at java.net.SocketInputStream.read(SocketInputStream.java:121)
	at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:254)
	- locked <0x29cec1d8> (a java.io.BufferedInputStream)
	at java.io.FilterInputStream.read(FilterInputStream.java:83)
	at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:535)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:808)
	at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:667)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- <0x29c9a258> (a java.util.concurrent.ThreadPoolExecutor$Worker)

"JMX server connection timeout 45" daemon prio=6 tid=0x052fcc00 nid=0x1c80 in Object.wait() [0x0659f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x29c9a2f0> (a [I)
	at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:168)
	- locked <0x29c9a2f0> (a [I)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"RMI Scheduler(0)" daemon prio=6 tid=0x052fc400 nid=0x1e00 waiting on condition [0x079ef000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x29c9a378> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1043)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1103)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"RMI TCP Accept-0" daemon prio=6 tid=0x052fac00 nid=0x11b0 runnable [0x0778f000]
   java.lang.Thread.State: RUNNABLE
	at java.net.DualStackPlainSocketImpl.accept0(Native Method)
	at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:121)
	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:396)
	at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:183)
	- locked <0x29c9a688> (a java.net.SocksSocketImpl)
	at java.net.ServerSocket.implAccept(ServerSocket.java:522)
	at java.net.ServerSocket.accept(ServerSocket.java:490)
	at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:387)
	at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"DestroyJavaVM" prio=6 tid=0x052fb800 nid=0x1c2c waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"BayeuxServer@15929294" daemon prio=6 tid=0x052fb000 nid=0x2f8 in Object.wait() [0x0759f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x29c34b58> (a java.util.TaskQueue)
	at java.util.TimerThread.mainLoop(Timer.java:552)
	- locked <0x29c34b58> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:505)

   Locked ownable synchronizers:
	- None

"Thread-" prio=6 tid=0x052fa400 nid=0x1b50 runnable [0x06c4f000]
   java.lang.Thread.State: RUNNABLE
	at java.util.Vector.size(Vector.java:311)
	- locked <0x29983048> (a java.util.Vector)
	at edu.mit.ll.sstbcore.webapps.RSCControl.find(RSCControl.java:381)
	at edu.mit.ll.sstbcore.webapps.RSCControl.isSystemFault(RSCControl.java:389)
	at edu.mit.ll.sstbcore.webapps.RadarManager.run(RadarManager.java:616)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"Thread-RemoteSentinelDisplay's UDPListener_1" prio=6 tid=0x052fa000 nid=0x19dc waiting on condition [0x0722f000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at edu.mit.ll.sstbcore.utilities.net.UDPListener.run(UDPListener.java:196)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"Thread-RemoteSentinel" prio=6 tid=0x052f9800 nid=0x1a5c waiting on condition [0x072cf000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at edu.mit.ll.sstbcore.webapps.SentinelWebServlet.run(SentinelWebServlet.java:349)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"Thread-localhost's_UDPListener's UDPListener_2" prio=6 tid=0x052f9400 nid=0x1488 runnable [0x070ff000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
	at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:295)
	at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:277)
	at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:158)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
	- locked <0x2997d8f8> (a sun.nio.ch.Util$2)
	- locked <0x2997d908> (a java.util.Collections$UnmodifiableSet)
	- locked <0x2997d880> (a sun.nio.ch.WindowsSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:102)
	at edu.mit.ll.sstbcore.utilities.net.UDPListener.run(UDPListener.java:144)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"Thread-FDLServlet" prio=6 tid=0x052f8c00 nid=0x1f24 in Object.wait() [0x0706f000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x299340d8> (a edu.mit.ll.sstbcore.webapps.FDLServlet)
	at edu.mit.ll.sstbcore.webapps.SentinelWebServlet.run(SentinelWebServlet.java:374)
	- locked <0x299340d8> (a edu.mit.ll.sstbcore.webapps.FDLServlet)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"Thread-localhost's_UDPListener's UDPListener_1" prio=6 tid=0x052f8400 nid=0x1364 waiting on condition [0x0694f000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at edu.mit.ll.sstbcore.utilities.net.UDPListener.run(UDPListener.java:196)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"Thread-RRDLServlet" prio=6 tid=0x052f8000 nid=0x1d00 in Object.wait() [0x06bff000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x29934340> (a edu.mit.ll.sstbcore.webapps.RRDLServlet)
	at edu.mit.ll.sstbcore.webapps.SentinelWebServlet.run(SentinelWebServlet.java:374)
	- locked <0x29934340> (a edu.mit.ll.sstbcore.webapps.RRDLServlet)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"Thread-RRDLServlet's UDPListener_1" prio=6 tid=0x050a8c00 nid=0x1cf4 waiting on condition [0x06a5f000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at edu.mit.ll.sstbcore.utilities.net.UDPListener.run(UDPListener.java:196)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"HttpClient-29" daemon prio=6 tid=0x050a8800 nid=0x1f20 waiting on condition [0x06b8f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x298e0b98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"HttpClient-28" daemon prio=6 tid=0x050a8000 nid=0x1e3c waiting on condition [0x06aaf000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x298e0b98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"HttpClient-27" daemon prio=6 tid=0x050a7800 nid=0x1710 waiting on condition [0x06a0f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x298e0b98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"HttpClient-26" daemon prio=6 tid=0x050a7400 nid=0x1b58 waiting on condition [0x0672f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x298e0b98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"HttpClient-25" daemon prio=6 tid=0x050a6c00 nid=0x69c waiting on condition [0x0683f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x298e0b98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"HttpClient-24" daemon prio=6 tid=0x050a6800 nid=0x1ff0 waiting on condition [0x067ef000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x298e0b98> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"HttpClient-23" daemon prio=6 tid=0x050a6000 nid=0xeac waiting on condition [0x065ef000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
	at java.lang.Thread.sleep(Native Method)
	at org.eclipse.jetty.client.HttpClient$1.run(HttpClient.java:456)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"HttpClient-22" daemon prio=6 tid=0x050a5c00 nid=0x1d20 runnable [0x0667f000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
	at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:295)
	at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:277)
	at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:158)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
	- locked <0x298e4008> (a sun.nio.ch.Util$2)
	- locked <0x298e4018> (a java.util.Collections$UnmodifiableSet)
	- locked <0x298e3f90> (a sun.nio.ch.WindowsSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
	at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:487)
	at org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:225)
	at org.eclipse.jetty.client.SelectConnector.run(SelectConnector.java:127)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"HashSessionScavenger-1" daemon prio=6 tid=0x050a5400 nid=0x1db8 in Object.wait() [0x063ef000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x2986b3b0> (a java.util.TaskQueue)
	at java.util.TimerThread.mainLoop(Timer.java:552)
	- locked <0x2986b3b0> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:505)

   Locked ownable synchronizers:
	- None

"HashSessionScavenger-0" daemon prio=6 tid=0x0505ac00 nid=0x156c in Object.wait() [0x060bf000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x297daf78> (a java.util.TaskQueue)
	at java.util.TimerThread.mainLoop(Timer.java:552)
	- locked <0x297daf78> (a java.util.TaskQueue)
	at java.util.TimerThread.run(Timer.java:505)

   Locked ownable synchronizers:
	- None

"qtp9879367-19" prio=6 tid=0x05047400 nid=0x1498 waiting on condition [0x0586f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x297b15b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"qtp9879367-18 Selector1 SelectChannelConnector@0.0.0.0:8080 STARTED" prio=6 tid=0x05044c00 nid=0x1d44 runnable [0x0480f000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
	at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:295)
	at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:277)
	at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:158)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
	- locked <0x29c39728> (a sun.nio.ch.Util$2)
	- locked <0x29c39738> (a java.util.Collections$UnmodifiableSet)
	- locked <0x29c396b0> (a sun.nio.ch.WindowsSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
	at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:487)
	at org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:225)
	at org.eclipse.jetty.server.nio.SelectChannelConnector$1.run(SelectChannelConnector.java:268)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"qtp9879367-17" prio=6 tid=0x05044800 nid=0x1bac waiting on condition [0x05edf000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x297b15b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"qtp9879367-16" prio=6 tid=0x05044000 nid=0x19c8 waiting on condition [0x05d8f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x297b15b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"qtp9879367-15 Acceptor0 SelectChannelConnector@0.0.0.0:8080 STARTING" prio=6 tid=0x0502d000 nid=0x1980 waiting for monitor entry [0x05acf000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:210)
	- waiting to lock <0x29c3ccd0> (a java.lang.Object)
	at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:92)
	at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:780)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"qtp9879367-14 Selector0 SelectChannelConnector@0.0.0.0:8080 STARTED" prio=6 tid=0x0502c800 nid=0x1b90 runnable [0x05d1f000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll0(Native Method)
	at sun.nio.ch.WindowsSelectorImpl$SubSelector.poll(WindowsSelectorImpl.java:295)
	at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$400(WindowsSelectorImpl.java:277)
	at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:158)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
	- locked <0x29c3fe20> (a sun.nio.ch.Util$2)
	- locked <0x29c3fe30> (a java.util.Collections$UnmodifiableSet)
	- locked <0x29c3fda8> (a sun.nio.ch.WindowsSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
	at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:487)
	at org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:225)
	at org.eclipse.jetty.server.nio.SelectChannelConnector$1.run(SelectChannelConnector.java:268)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"qtp9879367-13" prio=6 tid=0x0502c400 nid=0xfd4 waiting on condition [0x05b9f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x297b15b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"qtp9879367-12" prio=6 tid=0x04d5a800 nid=0x1778 waiting on condition [0x05bef000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x297b15b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"qtp9879367-11 Acceptor1 SelectChannelConnector@0.0.0.0:8080 STARTING" prio=6 tid=0x04d5a000 nid=0x153c runnable [0x059af000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:226)
	- locked <0x29c3ccd0> (a java.lang.Object)
	at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:92)
	at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:780)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"qtp9879367-10" prio=6 tid=0x04d59c00 nid=0x19d0 waiting on condition [0x05a5f000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x297b15b8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
	at java.lang.Thread.run(Thread.java:722)

   Locked ownable synchronizers:
	- None

"Service Thread" daemon prio=6 tid=0x0211a800 nid=0x1764 runnable [0x00000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"C1 CompilerThread0" daemon prio=10 tid=0x02115c00 nid=0x1f48 waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"Attach Listener" daemon prio=10 tid=0x02114c00 nid=0x19b0 waiting on condition [0x00000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"Signal Dispatcher" daemon prio=10 tid=0x02111c00 nid=0x664 runnable [0x00000000]
   java.lang.Thread.State: RUNNABLE

   Locked ownable synchronizers:
	- None

"Finalizer" daemon prio=8 tid=0x02102c00 nid=0xba0 in Object.wait() [0x046df000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x296f28e8> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
	- locked <0x296f28e8> (a java.lang.ref.ReferenceQueue$Lock)
	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:177)

   Locked ownable synchronizers:
	- None

"Reference Handler" daemon prio=10 tid=0x02101400 nid=0x1f68 in Object.wait() [0x0459f000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x296f2970> (a java.lang.ref.Reference$Lock)
	at java.lang.Object.wait(Object.java:503)
	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
	- locked <0x296f2970> (a java.lang.ref.Reference$Lock)

   Locked ownable synchronizers:
	- None

"VM Thread" prio=10 tid=0x020ff400 nid=0x1fd4 runnable 

"VM Periodic Task Thread" prio=10 tid=0x0212d800 nid=0x820 waiting on condition 

JNI global references: 162

Comment 7 Ryan Daley

2011-09-16 16:49:42 EDT

Created attachment 203529 [details]
screenshot of cpu analysis

Comment 8 Ryan Daley

2011-09-16 16:50:19 EDT

Created attachment 203530 [details]
select CPU hotspot

Comment 9 Ryan Daley

2011-09-16 16:51:06 EDT

Created attachment 203531 [details]
select CPU backtrace

Comment 10 Greg Wilkins

2011-09-20 01:07:16 EDT

Ryan,

Any chance you can try 7.5.2-SNAPSHOT.  This has some extra debugging code in it to try to detect endpoints that are being dispatched by the selector, but are making no progress.

The other thing that would be useful, is to run with JMX enabled and to call the dumpToStdErr method via jconsole using the mbean of the server.  This produces detailed information about the state of the selector and endpoints.

Comment 11 Greg Wilkins

2011-10-02 23:53:50 EDT

We were able to reproduce 100% CPU with SSL on an accessible server.   Investigation confirmed that there were problems associated with closing the SSL connection and the tweaks we had made in 7.5.2-SNAPSHOT to date had not helped.

Thus we have considerably refactored the SSL select channel endpoint to have the close handling use a common async process as all other handshake conversations.   This appears to have been successful in avoiding the 100% CPU on the accessible server.

This work was done in the sslClose branch which is now merged into master and will shortly be pushed.

It would be great if you could test this branch (or soon to be master) in the next few days to be sure that the upcoming 7.5.2 addresses your issue.

Comment 12 Greg Wilkins

2011-12-29 17:52:18 EST

A partial fix was released in 7.5.4
A full fix is in 7.6.0.RC2

Comment 13 SRIRAM M

2014-05-18 06:15:22 EDT

Hi Greg,
I am using Jetty 7.0.1 and I need a backport for this fix.
Is that possible?
Could you atleast please give a fix description/ files functions modified and the fix so that I could do the backport myself?
Thanks,
Sriram

Comment 14 SRIRAM M

2014-05-18 06:15:42 EDT

Hi Greg,
I am using Jetty 7.0.1 and I need a backport for this fix.
Is that possible?
Could you atleast please give a fix description/ files functions modified and the fix so that I could do the backport myself?
Thanks,
Sriram