Bug 168138

Description Joe Toomey 2006-12-14 17:50:15 EST

When migrating to the new Agent Controller, a noticable slowdown in test launch speed
was observed by many people, and was attributed to the compatibility layer on top of the
new AC.  More recently, as tests have been written to measure the stability, memory usage
and performance of test execution, I observed that test launches generally take around
20 seconds, but on occasion take only 5 seconds.  In runs of 200 tests, there was a clear
pattern where roughly 30% of the tests ran in 5 seconds, while the rest ran in roughly
20 seconds.

I have investigated the cause of this, and it turns out to be a race condition
in communication between the workbench and the runner's agent.  This race condition was
present with the old agent controller as well, but it appears that the performance characteristics
of certain commands with the new agent controller bias us much more heavily towards losing
the race and paying the performance penalty.

There is a 2-way handshake that takes
place when a test is launched.  The workbench launches the test via the AC and then waits
for the test process to become active.  There is no notification available for this, so
the workbench polls for the process.  On the other side, when the test process comes up,
the runner agent registers itself with the AC and then blocks, waiting to be signaled
by the workbench that it is ready for the test to start.

The race condition occurs
as a side effect of the NodeImpl.listProcesses() method, which despite its innocuous name,
does a lot more than list the processes from the node.  After querying the AC (sending
a command to list the processes from the node), it then queries again to list every agent
within each process, and yet again to get the details of every agent.  It is the last
of these queries that causes the race condition.  The query to list the agent's details
is sent as a command to the agent itself.  If the (runner) agent has already blocked waiting
to hear from the workbench that it is okay to start, then the command will not receive
a response because the agent is blocked.  This results in a (15 second) timeout waiting
for the details of the runner agent to be returned.

Interestingly, this does not cause
a failure in test launch, because we didn't need those details in the first place.