Bug 108646

Description amehrega 2005-09-01 19:16:52 EDT

This enhancement is opened in response to a message posted on the newsgroup:
-----------------------------------------------------
“More strongly, I would argue that the current TPTP profiling method that 
collects complete execution history is useless for the following 
reason:  Collecting data about each method call in a new data record 
each time increases the runtime of the profiled application by at least 
a factor of 5.  This means that when your application runs under the 
profiler, over 80% of the running time is not application runtime, but 
profiling overhead.  This means that your profiling data essentially 
profiles the profiler itself and not the application to be profiled.

The profiling overhead is essentially proportional to the number of 
profiled method calls.  This is not at all proportional to the true 
running time of the methods; especially when the methods cause external 
communication like database access.

 From my experience (over two years profiling and optimizing 
applications in C++ and Java), I would say that the profiling overhead 
must never exceed 100% to obtain a meaningful profile, i. e. the runtime 
under the profiler must not be more than 2x the runtime without 
profiler. Overhead of 10% to 50% is even better, as then the actual 
running time dominates the profiler overhead. Collecting aggreate 
statistics helps a lot, because per profiled method call only a few 
system calls plus a few arithmetic operations are needed, rather than 
the generation and storage of a new data record for the complete 
execution history (my guess is that this can reduce the profiling 
overhead by a factor of 10 to 100, thus making it much easier to reach 
the goal of low profiling overhead compared to the normal running time 
of the application).

I have found the Hyades profiler 3.0.0 useless for this reason, and 
currently work with the Eclipse Profiler 
(http://eclipsecolorer.sourceforge.net/index_profiler.html), which does 
exactly the aggregate data collection that Pratt describes, which works 
very well for me too.

Note that there are several bug reports related to the extraordinary 
memory overhead of complete execution history, like bugs 56645, 75266, 
and 88917, and the following trivial example numbers make it clear why 
approaches that improve only linearly, like improved storage format or 
database storage of execution history will not solve the problem:  When 
method A is called 1000 times and calls method B 1000 times each, the 
complete execution history will contain over a million call records (and 
these numbers are still small!).  The aggregate statistics only show 2 
records, namely

    Method A:      1000 calls, total real time: ... total CPU time: ...
    Method B: 1000000 calls, total real time: ... total CPU time: ...

There is no way the million-fold extra data storage overhead of the 
complete execution history can be compensated for by improved data 
storage technology; the only viable solution to reduce both storage and 
runtime overhead of profiling is to collect aggregate statistics.

At the moment, I can only recommend the Eclipse Profiler for serious 
performance work, although it seems to be unmaintained and can cause 
deadlocks on application server startup occasionally.

Regards,

Oliver Schoett”

--------------------------------------------------------------

This is a convincing argument.  There are many users that are not satisfied of
TPTP’s profiler because of its scalability issues on the workbench side.  The
loaders simply take a lot of time to process the events that are received from
the Java agent profiler.  Collecting aggregate statistics can significantly
reduce the amount of memory and processing that is performed on the client side.
 I propose creating a new profiling set called “Aggregate Result” that will only
increment counters based on the events that it receives from the workbench.  The
data model and the loaders will significantly be lighter than what is currently
present in TPTP.  This will also help in improving the scalability of the product.