Community
Participate
Working Groups
The XML based loader is consuming considerable cycles and is slow for both Offline and Online processing. A CSV based trace fragment or a binary format is needed for the TraceModel
We already have an enhancement request to provide -additional- event fragment format support such as binary. I am guessing you are opening this and a few other requests as a result of the JVMTI meetings. We can reconcile these once we have shut down 4.2 and are reviewing the 4.3 plan. The key for this enhancement is resources to develop and maintain the code. BTW, we need the XML fragments for several use cases and will not remove this support. Also concrete numbers are need to quantify the differences between the alternatives.
That's correct Harm, the enhancement is following the JVMTI meeting to improve the scalability and improve improve the performance of the model. We have an action item to call out the performance numbers for the need. In the short term an additional bugzilla for enhancement to improve the model performance and not having an inmemory model will get us closer to the needed performance
*** Bug 147659 has been marked as a duplicate of this bug. ***
So, we already have the infrastructure to go to a paged model, and we already have support for the log model using an RDB that shows the capabilites as we evolve them. The bigger design work is to make the UI present a query and analysis style rather that an expose the data style. Paging is only part of the changes.
(In reply to comment #2) > That's correct Harm, the enhancement is following the JVMTI meeting to improve > the scalability and improve improve the performance of the model. > We have an action item to call out the performance numbers for the need. > In the short term an additional bugzilla for enhancement to improve the model > performance and not having an inmemory model will get us closer to the needed > performance > I agree on the XML event processing speed and we already had a few enhancements open on this topic, I see this one as a dup of bug 97886 and bug 93452. See also bug 107521. I would say that the TPTP model (and related UI/analysis/reporting components) is performant, but not scalable to large data sets, actually works pretty well for small to medium scenarios (which most of the time could be achieved through intelligent raw data filtering, import time filtering speed is by the XML ). You need to be aware that moving the current model out of memory (through query and paging mechanism) will not gain performance in the current supported scenarios, but will allow better scalability. Making the event structure smaller and fast parsable improves only one side of the problem, the generation and event parsing performance, it would not provide any scalability benefits other then possible aggregated events and corresponding aggregated model structures. When we introduced filtering in the Trace views (TPTP 4.0) I made sure that we use the same (generic) query service (model and engine) as in the Log views (and later in some Test and Symptom views), so some of the Trace views are already query based but not UI pageble, they still load the whole query result set in the UI instead of paging through it like Log view does or using SWT virtual tree/table support (like Test will do in 4.4) to avoid the creation of more UI items than the visible ones. The same query can be executed (tested at least in Log resources case) against resources from both persistence mechanisms XMI or RDB. The new DMS designed as part of bug 169353, is intended to be the long term strategic mechanism (which could be shorted if we decide to pull all the required parties to focus on achieving that goal) to make the model scalable and achieve the required performance. The new DMS infrastructure will require extensive changes to our current code base, it basically changes the programming model by enforcing a more careful and controllable data handling behavior. Beside of code change impact we also need to maintain backward compatibility with previous UI, model, analysis and reporting services, unless we do a total split (between old model and new DMS and don't support mix-in use cases) and we rewrite them on top of the new DMS. Because of the new DMS high impact, in the mean time I've been working on enabling the Large Resource Support (TPTP EMF to RDB mapping framework used in Log case) for Test model (as Test was in the scalability focus for 4.4) and I also enabled it for Statistical and Trace models. As this behavior will not be part of 4.4 GA, after 4.4 we should do a feasibility study for both methods (current RDB based mechanism and new DMS) and decide what approach to use for TPTP 4.5. If for TPTP 4.5 we decide to enable the current RDB support for remaining models (including paging lists), beside of the II/analysis/reporting changes we also need to re factor the loaders to make them scalable (as of today we keep growing in-memory helper structures for the event load process), beside of alternate loading methods like those engaged in CBEtoCSVOutputter or new batch load events.
"import time filtering speed is by the XML" means "import with filtering speed is mostly affected by the XML parsing speed"
suggest promoting to 4.5 for discussion. relates to 196713.
To get an Enhancement considered as a candidate for a release we set the version as the requested target. Once approved for the plan of a gfiven release, we set the milestone target. I think you were after getting this on the list of candidates for 4.5. We also need to be clear to separate adding new event formats and adding alternate storage mechanisms. Both can impact the performance and scale, but both can be done somewhat separately. Generally we have not referred to the events as the trace model. The term trace model most often is the EMF model we use under the UI and currently for serialization. The particular request is about the events.
yes, this was my intent: to get version number set to 4.5 so it shows up in the voting query. Since I'm not a committer or the bugzilla filer, I couldn't change the version to 4.5 myself. I think there are two different clusters of potential work in this area. One relates to streamlining the data event format and its storage (e.g., in a trace file). The other relates to actually evolving the model (e.g., use of EMF). It sounds like you suggest that what I consider the first area to actually be two different areas. 1a) Alternate encoding/decoding of existing event format (i.e., same fields and field ranges but with say binary instead of XML) 1b) Event structure changes (e.g., removing unused fields, changing data ranges, etc). Is this what you mean? Thanks, Chris
As of TPTP 4.6.0, TPTP is in maintenance mode and focusing on improving quality by resolving relevant defects and increasing test coverage through test creation, automation, Build Verification Tests (BVTs), and expanded run-time execution. As such, TPTP is not delivering enhancements. As part of the TPTP Bugzilla housecleaning process (see http://wiki.eclipse.org/Bugzilla_Housecleaning_Processes), this enhancement is resolved as WONTFIX. For this enhancement to be considered, please re-open with an attached patch including the Description Document (see http://www.eclipse.org/tptp/home/documents/process/development/description_documents.html), code (see http://www.eclipse.org/tptp/home/documents/resources/TPTPDevGuide.htm), and test cases (see http://www.eclipse.org/tptp/home/documents/process/TPTP_Testing_Strategy.html).
As of TPTP 4.6.0, TPTP is in maintenance mode and focusing on improving quality by resolving relevant enhancements/defects and increasing test coverage through test creation, automation, Build Verification Tests (BVTs), and expanded run-time execution. As part of the TPTP Bugzilla housecleaning process (see http://wiki.eclipse.org/Bugzilla_Housecleaning_Processes), this enhancement/defect is verified/closed by the Project Lead since this enhancement/defect has been resolved and unverified for more than 1 year and considered to be fixed. If this enhancement/defect is still unresolved and reproducible in the latest TPTP release (http://www.eclipse.org/tptp/home/downloads/), please re-open.
