Community
Participate
Working Groups
Created attachment 89016 [details] Input stream. Report generation is very slow with a larger input stream. Using the M-R1-2.3M4-200801090800 driver, generating a BIRT Test report with a larger input stream (see enlcosed XML) is very slow (appearing to hang the workbench) hangs. On Windows, this report geenration ran for over 9+ hours without completing. Interesitng, the same report generation completed on Linux, but still very slow (1+ hour). Running through the debugging and pausing the execution during the apparent hang renders the following stack trace on the main thread: org.eclipse.datatools.enablement.oda.xml.util.SaxParserConsumer(java.lang.Object).wait() line: 199 [local variables unavailable] org.eclipse.datatools.enablement.oda.xml.util.SaxParserConsumer.next() line: 475 org.eclipse.datatools.enablement.oda.xml.impl.ResultSet.next() line: 124 org.eclipse.datatools.connectivity.oda.consumer.helper.OdaResultSet.next() line: 177 org.eclipse.birt.data.engine.odaconsumer.ResultSet.fetch() line: 115 org.eclipse.birt.data.engine.executor.cache.OdiAdapter.fetch(org.eclipse.birt.data.engine.impl.StopSign) line: 145 org.eclipse.birt.data.engine.executor.cache.RowResultSet.next(org.eclipse.birt.data.engine.impl.StopSign) line: 93 org.eclipse.birt.data.engine.executor.cache.ExpandableRowResultSet.next(org.eclipse.birt.data.engine.impl.StopSign) line: 64 org.eclipse.birt.data.engine.executor.cache.SmartCacheHelper.populateData(org.eclipse.birt.data.engine.executor.cache.IRowResultSet, org.eclipse.birt.data.engine.odi.IResultClass, org.eclipse.birt.data.engine.executor.cache.SortSpec, org.eclipse.birt.data.engine.impl.StopSign) line: 299 org.eclipse.birt.data.engine.executor.cache.SmartCacheHelper.initInstance(org.eclipse.birt.data.engine.executor.cache.CacheRequest, org.eclipse.birt.data.engine.executor.cache.OdiAdapter, org.eclipse.birt.data.engine.odi.IResultClass, org.eclipse.birt.data.engine.impl.StopSign) line: 271 org.eclipse.birt.data.engine.executor.cache.SmartCacheHelper.initOdaResult(org.eclipse.birt.data.engine.executor.cache.CacheRequest, org.eclipse.birt.data.engine.odaconsumer.ResultSet, org.eclipse.birt.data.engine.odi.IResultClass, org.eclipse.birt.data.engine.impl.StopSign) line: 155 org.eclipse.birt.data.engine.executor.cache.SmartCacheHelper.getResultSetCache(org.eclipse.birt.data.engine.executor.cache.CacheRequest, org.eclipse.birt.data.engine.odaconsumer.ResultSet, org.eclipse.birt.data.engine.odi.IResultClass, org.eclipse.birt.data.engine.impl.StopSign) line: 79 org.eclipse.birt.data.engine.executor.cache.SmartCache.<init>(org.eclipse.birt.data.engine.executor.cache.CacheRequest, org.eclipse.birt.data.engine.odaconsumer.ResultSet, org.eclipse.birt.data.engine.odi.IResultClass, org.eclipse.birt.data.engine.impl.DataEngineSession, org.eclipse.birt.data.engine.impl.StopSign) line: 56 org.eclipse.birt.data.engine.executor.transform.pass.PassUtil.populateOdiResultSet(org.eclipse.birt.data.engine.executor.transform.ResultSetPopulator, org.eclipse.birt.data.engine.executor.transform.OdiResultSetWrapper, org.eclipse.birt.data.engine.executor.cache.SortSpec, org.eclipse.birt.data.engine.impl.StopSign) line: 99 org.eclipse.birt.data.engine.executor.transform.pass.PassUtil.pass(org.eclipse.birt.data.engine.executor.transform.ResultSetPopulator, org.eclipse.birt.data.engine.executor.transform.OdiResultSetWrapper, boolean, org.eclipse.birt.data.engine.impl.StopSign) line: 62 org.eclipse.birt.data.engine.executor.transform.pass.PassManager.doSinglePass(org.eclipse.birt.data.engine.executor.transform.OdiResultSetWrapper, org.eclipse.birt.data.engine.impl.StopSign) line: 186 org.eclipse.birt.data.engine.executor.transform.pass.PassManager.pass(org.eclipse.birt.data.engine.executor.transform.OdiResultSetWrapper, org.eclipse.birt.data.engine.impl.StopSign) line: 90 org.eclipse.birt.data.engine.executor.transform.pass.PassManager.populateResultSet(org.eclipse.birt.data.engine.executor.transform.ResultSetPopulator, org.eclipse.birt.data.engine.executor.transform.OdiResultSetWrapper, org.eclipse.birt.data.engine.impl.StopSign) line: 71 org.eclipse.birt.data.engine.executor.transform.ResultSetPopulator.populateResultSet(org.eclipse.birt.data.engine.executor.transform.OdiResultSetWrapper, org.eclipse.birt.data.engine.impl.StopSign) line: 196 org.eclipse.birt.data.engine.executor.transform.CachedResultSet.<init>(org.eclipse.birt.data.engine.executor.BaseQuery, org.eclipse.birt.data.engine.odi.IResultClass, org.eclipse.birt.data.engine.odaconsumer.ResultSet, org.eclipse.birt.data.engine.odi.IEventHandler, org.eclipse.birt.data.engine.impl.DataEngineSession, org.eclipse.birt.data.engine.impl.StopSign) line: 86 org.eclipse.birt.data.engine.executor.DataSourceQuery.execute(org.eclipse.birt.data.engine.odi.IEventHandler, org.eclipse.birt.data.engine.impl.StopSign) line: 511 org.eclipse.birt.data.engine.impl.PreparedOdaDSQuery$OdaDSQueryExecutor.executeOdiQuery(org.eclipse.birt.data.engine.odi.IEventHandler, org.eclipse.birt.data.engine.impl.StopSign) line: 348 org.eclipse.birt.data.engine.impl.PreparedOdaDSQuery$OdaDSQueryExecutor(org.eclipse.birt.data.engine.impl.QueryExecutor).execute(org.eclipse.birt.data.engine.odi.IEventHandler, org.eclipse.birt.data.engine.impl.StopSign) line: 782 org.eclipse.birt.data.engine.impl.ServiceForQueryResults.executeQuery(org.eclipse.birt.data.engine.impl.StopSign) line: 232 org.eclipse.birt.data.engine.impl.QueryResults.getResultIterator() line: 158 org.eclipse.birt.report.engine.data.dte.QueryResultSet.<init>(org.eclipse.birt.report.engine.data.IDataEngine, org.eclipse.birt.report.engine.executor.ExecutionContext, org.eclipse.birt.report.engine.extension.IBaseResultSet, org.eclipse.birt.data.engine.api.IQueryDefinition, org.eclipse.birt.data.engine.api.IQueryResults) line: 109 org.eclipse.birt.report.engine.data.dte.DteDataEngine.doExecuteQuery(org.eclipse.birt.report.engine.extension.IBaseResultSet, org.eclipse.birt.data.engine.api.IQueryDefinition, boolean) line: 137 org.eclipse.birt.report.engine.data.dte.DteDataEngine(org.eclipse.birt.report.engine.data.dte.AbstractDataEngine).execute(org.eclipse.birt.report.engine.extension.IBaseResultSet, org.eclipse.birt.data.engine.api.IDataQueryDefinition, boolean) line: 224 org.eclipse.birt.report.engine.executor.ExecutionContext.executeQuery(org.eclipse.birt.report.engine.extension.IBaseResultSet, org.eclipse.birt.data.engine.api.IDataQueryDefinition, boolean) line: 1712 org.eclipse.birt.report.engine.executor.TableItemExecutor(org.eclipse.birt.report.engine.executor.QueryItemExecutor).executeQuery() line: 77 org.eclipse.birt.report.engine.executor.TableItemExecutor.execute() line: 62 org.eclipse.birt.report.engine.internal.executor.dup.SuppressDuplicateItemExecutor.execute() line: 42 org.eclipse.birt.report.engine.internal.executor.l18n.LocalizedReportItemExecutor(org.eclipse.birt.report.engine.internal.executor.wrap.WrappedReportItemExecutor).execute() line: 45 org.eclipse.birt.report.engine.internal.executor.l18n.LocalizedReportItemExecutor.execute() line: 33 org.eclipse.birt.report.engine.layout.html.HTMLBlockStackingLM.layoutNodes() line: 63 org.eclipse.birt.report.engine.layout.html.HTMLBlockStackingLM(org.eclipse.birt.report.engine.layout.html.HTMLStackingLM).layoutChildren() line: 27 org.eclipse.birt.report.engine.layout.html.HTMLBlockStackingLM(org.eclipse.birt.report.engine.layout.html.HTMLAbstractLM).layout() line: 133 org.eclipse.birt.report.engine.layout.html.HTMLRowLM(org.eclipse.birt.report.engine.layout.html.HTMLInlineStackingLM).resumeLayout() line: 104 org.eclipse.birt.report.engine.layout.html.HTMLRowLM(org.eclipse.birt.report.engine.layout.html.HTMLInlineStackingLM).layoutNodes() line: 153 org.eclipse.birt.report.engine.layout.html.HTMLRowLM(org.eclipse.birt.report.engine.layout.html.HTMLStackingLM).layoutChildren() line: 27 org.eclipse.birt.report.engine.layout.html.HTMLRowLM(org.eclipse.birt.report.engine.layout.html.HTMLAbstractLM).layout() line: 133 org.eclipse.birt.report.engine.layout.html.HTMLTableBandLM(org.eclipse.birt.report.engine.layout.html.HTMLBlockStackingLM).layoutNodes() line: 68 org.eclipse.birt.report.engine.layout.html.HTMLTableBandLM(org.eclipse.birt.report.engine.layout.html.HTMLStackingLM).layoutChildren() line: 27 org.eclipse.birt.report.engine.layout.html.HTMLTableBandLM.layoutChildren() line: 65 org.eclipse.birt.report.engine.layout.html.HTMLTableBandLM(org.eclipse.birt.report.engine.layout.html.HTMLAbstractLM).layout() line: 133 org.eclipse.birt.report.engine.layout.html.HTMLTableLM(org.eclipse.birt.report.engine.layout.html.HTMLBlockStackingLM).layoutNodes() line: 68 org.eclipse.birt.report.engine.layout.html.HTMLTableLM(org.eclipse.birt.report.engine.layout.html.HTMLStackingLM).layoutChildren() line: 27 org.eclipse.birt.report.engine.layout.html.HTMLTableLM.layoutChildren() line: 76 org.eclipse.birt.report.engine.layout.html.HTMLTableLM(org.eclipse.birt.report.engine.layout.html.HTMLAbstractLM).layout() line: 133 org.eclipse.birt.report.engine.layout.html.HTMLBlockStackingLM.layoutNodes() line: 68 org.eclipse.birt.report.engine.layout.html.HTMLBlockStackingLM(org.eclipse.birt.report.engine.layout.html.HTMLStackingLM).layoutChildren() line: 27 org.eclipse.birt.report.engine.layout.html.HTMLBlockStackingLM(org.eclipse.birt.report.engine.layout.html.HTMLAbstractLM).layout() line: 133 org.eclipse.birt.report.engine.layout.html.HTMLRowLM(org.eclipse.birt.report.engine.layout.html.HTMLInlineStackingLM).resumeLayout() line: 104 org.eclipse.birt.report.engine.layout.html.HTMLRowLM(org.eclipse.birt.report.engine.layout.html.HTMLInlineStackingLM).layoutNodes() line: 153 org.eclipse.birt.report.engine.layout.html.HTMLRowLM(org.eclipse.birt.report.engine.layout.html.HTMLStackingLM).layoutChildren() line: 27 org.eclipse.birt.report.engine.layout.html.HTMLRowLM(org.eclipse.birt.report.engine.layout.html.HTMLAbstractLM).layout() line: 133 org.eclipse.birt.report.engine.layout.html.HTMLTableLM(org.eclipse.birt.report.engine.layout.html.HTMLBlockStackingLM).layoutNodes() line: 68 org.eclipse.birt.report.engine.layout.html.HTMLTableLM(org.eclipse.birt.report.engine.layout.html.HTMLStackingLM).layoutChildren() line: 27 org.eclipse.birt.report.engine.layout.html.HTMLTableLM.layoutChildren() line: 76 org.eclipse.birt.report.engine.layout.html.HTMLTableLM(org.eclipse.birt.report.engine.layout.html.HTMLAbstractLM).layout() line: 133 org.eclipse.birt.report.engine.layout.html.HTMLPageLM(org.eclipse.birt.report.engine.layout.html.HTMLBlockStackingLM).layoutNodes() line: 68 org.eclipse.birt.report.engine.layout.html.HTMLPageLM.layout() line: 85 org.eclipse.birt.report.engine.layout.html.HTMLReportLayoutEngine.layout(org.eclipse.birt.report.engine.executor.IReportExecutor, org.eclipse.birt.report.engine.content.IReportContent, org.eclipse.birt.report.engine.emitter.IContentEmitter, boolean) line: 106 org.eclipse.birt.report.engine.api.impl.RunAndRenderTask.doRun() line: 138 org.eclipse.birt.report.engine.api.impl.RunAndRenderTask.run() line: 66 org.eclipse.tptp.platform.report.birt.core.provisional.BIRTReportGenerator.generateHTMLReportFragment(java.io.OutputStream, java.lang.String, java.lang.String, java.io.InputStream, java.io.InputStream, boolean, org.eclipse.tptp.platform.report.birt.core.internal.impl.BIRTResourceLocator) line: 293 org.eclipse.tptp.platform.report.birt.core.provisional.BIRTReportGenerator.generateHTMLReport(java.io.OutputStream, java.io.InputStream, java.io.InputStream, boolean, java.lang.String, java.lang.String, org.eclipse.tptp.platform.report.birt.core.internal.impl.BIRTResourceLocator) line: 190 org.eclipse.tptp.test.report.birt.internal.elements.TabularTPFTestSuiteReportElement.generate() line: 263 This defect can be reproduced by using the following steps: 1) Download TPTP (http://www.eclipse.org/tptp/home/downloads/?buildId=TPTP-4.5.0-200802040400), EMF/BIRT/GMF/DTP, and the TPTP BIRT Test Reports (see the as-is section). 2) Extract all of the TPTP test plug-ins (see http://www.eclipse.org/tptp/home/documents/process/TPTP_Testing_Strategy.html): http://dev.eclipse.org/viewcvs/index.cgi/test-results/monitor/?root=TPTP_Project http://dev.eclipse.org/viewcvs/index.cgi/test-results/test/?root=TPTP_Project http://dev.eclipse.org/viewcvs/index.cgi/test-results/platform/?root=TPTP_Project http://dev.eclipse.org/viewcvs/index.cgi/test-results/trace/?root=TPTP_Project 3) Open the Test Perspective and multiple select the following files in the Test Navigator: http://dev.eclipse.org/viewcvs/index.cgi/test-results/platform/org.eclipse.hyades.tests/TP1/?root=TPTP_Project 4) Select Report, select HTML Tabular Test Report (BIRT), and specify a report window from Dec 10 - today (see http://help.eclipse.org/help33/topic/org.eclipse.tptp.test.report.birt.doc.user/tasks/treporttest.xhtml). 5) Click Finish and note how the apparent hang when generaitng the CF8BB642624A64605E98A7A0920711DA.html file in the outut directory.
The input stream XML file is 240kb, what is the reason it takes such a long time to parse? Or is the same XML data source parsed many many times because it is used as a inner loop sub query?
Paul says this is a new issue (regression?) in M5.
Hi Paul, Could you please see the attached doc. I cannot reproduce the issue under the environment you've described.Mmm...there is maybe some conflict caused the failing load tptp plugin file in my environment. So I had to try another tptp version(tptp.runtime-TPTP-4.4.1-200802061217), it works ok in my environment, the report generation ran for about 3+ minutes.Is there anything different between tptp.runtime-TPTP-4.4.1-200802061217 and TPTP-4.5.0-200802040400? Did this bug only occurs in the environment you described? If so, I cannot exclude the possibility there is something wrong in the TPTP side. So could you please use the tptp.runtime-TPTP-4.4.1-200802061217 for another try? Thanks Mingxia
Created attachment 89826 [details] The result under different environment
Paul, As we still have not get your response either here or by email, we would like to postpone this bug to M6 so that we can catch up with M5 release schedule. Thanks. Lin
(In reply to comment #3) > Hi Paul, > > Could you please see the attached doc. > > I cannot reproduce the issue under the environment you've described.Mmm...there > is maybe some conflict caused the failing load tptp plugin file in my > environment. So I had to try another tptp > version(tptp.runtime-TPTP-4.4.1-200802061217), it works ok in my environment, > the report generation ran for about 3+ minutes.Is there anything different > between tptp.runtime-TPTP-4.4.1-200802061217 and TPTP-4.5.0-200802040400? Did > this bug only occurs in the environment you described? If so, I cannot exclude > the possibility there is something wrong in the TPTP side. So could you please > use the tptp.runtime-TPTP-4.4.1-200802061217 for another try? > > > Thanks > Mingxia > I would assume that you are not downloading the TPTP BIRT Test Reports from the as-is section of the TPTP 4.5 download site (see step #1 from the above Description). I tried the latest BIRT 2.3/DTP 1.6 builds this afternoon and was blocked by https://bugs.eclipse.org/bugs/show_bug.cgi?id=218495#c11.
Created attachment 90159 [details] report in TPTP-4.5.0-200802040400
Created attachment 90160 [details] report in TPTP-4.4.1-200802061217
Hi Paul, The report design in tptp.runtime-TPTP-4.4.1-200802061217 and TPTP-4.5.0-200802040400 is totally different, see attached report file. Here we can see there is a newly added nested table under 'Defects' cell in the TPTP-4.5.0-200802040400 report, which will bring many times query execution. According to my coarse estimate, there is about 1500+ records in the outer table, to execute the nested query, it will bring 1500+ query execution. During execution,the xml file will be parsed 1500+ times, that is, we should handle about 460M xml source even the xml is only 240k(1500*240k). This is definitely time consuming. But in tptp.runtime-TPTP-4.4.1-200802061217, there is no nested query, so the query will be executed only once to get the result. Thanks Mingxia
(In reply to comment #9) > Hi Paul, > > The report design in tptp.runtime-TPTP-4.4.1-200802061217 and > TPTP-4.5.0-200802040400 is totally different, see attached report file. > > Here we can see there is a newly added nested table under 'Defects' cell in the > TPTP-4.5.0-200802040400 report, which will bring many times query execution. > According to my coarse estimate, there is about 1500+ records in the outer > table, to execute the nested query, it will bring 1500+ query execution. During > execution,the xml file will be parsed 1500+ times, that is, we should handle > about 460M xml source even the xml is only 240k(1500*240k). This is definitely > time consuming. > But in tptp.runtime-TPTP-4.4.1-200802061217, there is no nested query, so the > query will be executed only once to get the result. > > Thanks > Mingxia > Hi Mingxia, Yes, the Defects query was added in to TPTP 4.5 but we have been running this report template using the BIRT/DTP M4 driver since early January without the symptoms noted in this defect. Could BIRT or DTP have introduced a performance regression in handling nested tables? Can you suggest a better design for the report template to reduce the number of times the XML data source is parsed? Either way, this is blocked by 218578.
(In reply to comment #10) > Hi Mingxia, > Yes, the Defects query was added in to TPTP 4.5 but we have been running this > report template using the BIRT/DTP M4 driver since early January without the > symptoms noted in this defect. Could BIRT or DTP have introduced a performance > regression in handling nested tables? Can you suggest a better design for the > report template to reduce the number of times the XML data source is parsed? > Either way, this is blocked by 218578. Hi Paul, I just test it with M-R1-2.3M4-20071221 with DTPM4, it still has this performance issue. Could you please provide the build version which works fine if you are sure that this is a regression bug. As I explained before, the poor performance is just due to thousands of nested query excution. I recommend a better design for this report template FYI. Please see the attached, there I got rid of the nested table instead of using joint dataset. It works ok in my test environment. Thanks Mingxia
Created attachment 90451 [details] Recommended design file
Created attachment 90452 [details] Recommended design file
Comment on attachment 90451 [details] Recommended design file Sorry,this is not correct attachment!!
(In reply to comment #11) > Hi Paul, > > I just test it with M-R1-2.3M4-20071221 with DTPM4, it still has this > performance issue. Could you please provide the build version which works fine > if you are sure that this is a regression bug. > As I explained before, the poor performance is just due to thousands of nested > query excution. I recommend a better design for this report template FYI. > Please see the attached, there I got rid of the nested table instead of using > joint dataset. It works ok in my test environment. > > Thanks > Mingxia Thanks Mingxia. We were using http://www.eclipse.org/tptp/home/downloads/?buildId=TPTP-4.5.0M4-200712211700 and its dependencies without an issue. Using the following configuration, the problem is resolved: birt-report-framework-2_3_0-N20080222.zip dtp-sdk_1.6M6_20080222.zip Resolving as fixed. We will review and integrate the changes you made to the report template. Thanks for taking the time to provide this attachment.
Closing.
