Community
Participate
Working Groups
We are using BIRT 2.5.2 to generate reports in a J2EE application. We have a report with nested tables which fails to generate when the number of records becomes too high (a few thousands records is enough to make it fails). The exception is "java.lang.OutOfMemoryError: unable to create new native thread". We noticed indeed that BIRT is creating a huge amount of threads when finalizing the report. This behaviour is due to the fact that each org.eclipse.birt.data.engine.executor.DataSource object registers a shutdown listener to the data engine. When the data engine initiates the shutdown at the end of the report, all listeners are notified and each of them will create a new thread to cleanup resources. With nested tables, since the number of datasources created can be high, this can lead to a situation where a very high number of threads is created (for an outer table with 300 records, containing a nested table with 10 records on average and containing again another nested table this means we'll have 3000 threads created at once). This is fatal for the JVM if there is not enough native memory to create all those threads. Please find below a simplified report demonstrating the problem + the stacktrace it generates + a fix suggestion. The suggested fix limits the number of created threads to 10. ##################################################### REPORT ##################################################### <?xml version="1.0" encoding="UTF-8"?> <report xmlns="http://www.eclipse.org/birt/2005/design" version="3.2.21" id="1"> <property name="comments">Copyright (c) 2006 <<Your Company Name here>></property> <property name="createdBy">Eclipse BIRT Designer Version 2.5.2.v20100208 Build <2.5.2.v20100210-0630></property> <property name="units">in</property> <property name="layoutPreference">auto layout</property> <data-sources> <script-data-source name="PojoDataSource" id="18"/> </data-sources> <data-sets> <script-data-set name="Message Details Data Set" id="19"> <property name="eventHandlerClass">com.xyz.reporting.test.cases.TestDS</property> <property name="newHandlerOnEachEvent">true</property> <structure name="cachedMetaData"/> <property name="dataSource">PojoDataSource</property> </script-data-set> <script-data-set name="Instances Data Set" id="827"> <property name="eventHandlerClass">com.xyz.reporting.test.cases.TestDS</property> <property name="newHandlerOnEachEvent">true</property> <structure name="cachedMetaData"/> <property name="dataSource">PojoDataSource</property> </script-data-set> <script-data-set name="Interventions Data Set" id="1224"> <property name="eventHandlerClass">com.xyz.reporting.test.cases.TestDS</property> <property name="newHandlerOnEachEvent">true</property> <structure name="cachedMetaData"/> <property name="dataSource">PojoDataSource</property> </script-data-set> </data-sets> <page-setup> <simple-master-page name="Search Details Master Page" id="2"> <property name="topMargin">1in</property> <property name="leftMargin">1.25in</property> <property name="bottomMargin">1in</property> <property name="rightMargin">1.25in</property> <page-footer> </page-footer> </simple-master-page> </page-setup> <body> <grid name="Message Details Section" id="2011"> <property name="width">100%</property> <column id="2012"/> <row id="2015"> <cell id="2016"> <label id="2017"> <text-property name="text">Messages</text-property> </label> </cell> </row> <row id="2013"> <cell id="2014"> <list name="Message Details" id="531"> <property name="dataSet">Message Details Data Set</property> <detail> <label id="121100"> <text-property name="text">Messages</text-property> </label> <grid name="Message" id="1756"> <property name="width">100%</property> <column id="1757"/> <row id="1758"> <cell id="1759"> <list name="Instances" id="826"> <property name="dataSet">Instances Data Set</property> <detail> <label id="12110"> <text-property name="text">Instances</text-property> </label> <list name="InterventionsSection" id="1203"> <detail> <label id="1211"> <text-property name="text">Interventions</text-property> </label> <list name="Interventions" id="1214"> <property name="dataSet">Interventions Data Set</property> <detail> <label id="999"> <text-property name="text">Test</text-property> </label> <!-- This makes it worse --> <!-- <list name="Intervention" id="1401"> <detail> </detail> </list> <list name="Appendix" id="1402"> <detail> </detail> </list> --> </detail> </list> </detail> </list> </detail> </list> </cell> </row> </grid> </detail> </list> </cell> </row> </grid> </body> </report> public class TestDS extends ScriptedDataSetEventAdapter { private int mRowcount; public TestDS() {} @Override public void open(IDataSetInstance pDataSet) throws ScriptException { mRowcount = 0; } @Override public boolean fetch(IDataSetInstance pDataSet, IUpdatableDataSetRow pRow) throws ScriptException { if (mRowcount++ > 40) { return false; } return true; } } ##################################################### STACKTRACE ##################################################### java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.$$YJP$$start0(Native Method) at java.lang.Thread.start0(Thread.java) at java.lang.Thread.start(Thread.java:597) at org.eclipse.birt.data.engine.executor.DataSource$ShutdownListener.dataEngineShutdown(DataSource.java:92) at org.eclipse.birt.data.engine.impl.DataEngineImpl.shutdown(DataEngineImpl.java:566) at org.eclipse.birt.report.data.adapter.impl.DataRequestSessionImpl.shutdown(DataRequestSessionImpl.java:455) at org.eclipse.birt.report.engine.data.dte.AbstractDataEngine.shutdown(AbstractDataEngine.java:348) at org.eclipse.birt.report.engine.executor.ExecutionContext.closeDataEngine(ExecutionContext.java:874) at org.eclipse.birt.report.engine.api.impl.RunAndRenderTask.doRun(RunAndRenderTask.java:175) at org.eclipse.birt.report.engine.api.impl.RunAndRenderTask.run(RunAndRenderTask.java:75) at com.xyz.report.server.birt.BIRTEngine.doRunAndRender(BIRTEngine.java:326) at com.xyz.report.server.birt.BIRTEngine.runReport(BIRTEngine.java:214) at com.xyz.report.server.birt.BIRTEngine.runReport(BIRTEngine.java:180) at com.xyz.reporting.test.cases.ReportingTestCase.runReport(ReportingTestCase.java:122) at com.xyz.reporting.test.cases.ReportingTestCase.testNestedDSReport(ReportingTestCase.java:222) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at junit.framework.TestCase.runTest(TestCase.java:164) at junit.framework.TestCase.runBare(TestCase.java:130) at junit.framework.TestResult$1.protect(TestResult.java:106) at junit.framework.TestResult.runProtected(TestResult.java:124) at junit.framework.TestResult.run(TestResult.java:109) at junit.framework.TestCase.run(TestCase.java:120) at org.eclipse.jdt.internal.junit.runner.junit3.JUnit3TestReference.run(JUnit3TestReference.java:130) at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390) at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197) ##################################################### SUGGESTED FIX ##################################################### package org.eclipse.birt.data.engine.impl; public class DataEngineImpl extends DataEngine { (...) public void shutdown( ) { logger.entering( "DataEngineImpl", "shutdown" ); if ( dataSources == null ) { // Already shutdown logger.fine( "The data engine has already been shutdown" ); return; } // Close all open data sources Collection col = dataSources.values( ); Iterator it = col.iterator( ); while ( it.hasNext( ) ) { DataSourceRuntime ds = (DataSourceRuntime) it.next( ); try { closeDataSource( ds ); } catch ( DataException e ) { if ( logger.isLoggable( Level.FINE ) ) logger.log( Level.FINE, "The data source (" + ds + ") fails to shut down", e ); } } this.dataSourceManager.close( ); releaseValidationContexts( ); if ( shutdownListenerList != null ) { //FIX notifyShutdownListeners(shutdownListenerList); //for ( int i = 0; i < shutdownListenerList.size( ); i++ ) //{ // ( (IShutdownListener) shutdownListenerList.get( i ) ).dataEngineShutdown( ); //} shutdownListenerList.clear( ); } logger.logp( Level.FINE, DataEngineImpl.class.getName( ), "shutdown", "Data engine shuts down" ); dataSetDesigns = null; dataSources = null; clearTempFile( ); logger.exiting( DataEngineImpl.class.getName( ), "shutdown" ); } /* Let's make sure we create only a fixed amount of threads. * (We could have used a ThreadPoolExecutor but we don't want * to change the way threads are created so we would need a * custom ThreadFactory and we would not save that much code). * */ private static final int MAX_THREAD_POOL_SIZE = 10; private void notifyShutdownListeners(List listeners) { final List list = new LinkedList(listeners); final int workerThreadCount = Math.min(list.size(), MAX_THREAD_POOL_SIZE); for (int i = 0; i < workerThreadCount; i++) { ThreadSecurity.createThread(new Runnable() { public void run() { while (true) { Object obj = null; synchronized (list) { if (!list.isEmpty()) { obj = list.remove(0); } else { break; } } process(obj); } } private void process(Object pObj) { try { ((IShutdownListener) pObj).dataEngineShutdown( ); } catch (Exception e) { //Let the thread process the remaining listeners e.printStackTrace(); } } }).start(); } } (...) } ##################################################### package org.eclipse.birt.data.engine.executor; (...) class DataSource implements IDataSource { public DataSource( String driverName, Map connProperties, DataEngineSession session ) { this.driverName = driverName; if ( connProperties != null ) this.connectionProps.putAll( connProperties ); this.session = session; this.session.getEngine( ).addShutdownListener( new ShutdownListener( session )); } private class ShutdownListener implements IShutdownListener { private DataEngineSession session; public ShutdownListener( DataEngineSession session ) { this.session = session; } public void dataEngineShutdown( ) { //FIX //Thread thread = ThreadSecurity.createThread( new ConnectionReleaser( this.session) ); //thread.start( ); new ConnectionReleaser( this.session).run(); } (...) }
Would you please try the latest 2.6.2 release build? This issue should have been fixed in the 2.6.x build already.
Set as fixed based on previous comments. Please feel free to reopen if you still have this issue in latest relese build.
Indeed, fixed in 2.6.2. Thanks for the support.
