Community
Participate
Working Groups
The apitooling.apiconsumer_reportconversion kept running out of memory while converting a full use scan of Eclipse 3.6. I tried even with a filter but same result. The task generated 120MB of (78) html files before running out of memory. The heap settings were default with a free RAM of over a GB. Following is the most relevant exception trace I could find eclipse.buildId=I20101026-2000 java.fullversion=J2RE 1.5.0 IBM J9 2.3 Windows XP x86-32 j9vmwi3223ifx-20080811 (JIT enabled) J9VM - 20080809_21892_lHdSMr JIT - 20080620_1845_r8 GC - 200806_19 BootLoader constants: OS=win32, ARCH=x86, WS=win32, NL=en_US Command-line arguments: -os win32 -ws win32 -arch x86 Error Thu Oct 28 17:59:46 IST 2010 Failure of Background Ant Build org.eclipse.core.runtime.CoreException: C:\PDE\workspaces\workspace.new3\test\anttask.xml:45: java.lang.OutOfMemoryError at org.eclipse.ant.core.AntRunner.handleInvocationTargetException(AntRunner.java:452) at org.eclipse.ant.core.AntRunner.run(AntRunner.java:384) at org.eclipse.ant.internal.launching.launchConfigurations.AntLaunchDelegate$1.run(AntLaunchDelegate.java:298) at java.lang.Thread.run(Thread.java:810) Caused by: C:\PDE\workspaces\workspace.new3\test\anttask.xml:45: java.lang.OutOfMemoryError at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:116) at org.apache.tools.ant.Task.perform(Task.java:348) at org.apache.tools.ant.Target.execute(Target.java:357) at org.apache.tools.ant.Target.performTasks(Target.java:385) at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1337) at org.apache.tools.ant.Project.executeTarget(Project.java:1306) at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41) at org.eclipse.ant.internal.core.ant.EclipseDefaultExecutor.executeTargets(EclipseDefaultExecutor.java:32) at org.apache.tools.ant.Project.executeTargets(Project.java:1189) at org.eclipse.ant.internal.core.ant.InternalAntRunner.run(InternalAntRunner.java:662) at org.eclipse.ant.internal.core.ant.InternalAntRunner.run(InternalAntRunner.java:495) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:618) at org.eclipse.ant.core.AntRunner.run(AntRunner.java:378) ... 2 more Caused by: java.lang.OutOfMemoryError at java.lang.String.<init>(String.java:295) at org.apache.xerces.xni.XMLString.toString(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanAttribute(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source) at javax.xml.parsers.SAXParser.parse(Unknown Source) at org.eclipse.pde.api.tools.internal.search.UseScanParser.parse(UseScanParser.java:246) at org.eclipse.pde.api.tools.internal.search.ConsumerReportConvertor.parse(ConsumerReportConvertor.java:467) at org.eclipse.pde.api.tools.internal.search.UseReportConverter.convert(UseReportConverter.java:756) at org.eclipse.pde.api.tools.internal.tasks.ApiConsumerUseReportConversionTask.execute(ApiConsumerUseReportConversionTask.java:118) at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:618) at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106) ... 17 more
This is most likely caused by the UseScanParser's poor choice of opening every xml file even if the visitor says not to open any children (bug 328179).
(In reply to comment #1) > This is most likely caused by the UseScanParser's poor choice of opening every > xml file even if the visitor says not to open any children (bug 328179). I have a patch in hand that avoids opening the xml file. While this definitely improves the performance of the conversion, the task still uses significantly more memory than the standard use scan converter. It would be worthwhile to profile the operation to see what areas are best to target.
Created attachment 183411 [details] Fix I discovered that the entire tree of references was being stored and could not be GC'd between the visitors being run. Solution was to move the location where the producers were listed to inside the visitor where they are cleared after that visitor's consumer report is written.
The memory usage/time to complete for a 300k reference use scan (280Mb of HTML) is very similar what the standard use scan converter took for the same scan.
Applied the fix to HEAD.
Ankur, please verify.
Verified in N20101204-2000
