Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 328967

Summary: Reduce memory consumption for API Use Scan consumer report conversion ant task
Product: [Eclipse Project] PDE Reporter: Ankur Sharma <ankur_sharma>
Component: API ToolsAssignee: PDE API Tools Inbox <pde-apitools-inbox>
Status: VERIFIED FIXED QA Contact:
Severity: normal    
Priority: P3 CC: curtis.windatt.public, darin.eclipse, Michael_Rennie
Version: 3.7Flags: curtis.windatt.public: review? (ankur_sharma)
Target Milestone: 3.7 M4   
Hardware: PC   
OS: Windows XP   
Whiteboard:
Attachments:
Description Flags
Fix none

Description Ankur Sharma CLA 2010-10-28 15:16:09 EDT 
The apitooling.apiconsumer_reportconversion kept running out of memory while converting a full use scan of Eclipse 3.6. I tried even with a filter but same result.

The task generated 120MB of (78) html files before running out of memory. The heap settings were default with a free RAM of over a GB.


Following is the most relevant exception trace I could find


eclipse.buildId=I20101026-2000
java.fullversion=J2RE 1.5.0 IBM J9 2.3 Windows XP x86-32 j9vmwi3223ifx-20080811 (JIT enabled)
J9VM - 20080809_21892_lHdSMr
JIT  - 20080620_1845_r8
GC   - 200806_19
BootLoader constants: OS=win32, ARCH=x86, WS=win32, NL=en_US
Command-line arguments:  -os win32 -ws win32 -arch x86


Error
Thu Oct 28 17:59:46 IST 2010
Failure of Background Ant Build

org.eclipse.core.runtime.CoreException: C:\PDE\workspaces\workspace.new3\test\anttask.xml:45: java.lang.OutOfMemoryError
at org.eclipse.ant.core.AntRunner.handleInvocationTargetException(AntRunner.java:452)
at org.eclipse.ant.core.AntRunner.run(AntRunner.java:384)
at org.eclipse.ant.internal.launching.launchConfigurations.AntLaunchDelegate$1.run(AntLaunchDelegate.java:298)
at java.lang.Thread.run(Thread.java:810)
Caused by: C:\PDE\workspaces\workspace.new3\test\anttask.xml:45: java.lang.OutOfMemoryError
at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:116)
at org.apache.tools.ant.Task.perform(Task.java:348)
at org.apache.tools.ant.Target.execute(Target.java:357)
at org.apache.tools.ant.Target.performTasks(Target.java:385)
at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1337)
at org.apache.tools.ant.Project.executeTarget(Project.java:1306)
at org.apache.tools.ant.helper.DefaultExecutor.executeTargets(DefaultExecutor.java:41)
at org.eclipse.ant.internal.core.ant.EclipseDefaultExecutor.executeTargets(EclipseDefaultExecutor.java:32)
at org.apache.tools.ant.Project.executeTargets(Project.java:1189)
at org.eclipse.ant.internal.core.ant.InternalAntRunner.run(InternalAntRunner.java:662)
at org.eclipse.ant.internal.core.ant.InternalAntRunner.run(InternalAntRunner.java:495)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:618)
at org.eclipse.ant.core.AntRunner.run(AntRunner.java:378)
... 2 more
Caused by: java.lang.OutOfMemoryError
at java.lang.String.<init>(String.java:295)
at org.apache.xerces.xni.XMLString.toString(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanAttribute(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source)
at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source)
at org.apache.xerces.parsers.XMLParser.parse(Unknown Source)
at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source)
at org.apache.xerces.jaxp.SAXParserImpl.parse(Unknown Source)
at javax.xml.parsers.SAXParser.parse(Unknown Source)
at org.eclipse.pde.api.tools.internal.search.UseScanParser.parse(UseScanParser.java:246)
at org.eclipse.pde.api.tools.internal.search.ConsumerReportConvertor.parse(ConsumerReportConvertor.java:467)
at org.eclipse.pde.api.tools.internal.search.UseReportConverter.convert(UseReportConverter.java:756)
at org.eclipse.pde.api.tools.internal.tasks.ApiConsumerUseReportConversionTask.execute(ApiConsumerUseReportConversionTask.java:118)
at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:288)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:79)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:618)
at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
... 17 more
Comment 1 Curtis Windatt CLA 2010-11-16 15:25:10 EST 
This is most likely caused by the UseScanParser's poor choice of opening every xml file even if the visitor says not to open any children (bug 328179).
Comment 2 Curtis Windatt CLA 2010-11-17 17:17:13 EST 
(In reply to comment #1)
> This is most likely caused by the UseScanParser's poor choice of opening every
> xml file even if the visitor says not to open any children (bug 328179).

I have a patch in hand that avoids opening the xml file.  While this definitely improves the performance of the conversion, the task still uses significantly more memory than the standard use scan converter.  It would be worthwhile to profile the operation to see what areas are best to target.
Comment 3 Curtis Windatt CLA 2010-11-18 14:07:26 EST 
Created attachment 183411 [details]
Fix

I discovered that the entire tree of references was being stored and could not be GC'd between the visitors being run.  Solution was to move the location where the producers were listed to inside the visitor where they are cleared after that visitor's consumer report is written.
Comment 4 Curtis Windatt CLA 2010-11-18 14:09:48 EST 
The memory usage/time to complete for a 300k reference use scan (280Mb of HTML) is very similar what the standard use scan converter took for the same scan.
Comment 5 Curtis Windatt CLA 2010-11-19 11:31:21 EST 
Applied the fix to HEAD.
Comment 6 Curtis Windatt CLA 2010-11-19 14:47:08 EST 
Ankur, please verify.
Comment 7 Ankur Sharma CLA 2010-12-07 07:22:07 EST 
Verified in N20101204-2000