Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 573239

Summary: [performance] improve Util.getInputStreamAsByteArray
Product: [Eclipse Project] JDT Reporter: Jörg Kubitz <jkubitz-eclipse>
Component: CoreAssignee: Jörg Kubitz <jkubitz-eclipse>
Status: VERIFIED FIXED QA Contact:
Severity: enhancement    
Priority: P3 CC: jarthana, loskutov, manoj.palat
Version: 4.20   
Target Milestone: 4.21 M1   
Hardware: All   
OS: All   
See Also: https://git.eclipse.org/r/c/jdt/eclipse.jdt.core/+/179974
https://git.eclipse.org/c/jdt/eclipse.jdt.core.git/commit/?id=574cc2ed536eaea4089c76940d123e90ce2894b0
https://git.eclipse.org/r/c/jdt/eclipse.jdt.core/+/182165
https://git.eclipse.org/c/jdt/eclipse.jdt.core.git/commit/?id=2b7c3edd9892af020cb0925039813d7383aacb63
Whiteboard:
Bug Depends on:    
Bug Blocks: 572372    
Attachments:
Description Flags
source of jmh benchmark: ReadAllBytes.java none

Description Jörg Kubitz CLA 2021-04-29 06:26:02 EDT
Created attachment 286264 [details]
source of jmh benchmark: ReadAllBytes.java

org.eclipse.jdt.internal.compiler.util.Util.getInputStreamAsByteArray
is mainly used for reading files from the local file system.
JDT never uses the length parameter for anything but providing the full length of the file stream.
There are two different implementations whether the length is known in advance or not (length=-1). When the length of file is given in advance a buffered copy is avoided.
If the input stream is a FileInputStream (for example .class file) the file size can be read with ((FileInputStream) input).getChannel().size(). This is cheap because it uses the already open filedescriptor.

Also: If the file size is not known in advance the array resize after each read can be avoided by a final concatenation of a list of buffers (idea taken from https://bugs.openjdk.java.net/browse/JDK-8193832).

Microbenchmark times ±2 in us/getInputStreamAsByteArray (less is better) on Windows:

  (size)  eclipseM1  eclipseWithLength  suggested  suggestedConcatenation  
       0     31,538             34,855    +20,930                  30,676
    1000     37,689             42,598    +27,413                  33,417
    5000     38,122             44,942    +29,285                  34,005
   10000     40,275             44,542    +30,652                  38,168
   50000     56,150             50,910    +36,023                  44,380
  100000     74,332             59,713    +43,665                  51,853
  500000    197,321            113,714    +98,603                 107,508

=>The suggested changes are significant faster throughout all file sizes.
Comment 1 Eclipse Genie CLA 2021-04-29 06:27:15 EDT
New Gerrit change created: https://git.eclipse.org/r/c/jdt/eclipse.jdt.core/+/179974
Comment 2 Andrey Loskutov CLA 2021-04-29 07:02:03 EDT
(In reply to Eclipse Genie from comment #1)
> New Gerrit change created:
> https://git.eclipse.org/r/c/jdt/eclipse.jdt.core/+/179974

Interesting, I will try to review / check on Linux soon, right now I run tests for bug 573161.

Jörg, how do you setup JMH Eclipse project / run JMH from inside Eclipse? I'm failed to setup JMH project in plain SDK so I use command line maven tools / plain text editor which is awkward.
Comment 3 Jörg Kubitz CLA 2021-04-29 07:40:42 EDT
> Interesting, I will try to review / check on Linux soon, right now I run
> tests for bug 573161.

great. Do not expect major speedup on a full build. This is mainly a optimization for a thread that is not on the criticical path (at least in my environment).

> Jörg, how do you setup JMH Eclipse project / run JMH from inside Eclipse?

good question! Was hard for me too. And i did not document the working steps :-(
As far as i remember i finally followed this guide: http://www.angelikalanger.com/Articles/EffectiveJava/90.Performance.JMH-Micro-Benchmark-Harness/90.Performance.JMH-Micro-Benchmark-Harness.appendix.html
and created a example project with a wizard. I had to manually fix the package structure in the autocreated example project though.

If it helps i could send you the full project or workspace.

Once installed i run the main method as normal (Run As .. Java application).  

> I'm failed to setup JMH project in plain SDK so I use command line maven
> tools / plain text editor which is awkward.

As an IDE petted developer i totally agree.
Comment 5 Manoj N Palat CLA 2021-06-11 02:33:13 EDT
Thanks Joerg for the patch and thanks Sravan and Andrey for the review. Resolving.
Comment 6 Andrey Loskutov CLA 2021-06-18 07:23:11 EDT
I've managed to get ~10 exception in the log in one shot during class path changes/rebuilds/code changes, all very similar, like below one, and the stack trace lead me to this bug.

The interruption of hover operations is "expected" (not that I like the way it is done), but I haven't seen so many errors before in such case, so I assume that some behavior could be changed now.

@Jörg: could you please check that?

eclipse.buildId=4.21.0.I20210617-1800
java.version=11.0.10
java.vendor=Red Hat, Inc.
BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US
Command-line arguments:  -data /data/4x_platform_workspace -os linux -ws gtk -arch x86_64

org.eclipse.jdt.core
Error
Fri Jun 18 13:16:01 CEST 2021
File not found: '/org.eclipse.jdt.core/compiler/org/eclipse/jdt/core/compiler/IProblem.java'

java.nio.channels.ClosedByInterruptException
	at java.base/java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:199)
	at java.base/sun.nio.ch.FileChannelImpl.endBlocking(FileChannelImpl.java:162)
	at java.base/sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:388)
	at org.eclipse.jdt.internal.compiler.util.Util.getInputStreamAsByteArray(Util.java:479)
	at org.eclipse.jdt.internal.compiler.util.Util.getInputStreamAsCharArray(Util.java:567)
	at org.eclipse.jdt.internal.core.util.Util.getResourceContentsAsCharArray(Util.java:1198)
	at org.eclipse.jdt.internal.core.CompilationUnit.getContents(CompilationUnit.java:680)
	at org.eclipse.jdt.internal.compiler.parser.SourceTypeConverter.getSource(SourceTypeConverter.java:708)
	at org.eclipse.jdt.internal.compiler.parser.SourceTypeConverter.convertAnnotations(SourceTypeConverter.java:678)
	at org.eclipse.jdt.internal.compiler.parser.SourceTypeConverter.convert(SourceTypeConverter.java:295)
	at org.eclipse.jdt.internal.compiler.parser.SourceTypeConverter.convert(SourceTypeConverter.java:613)
	at org.eclipse.jdt.internal.compiler.parser.SourceTypeConverter.convert(SourceTypeConverter.java:183)
	at org.eclipse.jdt.internal.compiler.parser.SourceTypeConverter.buildCompilationUnit(SourceTypeConverter.java:100)
	at org.eclipse.jdt.internal.codeassist.impl.Engine.accept(Engine.java:131)
	at org.eclipse.jdt.internal.codeassist.SelectionEngine.accept(SelectionEngine.java:372)
	at org.eclipse.jdt.internal.compiler.lookup.LookupEnvironment.askForType(LookupEnvironment.java:342)
	at org.eclipse.jdt.internal.compiler.lookup.PackageBinding.getTypeOrPackage(PackageBinding.java:276)
	at org.eclipse.jdt.internal.compiler.lookup.CompilationUnitScope.findImport(CompilationUnitScope.java:551)
	at org.eclipse.jdt.internal.compiler.lookup.CompilationUnitScope.findSingleImport(CompilationUnitScope.java:623)
	at org.eclipse.jdt.internal.compiler.lookup.CompilationUnitScope.faultInImports(CompilationUnitScope.java:452)
	at org.eclipse.jdt.internal.compiler.lookup.CompilationUnitScope.faultInTypes(CompilationUnitScope.java:525)
	at org.eclipse.jdt.internal.codeassist.SelectionEngine.select(SelectionEngine.java:1074)
	at org.eclipse.jdt.internal.core.Openable.codeSelect(Openable.java:167)
	at org.eclipse.jdt.internal.core.CompilationUnit.codeSelect(CompilationUnit.java:389)
	at org.eclipse.jdt.internal.core.CompilationUnit.codeSelect(CompilationUnit.java:382)
	at org.eclipse.jdt.internal.ui.text.java.hover.AbstractJavaEditorTextHover.getJavaElementsAt(AbstractJavaEditorTextHover.java:121)
	at org.eclipse.jdt.internal.ui.text.java.hover.JavadocHover.internalGetHoverInfo(JavadocHover.java:662)
	at org.eclipse.jdt.internal.ui.text.java.hover.JavadocHover.getHoverInfo2(JavadocHover.java:658)
	at org.eclipse.jdt.internal.ui.text.java.hover.BestMatchHover.getHoverInfo2(BestMatchHover.java:163)
	at org.eclipse.jdt.internal.ui.text.java.hover.BestMatchHover.getHoverInfo2(BestMatchHover.java:130)
	at org.eclipse.jdt.internal.ui.text.java.hover.JavaEditorTextHoverProxy.getHoverInfo2(JavaEditorTextHoverProxy.java:89)
	at org.eclipse.jface.text.TextViewerHoverManager$1.run(TextViewerHoverManager.java:155)
Comment 7 Jörg Kubitz CLA 2021-06-18 07:45:57 EDT
(In reply to Andrey Loskutov from comment #6)
> I've managed to get ~10 exception in the log in one shot during class path
> changes/rebuilds/code changes, all very similar, like below one, and the
> stack trace lead me to this bug.
> 
> The interruption of hover operations is "expected" (not that I like the way
> it is done), but I haven't seen so many errors before in such case, so I
> assume that some behavior could be changed now.
> 
> @Jörg: could you please check that?

I did not get such errors yet so its difficult to say. 
ClosedByInterruptException is an IOException and should have been transformed to JavaModelException at org.eclipse.jdt.internal.core.util.Util:1200
Did you get an JavaModelException too? 
The JavaModelException again should be catched in CompilationUnit.java:680. Both catches in your stacktrace. 
Please check if ClosedByInterruptException  is IOException in you jdk (which is it?).
Also please share where the interruption is done.

As change 182165 will remove my code anyway please reevaluate after submitting that. If possible also test it with JDK 17 preview (https://jdk.java.net/17/) then.
Comment 9 Jörg Kubitz CLA 2021-06-30 04:05:13 EDT
(In reply to Andrey Loskutov from comment #6)
> I've managed to get ~10 exception in the log in one shot during class path
> changes/rebuilds/code changes, all very similar, like below one, and the
> stack trace lead me to this bug.

@Andrey can you still reproduce it (with JDK 17?) otherwise lets close this
Comment 10 Andrey Loskutov CLA 2021-06-30 16:53:00 EDT
(In reply to Jörg Kubitz from comment #9)
> (In reply to Andrey Loskutov from comment #6)
> > I've managed to get ~10 exception in the log in one shot during class path
> > changes/rebuilds/code changes, all very similar, like below one, and the
> > stack trace lead me to this bug.
> 
> @Andrey can you still reproduce it (with JDK 17?) otherwise lets close this

Haven't used IDE for code writing much, but no, I don't see same errors anymore. Let's hope the last patch did it.
Comment 11 Jay Arthanareeswaran CLA 2021-07-08 01:30:30 EDT
Verified for 4.21 M1.