Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 578211

Summary: OOM in HashtableOfObject: Requested array size exceeds VM limit
Product: [Eclipse Project] JDT Reporter: Andrey Loskutov <loskutov>
Component: CoreAssignee: Andrey Loskutov <loskutov>
Status: VERIFIED FIXED QA Contact:
Severity: normal    
Priority: P3 CC: jkubitz-eclipse, sravankumarl
Version: 4.12   
Target Milestone: 4.23 M2   
Hardware: PC   
OS: All   
See Also: https://bugs.eclipse.org/bugs/show_bug.cgi?id=474323
https://git.eclipse.org/r/c/jdt/eclipse.jdt.core/+/189648
https://git.eclipse.org/c/jdt/eclipse.jdt.core.git/commit/?id=d041f3803d811a0e9790d305639e77e557feceab
https://bugs.eclipse.org/bugs/show_bug.cgi?id=569069
Whiteboard:

Description Andrey Loskutov CLA 2022-01-14 04:33:31 EST 
Since yesterday I constantly see following error in my main SDK workspace.
I can rebuild index, restart, do some work and it will appear again, tried multiple times. I do not see related changes, so I assume the problem is not a recent regression (see bug 474323), but might be related to the increased index size (I've added one or two additional JDK installations, might be that is it).

Two things:
1) I can't see which file causes it, I can't check this file
2) May be a trivial size check could be added?

java.lang.OutOfMemoryError: Requested array size exceeds VM limit
	at org.eclipse.jdt.internal.compiler.util.HashtableOfObject.<init>(HashtableOfObject.java:41)
	at org.eclipse.jdt.internal.core.index.DiskIndex.readCategoryTable(DiskIndex.java:679)
	at org.eclipse.jdt.internal.core.index.DiskIndex.mergeCategory(DiskIndex.java:483)
	at org.eclipse.jdt.internal.core.index.DiskIndex.mergeCategories(DiskIndex.java:475)
	at org.eclipse.jdt.internal.core.index.DiskIndex.mergeWith(DiskIndex.java:577)
	at org.eclipse.jdt.internal.core.index.Index.save(Index.java:225)
	at org.eclipse.jdt.internal.core.search.indexing.IndexManager.saveIndex(IndexManager.java:1139)
	at org.eclipse.jdt.internal.core.search.indexing.IndexManager.saveIndexes(IndexManager.java:1192)
	at org.eclipse.jdt.internal.core.search.indexing.IndexManager.notifyIdle(IndexManager.java:818)
	at org.eclipse.jdt.internal.core.search.processing.JobManager.run(JobManager.java:422)
	at java.base/java.lang.Thread.run(Thread.java:834)
Comment 1 Eclipse Genie CLA 2022-01-14 10:16:13 EST 
New Gerrit change created: https://git.eclipse.org/r/c/jdt/eclipse.jdt.core/+/189648
Comment 3 Andrey Loskutov CLA 2022-01-17 06:59:32 EST 
(In reply to Andrey Loskutov from comment #0)
> Two things:
> 1) I can't see which file causes it, I can't check this file

Error handling / logging improved

> 2) May be a trivial size check could be added?

Size checks / calculation of the next table size improved / tests added documenting current expectations / limitations.

Ideally we should simply throw away HashtableOfObject (& Co) and replace it with HashMap, similar to the work done in bug 569069.
Comment 4 Andrey Loskutov CLA 2022-01-17 07:01:48 EST 
*** Bug 474323 has been marked as a duplicate of this bug. ***
Comment 5 Jörg Kubitz CLA 2022-01-17 11:04:52 EST 
> java.lang.OutOfMemoryError: Requested array size exceeds VM limit

Such error can happen before Integer.MAX_VALUE-2. 

see:
java.io.InputStream.MAX_BUFFER_SIZE,
java.util.Hashtable.MAX_ARRAY_SIZE
    /**
     * The maximum size of array to allocate.
     * Some VMs reserve some header words in an array.
     * Attempts to allocate larger arrays may result in
     * OutOfMemoryError: Requested array size exceeds VM limit
     */
    private static final int MAX_BUFFER_SIZE = Integer.MAX_VALUE - 8;

also 
 int extraRoom = (int) (size * 1.75f);
could already overflow.

however i wonder how that many objects are stored in the map. Even if only half of maximum number of objects caused that error. 1_000_000_000 objects sounds implausible many. that file would have to be gigantic. To me it sound more like the saved file was corrupted so that the wrong amount of objects was tried to read.
Comment 6 Sravan Kumar Lakkimsetti CLA 2022-02-16 03:53:30 EST 
Verified in 
Eclipse SDK
Version: 2022-03 (4.23)
Build id: I20220215-1800
OS: Windows 10, v.10.0, x86_64 / win32
Java vendor: Eclipse Adoptium
Java runtime version: 17.0.1+12
Java version: 17.0.1