Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 365170

Summary: /api/projects?query=foobar results in OutOfMemory error
Product: z_Archived Reporter: Michael Ochmann <michael.ochmann>
Component: SkalliAssignee: Michael Ochmann <michael.ochmann>
Status: RESOLVED FIXED QA Contact:
Severity: major    
Priority: P3    
Version: unspecified   
Target Milestone: ---   
Hardware: All   
OS: All   
Whiteboard:
Attachments:
Description Flags
proposed patch
none
proposed patch (fixed invalid schema) none

Description Michael Ochmann CLA 2011-11-30 04:14:51 EST 
reproducible: always

java.lang.OutOfMemoryError: Requested array size exceeds VM limit (failed to allocate 17179869200 bytes) (array length 2147483647)|
  at org.apache.lucene.util.PriorityQueue.initialize(PriorityQueue.java:105)|
  at org.apache.lucene.search.HitQueue.<init>(HitQueue.java:67)|
  at org.apache.lucene.search.TopScoreDocCollector.<init>(TopScoreDocCollector.java:126)|
  at org.apache.lucene.search.TopScoreDocCollector.<init>(TopScoreDocCollector.java:37)|
  at org.apache.lucene.search.TopScoreDocCollector$OutOfOrderTopScoreDocCollector.<init>(TopScoreDocCollector.java:74)|
  at org.apache.lucene.search.TopScoreDocCollector$OutOfOrderTopScoreDocCollector.<init>(TopScoreDocCollector.java:72)|
  at org.apache.lucene.search.TopScoreDocCollector.create(TopScoreDocCollector.java:115)|
  at org.eclipse.skalli.core.internal.search.LuceneIndex.search(LuceneIndex.java:341)|
  at org.eclipse.skalli.core.internal.search.LuceneIndex.search(LuceneIndex.java:307)|
  at org.eclipse.skalli.core.internal.search.SearchServiceImpl.findProjectsByQuery(SearchServiceImpl.java:94)|
  at org.eclipse.skalli.api.rest.internal.resources.ProjectsResource.retrieve(ProjectsResource.java:48)|
…
Server process shutting down with exit code [666]; memory allocation error [OutOfMemoryError]|
Comment 1 Michael Ochmann CLA 2011-11-30 04:21:15 EST 
Lucene seems not prepared to accept count=Integer.MAX_VALUE for search queries in org.eclipse.skalli.api.rest.internal.resources.ProjectsResource
Comment 2 Michael Ochmann CLA 2011-11-30 08:03:33 EST 
https://issues.apache.org/jira/browse/LUCENE-504

org.apache.lucene.util.PriorityQueue allocates an Object array with maxSize entries in advance even if the result set is much smaller than maxSize. Incredible waste of memory, but the way TopScoreDocCollector is implemented, it seems to be inevitable.

I'm going to add the "start" and "count" query parms for /api/projects?query=foobar so that one can retrieve search hits "page by page" like in the UI.

Furthermore, I will limit "count" to the overall number of projects since it is impossible to have more search hits.
Comment 3 Michael Ochmann CLA 2011-11-30 08:56:49 EST 
Created attachment 207720 [details]
proposed patch
Comment 4 Michael Ochmann CLA 2011-12-01 03:11:45 EST 
Created attachment 207777 [details]
proposed patch (fixed invalid schema)