Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.
Bug 365170 - /api/projects?query=foobar results in OutOfMemory error
Summary: /api/projects?query=foobar results in OutOfMemory error
Status: RESOLVED FIXED
Alias: None
Product: z_Archived
Classification: Eclipse Foundation
Component: Skalli (show other bugs)
Version: unspecified   Edit
Hardware: All All
: P3 major (vote)
Target Milestone: ---   Edit
Assignee: Michael Ochmann CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-11-30 04:14 EST by Michael Ochmann CLA
Modified: 2022-10-03 10:28 EDT (History)
0 users

See Also:

Attachments
proposed patch (15.91 KB, patch)
2011-11-30 08:56 EST, Michael Ochmann CLA 		no flags Details | Diff
proposed patch (fixed invalid schema) (15.92 KB, patch)
2011-12-01 03:11 EST, Michael Ochmann CLA 		no flags Details | Diff
Show Obsolete (1) View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Michael Ochmann CLA 2011-11-30 04:14:51 EST 
reproducible: always

java.lang.OutOfMemoryError: Requested array size exceeds VM limit (failed to allocate 17179869200 bytes) (array length 2147483647)|
  at org.apache.lucene.util.PriorityQueue.initialize(PriorityQueue.java:105)|
  at org.apache.lucene.search.HitQueue.<init>(HitQueue.java:67)|
  at org.apache.lucene.search.TopScoreDocCollector.<init>(TopScoreDocCollector.java:126)|
  at org.apache.lucene.search.TopScoreDocCollector.<init>(TopScoreDocCollector.java:37)|
  at org.apache.lucene.search.TopScoreDocCollector$OutOfOrderTopScoreDocCollector.<init>(TopScoreDocCollector.java:74)|
  at org.apache.lucene.search.TopScoreDocCollector$OutOfOrderTopScoreDocCollector.<init>(TopScoreDocCollector.java:72)|
  at org.apache.lucene.search.TopScoreDocCollector.create(TopScoreDocCollector.java:115)|
  at org.eclipse.skalli.core.internal.search.LuceneIndex.search(LuceneIndex.java:341)|
  at org.eclipse.skalli.core.internal.search.LuceneIndex.search(LuceneIndex.java:307)|
  at org.eclipse.skalli.core.internal.search.SearchServiceImpl.findProjectsByQuery(SearchServiceImpl.java:94)|
  at org.eclipse.skalli.api.rest.internal.resources.ProjectsResource.retrieve(ProjectsResource.java:48)|
…
Server process shutting down with exit code [666]; memory allocation error [OutOfMemoryError]|
Comment 1 Michael Ochmann CLA 2011-11-30 04:21:15 EST 
Lucene seems not prepared to accept count=Integer.MAX_VALUE for search queries in org.eclipse.skalli.api.rest.internal.resources.ProjectsResource
Comment 2 Michael Ochmann CLA 2011-11-30 08:03:33 EST 
https://issues.apache.org/jira/browse/LUCENE-504

org.apache.lucene.util.PriorityQueue allocates an Object array with maxSize entries in advance even if the result set is much smaller than maxSize. Incredible waste of memory, but the way TopScoreDocCollector is implemented, it seems to be inevitable.

I'm going to add the "start" and "count" query parms for /api/projects?query=foobar so that one can retrieve search hits "page by page" like in the UI.

Furthermore, I will limit "count" to the overall number of projects since it is impossible to have more search hits.
Comment 3 Michael Ochmann CLA 2011-11-30 08:56:49 EST 
Created attachment 207720 [details]
proposed patch
Comment 4 Michael Ochmann CLA 2011-12-01 03:11:45 EST 
Created attachment 207777 [details]
proposed patch (fixed invalid schema)