Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 442084

Summary:	Memory leak caused by SearchServlet
Product:	[ECD] Orion	Reporter:	Michael Ochmann <michael.ochmann>
Component:	Server	Assignee:	John Arthorne <john.arthorne>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	P2	CC:	pwebster
Version:	unspecified
Target Milestone:	7.0
Hardware:	All
OS:	All
Whiteboard:

Description Michael Ochmann

2014-08-19 12:26:05 EDT

I performed some loadrunner tests of the /filesearch API and reproducibly observed a more or less linear increase of the retained heap over time causing an OOM after some hours. Heap dump analysis revealed that almost all heap was consumed by Solr, i.e. there were in some cases several thousand instances of org.apache.solr.search.SolrIndexSearcher hold by a variable _searchers in org.apache.solr.core.SolrCore consuming several GB of heap.

Orion's SearchServlet uses a LocalSolrQueryRequest instance for rendering of the JSON response of a search query (SearchServlet#writeResponse()). It is not obvious, but debugging reveals that each LocalSolrQueryRequest instance internally calls SolrCore#getSearcher() and increases the reference count of the currently active SolrIndexSearcher. Upon next run of the Orion Indexer, Solr tries to release this SolrIndexSearcher, but can't because the reference count is not zero. This causes the accumulation of SolrIndexSearcher instances over time.

I'm going to push a patch for this issue to Gerrit.

Comment 1 Michael Ochmann

2014-08-19 12:28:19 EDT

https://git.eclipse.org/r/#/c/31906/

Comment 2 John Arthorne

2014-08-19 21:07:36 EDT

Great find!!

Comment 3 John Arthorne

2014-08-20 16:32:12 EDT

I have reviewed and released the patch. For anyone looking, the patch looks long but in fact it is just wrapping the block in try/finally and closing the request in the finally block, so this is a simple change.

http://git.eclipse.org/c/orion/org.eclipse.orion.server.git/commit/?id=f2e970fc809a5253510dd70bf7eb59347677ddf8

Thanks again Michael for your analysis and fix.