Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.
Bug 355542 - Massive memory spikes from solr
Summary: Massive memory spikes from solr
Status: RESOLVED FIXED
Alias: None
Product: Orion
Classification: ECD
Component: Server (show other bugs)
Version: unspecified   Edit
Hardware: PC Windows 7
: P3 normal (vote)
Target Milestone: 0.3 M2   Edit
Assignee: John Arthorne CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-08-23 13:42 EDT by John Arthorne CLA
Modified: 2011-08-23 14:41 EDT (History)
0 users

See Also:

Attachments
Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description John Arthorne CLA 2011-08-23 13:42:58 EDT 
While profiling an orion server I noticed large spikes in heap usage for brief periods (300MB). None of this is retained. This is coming from a background job that is looking for stale index entries.
Comment 1 John Arthorne CLA 2011-08-23 14:41:00 EDT 
The problem was the index purge job did a query for "everything" (in chunks of 1000 results at once), and then checked if each file was still present. By default solr returns all fields, including the contents of the files. So, we had the contents of 1000 files in memory during the query. In this case all we needed was the ID of each solr document, which vastly reduces the memory overhead. With this fix I am seeing a spike of 5MB rather than 300MB during index operations.

http://git.eclipse.org/c/orion/org.eclipse.orion.server.git/commit/?id=7a062e8c082fa90b42da20353f00c203a7790b91