Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 369941

Summary: [api][dstore] cancelable threads not removed fast enough from Hashmap, resulting in OOM
Product: [Tools] Target Management Reporter: Onno Van den Troost <onno>
Component: RSEAssignee: David McKnight <dmcknigh>
Status: RESOLVED FIXED QA Contact: Martin Oberhuber <mober.at+eclipse>
Severity: normal    
Priority: P3 CC: dmcknigh
Version: 3.0.3Keywords: api
Target Milestone: 3.4 M6   
Hardware: Other   
OS: other   
Whiteboard:
Bug Depends on:    
Bug Blocks: 369950, 369951    
Attachments:
Description Flags
patch to automatically update cancellable threads after a download
none
update to add api for HEAD stream change none

Description Onno Van den Troost CLA 2012-01-27 10:23:49 EST 
A customer reported an Out-Of-Memory error, and heapdump analysis showed that ServerCommandHandler was the main leak suspect.

Our developer already communicated with David M. about this:

<quote>
Looking at the heap dump, this part of heap occupation
is narrowed down to HashMap which stores the cancellable threads of UniversalDownloadHandler. 
The UniversalDownloadHandler instance is not removed from the HashMap soon after the download is finished.
The following actions are defined as the cancellable threads:
  Copy
  CreateFile
  CreateFolder
  Delete
  Query
  Rename
  Download
  Search
The timing when the thread instance is removed from the HashMap is the time when the thread is gone and the next cancellable command has come to the UniversalFileSystemMiner from a client.

Looking at the server log, about 15 clients were working at one process.
The total size of instances which the above HashMap has is about 150MB.
That is, about 15MB per client in average.
When all of the clients download some USS file and then works for MVS miners, all of the UniversalDownloadHandler instances are not removed.
In fact, the heap core dump said that the current thread was MVS miner.
 
After all, I think that Open RSE base should remove the  cancellable threads from the HashMap soon after the UniversalFileSystemMiner finishes the process for the corresponding command.

David,
What do you think about this problem ?
---
Completed threads are supposed to be removed from the cancellable threads HashMap on subsequent commands so I'm not sure why there would be an accumulation.  I've attached a dstore_miners.jar that automatically involves updateCancellableThreads() after download completion.
</quote>

The customer confirmed that the test fix worked.

We would like a backport of the fix to 30x and 32x.
Comment 1 David McKnight CLA 2012-01-27 10:41:54 EST 
Created attachment 210192 [details]
patch to automatically update cancellable threads after a download
Comment 2 David McKnight CLA 2012-01-27 10:42:37 EST 
To avoid API changes, I've used reflection to allow the UniversalDownloadHandler to access UniversalFileSystemMiner.updateCancellableThreads().
Comment 3 David McKnight CLA 2012-01-27 11:09:55 EST 
I've committed the change to the HEAD stream and opened bug 369950 and bug 369951 for backporting.
Comment 4 David McKnight CLA 2012-02-02 09:25:13 EST 
Created attachment 210446 [details]
update to add api for HEAD stream change
Comment 5 David McKnight CLA 2012-02-02 09:26:36 EST 
For the HEAD stream change, I've changed private method UniversalFileSystemMiner.updateCancellableThreads() to be public so reflection isn't needed to make the call immediately after a download request completes.