Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 324175

Summary:

Performance issues when querying large dirty resources

Product:

[Modeling] EMF

Reporter:

Axel Uhl <eclipse>

Component:

Query2

Assignee:

Project Inbox <emf.query2-inbox>

Status:

RESOLVED FIXED

QA Contact:

Severity:

normal

Priority:

CC:

eclipse, manasa.priya.c, saurav.sarkar1

Version:

unspecified

Target Milestone:

---

Hardware:

OS:

Windows XP

Whiteboard:

Attachments:

Description	Flags
Test case for delete Resources	none
Test plugin for index updation on resource deletion and its dependency plugin	none

Description Axel Uhl

2010-09-01 05:57:49 EDT

Build Identifier: Query2 Prototype

We face a serious problem with the current query2 implementation. We use the IndexFactory to index several dirty in-memory resources over which then we run some MQL queries. We then discard the transient resources and construct new ones, repeating the whole process many times. This all happens during the course of a performance benchmarking exercise.

During this, the memory consumption of the index increases to several GB, ultimately letting the heap exceed any limit we set. The way it looks, the index doesn't evict entries from no longer existing resources. This is a serious problem.

Additionally, it seems that the index doesn't successfully evict parts of the index from memory. So even if our models *were* that huge, I think the index should scale properly and load parts of the index from disk.


Reproducible: Always

Comment 1 manasa

2010-12-20 23:59:45 EST

Created attachment 185611 [details]
Test case for delete Resources

Comment 2 manasa

2010-12-21 00:01:52 EST

There are 2 scenarios that can be considered for the issue mentioned below.

1.	Integrated Query2 IndexBuilder 
When the index builder is integrated with the workspace,  index updation is taken care of, when new resources are added/deleted( including transient resources). Internally, IndexUpdater gets notification on deleting a resource, indexes are updated accordingly.

2.	Indexes created manually (without using IndexBuilder)
When indexes are created manually, like in the testcase attached, deleting a resource using resource.delete , would not update the indexes, as the IndexUpdater is not notified about changes in the eclipse workspace.
In this case, the updation of indexes has to be explicitly called after deleting the resource, without which the indexes of the deleted resources would still exist.

Scenario2 could be the possible reason for the issue below.

Comment 3 Axel Uhl

2010-12-21 03:16:18 EST

(In reply to comment #2)
Ok, then we have to ensure we clean up after explicitly indexing transient resources. If you can demonstrate by a test case that you can do this many times in a row without exceeding memory limits that should suffice.

Thanks.

Comment 4 manasa

2010-12-30 05:45:17 EST

Created attachment 185905 [details]
Test plugin for index updation on resource deletion and its dependency plugin

Comment 5 manasa

2010-12-30 05:46:25 EST

Hi Axel,

Please find attached the plugin containing the test case for the index updation. Along with it, is the plugin it depends on.

The scenario in the test case is creation of huge number of resources and indexing them,  followed by deletion of them and updating indexes.
This is repeated a few number of times, after each iteration, global tables of index gets updated with deleted resources.
We have tried upto creation of 95000 resources, updating and deleting the indexes, without exceeding the memory limits. 

Below given are the VM arguments used.

-Xms32m
-Xmx128m
-XX:PermSize=256M

Please let us know if you need any further details.

Comment 6 manasa

2011-01-06 02:23:48 EST

I hope the test plugin attached describes the resolution to your problem.
I shall close this bug on that note and please revert back in case you find the same issue again.

Regards,
Manasa

Comment 7 saurav sarkar

2011-01-06 23:50:15 EST

Closing the bug