Bugzilla – Full Text Bug Listing
| Summary: | [query] Repository query takes a long time when deleting a lot of IUs (from dropins) | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Eclipse Project] Equinox | Reporter: | DJ Houghton <dj.houghton> | ||||||||
| Component: | p2 | Assignee: | P2 Inbox <equinox.p2-inbox> | ||||||||
| Status: | RESOLVED FIXED | QA Contact: | |||||||||
| Severity: | normal | ||||||||||
| Priority: | P3 | CC: | irbull, thomas | ||||||||
| Version: | 3.7 | ||||||||||
| Target Milestone: | 3.7 M3 | ||||||||||
| Hardware: | PC | ||||||||||
| OS: | Mac OS X - Carbon (unsup.) | ||||||||||
| Whiteboard: | |||||||||||
| Bug Depends on: | |||||||||||
| Bug Blocks: | 332095 | ||||||||||
| Attachments: |
|
||||||||||
|
Description
DJ Houghton
(In reply to comment #0) > Thomas, is there a way we could re-write the query or do something differently > here? I have a hard time understanding how that query can take more then a few milliseconds to execute over 6000 rows. If it executes millions of times, then sure, that would take some time. How many times does this query run during the 10 minutes? How much of that time is spent in the query? - the query is run once - about 470,000 ms of time is spent in MatchExpression.evaluate - the attachment has the profile output - the removedFiles arg has 5998 elements in it Sorry, I missed the profile. One thing I noticed there is that the java.io.File.equals() is called which means that in order to do the comparison, the property (always a string) is first converted into a File. This makes sense if the list that you pass in as $0 is a list of java.io.File instances. p2QL uses the same semantics as an OSGi LDAP filter in this case and makes an attempt to find a constructor for the corresponding class that takes a String argument (i.e. new File(String) in this case). Try converting the argument into a Collection<String> list using x.getPath() on each element before you pass it the query. That will make the comparisons a lot faster since they don't need to try and find a String constructor, create a File instance, and the compare for each attempt. The number of comparisons will be total number of ius in the repository * number of properties in each iu * 6000. Created attachment 178173 [details]
profile output 2
Here is the profile output with the collection changed to be Strings instead of Files. Unfortunately there isn't much of a change in the timings, the majority of the time is taken evaluating the expression.
Collection<String> paths = new HashSet<String>(removedFiles.size());
for (File file : removedFiles)
paths.add(file.getAbsolutePath());
IQuery<IInstallableUnit> removeQuery = QueryUtil.createMatchQuery( //
"$1.exists(x | properties[$0] == x)", FILE_NAME, paths); //$NON-NLS-1$
IQueryResult<IInstallableUnit> toRemove = metadataRepository.query(removeQuery, null);
metadataRepository.removeInstallableUnits(toRemove.toUnmodifiableSet());
Looking at the new profile output I still see the calls to File.equals() ... Oops. I exported and replaced the wrong version, sorry. Will try again. Ahh... much better. That part of the operation went from 460s down to 13s. Thanks, Thomas! Created attachment 178182 [details]
new profiler output
I've attached the new profiler output for those who might be interested.
Fix released to HEAD. Thanks for your help, Thomas. There was a problem with tagging HEAD so this fix won't appear in integration builds until the first build after 3.7 M2. |