Community
Participate
Working Groups
Created attachment 178100 [details] profile output This is a bug to address part 2 of bug 313953. Thomas, sorry I missed your questions in the original bug. Yes, there are almost 6000 files on disk which have been removed. - copy test data (3000 features, 3000 plugins) to dropins folder - start Eclipse - shutdown - delete all plugins and features except 1 of each - start Eclipse - this start takes almost 10min. Here is the query that we are performing: // create a query that will identify all ius related to removed files. // It's safe to compare a String with a File since the auto coercion will // first convert the String into a File. IQuery<IInstallableUnit> removeQuery = QueryUtil.createMatchQuery( // "$1.exists(x | properties[$0] == x)", FILE_NAME, removedFiles); //$NON-NLS-1$ IQueryResult<IInstallableUnit> toRemove = metadataRepository.query(removeQuery, null); metadataRepository.removeInstallableUnits(toRemove.toUnmodifiableSet()); Thomas, is there a way we could re-write the query or do something differently here?
(In reply to comment #0) > Thomas, is there a way we could re-write the query or do something differently > here? I have a hard time understanding how that query can take more then a few milliseconds to execute over 6000 rows. If it executes millions of times, then sure, that would take some time. How many times does this query run during the 10 minutes? How much of that time is spent in the query?
- the query is run once - about 470,000 ms of time is spent in MatchExpression.evaluate - the attachment has the profile output - the removedFiles arg has 5998 elements in it
Sorry, I missed the profile. One thing I noticed there is that the java.io.File.equals() is called which means that in order to do the comparison, the property (always a string) is first converted into a File. This makes sense if the list that you pass in as $0 is a list of java.io.File instances. p2QL uses the same semantics as an OSGi LDAP filter in this case and makes an attempt to find a constructor for the corresponding class that takes a String argument (i.e. new File(String) in this case). Try converting the argument into a Collection<String> list using x.getPath() on each element before you pass it the query. That will make the comparisons a lot faster since they don't need to try and find a String constructor, create a File instance, and the compare for each attempt. The number of comparisons will be total number of ius in the repository * number of properties in each iu * 6000.
Created attachment 178173 [details] profile output 2 Here is the profile output with the collection changed to be Strings instead of Files. Unfortunately there isn't much of a change in the timings, the majority of the time is taken evaluating the expression. Collection<String> paths = new HashSet<String>(removedFiles.size()); for (File file : removedFiles) paths.add(file.getAbsolutePath()); IQuery<IInstallableUnit> removeQuery = QueryUtil.createMatchQuery( // "$1.exists(x | properties[$0] == x)", FILE_NAME, paths); //$NON-NLS-1$ IQueryResult<IInstallableUnit> toRemove = metadataRepository.query(removeQuery, null); metadataRepository.removeInstallableUnits(toRemove.toUnmodifiableSet());
Looking at the new profile output I still see the calls to File.equals() ...
Oops. I exported and replaced the wrong version, sorry. Will try again.
Ahh... much better. That part of the operation went from 460s down to 13s. Thanks, Thomas!
Created attachment 178182 [details] new profiler output I've attached the new profiler output for those who might be interested.
Fix released to HEAD. Thanks for your help, Thomas.
There was a problem with tagging HEAD so this fix won't appear in integration builds until the first build after 3.7 M2.
