Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.
Bug 208488 - [performance] streamline synch after task submit
Summary: [performance] streamline synch after task submit
Status: RESOLVED FIXED
Alias: None
Product: z_Archived
Classification: Eclipse Foundation
Component: Mylyn (show other bugs)
Version: unspecified   Edit
Hardware: PC Windows Vista
: P1 enhancement (vote)
Target Milestone: 2.2   Edit
Assignee: Steffen Pingel CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2007-11-01 17:57 EDT by Mik Kersten CLA
Modified: 2007-11-07 17:38 EST (History)
1 user (show)

See Also:


Attachments
patch (9.79 KB, patch)
2007-11-01 19:41 EDT, Steffen Pingel CLA
no flags Details | Diff
mylyn/context/zip (5.58 KB, application/octet-stream)
2007-11-01 19:41 EDT, Steffen Pingel CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Mik Kersten CLA 2007-11-01 17:57:17 EDT
Currently after a task is submitted, a full synch is invoked.  Instead, let's do the following:
* Synch the task that was submitted (already done afaik)
* Synch all queries, do NOT check for change tasks
Comment 1 Mik Kersten CLA 2007-11-01 17:59:20 EDT
This is a big deal because it's queuing a huge number of queries. 
Comment 2 Eugene Kuleshov CLA 2007-11-01 18:50:43 EDT
Why do we need to sync queries? They will be synched on regular schedule anyways, but this full sync will significantly increase load to the issue tracking servers.

BTW, this might be related to the issue when query sync is silently throwing not matching tasks out of the query results without user's preview. Once we got that right, there probably won't be need to resync queries.
Comment 3 Steffen Pingel CLA 2007-11-01 19:41:28 EDT
Created attachment 81904 [details]
patch

Adds a flag for doing full synchronization.
Comment 4 Steffen Pingel CLA 2007-11-01 19:41:30 EDT
Created attachment 81905 [details]
mylyn/context/zip
Comment 5 Mik Kersten CLA 2007-11-01 20:22:44 EDT
Steffen: Awesome. 

Rob: please review.

(In reply to comment #2)
> Why do we need to sync queries? They will be synched on regular schedule
> anyways, but this full sync will significantly increase load to the issue
> tracking servers.

Submitting the task can cause it to move between or out of of queries, and the user needs near-immediate feedback on that in order to avoid thinking that something about the submission failed.
Comment 6 Steffen Pingel CLA 2007-11-01 21:34:13 EDT
The problem that with the patch is that Bugzilla's query result feed does not contain modification time stamps. That means although we can synchronize the query we don't get information which tasks have changed. This means a user will see queries being synchronized will not see new incoming indications (for Bugzilla). For the task that was edited we already know it was changed so that will get properly synchronized whereas other changed tasks will not get updated until the next background synchronization. Given the significant performance of skipping the check for changed tasks this could still be worth merging. 
Comment 7 Robert Elves CLA 2007-11-02 13:06:06 EDT
 (In reply to comment #6)
> Given the significant performance of
> skipping the check for changed tasks this could still be worth merging.
I agree. Lets try this out since background synch will take care of bringing in any new incoming. Patch applied.
Comment 8 Eugene Kuleshov CLA 2007-11-02 15:04:58 EDT
(In reply to comment #5)
> Submitting the task can cause it to move between or out of of queries, and the
> user needs near-immediate feedback on that in order to avoid thinking that
> something about the submission failed.

There are examples like that in other Eclipse components (i.e. in JUnit results and in Debug view). Basically you can mark/decorate query as "possible out of sync", so user can decide to manually synchronize it if here ally need that. But I think your evaluation about "near-immediate feedback" is overrated (there is editor opened after submission that would had a error status if anything went wrong).
Comment 9 Steffen Pingel CLA 2007-11-05 23:54:12 EST
Unfortunately the implementation of the patch "optimized" a bit too much and only checks for incomings when "Synchronize Changed" is invoked from the task list menu. Fixed in CVS.
Comment 10 Eugene Kuleshov CLA 2007-11-06 00:56:49 EST
(In reply to comment #5)
> Submitting the task can cause it to move between or out of of queries, and the
> user needs near-immediate feedback on that in order to avoid thinking that
> something about the submission failed.

The more I think about it the scarier it look. Say I have 20 task to triage. I can change priority, reassign or close each of them. Now, after each change whole query will be synchronized and that will happens 20 times in the row. Can I at least turn this off?
Comment 11 maarten meijer CLA 2007-11-06 09:02:19 EST
 (In reply to comment #10)
> The more I think about it the scarier it look. Say I have 20 task to triage. I
> can change priority, reassign or close each of them. Now, after each change
> whole query will be synchronized and that will happens 20 times in the row. Can
> I at least turn this off?
There are other alternatives between yes or nio full sync:

One is to only synchronize the task AND move a duplicate to a special category: 'In progress' or 'Changed today' similar to Archve and Uncategorized.
That means they will still be accessible in the short term, but the query can just synchronize on the regular schedule.

Alternatively, each change to a task in a query will pull closer the next sync by a fixed or proportional amount of time. So rather than causing a full synch each time, you're just raising the urgency of the next sync by a bit.   
Comment 12 Mik Kersten CLA 2007-11-06 12:31:53 EST
I don't want us exposing implementation details of synchronization to users.  That has been the goal all along, and it has helped us prioritize and resolve performance problems while making the tool usable to non-experts who have no interest in such details.  So let's first continue to collect the current performance problems and address them by making the synch mechanism smarter instead of overloading the user.  This bug is a good example of that effort, and has reduced synchronization load and times dramatically while maintaining user expectations.
Comment 13 Eugene Kuleshov CLA 2007-11-06 14:38:37 EST
Simple decoration or "changed" marker on query node would help to address the visibility issues.

On the other hand, it somehow feels that you are trying to address needs of non-developers and generalize some other connector (perhaps outlook) into the same synchronization workflow. Maybe this synchronization aspect (i.e. should it sync the whole thing or not) should be controlled by connector type, so connectors like bugzilla won't use that.
Comment 14 Steffen Pingel CLA 2007-11-06 14:46:23 EST
(In reply to comment #10)
> The more I think about it the scarier it look. Say I have 20 task to triage. I
> can change priority, reassign or close each of them. Now, after each change
> whole query will be synchronized and that will happens 20 times in the row. Can
> I at least turn this off?

This is a bug in the current implementation. Synchronizations should not be queued up when a synchronization is already in progress. 

Comment 15 Mik Kersten CLA 2007-11-07 17:38:51 EST
 (In reply to comment #13)
> On the other hand, it somehow feels that you are trying to address needs of
> non-developers and generalize some other connector (perhaps outlook) into the
> same synchronization workflow. Maybe this synchronization aspect (i.e. should it
> sync the whole thing or not) should be controlled by connector type, so
> connectors like bugzilla won't use that.

While we could prioritize such connectors higher, we haven't done any of that to this point.  Connectors like Outlook are different because they have near-instant access to the repository and no network lag or potential for things like mid-air collisions.  If we ever put API improvements for such connectors it will be very explicit in the naming, and the Local task repository will serve well as the reference implementation.  For now I believe that such connectors are fine using the current synchronization infrastructure even though the need it all.  In terms of UI, I think the same constraints apply.