Bugzilla – Full Text Bug Listing
| Summary: | [performance] submitting a trac ticket change always triggers a full synchronization | ||||||
|---|---|---|---|---|---|---|---|
| Product: | z_Archived | Reporter: | Jörg Thönnes <jtk499> | ||||
| Component: | Mylyn | Assignee: | Steffen Pingel <steffen.pingel> | ||||
| Status: | RESOLVED DUPLICATE | QA Contact: | |||||
| Severity: | major | ||||||
| Priority: | P3 | ||||||
| Version: | 3.2 | ||||||
| Target Milestone: | --- | ||||||
| Hardware: | PC | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Attachments: |
|
||||||
|
Description
Jörg Thönnes
Created attachment 147881 [details]
Screen shot showing the unexpected sync job
Here you see the job starting immediately after the submit job is finished.
See also bug 239182 which talks about the blocked jobs, but this ticket is more about __why__ the job is started after the submit. The mentioned bug would help a bit though. The sync job runs to ensure that all queries accurately reflect the new repository state. The submission may cause a bug to move between queries or to drop out of a query and users generally expect that to be visible instantly. On the other hand I see your point that running all queries generates significant load and these queries can queue up which has a further impact on performance (bug 239182). I am considering two optimizations at the moment: * Run lightweight queries only, i.e. do not check for changed tasks * Only include queries that are marked for automatic synchronization I am surprised that the triggered sync takes a few minutes. I would expect this to take seconds particularly if you have a local server and not be much slower than a web sync. I assume that you are running 3.2.0 or later which has the fixes for bug 264930? Hi Steffen, thanks for the detailed reply. (In reply to comment #3) > The sync job runs to ensure that all queries accurately reflect the new > repository state. The submission may cause a bug to move between queries or to > drop out of a query and users generally expect that to be visible instantly. On > the other hand I see your point that running all queries generates significant > load and these queries can queue up which has a further impact on performance > (bug 239182). I am considering two optimizations at the moment: > * Run lightweight queries only, i.e. do not check for changed tasks What do you mean by light-weight queries? > * Only include queries that are marked for automatic synchronization Yes, I would expect "manual" as "never triggered automatically." > I am surprised that the triggered sync takes a few minutes. I would expect this > to take seconds particularly if you have a local server and not be much slower > than a web sync. I assume that you are running 3.2.0 or later which has the > fixes for bug 264930? Yes, we are using the lastest stable release 3.2.1v20090722-0200. And we use Trac 0.11.1 + lastest XML RPC 1.06. I wonder how the queries are processed. I would expect two phases: 1. Run all queries and fetch the ids of all changed tasks 2. For every returned id update the task information In this way, if several queries return the same id, only one query has to be made in step 2. In addition, you could also use timestamps for the last update and the id returned by a query run and skip the update if the time difference is less than a "fudge" value. But I guess something like that is already done by bug 264930. And yes, the web queries are much faster, so that most of my team members avoid using the XML-RPC mode. I would very much like to help you debugging this. Which kind of information would you need? Please outline some steps. Thanks a lot, Jörg (In reply to comment #3) > The sync job runs to ensure that all queries accurately reflect the new > repository state. The submission may cause a bug to move between queries or to > drop out of a query and users generally expect that to be visible instantly. OK, but I am experience is that every submit starts a __new__ sync job while the previous one is not completed yet. So I always have the "Progress" view open and cancel old jobs by hand. Is there a way to reduce the amount of sync jobs, i.e. by automatically cancelling old ones etc? Submitting several task in short succession creates a bunch on sync job slowing down Eclipse and creating a high load on the server. Could you increase the prio to P2 therefore? We'll fix the multiple jobs running in parallel as part of bug 239182 . (In reply to comment #6) > We'll fix the multiple jobs running in parallel as part of bug 239182 . Thanks, I just installed the 3.3RC1 together with Tasktop 1.5.2 and could verify that only one Synchronize job is started. This is good. But the job is still taking quite long: We have 2300 tickets in the database and it runs several minutes on a local 1GBit net. Could you provide some assistance how I could debug this? It should only take that long on the first run and then do incremental updates for changed bugs. It is synchronizing all 2300 bugs each time it runs? (In reply to comment #8) > It should only take that long on the first run and then do incremental updates > for changed bugs. It is synchronizing all 2300 bugs each time it runs? Yes, it did so. But now I guess I found the error: One of the tickets in the trac repository has a comment entry with an ASCII(1) character (Ctrl-A). The XML-RPC plugin fails to return this character as CDATE, but returns it directly rendering invalid XML. The processing of this ticket fails with an XML parser exception. Now I have deleted the offending comment from the Trac sqlite database directly. And now the update is very fast, i.e. the incremental updates do work. Even with 3.2.2 this now works satisfying, the accumulation of blocking task does not happen. Now my guess is that any exception during the task list refresh triggers a complete refresh even if the exception is local to one specific task and all other task could be updated without any problems. While the above behaviour relates to an XML-RPC bug, Mylyn should safeguard against such sort of failures since they could affect performance. To reproduce, just add a Ctrl-A character as a comment to one of the test trac tickets. Will try to reproduce myself if time allows. What do you think about this suspect? Great that you were able to track that down Joerg. Your explanation sounds right to me. It is correct that incremental synchronizations will fail if a single task fails to synchronize. I have added a comment to bug 251658 to handle that case more gracefully. Thanks for getting back. Is it okay to close this bug and track further improvements on bug 251658 since this needs to be addressed on the framework level rather than in the Trac connector? (In reply to comment #10) > Great that you were able to track that down Joerg. Your explanation sounds > right to me. It is correct that incremental synchronizations will fail if a > single task fails to synchronize. I am also happy. Now the Mylyn is much more usable. I will see whether the high load on Trac server which caused most of my team to use the Web-based mode instead is now gone. This issue also explains why the web-mode worked much better. > I have added a comment to bug 251658 to handle that case more gracefully. So did I :-) > Thanks for getting back. Is it okay to close this bug and track further > improvements on bug 251658 since this needs to be addressed on the framework > level rather than in the Trac connector? Yes, closing as duplicate. *** This bug has been marked as a duplicate of bug 251658 *** On the good side: This bug also helped to resolve bug 239182. |