Community
Participate
Working Groups
Sorry for not being more specific, but Bugzilla is too slow for effectively working with it, too often. I am not talking about operations that by their very nature (e.g. queries) can take some time. One operation that needs to be fast is when I click on a link to a Bugzilla that I get the single bug page back in a reasonable response time. Ideally, that response time should be below a second. Another operation that should be fast is adding a comment, changing a target milestone, or adding someone to the cc list. I need to wait for the resulting page to make sure there is no conflict, and to check for misspelled email addresses when adding to the cc list. It just happens too often that I end up waiting for 5-10 seconds. That might not sound like very much, but if your main task is to look at incoming bugzilla emails, you need to look at a good number of bugzilla entries for context, and having to wait 5-10 seconds for each bug kills productivity (and makes this part of my work even less enjoyable). Do you know the causes of the slowness? Just a wild guess, but perhaps some people are running too many queries too often?
> Do you know the causes of the slowness? Just a wild guess, but perhaps some > people are running too many queries too often? Ah, I was expecting you. Yes, I do know why Bugzilla is slow. I always know. I know why it's slow before you even realize it's slow :-) The main reason is database-related. More specifically (in no particular order): - Because the BZ tables are MyISAM, table locking can introduce tremendous lag when some folks run large searches. The table types are changed in Bugzilla 3.2, so I'm making this bug depend on the upgrade (bug 244441). - I see a fair amount of Mylyn queries that come in at the same time. It's not uncommon to see six to nine MySQL threads working on Mylyn-related searches that usually end up being killed for taking too long. Of course, while these queries run, the database gets slower, and incoming changes are blocked by locked tables from the point above. I'm working with the Mylyn team so that we can optimize some things via some open bugs. - Our database slave server, which handles Bugzilla searches most of the time, is quite busy since it's our primary file server. This was not a problem before we had a local build server and local project vservers, but now with LAN access to CVS and downloads, some projects are building constantly, which uses resources. - Disk performance on our database servers is suffering since our ageing servers require maintenance. Right now, all our RAID controller batteries are dead, which means disk write cache is disabled. Sourcing new batteries that don't cost an arm is in the works, but replacing batteries requires the machines be powered down. This was a no-go prior to galileo, but fixing this is on our plate for Q3. That pretty much sums it up.. Sorry if there's no silver bullet.
Thanks for the detailed comments. I'm impressed.
Denis, thanks for that info. During our annual release retrospective of the Eclipse top-level project the bugzilla slowness was mentioned as an issue by several teams, which is natural given bugzilla is one of our main working tools. So, any progress in this is really much appreciated and will have good impact on all eclipse developers. Regarding the Mylyn problem: would it be possible to implement something along the lines of QoS and give Mylyn queries lower priority before they even reach the bugzilla server? Or to be more general give higher priority to simple and direct bugzilla requests/queries?
And of course the new asynchronous email sending feature will hopefully improve things.
(In reply to comment #1) > - Disk performance on our database servers is suffering since our ageing > servers require maintenance. Right now, all our RAID controller batteries are > dead, which means disk write cache is disabled. This was taken care of last weekend, and our disk performance is definitely better this morning.
> - I see a fair amount of Mylyn queries that come in at the same time. bug 281893 c7 clearly illustrates the Mylyn problem.
That should read, bug 281893 comment 7.
I just re-enabled mod_perl. BEFORE (CGI): ============= time wget --header="Accept-encoding: gzip,deflate" https://bugs.eclipse.org/bugs/show_bug.cgi?id=12345 real 0m1.678s user 0m0.040s sys 0m0.006s AFTER (MOD_PERL): ================= time wget --header="Accept-encoding: gzip,deflate" https://bugs.eclipse.org/bugs/show_bug.cgi?id=12345 real 0m0.702s user 0m0.038s sys 0m0.009s I've also moved the Bugzilla app files to local disks to reduce the load (and latency) of pulling them from NFS. We' started doing this some time ago, and it's helped performance greatly. I've also confirmed that the table lock problem has gone away. Bugzilla doesn't need to wait for seconds while a table lock is in place, so there shouldn't be any random 30-second response times. The only think I have left is to enable asynchronous email.
(In reply to comment #8) > The only think I have left is to enable asynchronous email. woot :-) Thanks a lot!
(In reply to comment #8) > The only think I have left is to enable asynchronous email. Asynchronous email has been set up. I noticed that this does introduce a slight lag in sending mail (up to 1 minute) since mail is batched together...
>Asynchronous email has been set up. I noticed that this does introduce a >slight lag in sending mail (up to 1 minute) since mail is batched together... I think that's acceptable: bugzilla performance for people who make chances is more important than immediate notification.
Thanks for the improvements so far, much appreciated! Just in case this helps finding more causes for slowness: I just had to wait approx. 30 seconds on a bug (https://bugs.eclipse.org/bugs/show_bug.cgi?id=277947 at 3:13pm EST on Sep 9, 2009) But overall Bugzilla has gotten much more consistent, and faster, so no big deal at all that I had to wait one time for a little longer.
Sometime in early 2010 we'll be getting some dedicated (and faster) hardware for our database servers. Our DB servers are currently coexisting with the file servers, which is far from ideal. Once that is done, and when bug 281893 gets fixed, I think Bugzilla will be in great shape.
> Sometime in early 2010 we'll be getting some dedicated (and faster) hardware > for our database servers. The new database servers are configured and ready to be deployed. Just waiting for some rackmount hardware.
New database servers were put into production. Even at peak times, their load average is basically just above idle. Marking fixed since I've done everything I can to make Bugzilla as fast as it can be. I'm also removing bug 281893 as a blocker since I've implemented a workaround (see bug 281893 comment 22).
