Bugzilla – Full Text Bug Listing
| Summary: | A lot of "Bad Gateways" with Sonar | ||
|---|---|---|---|
| Product: | Community | Reporter: | Mickael Istria <mistria> |
| Component: | Sonar | Assignee: | Generic Inbox <sonar-inbox> |
| Status: | CLOSED FIXED | QA Contact: | |
| Severity: | minor | ||
| Priority: | P3 | CC: | akurtakov, denis.roy, jan.sievers, jfaltermeier, leo.dos.santos, lorenzo.bettini, malaperle, mikael.barbero, sam.davis, sewe, thanh.ha, webmaster, zulus |
| Version: | unspecified | ||
| Target Milestone: | --- | ||
| Hardware: | All | ||
| OS: | All | ||
| Whiteboard: | |||
| Bug Depends on: | |||
| Bug Blocks: | 432376 | ||
|
Description
Mickael Istria
This has become a blocker for Platform build (which always gets a 502). Can you provide details as to what it's trying to do? What has changed for this to start happening? It's just trying to fetch files. It started happening more and more since update of SonarQube and setting of reverse proxy. I can sometimes reproduce it from my browser, just by browsing the interface. Sorry, I still don't understand how to reproduce this. I went to https://dev.eclipse.org/sonar and clicked around for a few minutes but everything worked. *** Bug 419945 has been marked as a duplicate of this bug. *** As I understand it, this worked fine before the recent Sonar upgrade, correct? Yes, there was no such problem before Sonar upgrade & usage of reverse-proxy for HTTPS (bug 418502). Reducing priority as this did not happen for a while AFAIK. For ECP and EMFStore the sonar jobs still fail with 502 errors frequently. https://hudson.eclipse.org/ecp/job/emf-emfclient-sonar/ https://hudson.eclipse.org/emfstore/job/emf-emfstore-sonar/ Yep, I could notice the same issues with some other jobs. @Webmaster: could anything have changed in network or Sonar VM that could be a cause and a clue for a potential permanent fix? I'm not aware of any changes to the network/proxy or the sonar VM(I don't have access). -M. This seems to have been consistently occurring on linuxtools hipp. See: https://hudson.eclipse.org/linuxtools/job/linuxtools-sonar/ This is consistently happening for several Mylyn builds as well, e.g. https://hudson.eclipse.org/hudson/view/Mylyn/job/mylyn-commons-nightly/338/console. The error message is "Fail to download [https://dev.eclipse.org/sonar/batch_bootstrap/properties?project=org.eclipse.mylyn.commons:org.eclipse.mylyn.commons.repositories.ui&dryRun=false]. Response code: 502." Is it possible there is a network issue between hudson.eclipse.org and dev.eclipse.org? We have the same problem at Code Recommenders [1]; this has plagued us for 2-3 weeks now. <https://hudson.eclipse.org/recommenders/job/sonar.origin.master.kepler/41/console> Today we started using Sonar for EMF Parsley, and we run in the same issue: https://hudson.eclipse.org/emf-parsley/job/emf-parsley-sonar/3/console Here's the main failure in the Hudson Job: [ERROR] Failed to execute goal org.codehaus.mojo:sonar-maven-plugin:2.3:sonar (default-cli) on project org.eclipse.emf.parsley.parent: Can not execute SonarQube analysis: Fail to execute request [code=502, url=https://dev.eclipse.org/sonar/batch_bootstrap/properties?dryRun=false]: Fail to download [https://dev.eclipse.org/sonar/batch_bootstrap/properties?dryRun=false]. Response code: 502 -> [Help 1] org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.codehaus.mojo:sonar-maven-plugin:2.3:sonar (default-cli) on project org.eclipse.emf.parsley.parent: Can not execute SonarQube analysis FWIW, as of sonar-maven-plugin 2.3.0, the Maven plugin embeds the sonar runner [1]. From what I can see, this makes the Bad Gateways goes away, as the sonar plugin doesn't talk to the SonarQube server as much. So it may be worth adding <plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>sonar-maven-plugin</artifactId> <version>2.3.1</version> </plugin> to your <pluginManagement> (this causes the Hudson sonar plugin to use version 2.3.1 rather than the default one, which for us is 2.1). However, at least for us at Code Recommenders the newer sonar-maven-plugin has then problems picking up the Java source files from our Tycho build. Can anyone else give upgrading the sonar-maven-plugin a try and report back here? [1] <http://jira.codehaus.org/browse/MSONAR-60> (In reply to Andreas Sewe from comment #17) Is there no way of upgrading the Hudson plugin directly in Hudson? I tried to see if there was updates and it seems 2.0.1 is the latest version at least as detected by Hudson's plugin manager anyway... (In reply to Thanh Ha from comment #18) > (In reply to Andreas Sewe from comment #17) > Is there no way of upgrading the Hudson plugin directly in Hudson? > > I tried to see if there was updates and it seems 2.0.1 is the latest version > at least as detected by Hudson's plugin manager anyway... I think there's some confusion: There is the *Hudson* plugin [1], whose latest version is 2.0.1, and there is the *Maven* plugin [2], whose latest version is 2.3.1. As far as I can tell, the two are quite independent from each other. The former simply invokes the latter with "mvn sonar:sonar" during the Hudson job's execution. As the Hudson plugin kicks off Maven using a "mvn" command line, the exact version of the sonar-maven-plugin used is determined by your POM. Hence, if your POM's <pluginManagement> specifies a different version of sonar-maven-plugin then that version is used. Hope that clarifies things. [1] <http://docs.codehaus.org/display/SONAR/Hudson+Plugin> [2] <http://mojo.codehaus.org/sonar-maven-plugin/> (In reply to Andreas Sewe from comment #19) > Hope that clarifies things. Ah, that clarifies it quite a bit thanks! I set this up for CBI [1] and it seems to be working but I don't use sonar much so I'm not sure what to look for. [1] https://hudson.eclipse.org/cbi/job/bug419172/lastBuild/console (In reply to Thanh Ha from comment #20) > (In reply to Andreas Sewe from comment #19) > > Hope that clarifies things. > > Ah, that clarifies it quite a bit thanks! > > I set this up for CBI [1] and it seems to be working but I don't use sonar > much so I'm not sure what to look for. > > [1] https://hudson.eclipse.org/cbi/job/bug419172/lastBuild/console Good to see that this works for you; hope that solves the Bad Gateway problem for you in the future. Unlike your build, however, ours is Tycho-based so you don't run into the same "cannot find source files" problem than we do; according to [1], the sonar-maven-plugin 2.3.1 picks up the sources just fine. It would thus be interesting to hear from other Tycho-using projects whether moving to sonar-maven-plugin 2.3.1 - fixes the Bad Gateway problem - fails to pick up the source files. (In reply to Andreas Sewe from comment #21) > It would thus be interesting to hear from other Tycho-using projects whether > moving to sonar-maven-plugin 2.3.1 > > - fixes the Bad Gateway problem > - fails to pick up the source files. Here is a PDT sonar build output [1] [2] based on bug_419172 branch [3] First [1] work correctly. Second [2] again Bad Gateway [1] - https://hudson.eclipse.org/pdt/job/bug_419172/1/console [2] - https://hudson.eclipse.org/pdt/job/bug_419172/2/console [3] - http://git.eclipse.org/c/pdt/org.eclipse.pdt.git/log/?h=bug_419172 Spent sometime this morning investigating this issue from the server side. At the Apache level (We're using proxypass) I am seeing logs such as the ones below when something fails. From my investigation it can fail at any stage so the file being retrieved by sonar plugin is different almost every time the failure happens. [Thu Jun 19 11:47:41 2014] [error] [client x.x.x.x] (104)Connection reset by peer: proxy: error reading status line from remote server x.x.x.x [Thu Jun 19 11:47:41 2014] [error] [client x.x.x.x] proxy: Error reading from remote server returned by /sonar/deploy/bootstrap/index.txt [Thu Jun 19 11:48:39 2014] [error] [client x.x.x.x] (104)Connection reset by peer: proxy: error reading status line from remote server x.x.x.x [Thu Jun 19 11:48:39 2014] [error] [client x.x.x.x] proxy: Error reading from remote server returned by /sonar/batch_bootstrap/properties Unfortunately on the Sonar end I can't see any useful logs, in fact sonar seems to log very little... So it's hard to tell what's happening. I've been running tests over and over using my test Sonar job [1] trying to find clues. It seems to fail at a pretty high rate in my opinion maybe 1/4 of jobs I run will fail (just an eyeball figure). Per bug 437780 I upgraded a few things but it didn't seem to make the Bad Gateway issue go away. I did notice during my runs that the Sonar Java process takes up 80% - 100% CPU time when the sonar plugin on Hudson connects to it. Maybe it just gets too busy sometimes? I'm trying to find if there's anything we can tune to make this better. [1] https://hudson.eclipse.org/cbi/job/bug419172/ Thanh Ha: Looks like this bug is similar to apache bug? [1] [1] - https://issues.apache.org/bugzilla/show_bug.cgi?id=37770 (In reply to Dawid Pakula from comment #25) > Thanh Ha: Looks like this bug is similar to apache bug? [1] > > [1] - https://issues.apache.org/bugzilla/show_bug.cgi?id=37770 Thanks Dawid, I had a look and tried the solution proposed in that bug but it does not seem to help. I am still getting the 502 errors even with the extra configuration. (In reply to Andreas Sewe from comment #17) > FWIW, as of sonar-maven-plugin 2.3.0, the Maven plugin embeds the sonar > runner [1]. From what I can see, this makes the Bad Gateways goes away, as > the sonar plugin doesn't talk to the SonarQube server as much. So it may be > worth adding > > <plugin> > <groupId>org.codehaus.mojo</groupId> > <artifactId>sonar-maven-plugin</artifactId> > <version>2.3.1</version> > </plugin> > > to your <pluginManagement> (this causes the Hudson sonar plugin to use > version 2.3.1 rather than the default one, which for us is 2.1). > > However, at least for us at Code Recommenders the newer sonar-maven-plugin > has then problems picking up the Java source files from our Tycho build. > > Can anyone else give upgrading the sonar-maven-plugin a try and report back > here? > > [1] <http://jira.codehaus.org/browse/MSONAR-60> Hi I tried to add that plugin section in pluginManagement, but the problem is still there: https://hudson.eclipse.org/emf-parsley/job/emf-parsley-sonar/19/console However, from looking at the previous run, version 2.3.1 was already being used https://hudson.eclipse.org/emf-parsley/job/emf-parsley-sonar/18/consoleFull or did I get something wrong? Thanh, I've just seen this http://stackoverflow.com/questions/21244235/jenkins-sonar-error-unable-to-evict-preview-database-batch-bootstrap-evic I don't know if that's strictly related, but it says: "We have not seen this exception in any of the sonar or jenkins logs before. So we changed the open file limit from 1024 (default on ubuntu) to 4096 and restartet Jenkins, Sonar and the PostgreSQL Server. Since then, all builds pass." The THyM - Sonar job also have this issue. (In reply to Lorenzo Bettini from comment #28) > I don't know if that's strictly related, but it says: > > "We have not seen this exception in any of the sonar or jenkins logs before. > So we changed the open file limit from 1024 (default on ubuntu) to 4096 and > restartet Jenkins, Sonar and the PostgreSQL Server. Since then, all builds > pass." Thanks Lorenzo, I don't think this is related but I tried it anyway and it didn't help. I'm still getting the issue. However the first comment to the one you linked mentioned something I over looked "This kind of helped identifying the root cause for my problem. I tried changing the open file limit and then also it failed after the first success. My Mac server had only 2G of RAM and it had mysql, memcached, jenkins and sonar running on the same machine. The builds were all successful after the RAM was increased to 8GB." So I took a look and found that the free ram on this server is pretty low: total used free shared buffers cached Mem: 4210736 4044052 166684 0 101700 2838404 -/+ buffers/cache: 1103948 3106788 Swap: 867472 14484 852988 On this server we're running both PostgreSQL and Sonar. Maybe the answer is we need more ram on this server. CDT is also affected. See https://hudson.eclipse.org/cdt/job/cdt-sonar/232/ (In reply to Thanh Ha from comment #30) > > On this server we're running both PostgreSQL and Sonar. Maybe the answer is > we need more ram on this server. Wouldn't it be the case to try and upgrade to Sonarqube 4.3.2? I personally run it on a very old machine with only 1Gb of RAM; this also hosts MySql and Jenkins. I tried to invoke sonar while there was also another Jenkins job running... it took time, but everything completed in the end. Moreover, together with sonar-maven-plugin 2.4, this also supports the -pl (--projects) advanced option so that it's much easier to analyze only the desired modules... (In reply to Lorenzo Bettini from comment #32) > (In reply to Thanh Ha from comment #30) > > > > On this server we're running both PostgreSQL and Sonar. Maybe the answer is > > we need more ram on this server. > > Wouldn't it be the case to try and upgrade to Sonarqube 4.3.2? > > I personally run it on a very old machine with only 1Gb of RAM; this also > hosts MySql and Jenkins. I tried to invoke sonar while there was also > another Jenkins job running... it took time, but everything completed in the > end. I guess this highly depends on the size and number of projects analyzed by SonarQube. The instance at Eclipse.org has become a very big one. Maybe MySQL is better than PostgreSQL for SonarQube, though. > Moreover, together with sonar-maven-plugin 2.4, this also supports the -pl > (--projects) advanced option so that it's much easier to analyze only the > desired modules... I wouldn't recommend making it a default. There is no reason why not analyzing a module (even a test): clean code has to be everywhere, if you start disabling module, then you're on a dangerous slope that will make that one day, you won't analyze anything ;) I've been trying to reproduce numerous times against the new Sonar instance that we are in the process of setting up per bug 438856. My test job: https://hudson.eclipse.org/cbi/job/bug419172/ I haven't been able to reproduce yet but on the other hand I have a lot less load than the existing sonar server and I am not using the ProxyPass through Apache. We probably won't know more until we make the switch to the new server. The new sonar instance is now the default Sonar instance since yesterday. I haven't been able to reproduce this problem since and all the HIPP instances have been updated to use this new instance now. Please report if anyone runs into this issue with the new instance. Thank you for making this happen. The bad gateway error seems to have disappeared. However, when navigating the instance I get this error from time to time: Proxy Error The proxy server received an invalid response from an upstream server. The proxy server could not handle the request GET /sonar/drilldown/measures/24730. Reason: Error reading from remote server A refresh or two get rid of it and eventually display the proper page. With the update to the latest LTS version, these errors seems to have disappeared. |