Bugzilla – Full Text Bug Listing
| Summary: | "Guice provision errors" when pushing via HTTPS | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Community | Reporter: | Matthias Sohn <matthias.sohn> | ||||||
| Component: | Gerrit | Assignee: | Eclipse Webmaster <webmaster> | ||||||
| Status: | RESOLVED FIXED | QA Contact: | |||||||
| Severity: | normal | ||||||||
| Priority: | P3 | CC: | alvaro.sanchez-leon, daniel_megert, donald.g.dunne, edwin.kempin, jan.sievers, lmcbout, marc.khouzam, marcel.bruch, michael.wenz, mober.at+eclipse, roberto.e.escobar, robin, sasa.zivkov, sebastien.dubois, sop, t-oberlies | ||||||
| Version: | unspecified | ||||||||
| Target Milestone: | --- | ||||||||
| Hardware: | All | ||||||||
| OS: | All | ||||||||
| Whiteboard: | |||||||||
| Attachments: |
|
||||||||
|
Description
Matthias Sohn
Same for me and also on other projects, e.g. https://git.eclipse.org/r/#/q/status:open+project:platform/eclipse.platform.team,n,z *** Bug 393746 has been marked as a duplicate of this bug. *** I suspect something is wrong with LDAP groups used to define committer permissions. I checked the permission definitions for several projects and they seem to be correct. Maybe caused by bug 377123 comment 19? What we see on the TCF project is that we cannot "push through" to master anymore since yesterday, could that be caused by the same issue: 190 mober@szg-qa-lx5~/git/org.eclipse.tcf$git push Counting objects: 11, done. Delta compression using up to 2 threads. Compressing objects: 100% (6/6), done. Writing objects: 100% (6/6), 700 bytes, done. Total 6 (delta 5), reused 0 (delta 0) remote: Resolving deltas: 0% (0/5) To ssh://moberhuber@git.eclipse.org:29418/tcf/org.eclipse.tcf.git ! [remote rejected] master -> master (prohibited by Gerrit) error: failed to push some refs to 'ssh://moberhuber@git.eclipse.org:29418/tcf/org.eclipse.tcf.git' This worked fine until yesterday. It's a blocker for our workflows since we need p2 repositories built by Hudson as artifcats to pass from team to team ... and Hudson won't build anything unless it's in the Eclipse git repo. (In reply to comment #5) > What we see on the TCF project is that we cannot "push through" to master > anymore since yesterday, could that be caused by the same issue: Not sure what you mean by "push through", but on the project mentioned in comment 1, while I can't push via Gerrit, I can still push directly to 'master'. Note that (unlike in other setups I know) Eclipse has both a native and a Gerrit server serving the same Git repository. So direct pushes to a branch may work via the native server, but probably don't work via the Gerrit server. (In reply to comment #6) > while I can't push via Gerrit, I can still push directly to 'master'. This doesn't work on my repos since yesterday, with the error message from comment 5. Same problem on org.eclipse.tm.git . (In reply to comment #7) > Note that (unlike in other setups I know) Eclipse has both a native and a > Gerrit server serving the same Git repository. Direct pushes to the native server were disabled for us when the project had Gerrit support enabled. See this E-Mail thread for instance: http://dev.eclipse.org/mhonarc/lists/tcf-dev/msg00271.html One difference I've noticed is that my repos are marked as "Owner: Code Review" whereas most others are marked as owned by a real person: http://git.eclipse.org/c/?q=tcf (In reply to comment #3) > I suspect something is wrong with LDAP groups used to define committer > permissions. I checked the permission definitions for several projects and > they seem to be correct. Yesterday, I upgraded the Gerrit sandbox. The sandbox does point to the "real" Git repos used by the production Gerrit. I guess something has changed in Gerrit in that groups now have an "ldap/" prefix in front of them. ldap/technology.tycho Instead of technology.tycho I believe Gerrit stores permissions within the Git repo itself. From here, I'll probably have to upgrade the production instance to 2.5 RC1 and disable the Gitweb option. Can you enable direct push to git while this is in progress ? These 2 projects / repositories are currently blocked completely: - tm/org.eclipse.tm.git - tcf/org.eclipse.tcf.git I couldn't find an easy way of undoing this, so I upgraded the live site to 2.5 RC1. Please let me know how it works out. Looks good to me. Push is working again for me to tcf/org.eclipse.tcf.git. Thanks! (In reply to comment #11) > I couldn't find an easy way of undoing this, so I upgraded the live site to > 2.5 RC1. Please let me know how it works out. while I can vote +2 again now, it seems that push via https is broken now (HTTP 500): $ git push -v --progress origin HEAD:refs/for/master Pushing to https://jsievers@git.eclipse.org/r/p/tycho/org.eclipse.tycho.extras Password for 'https://jsievers@git.eclipse.org': Counting objects: 34, done. Delta compression using up to 8 threads. Compressing objects: 100% (12/12), done. Writing objects: 100% (19/19), 2.67 KiB, done. Total 19 (delta 6), reused 0 (delta 0) POST git-receive-pack (2866 bytes) efrror: RPC failed; result=22, HTTP code = 500 atal: The remote end hung up unexpectedly fatal: The remote end hung up unexpectedly Everything up-to-date > https://jsievers@git.eclipse.org/r/p/tycho/org.eclipse.tycho.extras
Ick.
[2012-11-07 09:19:00,987] WARN org.eclipse.jetty.util.log : /r/p/tycho/org.eclipse.tycho.extras/git-receive-pack
com.google.inject.ProvisionException: Guice provision errors:
1) Error in custom provider, java.lang.IllegalStateException: Request has already been cleaned up
while locating com.google.gerrit.server.config.RequestScopedReviewDbProvider
at com.google.gerrit.server.config.GerritRequestModule.configure(GerritRequestModule.java:68)
while locating com.google.gerrit.reviewdb.server.ReviewDb
for parameter 0 at com.google.gerrit.server.git.ReceiveCommits.<init>(ReceiveCommits.java:308)
while locating com.google.gerrit.server.git.ReceiveCommits annotated with interface com.google.inject.assistedinject.Assisted
at com.google.gerrit.server.git.AsyncReceiveCommits.<init>(AsyncReceiveCommits.java:148)
while locating com.google.gerrit.server.git.AsyncReceiveCommits annotated with interface com.google.inject.assistedinject.Assisted
Caused by: java.lang.IllegalStateException: Request has already been cleaned up
(In reply to comment #11) > I couldn't find an easy way of undoing this, so I upgraded the live site to > 2.5 RC1. Please let me know how it works out. You need to upgrade to 2.5 which was released last weekend since this fixes a few problems which exist in 2.5 RC1 [1]. One of these fixes is fixing push over HTTP [2] :) I think the problem was caused by running two different Gerrit versions against the same repositories. Since Gerrit stores permissions and group definitions in Git and might change the way how it does that in newer versions this may lead to such problems. [1] https://groups.google.com/forum/#!topic/repo-discuss/QRqCyJbEmJo/discussion [2] https://gerrit.googlesource.com/gerrit/+/31f124e8ac837fb427c1b015ede96cb03bfc0a24 It seems to work again: I was able to submit a change via Gerrit to EGit. > You need to upgrade to 2.5 which was released last weekend since this fixes > a few problems which exist in 2.5 RC1 [1]. One of these fixes is fixing push > over HTTP [2] :) Would this be one of those problems? I am trying to push a new branch to origin = https://git.eclipse.org/r/p/osee/org.eclipse.osee.git. I get the following: $ git push origin 0.10.4 Counting objects: 65, done. Delta compression using up to 12 threads. Compressing objects: 100% (17/17), done. Writing objects: 100% (38/38), 5.77 KiB, done Total 38 (delta 15), reused 25 (delta 6) error: RPC failed; result=22, HTTP code = 500 fatal: The remote end hung up unexpectedly fatal: The remote end hung up unexpectedly (In reply to comment #17) > > You need to upgrade to 2.5 which was released last weekend since this fixes > > a few problems which exist in 2.5 RC1 [1]. One of these fixes is fixing push > > over HTTP [2] :) > > Would this be one of those problems? > > I am trying to push a new branch to origin = > https://git.eclipse.org/r/p/osee/org.eclipse.osee.git. I get the following: > > $ git push origin 0.10.4 > Counting objects: 65, done. > Delta compression using up to 12 threads. > Compressing objects: 100% (17/17), done. > Writing objects: 100% (38/38), 5.77 KiB, done > Total 38 (delta 15), reused 25 (delta 6) > error: RPC failed; result=22, HTTP code = 500 > fatal: The remote end hung up unexpectedly > fatal: The remote end hung up unexpectedly could you check the error_log, I think internals server error should log some details Same as comment 14: 2012-11-08 15:36:24,300] WARN org.eclipse.jetty.util.log : /r/p/osee/org.eclipse.osee.git/git-receive-pack com.google.inject.ProvisionException: Guice provision errors: 1) Error in custom provider, java.lang.IllegalStateException: Request has already been cleaned up while locating com.google.gerrit.server.config.RequestScopedReviewDbProvider at com.google.gerrit.server.config.GerritRequestModule.configure(GerritRequestModule.java:68) while locating com.google.gerrit.reviewdb.server.ReviewDb for parameter 0 at com.google.gerrit.server.git.ReceiveCommits.<init>(ReceiveCommits.java:287) while locating com.google.gerrit.server.git.ReceiveCommits annotated with interface com.google.inject.assistedinject.Assisted at com.google.gerrit.server.git.AsyncReceiveCommits.<init>(AsyncReceiveCommits.java:148) while locating com.google.gerrit.server.git.AsyncReceiveCommits annotated with interface com.google.inject.assistedinject.Assisted Caused by: java.lang.IllegalStateException: Request has already been cleaned up at com.google.gerrit.server.RequestCleanup.add(RequestCleanup.java:42) at com.google.gerrit.server.config.RequestScopedReviewDbProvider.get(RequestScopedReviewDbProvider.java:48) at com.google.gerrit.server.config.RequestScopedReviewDbProvider.get(RequestScopedReviewDbProvider.java:27) at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:55) at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031) at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) at com.google.inject.servlet.ServletScopes$1$1.get(ServletScopes.java:88) at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:40) at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:84) at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:254) at com.google.inject.internal.InjectorImpl$4$1.call(InjectorImpl.java:978) at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1031) (In reply to comment #15) > (In reply to comment #11) > > I couldn't find an easy way of undoing this, so I upgraded the live site to > > 2.5 RC1. Please let me know how it works out. > > You need to upgrade to 2.5 which was released last weekend since this fixes > a few problems which exist in 2.5 RC1 [1]. One of these fixes is fixing push > over HTTP [2] :) nope, obviously update to gerrit 2.5 did not solve my problem with HTTP 500 while trying to push via HTTPS. What is the httpd.listenUrl? What is httpd.requestLog? We have seen this bug before but its usually only showing up when Jetty is logging. In most installations you want to disable the Jetty log, as the frontend web server should be handling that. (In reply to comment #21) > What is the httpd.listenUrl? > What is httpd.requestLog? [httpd] listenUrl = proxy-https://(private IP removed):8080/r/ maxthreads = 300 requestLog = true So I've set the requestLog to false as we do proxy through httpd. (In reply to comment #22) > So I've set the requestLog to false as we do proxy through httpd. I just tried to push to https://git.eclipse.org/r/p/tycho/org.eclipse.tycho.extras.git using EGit, and I'm getting the same server response as in comment #19. On the console, it also fails: $ git push origin master:refs/for/master Password for 'https://toberlies@git.eclipse.org': Counting objects: 76, done. Delta compression using up to 8 threads. Compressing objects: 100% (40/40), done. Writing objects: 100% (41/41), 6.54 KiB, done. Total 41 (delta 26), reused 0 (delta 0) error: RPC failed; result=22, HTTP code = 500 fatal: The remote end hung up unexpectedly fatal: The remote end hung up unexpectedly Do we have an estimate for when this bug will be resolved? @Shaun: The Eclipse Gerrit still has the "Guice provision errors" problem (see comment #19). I was told that you are the expert for Guice in the Gerrit project. Could you please help analysing this problem? > Do we have an estimate for when this bug will be resolved?
If the issue is with Gerrit itself (which, I believe it is), then I cannot provide that answer. If this issue is blocking you (ie, you cannot push through SSH) then perhaps we can disable Gerrit for your project temporarily and allow you to push to https directly to the repo.
I apologize for the situation -- we're typically very conservative with our software updates. Unfortunately, I never expected that upgrading the sandbox would impact the live data this way. Lesson learned.
Just let me know if you'd like to push straight https and we'll make it happen.
Denis
> You need to upgrade to 2.5 which was released last weekend since this fixes
> a few problems which exist in 2.5 RC1 [1]. One of these fixes is fixing push
> over HTTP [2] :)
Actually, I just noticed this gem while re-reading though the comments.
I'll upgrade from 2.5 RC1 to 2.5 in the hopes this makes it better.
(In reply to comment #26) > > Do we have an estimate for when this bug will be resolved? > > If the issue is with Gerrit itself (which, I believe it is), then I cannot > provide that answer. If this issue is blocking you (ie, you cannot push > through SSH) then perhaps we can disable Gerrit for your project temporarily > and allow you to push to https directly to the repo. > > I apologize for the situation -- we're typically very conservative with our > software updates. Unfortunately, I never expected that upgrading the > sandbox would impact the live data this way. Lesson learned. > > Just let me know if you'd like to push straight https and we'll make it > happen. > > Denis Yes, please disable Gerrit for eclipse.org/osee and we'll attempt to push to see if that resolves our issues. We would definitely want to switch back as soon as Gerrit is back up and running. Thanks, Don(In reply to comment #27) > > You need to upgrade to 2.5 which was released last weekend since this fixes > > a few problems which exist in 2.5 RC1 [1]. One of these fixes is fixing push > > over HTTP [2] :) > > Actually, I just noticed this gem while re-reading though the comments. > > I'll upgrade from 2.5 RC1 to 2.5 in the hopes this makes it better. Great, let us know when that is completed and we'll try our push. Don eclipse.org/osee > Great, let us know when that is completed and we'll try our push. Looks like we're already running 2.5 on the production Gerrit, so it still seems to be affected. The OSEE repo is accessible from plain https, all you need to do is change your remote to point to https://git.eclipse.org/gitroot/osee/org.eclipse.osee.git (In reply to comment #29) > > Great, let us know when that is completed and we'll try our push. > > Looks like we're already running 2.5 on the production Gerrit, so it still > seems to be affected. > > The OSEE repo is accessible from plain https, all you need to do is change > your remote to point to > https://git.eclipse.org/gitroot/osee/org.eclipse.osee.git Doesn't work for me using https://ddunne@git.eclipse.org/gitroot/osee/org.eclipse.osee.git Verified user/pw by logging into portal at https://dev.eclipse.org/portal/myfoundation/portal/portal.php. Counting objects: 40, done. Delta compression using up to 4 threads. Compressing objects: 100% (11/11), done. Writing objects: 100% (21/21), 1.51 KiB, done. Total 21 (delta 10), reused 0 (delta 0) remote: Starting policy checks... remote: Finished checking policies remote: remote: error: git.eclipse.org does not know this committer, remote: or this this committer cannot commit to this project (gid=8000 repo:org.eclipse.osee): ddunne. remote: remote: denied: You (ddunne) cannot push changes that were not committed by you or members of your project. Please fix your repository see http://wiki.eclipse.org/Git for information on configuring your Git environment. remote: remote: error: hook declined to update refs/heads/dev The update hook looks at the group ownership to determine the committers. It was still owned by gerrit. Please try again. (In reply to comment #31) > The update hook looks at the group ownership to determine the committers. > It was still owned by gerrit. Please try again. That did it, thanks! Is there an estimate on when Gerritt will be back working? Don (In reply to comment #26) >Just let me know if you'd like to push straight https and we'll make it happen. @Denis: could you please disable Gerrit for gmp/org.eclipse.gmp.graphiti as well? I'm also getting the not authorized error Donald got. We would also want to switch back as soon as Gerrit is back up and running. @All: Please let's concentrate on fixing this issue here. If you have operation requests for temporary workarounds, please open new bugs. > (In reply to comment #26) > @Denis: could you please disable Gerrit for gmp/org.eclipse.gmp.graphiti as > well? Ownership has changed; you should be able to push directly to https://git.eclipse.org/gitroot/gmp/org.eclipse.gmp.graphiti.git if you update your remote. > @All: Please let's concentrate on fixing this issue here. I'm sure the Gerrit team is tracking this issue on their own bug tracker. (In reply to comment #35) > I'm sure the Gerrit team is tracking this issue on their own bug tracker. I don't care about the Gerrit project. The eclipse.org infrastructure is broken, and I expect someone from the Foundation to feel responsible for fixing this problem. judging from the stacktrace in the server logs, this seems to be gerrit issue http://code.google.com/p/gerrit/issues/detail?id=1408 (In reply to comment #36) > (In reply to comment #35) >. The eclipse.org infrastructure is broken, Ok, at this time the only fix seems to be removing your project from Gerrit. If that's what you'd like I can certainly do that. -M. (In reply to comment #36) > (In reply to comment #35) > > I'm sure the Gerrit team is tracking this issue on their own bug tracker. > I don't care about the Gerrit project. The eclipse.org infrastructure is > broken, and I expect someone from the Foundation to feel responsible for > fixing this problem. The tone of your message is very unfortunate. So is this Gerrit bug. However, at this time, I cannot control either. I've already assumed responsibility in comment 26 for the eclipse.org infrastructure part of the issue and I've already offered numerous workarounds: - you have the option of pushing to Gerrit via SSH, which is by far the method we prefer . If your corporate firewall prohibits that, open a ticket with your local IT team to have the port opened. In the end, that is the best solution. - if you're behind certain types of proxies, you push via SSH to proxy.eclipse.org:443 I have heard from other SAP members that this works wonders. - you can bypass Gerrit and push to https directly if you must use https I'll once again apologize for the situation. However, the Gerrit team has always been quite responsive, so I'm sure that with a bit of patience they'll get this resolved and we'll all be back in our happy places again. (In reply to comment #39) > The tone of your message is very unfortunate. So is this Gerrit bug. However, > at this time, I cannot control either. Sorry for the rude comment. To me it was not clear that the Eclipse Gerrit problem is really a Gerrit bug - it originally (comment #21) sounded rather like a configuration issue. Also, there was no visible progress here (and until Jan's comment #37 no pointers to a potential root cause), so I got the impression that no-one is working on resolving the https problem. I'll look into how I can bypass our corporate firewall, but right now the only solution seems to be to carry the bits outside the network. I need to use Gerrit, because this is where we work on contributions. This is a tricky problem involving multiple threads. We failed to reproduce the problem here (using a local Gerrit 2.5 test instance) so we think the best way to gain more insight would be to deploy a patched Gerrit version which adds some extra tracing for this problem. Sasa has created a new version of Gerrit 2.5 which captures additional thread information about the thread interfering with the failing cleanup which will be logged in case this exception is thrown. He'll upload the patch to Gerrit's Gerrit server soon so that you can have a look at the patch. I have uploaded this Gerrit version to build.eclipse.org:/home/data/users/msohn/gerrit-full-2.5-1-g42ded0e.war So if you could update to this version we could retry the failing push over HTTPS and hopefully the additional trace will help to identify the cause of the problem. As soon as this trace is available you can downgrade to the original 2.5 version. This is the patch mentioned by Matthias: https://gerrit-review.googlesource.com/39684 I've installed the patched version on the production site. I'll create a bogus repo and push via https and report back. Created attachment 223919 [details]
Trace
Here's what I'm seeing in the error_log file when I attempt to push something.
Thanks for providing us the stack trace! From this stack trace we see which thread cleaned up the request but we don't see why. Therefore, we are providing another custom build based on this change [1] in hope to get more insights into the issue. Matthias will upload the custom build as he did before. Please deploy this new custom build, reproduce the issue and provide us with the both entries from the error log (one will contains '====' in the message and the other will contain '----' in the message). [1] https://gerrit-review.googlesource.com/39684 I have uploaded this Gerrit version to build.eclipse.org:/home/data/users/msohn/gerrit-full-2.5-1-g9bb651d.war please follow Sasa's instructions and provide a new trace Created attachment 224025 [details]
New trace
Here is the new trace. I only performed a single https push, so I attached the complete trace in case the obvious repetition could provide additional clues.
Can it be that for some reason we have the RequestContextFilter twice in the chain of filters? The inner RequestContextFilter succeeds and cleans up the request in its finally block and then the finally block of the outer RequestContextFilter fails with the 'Request has already been cleaned up' exception? I finally found out why some Gerrit users have this issue and some not. In short: - pushing to http://hostname/my/project works - pushing to http://hostname/p/my/project fails (note the "/p" in the path) For those having the issue: please try pushing again and remove the "/p" from the path. More details in this Gerrit discussion thread: https://groups.google.com/d/topic/repo-discuss/OP6hg1KYah8/discussion (In reply to comment #49) > For those having the issue: please try pushing again and remove the "/p" > from the path. thanks, this works :-), I was able to push change [1] over https from EGit using the https URL without /p, I did the EGit equivalent of git push https://git.eclipse.org/r/egit/egit.git HEAD:refs/for/master [1] https://git.eclipse.org/r/#/c/8925/ pushed https://git.eclipse.org/r/#/c/8926/ for review to teach EGit not to use /p for Gerrit http URLs anymore (In reply to comment #49) > For those having the issue: please try pushing again and remove the "/p" > from the path. great. I just verified this works for https://git.eclipse.org/r/tycho/org.eclipse.tycho Thanks! reducing priority as push over https can be done with the alternate URL (In reply to comment #49) > For those having the issue: please try pushing again and remove the "/p" > from the path. Works for CDT. Thanks a lot! The problem is now fixed with Gerrit 2.5.1: http://gerrit-documentation.googlecode.com/svn/ReleaseNotes/ReleaseNotes-2.5.1.html (In reply to comment #55) > The problem is now fixed with Gerrit 2.5.1: > > http://gerrit-documentation.googlecode.com/svn/ReleaseNotes/ReleaseNotes-2.5. > 1.html @webmaster: could you upgrade to that fix version ? > @webmaster: could you upgrade to that fix version ?
I'm afraid of the collateral damage :)
I'll look into this early next year.
We're fixed here. |