Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 568904

Summary: Gerrit / jgit protocol version 2 broken since Gerrit 3.2.5
Product: Community Reporter: Jonah Graham <jonah>
Component: GerritAssignee: Eclipse Webmaster <webmaster>
Status: RESOLVED FIXED QA Contact:
Severity: major    
Priority: P2 CC: alex.blewitt, Arthur.vanDorp, d.ostrovsky, denis.roy, frederic.gurr, jonah, luca.milanesio, matthias.sohn, mikael.barbero, twolf, webmaster
Version: unspecified   
Target Milestone: ---   
Hardware: PC   
OS: Linux   
See Also: https://bugs.eclipse.org/bugs/show_bug.cgi?id=568950
Whiteboard:
Attachments:
Description Flags
git fetch logs none

Description Jonah Graham CLA 2020-11-17 15:10:46 EST
CDT's gerrit builds on CI are failing with the following error:

15:06:35  + git fetch --no-tags --force --progress -- https://git.eclipse.org/r/cdt/org.eclipse.cdt.git +refs/heads/*:refs/remotes/origin/*
15:06:35  fatal: expected 'packfile', received 'acknowledgments'

(from https://ci.eclipse.org/cdt/job/cdt-verify-test-cdt-ui-only-pipeline/2663/console)

A quick google search gave me no clues.

I have done this for two different gerrits, here is the output from another one:

https://ci.eclipse.org/cdt/job/cdt-verify-test-cdt-ui-only-pipeline/2664/console
Comment 1 Jonah Graham CLA 2020-11-17 15:11:29 EST
I tried a change on the simrel project and it worked https://ci.eclipse.org/simrel/job/simrel.runaggregator.VALIDATE.gerrit/1824/console
Comment 2 Jonah Graham CLA 2020-11-17 20:01:41 EST
I didn't dig into this enough before. It looks like the problem is just on the Git side:

$ git clone https://git.eclipse.org/r/cdt/org.eclipse.cdt.git
Cloning into 'org.eclipse.cdt'...
fatal: expected 'packfile', received 'acknowledgments'

$ git clone "ssh://jograham@git.eclipse.org:29418/cdt/org.eclipse.cdt"
Cloning into 'org.eclipse.cdt'...
fatal: internal server error

# this fetch done in a CDT repo I already had cloned:
$ git fetch origin 
fatal: internal server error
fatal: unable to write request to remote: Broken pipe

I can see a similar problem for CDT's website:

$ git clone "ssh://jograham@git.eclipse.org:29418/www.eclipse.org/cdt" 
Cloning into 'cdt'...
fatal: internal server error
fatal: remote error: unknown command peel


# this fetch done in a CDT repo I already had cloned:
$ git fetch
From ssh://git.eclipse.org:29418/www.eclipse.org/cdt
   522ac15..3ecb5b8  master     -> origin/master
fatal: internal server error
Comment 3 Jonah Graham CLA 2020-11-17 20:05:39 EST
The git protocol URL for CDT is working though: git://git.eclipse.org/gitroot/cdt/org.eclipse.cdt.git
Comment 4 Alex Blewitt CLA 2020-11-18 03:43:35 EST
It looks like the SSH daemon is down.

ssh -p 29148 ablewitt@git.eclipse.org
ssh: connect to host git.eclipse.org port 29148: Operation timed out

There was a message yesterday:

https://twitter.com/Eclipse_Status/status/1328776033693171712

According to the status page the maintenance has finished:

https://status.eclipse.org/incidents/3tn40vgwr845?u=yvg8t8k2dlhm

However, the sshd part doesn't seem to be working, even though the UI is.
Comment 5 Denis Roy CLA 2020-11-18 08:28:05 EST
> ssh -p 29148 ablewitt@git.eclipse.org
> ssh: connect to host git.eclipse.org port 29148: Operation timed out

It is on port 29418 ;) 


(In reply to Jonah Graham from comment #2)
> I didn't dig into this enough before. It looks like the problem is just on
> the Git side:
> $ git clone "ssh://jograham@git.eclipse.org:29418/www.eclipse.org/cdt" 
> Cloning into 'cdt'...
> fatal: internal server error
> fatal: remote error: unknown command peel


I'm seeing a few of those in the Gerrit logs:

[2020-11-18T08:02:00.806-0500] [SSH git-upload-pack /www.eclipse.org/cognicrypt.git ] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error  during git-upload-pack '/www.eclipse.org/cognicrypt.git'
org.eclipse.jgit.transport.UploadPackInternalServerErrorException
        at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:771)
        at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:101)
        at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:108)
        at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:32)
        at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:73)
        at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:473)
        at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:110)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:610)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
        at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: org.eclipse.jgit.errors.PackProtocolException: unknown command peel
        at org.eclipse.jgit.transport.UploadPack.serveOneCommandV2(UploadPack.java:1282)
        at org.eclipse.jgit.transport.UploadPack.serviceV2(UploadPack.java:1318)
        at org.eclipse.jgit.transport.UploadPack.uploadWithExceptionPropagation(UploadPack.java:832)
        at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:748)
Comment 6 Denis Roy CLA 2020-11-18 08:52:40 EST
I've filed https://bugs.chromium.org/p/gerrit/issues/detail?id=13694
Comment 7 Jonah Graham CLA 2020-11-18 10:32:55 EST
As reported on cross-project-issues-dev forcing protocol to version 1 makes stuff work:

$ git fetch origin 
fatal: internal server error
fatal: unable to write request to remote: Broken pipe

$ git config --global protocol.version 1

$ git fetch origin 
remote: Counting objects: 16023, done
remote: Finding sources: 100% (9/9)
remote: Total 9 (delta 0), reused 7 (delta 0)
Unpacking objects: 100% (9/9), 5.60 KiB | 819.00 KiB/s, done.
From ssh://git.eclipse.org:29418/cdt/org.eclipse.cdt
   9a70f31c4e..dc5b05442e  master     -> origin/master
Comment 8 Jonah Graham CLA 2020-11-18 10:52:10 EST
I rolled out the git protocol version change on CDT's infra: https://github.com/eclipse-cdt/cdt-infra/commit/49d90a022d32cc7b6c367aa3c903aa358a9c8e9e and it looks like things are moving.

Thanks @Arthur for providing the workaround!

(In reply to Denis Roy from comment #6)
> I've filed https://bugs.chromium.org/p/gerrit/issues/detail?id=13694
I commented on here about the workaround.


I have lowered the severity because CDT project is no longer blocked. I suspect many other contributors will be blocked until they discover this workaround. @Denis I leave it to you if there is more communication on this needed to the wider community.
Comment 9 Thomas Wolf CLA 2020-11-18 10:58:49 EST
(In reply to Jonah Graham from comment #7)
> As reported on cross-project-issues-dev forcing protocol to version 1 makes
> stuff work:

So it's definitely a git protocol V2 bug. Question is -- in C git or in JGit?
One thing is clear: "peel" is indeed not a valid command.

Could you run this with as in

  GIT_TRACE_PACKET=1 git -c protocol.version=2 fetch

If possible twice, once each for https and ssh. And then of course a comparison with the same done from the same repo at the Github mirror would be very helpful.
Comment 10 Jonah Graham CLA 2020-11-18 11:03:44 EST
(In reply to Thomas Wolf from comment #9)
> (In reply to Jonah Graham from comment #7)
> > As reported on cross-project-issues-dev forcing protocol to version 1 makes
> > stuff work:
> 
> So it's definitely a git protocol V2 bug. Question is -- in C git or in JGit?
> One thing is clear: "peel" is indeed not a valid command.
> 
> Could you run this with as in
> 
>   GIT_TRACE_PACKET=1 git -c protocol.version=2 fetch
> 
> If possible twice, once each for https and ssh. And then of course a
> comparison with the same done from the same repo at the Github mirror would
> be very helpful.

OK - the logs are very long. Is there anything private in them? AFAICT there isn't but I want to know if I should be censoring anything before uploading.
Comment 11 Arthur van Dorp CLA 2020-11-18 11:06:46 EST
> So it's definitely a git protocol V2 bug. Question is -- in C git or in JGit?

In the Scout project we see the problem with the stock git-for-windows client. Only those few that are older than protocol v2 enablement (older than approx. March 2020) don't have a problem. For whatever reason users using Linux or Windows subsystem for Linux don't seem to have a problem (opposed to what you are seeing on some jenkins jobs).
Comment 12 Thomas Wolf CLA 2020-11-18 11:09:04 EST
Not as far as I know of; unless you consider branch or tag names sensitive.
Comment 13 Thomas Wolf CLA 2020-11-18 11:10:25 EST
Sorry, forgot the reply button. Here's comment 12 again as it was intended:

(In reply to Jonah Graham from comment #10)
> OK - the logs are very long. Is there anything private in them? AFAICT there
> isn't but I want to know if I should be censoring anything before uploading.

Not as far as I know of; unless you consider branch or tag names sensitive.
Comment 14 Jonah Graham CLA 2020-11-18 11:15:42 EST
Created attachment 284808 [details]
git fetch logs

Please find attached the logs. 

GIT_TRACE_PACKET=1 git -c protocol.version=2 fetch https://jograham@git.eclipse.org/r/a/cdt/org.eclipse.cdt >&https-gerrit.log
GIT_TRACE_PACKET=1 git -c protocol.version=2 fetch git@github.com:eclipse-cdt/cdt.git >&git-github.log
GIT_TRACE_PACKET=1 git -c protocol.version=2 fetch https://github.com/eclipse-cdt/cdt.git >&https-github.log
GIT_TRACE_PACKET=1 git -c protocol.version=2 fetch  "ssh://jograham@git.eclipse.org:29418/cdt/org.eclipse.cdt" >&ssh-gerrit.log

Only the last one has the internal error.
Comment 15 Thomas Wolf CLA 2020-11-18 11:41:52 EST
(In reply to Jonah Graham from comment #14)
> Only the last one has the internal error.

I don't see any problem? The last one also appears to have set FETCH_HEAD successfully? Of course, your clone is up-to-date now. Might be more interesting when there's actually something to fetch.
Comment 16 Arthur van Dorp CLA 2020-11-18 11:43:59 EST
(In reply to Thomas Wolf from comment #15)
> (In reply to Jonah Graham from comment #14)
> Might be more
> interesting when there's actually something to fetch.
Fetching or pulling when you're already up to date still gives the error.
Comment 17 Jonah Graham CLA 2020-11-18 11:45:44 EST
(In reply to Thomas Wolf from comment #15)
> (In reply to Jonah Graham from comment #14)
> > Only the last one has the internal error.
> 
> I don't see any problem? The last one also appears to have set FETCH_HEAD
> successfully? Of course, your clone is up-to-date now. Might be more
> interesting when there's actually something to fetch.

There is a server error in the log in the middle somewhere:

$ grep error *.log
ssh-gerrit.log:fatal: internal server error
Comment 18 Denis Roy CLA 2020-11-18 12:28:59 EST
cat etc/jgit.config 
[protocol]
      version = 2


I can disable protocol version 2 on the server, which would require a restart.  I'm thinking it's worth it.
Comment 19 Thomas Wolf CLA 2020-11-18 12:30:37 EST
(In reply to Jonah Graham from comment #17)
> There is a server error in the log in the middle somewhere:
> 
> $ grep error *.log
> ssh-gerrit.log:fatal: internal server error

The ssh-gerrit.log has at the beginning:

11:12:53.021240 pkt-line.c:80           packet:        fetch> command=ls-refs
11:12:53.021262 pkt-line.c:80           packet:        fetch> 0001
11:12:53.021280 pkt-line.c:80           packet:        fetch> peel
11:12:53.021297 pkt-line.c:80           packet:        fetch> symrefs
11:12:53.021313 pkt-line.c:80           packet:        fetch> 0000

This is correct. It's C git telling the server to run ls-ref, no server options or agent, a 0001 delimiter, and then C git tells the server to peel annotated tags and show the underlying refs for symbolic references like HEAD. The 0000 terminates the command request.

Next ls-ref returns results:

11:12:53.112581 pkt-line.c:80           packet:        fetch< dc5b05442eddf3e53dcdc3dfa3dd11a2bb690b2a HEAD

Oops. Gerrit did *not* show the symbolic ref.

11:12:53.112658 pkt-line.c:80           packet:        fetch< 4c594b1ae7919a172d8b2fb3b5ba50f84cf5e0a3 refs/cache-automerge/53/ba4006b5493c6563bedcc051c7ec03257d55f2

The rest is OK, except that it also does not peel annotated tags. (And of course that error reported in the middle...)

Conclusion: for some reason, Gerrit/JGit did not parse the parameters for ls-ref (peel and symrefs) but considered the command complete after the 0001. In the next round of its command loop on that ssh connection, it then finds "peel", and complains because that is not a valid git command.

So the real question is why did Gerrit stop parsing the ls-refs command at 0001? In the JGit code I don't see how that could happen...
Comment 20 Denis Roy CLA 2020-11-18 12:30:46 EST
Gerrit has been restarted with protocol version 1.  Can you all try resetting your git client config to see if this still fails?

I suspect this could be a bug with jGit.
Comment 21 Jonah Graham CLA 2020-11-18 12:34:16 EST
(In reply to Denis Roy from comment #20)
> Gerrit has been restarted with protocol version 1.  Can you all try
> resetting your git client config to see if this still fails?
> 
> I suspect this could be a bug with jGit.

git -c protocol.version=2 fetch

works.

and once I remove the protocol.version from my various config files things are working too.

Thanks for implementing a workaround for everyone on the server.
Comment 22 Arthur van Dorp CLA 2020-11-18 12:42:09 EST
(In reply to Denis Roy from comment #20)
> Gerrit has been restarted with protocol version 1.  Can you all try
> resetting your git client config to see if this still fails?

Works. Thanks!
Comment 23 Denis Roy CLA 2020-11-18 14:12:55 EST
Many thanks to all of you for your troubleshooting, which has led to a workaround.
Comment 24 Thomas Wolf CLA 2020-11-19 02:04:06 EST
(In reply to Denis Roy from comment #23)
> Many thanks to all of you for your troubleshooting, which has led to a
> workaround.

One question, though: what is Gerrit version 3.2.5?

According to https://www.gerritcodereview.com/ the latest release is 3.2.3.

Three, not five.

There are no release notes. 3.2.4 is in the works. There are no tags v3.2.4 or v.3.2.5 in the Gerrit git repository.

Only at https://git.eclipse.org/r/Documentation/index.html#_resources , when you click on the "Downloads" link, you get to https://gerrit-releases.storage.googleapis.com/index.html where there is indeed a 3.2.4 and a 3.2.5 war file.

Is this even an official release?
Comment 25 Thomas Wolf CLA 2020-11-19 03:39:35 EST
(In reply to Thomas Wolf from comment #24)
> (In reply to Denis Roy from comment #23)
> > Many thanks to all of you for your troubleshooting, which has led to a
> > workaround.

@Denis: what Java version is this Gerrit running on? What are the JVM arguments?

The "peel" error can basically occur only if the following condition is false:

  new StringBuilder(0).toString() != new StringBuilder(0).toString()

JGit relies on that returning two different objects. If that returns twice the same object, JGit would consider the "0001" and "0000" lines identical, which could explain why it stopped parsing too early.
Comment 26 Luca Milanesio CLA 2020-11-19 03:46:48 EST
> Is this even an official release?

Yes, see [1]. I haven't linked the release yet to the homepage because there is a known issue on Java 8 that we need to fix.

With regards to Git Protocol v2, we have passed the integration and E2E tests before releasing. It would be interesting to know what broke in the Eclipse's setup for the Git Protocol v2.

Thanks for investigating.

Luca.

[1] https://groups.google.com/g/repo-discuss/c/CBWTPpvjq6k
Comment 27 Mikaël Barbero CLA 2020-11-19 03:48:37 EST
(In reply to Thomas Wolf from comment #25)
> (In reply to Thomas Wolf from comment #24)
> > (In reply to Denis Roy from comment #23)
> > > Many thanks to all of you for your troubleshooting, which has led to a
> > > workaround.
> 
> @Denis: what Java version is this Gerrit running on? What are the JVM
> arguments?
> 
> The "peel" error can basically occur only if the following condition is
> false:
> 
>   new StringBuilder(0).toString() != new StringBuilder(0).toString()
> 
> JGit relies on that returning two different objects. If that returns twice
> the same object, JGit would consider the "0001" and "0000" lines identical,
> which could explain why it stopped parsing too early.

Here are the info:

GerritCodeReview -Dflogger.backend_factory=com.google.common.flogger.backend.log4j.Log4jBackendFactory#getInstance -Dflogger.logging_context=com.google.gerrit.server.logging.LoggingContext#getInstance -Xmx16g -jar <path/to>/bin/gerrit.war daemon -d <gerrit/home> --run-id=<id>

java is:

openjdk version "15.0.1" 2020-10-20
OpenJDK Runtime Environment (build 15.0.1+9-18)
OpenJDK 64-Bit Server VM (build 15.0.1+9-18, mixed mode, sharing)
Comment 28 Luca Milanesio CLA 2020-11-19 03:49:55 EST
> openjdk version "15.0.1" 2020-10-20

I never tested Gerrit on Java 15: can you reproduce the issue with Java 8 or Java 11?
Comment 29 Mikaël Barbero CLA 2020-11-19 03:55:08 EST
(In reply to Luca Milanesio from comment #28)
> > openjdk version "15.0.1" 2020-10-20
> 
> I never tested Gerrit on Java 15: can you reproduce the issue with Java 8 or
> Java 11?

I'll leave that to Denis. I don't know how Java 15 ended up on the machine.
Comment 30 Thomas Wolf CLA 2020-11-19 05:58:43 EST
(In reply to Luca Milanesio from comment #28)
> > openjdk version "15.0.1" 2020-10-20
> 
> I never tested Gerrit on Java 15: can you reproduce the issue with Java 8 or
> Java 11?

All right, that's exactly the problem:

thomas@localhost:~/tmp/javatest$ more ot.java
public class Ot {

  private static final String END = new StringBuilder(0).toString();

  private static final String DELIM = new StringBuilder(0).toString();

  public static void main(String... args) {
    System.out.println("END == DELIM = "+ Boolean.toString(END == DELIM));
  }

}
thomas@localhost:~/tmp/javatest$ java -version
openjdk version "11.0.1" 2018-10-16 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.1+13-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.1+13-LTS, mixed mode, sharing)
thomas@localhost:~/tmp/javatest$ java ot.java
END == DELIM = false
thomas@localhost:~/tmp/javatest$ ../jdk15/jdk-15.0.1+9/bin/java --version
openjdk 15.0.1 2020-10-20
OpenJDK Runtime Environment AdoptOpenJDK (build 15.0.1+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 15.0.1+9, mixed mode, sharing)
thomas@localhost:~/tmp/javatest$ ../jdk15/jdk-15.0.1+9/bin/java ot.java
END == DELIM = true
thomas@localhost:~/tmp/javatest$

So with Java 15, a basic assumption of JGit's PacketLineIn is violated.
Comment 31 David Ostrovsky CLA 2020-11-19 06:45:36 EST
(In reply to Thomas Wolf from comment #30)
> (In reply to Luca Milanesio from comment #28)
> > > openjdk version "15.0.1" 2020-10-20
> > 
> > I never tested Gerrit on Java 15: can you reproduce the issue with Java 8 or
> > Java 11?
> 
> All right, that's exactly the problem:
> 
> thomas@localhost:~/tmp/javatest$ more ot.java
> public class Ot {
> 
>   private static final String END = new StringBuilder(0).toString();
> 
>   private static final String DELIM = new StringBuilder(0).toString();
> 
>   public static void main(String... args) {
>     System.out.println("END == DELIM = "+ Boolean.toString(END == DELIM));
>   }
> 
> }
> thomas@localhost:~/tmp/javatest$ java -version
> openjdk version "11.0.1" 2018-10-16 LTS
> OpenJDK Runtime Environment 18.9 (build 11.0.1+13-LTS)
> OpenJDK 64-Bit Server VM 18.9 (build 11.0.1+13-LTS, mixed mode, sharing)
> thomas@localhost:~/tmp/javatest$ java ot.java
> END == DELIM = false
> thomas@localhost:~/tmp/javatest$ ../jdk15/jdk-15.0.1+9/bin/java --version
> openjdk 15.0.1 2020-10-20
> OpenJDK Runtime Environment AdoptOpenJDK (build 15.0.1+9)
> OpenJDK 64-Bit Server VM AdoptOpenJDK (build 15.0.1+9, mixed mode, sharing)
> thomas@localhost:~/tmp/javatest$ ../jdk15/jdk-15.0.1+9/bin/java ot.java
> END == DELIM = true
> thomas@localhost:~/tmp/javatest$
> 
> So with Java 15, a basic assumption of JGit's PacketLineIn is violated.

Thanks. I also found out, that Git protocol v2 integration test
is broken in gerrit on JDK 15: [1] on Gerrit CI, with exactly the
same stack trace as reported here.

I even filed this JGit issue: https://bugs.eclipse.org/bugs/show_bug.cgi?id=568950.

[1] https://gerrit-review.googlesource.com/c/gerrit/+/269382
Comment 32 Denis Roy CLA 2020-11-19 08:30:37 EST
> > I never tested Gerrit on Java 15: can you reproduce the issue with Java 8 or
> > Java 11?
> 
> I'll leave that to Denis. I don't know how Java 15 ended up on the machine.

init would not start with the Java 8 I was using for 3.2.2

Exception in thread "main" java.lang.NoSuchMethodError: java.nio.ByteBuffer.mark()Ljava/nio/ByteBuffer;
        at org.eclipse.jgit.util.RawParseUtils.decodeNoFallback(RawParseUtils.java:1140)
        at org.eclipse.jgit.util.RawParseUtils.decode(RawParseUtils.java:1106)
        at org.eclipse.jgit.util.RawParseUtils.decode(RawParseUtils.java:1065)
        at org.eclipse.jgit.util.RawParseUtils.decode(RawParseUtils.java:1044)
        at org.eclipse.jgit.storage.file.FileBasedConfig.load(FileBasedConfig.java:142)
        at com.google.gerrit.server.config.GerritServerConfigModule.getSecureStoreFromGerritConfig(GerritServerConfigModule.java:61)
        at com.google.gerrit.server.config.GerritServerConfigModule.getSecureStoreClassName(GerritServerConfigModule.java:37)
        at com.google.gerrit.pgm.util.SiteProgram.getConfiguredSecureStoreClass(SiteProgram.java:163)
        at com.google.gerrit.pgm.init.BaseInit.createSiteInit(BaseInit.java:228)
        at com.google.gerrit.pgm.init.BaseInit.run(BaseInit.java:100)
        at com.google.gerrit.pgm.util.AbstractProgram.main(AbstractProgram.java:61)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.google.gerrit.launcher.GerritLauncher.invokeProgram(GerritLauncher.java:224)
        at com.google.gerrit.launcher.GerritLauncher.mainImpl(GerritLauncher.java:120)
        at com.google.gerrit.launcher.GerritLauncher.main(GerritLauncher.java:65)


Assuming my Java was too old, I pulled the latest Java.

Java 15 did show me this warning:

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.google.gerrit.common.IoUtil (file:/home/data/users/gerrit/.gerritcodereview/tmp/gerrit_7643044267235376686_app/com_google_gerrit_common_libserver.jar) to method java.net.URLClassLoader.addURL(java.net.URL)
WARNING: Please consider reporting this to the maintainers of com.google.gerrit.common.IoUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release


Which I chose to ignore.
Comment 33 Thomas Wolf CLA 2020-11-19 12:07:08 EST
Anyway: the protocol V2 problem with Gerrit running on Java 15 was indeed a JGit bug, fixed now in bug 568950.
Comment 34 Matthias Sohn CLA 2020-11-21 05:12:49 EST
(In reply to Denis Roy from comment #32)
> > > I never tested Gerrit on Java 15: can you reproduce the issue with Java 8 or
> > > Java 11?
> > 
> > I'll leave that to Denis. I don't know how Java 15 ended up on the machine.
> 
> init would not start with the Java 8 I was using for 3.2.2

This was caused by another issue in the build chain used to release 3.2.5 which had the consequence the Java classes where built with Java 11 and can only run on Java 11 :-(. This was fixed in 3.2.5.1  Hence if you upgrade to 3.2.5.1 [1] Gerrit can run on both Java 8 and Java 11 and also protocol v2 should work properly on Java 8 and 11.

Thomas' JGit patch for the problem on Java 15 was downported to the JGit version used by Gerrit 3.2.x. The next 3.2.x release will contain this fix so that protocol v2 will also work on Java 15.

[1] https://www.gerritcodereview.com/3.2.html#3251
Comment 35 David Ostrovsky CLA 2020-11-21 05:49:10 EST
(In reply to Matthias Sohn from comment #34)
> (In reply to Denis Roy from comment #32)
> > > > I never tested Gerrit on Java 15: can you reproduce the issue with Java 8 or
> > > > Java 11?
> > > 
> > > I'll leave that to Denis. I don't know how Java 15 ended up on the machine.
> > 
> > init would not start with the Java 8 I was using for 3.2.2
> 
> This was caused by another issue in the build chain used to release 3.2.5
> which had the consequence the Java classes where built with Java 11 and can
> only run on Java 11 :-(. This was fixed in 3.2.5.1  Hence if you upgrade to
> 3.2.5.1 [1] Gerrit can run on both Java 8 and Java 11 and also protocol v2
> should work properly on Java 8 and 11.
> 
> Thomas' JGit patch for the problem on Java 15 was downported to the JGit
> version used by Gerrit 3.2.x. The next 3.2.x release will contain this fix
> so that protocol v2 will also work on Java 15.

Gerrit is now fixed on stable-3.2 branch to include Thomas' JGit ptach to fix git wire protocol v2 on JDK 15.

@Dennis, could you please update Gerrit version on Eclipse site, with 3.2.5.1, then you would have to use Java 8 or Java 11 and then you could re-enable git wire protocol v2 on Eclipse site. Note, that the official release 3.2.5.1 isn't work yet against JDK 15. git wire protocol is broken, as you noticed. 

Alternatively, I conducted intermediate Gerrit release: gerrit-v3.2.5.1-60-gbb9ea229fb: [1] on stable-3.2 branch with this patch included [2], basically release 3.2.5.1 + Thomas's JGit fix: [3], that Matthias cherry-picked to the right JGit branch 5.8 used on gerrit stable-3.2 branch. With that fix release, you could continue to use JDK 15 and git wire protocol v2 should just work. (Gerrit integration tests for Git wire protocol v2 on JDK 15 are green now with Thomas' JGit fix for JDK 15.)

There are ongoing efforts to extend Git wire protocol v2 support in JGit to client-side protocol v2 support for fetching: [4], so it would be really helpful if git wire protocol v2 would be re-enabled again on Eclipse site, so that EGit earlier adopter could give it a try using nightly JGit/EGit build artifacts.

[1] https://github.com/davido/gerrit/releases/tag/gerrit-v3.2.5.1-60-gbb9ea229fb
[2] https://gerrit-review.googlesource.com/c/gerrit/+/289430
[3] https://git.eclipse.org/r/c/jgit/jgit/+/172601
[4] https://git.eclipse.org/r/c/jgit/jgit/+/172595
Comment 36 Denis Roy CLA 2020-11-24 09:17:29 EST
> Alternatively, I conducted intermediate Gerrit release:
> gerrit-v3.2.5.1-60-gbb9ea229fb: [1] on stable-3.2 branch with this patch
> included [2], basically release 3.2.5.1 + Thomas's JGit fix: [3], that
> Matthias cherry-picked to the right JGit branch 5.8 used on gerrit
> stable-3.2 branch. With that fix release, you could continue to use JDK 15
> and git wire protocol v2 should just work. 

I have scheduled this for Friday PM, as we have a window for minor maintenance. I'd like to keep using JDK 15; seems Gerrit is stable (minus the bug).
Comment 37 Frederic Gurr CLA 2020-12-03 12:44:42 EST
I guess this issue can be closed?
Comment 38 David Ostrovsky CLA 2021-01-03 11:04:48 EST
(In reply to Denis Roy from comment #36)
> > Alternatively, I conducted intermediate Gerrit release:
> > gerrit-v3.2.5.1-60-gbb9ea229fb: [1] on stable-3.2 branch with this patch
> > included [2], basically release 3.2.5.1 + Thomas's JGit fix: [3], that
> > Matthias cherry-picked to the right JGit branch 5.8 used on gerrit
> > stable-3.2 branch. With that fix release, you could continue to use JDK 15
> > and git wire protocol v2 should just work. 
> 
> I have scheduled this for Friday PM, as we have a window for minor
> maintenance. I'd like to keep using JDK 15; seems Gerrit is stable (minus
> the bug).

@Denis, any news on Gerrit upgrade and re-enabling of git wire protocol v2 on eclipse.org?

Gerrit 3.2.6 was released: [1] with these fixes: [2], that includes Thomas's JGit fix: [3] for git wire protocol v2 on JDK 15.

This is particularly important now, that git wire protocol v2 support on JGit client side was merged: [4] (see bug 553083) and we would like to see early adopters to test this feature against Gerrit on eclipse.org. So that re-enabling of git wire protocol v2 on eclipse.org would be really appreciated.

[1] https://groups.google.com/g/repo-discuss/c/9MOEK_K_sCY/m/mvWmkywfAQAJ
[2] https://gerrit.googlesource.com/gerrit/+log/v3.2.5.1..v3.2.6?no-merges
[3] https://gerrit.googlesource.com/gerrit/+/bb9ea229fbc10fbe37658ae4b68e723d3095120a
[4] https://git.eclipse.org/r/c/jgit/jgit/+/172595
Comment 39 Denis Roy CLA 2021-01-04 11:39:23 EST
> @Denis, any news on Gerrit upgrade and re-enabling of git wire protocol v2
> on eclipse.org?
> 

I've scheduled it for Friday PM, since we have a maintenance window for such things.  It didn't get done as per comment 36 as that Friday was a stat holiday.
Comment 40 Luca Milanesio CLA 2021-01-04 11:46:12 EST
I would also consider to plan the upgrade to Gerrit v3.3.1 (see release notes at [1]).

We upgraded GerritHub.io before Christmas, and in addition to the attention-set (a *huge* code-review productivity feature) we have also experienced better performance from a UI perspective.

Gerrit v3.3.1 requires at least Java 11.

Luca.

[1] https://bugs.eclipse.org/bugs/show_bug.cgi?id=568904
Comment 41 Denis Roy CLA 2021-01-08 13:29:21 EST
We've upgraded to Gerrit 3.2.6 and re-enabled protocol v2.