Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 568950

Summary: Git wire protocol v2 is broken when running on JDK 15
Product: [Technology] JGit Reporter: David Ostrovsky <d.ostrovsky>
Component: JGitAssignee: Project Inbox <jgit.core-inbox>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: P3 CC: Arthur.vanDorp, jonah, matthias.sohn, twolf
Version: unspecified   
Target Milestone: 5.10   
Hardware: PC   
OS: Linux   
See Also: https://git.eclipse.org/r/c/jgit/jgit/+/172502
https://git.eclipse.org/c/jgit/jgit.git/commit/?id=7da0f0a8f37e35e9c3108588f1e6f7a7381d8f77
https://bugs.eclipse.org/bugs/show_bug.cgi?id=568904
https://git.eclipse.org/r/c/jgit/jgit/+/172535
https://git.eclipse.org/c/jgit/jgit.git/commit/?id=e2663a8b85cf92f6a84d72834257243a84066e9d
https://git.eclipse.org/r/c/jgit/jgit/+/172601
https://git.eclipse.org/c/jgit/jgit.git/commit/?id=9f3616dcb43246b883f9943af0de7cf67836c443
https://git.eclipse.org/r/c/jgit/jgit/+/173012
https://git.eclipse.org/c/jgit/jgit.git/commit/?id=c053b510b3b921c5859a69bc74b98bcdec9c2a17
Whiteboard:

Description David Ostrovsky CLA 2020-11-19 03:21:28 EST
When building Gerrit and JGit with JDK 15: [1], git wire protocol v2 doesn't work:

SEVERE: Internal server error (user admin account 1000000) during git-upload-pack '/foo'
org.eclipse.jgit.transport.UploadPackInternalServerErrorException
	at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:771)
	at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:101)
	at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:109)
	at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:33)
	at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:74)
	at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:475)
	at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:612)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
	at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: org.eclipse.jgit.errors.PackProtocolException: unknown command peel
	at org.eclipse.jgit.transport.UploadPack.serveOneCommandV2(UploadPack.java:1282)
	at org.eclipse.jgit.transport.UploadPack.serviceV2(UploadPack.java:1318)
	at org.eclipse.jgit.transport.UploadPack.uploadWithExceptionPropagation(UploadPack.java:832)
	at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:748)
	... 13 more

Nov 19, 2020 7:23:07 AM com.google.gerrit.sshd.BaseCommand handleError
SEVERE: Internal server error (user user account 1000001) during git-upload-pack '/foo'
org.eclipse.jgit.transport.UploadPackInternalServerErrorException
	at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:771)
	at com.google.gerrit.sshd.commands.Upload.runImpl(Upload.java:101)
	at com.google.gerrit.sshd.AbstractGitCommand.service(AbstractGitCommand.java:109)
	at com.google.gerrit.sshd.AbstractGitCommand.access$000(AbstractGitCommand.java:33)
	at com.google.gerrit.sshd.AbstractGitCommand$1.run(AbstractGitCommand.java:74)
	at com.google.gerrit.sshd.BaseCommand$TaskThunk.run(BaseCommand.java:475)
	at com.google.gerrit.server.logging.LoggingContextAwareRunnable.run(LoggingContextAwareRunnable.java:113)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
	at com.google.gerrit.server.git.WorkQueue$Task.run(WorkQueue.java:612)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630)
	at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: org.eclipse.jgit.errors.PackProtocolException: unknown command peel
	at org.eclipse.jgit.transport.UploadPack.serveOneCommandV2(UploadPack.java:1282)
	at org.eclipse.jgit.transport.UploadPack.serviceV2(UploadPack.java:1318)
	at org.eclipse.jgit.transport.UploadPack.uploadWithExceptionPropagation(UploadPack.java:832)
	at org.eclipse.jgit.transport.UploadPack.upload(UploadPack.java:748)
	... 13 more

Nov 19, 2020 7:23:07 AM com.google.gerrit.sshd.SshDaemon stop
INFO: Stopped Gerrit SSHD
Nov 19, 2020 7:23:07 AM com.google.gerrit.server.config.ScheduleConfig isInvalidOrMissing
INFO: No schedule configuration for "accountDeactivation".
Gerrit Server Shutdown

Time: 112.724
There were 2 failures:
1) testGitWireProtocolV2FetchIndividualRef(com.google.gerrit.integration.git.GitProtocolV2IT)
java.io.IOException: 07:22:19.497199 pkt-line.c:80           packet:          git< version 2
07:22:19.497333 pkt-line.c:80           packet:          git< version 2
07:22:19.497341 pkt-line.c:80           packet:          git< ls-refs
07:22:19.497345 pkt-line.c:80           packet:          git< fetch=shallow
07:22:19.497350 pkt-line.c:80           packet:          git< server-option
07:22:19.497355 pkt-line.c:80           packet:          git< 0000
07:22:19.501743 pkt-line.c:80           packet:        fetch< version 2
07:22:19.501892 pkt-line.c:80           packet:        fetch< ls-refs
07:22:19.501902 pkt-line.c:80           packet:        fetch< fetch=shallow
07:22:19.501910 pkt-line.c:80           packet:        fetch< server-option
07:22:19.501917 pkt-line.c:80           packet:        fetch< 0000
07:22:19.501924 pkt-line.c:80           packet:        fetch> command=ls-refs
07:22:19.501936 pkt-line.c:80           packet:        fetch> 0001
07:22:19.501954 pkt-line.c:80           packet:        fetch> peel
07:22:19.501961 pkt-line.c:80           packet:        fetch> symrefs
07:22:19.501971 pkt-line.c:80           packet:        fetch> ref-prefix refs/changes/01/1/1
07:22:19.501978 pkt-line.c:80           packet:        fetch> ref-prefix refs/refs/changes/01/1/1
07:22:19.501984 pkt-line.c:80           packet:        fetch> ref-prefix refs/tags/refs/changes/01/1/1
07:22:19.501991 pkt-line.c:80           packet:        fetch> ref-prefix refs/heads/refs/changes/01/1/1
07:22:19.501998 pkt-line.c:80           packet:        fetch> ref-prefix refs/remotes/refs/changes/01/1/1
07:22:19.502004 pkt-line.c:80           packet:        fetch> ref-prefix refs/remotes/refs/changes/01/1/1/HEAD
07:22:19.502011 pkt-line.c:80           packet:        fetch> ref-prefix refs/tags/
07:22:19.502017 pkt-line.c:80           packet:        fetch> 0000
07:22:19.512201 pkt-line.c:80           packet:          git< command=ls-refs
07:22:19.512231 pkt-line.c:80           packet:          git< 0001
07:22:19.512240 pkt-line.c:80           packet:          git< peel
07:22:19.512249 pkt-line.c:80           packet:          git< symrefs
07:22:19.512257 pkt-line.c:80           packet:          git< ref-prefix refs/changes/01/1/1
07:22:19.512266 pkt-line.c:80           packet:          git< ref-prefix refs/refs/changes/01/1/1
07:22:19.512275 pkt-line.c:80           packet:          git< ref-prefix refs/tags/refs/changes/01/1/1
07:22:19.512284 pkt-line.c:80           packet:          git< ref-prefix refs/heads/refs/changes/01/1/1
07:22:19.512293 pkt-line.c:80           packet:          git< ref-prefix refs/remotes/refs/changes/01/1/1
07:22:19.512301 pkt-line.c:80           packet:          git< ref-prefix refs/remotes/refs/changes/01/1/1/HEAD
07:22:19.512309 pkt-line.c:80           packet:          git< ref-prefix refs/tags/
07:22:19.512316 pkt-line.c:80           packet:          git< 0000
07:22:19.592656 pkt-line.c:80           packet:        fetch< e568232a36d8a8b35a637ec0af02e51fe5e986ea refs/changes/01/1/1
07:22:19.592698 pkt-line.c:80           packet:        fetch< b8c1cfdcfb11c16f2905c037a77f2c8dc9617d68 refs/changes/01/1/meta
07:22:19.592708 pkt-line.c:80           packet:        fetch< b155c4de9588883429b2fa03fd445b00f47633fb refs/meta/config
07:22:19.592715 pkt-line.c:80           packet:        fetch< 0000
07:22:19.593180 pkt-line.c:80           packet:        fetch> command=fetch
07:22:19.593191 pkt-line.c:80           packet:        fetch> 0001
07:22:19.593195 pkt-line.c:80           packet:        fetch> thin-pack
07:22:19.593199 pkt-line.c:80           packet:        fetch> no-progress
07:22:19.593204 pkt-line.c:80           packet:        fetch> ofs-delta
07:22:19.593230 pkt-line.c:80           packet:        fetch> want e568232a36d8a8b35a637ec0af02e51fe5e986ea
07:22:19.593235 pkt-line.c:80           packet:        fetch> done
07:22:19.593239 pkt-line.c:80           packet:        fetch> 0000
07:22:19.595444 pkt-line.c:80           packet:          git< command=fetch
07:22:19.595467 pkt-line.c:80           packet:          git< 0001
07:22:19.595476 pkt-line.c:80           packet:          git< thin-pack
07:22:19.595485 pkt-line.c:80           packet:          git< no-progress
07:22:19.595493 pkt-line.c:80           packet:          git< ofs-delta
07:22:19.595502 pkt-line.c:80           packet:          git< want e568232a36d8a8b35a637ec0af02e51fe5e986ea
07:22:19.595510 pkt-line.c:80           packet:          git< done
07:22:19.595516 pkt-line.c:80           packet:          git< 0000
07:22:19.668803 pkt-line.c:80           packet:        fetch< acknowledgments
fatal: expected 'packfile', received 'acknowledgments'

	at com.google.gerrit.acceptance.StandaloneSiteTest.execute(StandaloneSiteTest.java:254)
	at com.google.gerrit.acceptance.StandaloneSiteTest.execute(StandaloneSiteTest.java:216)
	at com.google.gerrit.integration.git.GitProtocolV2IT.execute(GitProtocolV2IT.java:376)
	at com.google.gerrit.integration.git.GitProtocolV2IT.testGitWireProtocolV2FetchIndividualRef(GitProtocolV2IT.java:325)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at com.google.gerrit.acceptance.StandaloneSiteTest$1.evaluate(StandaloneSiteTest.java:118)
	at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at com.google.testing.junit.runner.internal.junit4.CancellableRequestFactory$CancellableRunner.run(CancellableRequestFactory.java:108)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
	at com.google.testing.junit.runner.junit4.JUnit4Runner.run(JUnit4Runner.java:116)
	at com.google.testing.junit.runner.BazelTestRunner.runTestsInSuite(BazelTestRunner.java:159)
	at com.google.testing.junit.runner.BazelTestRunner.main(BazelTestRunner.java:85)


[1] https://gerrit-review.googlesource.com/c/gerrit/+/269382
Comment 1 Thomas Wolf CLA 2020-11-19 06:58:02 EST
From bug 568904 comment 30:

thomas@localhost:~/tmp/javatest$ more ot.java
public class Ot {

  private static final String END = new StringBuilder(0).toString();

  private static final String DELIM = new StringBuilder(0).toString();

  public static void main(String... args) {
    System.out.println("END == DELIM = "+ Boolean.toString(END == DELIM));
  }

}
thomas@localhost:~/tmp/javatest$ java -version
openjdk version "11.0.1" 2018-10-16 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.1+13-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.1+13-LTS, mixed mode, sharing)
thomas@localhost:~/tmp/javatest$ java ot.java
END == DELIM = false
thomas@localhost:~/tmp/javatest$ ../jdk15/jdk-15.0.1+9/bin/java --version
openjdk 15.0.1 2020-10-20
OpenJDK Runtime Environment AdoptOpenJDK (build 15.0.1+9)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 15.0.1+9, mixed mode, sharing)
thomas@localhost:~/tmp/javatest$ ../jdk15/jdk-15.0.1+9/bin/java ot.java
END == DELIM = true
thomas@localhost:~/tmp/javatest$

So with Java 15, a basic assumption of JGit's PacketLineIn is violated.

As a result, JGit cannot distinguish END and DELIM anymore, and parsing gets confused.

Unclear to me whether that's a bug in Java 15. Seems to me that two different allocations should _never_ return the same object. If only because client code might use the object's monitors to synchronize something. But OK, JGit is relying here on the implementation of StringBuilder. If this is intended behavior in Java 15, PacketLineIn needs to find a different way to distinguish the two.
Comment 2 Thomas Wolf CLA 2020-11-19 07:11:57 EST
This works as expected:

public class Ot {

  private static String END = new String();

  private static String DELIM = new String();

  public static void main(String... args) {
    System.out.println("END == DELIM = "+ Boolean.toString(END == DELIM));
  }

}

END != DELIM, also on Java 15.

Why was to done with new StringBuilder(0).toString() at all? There's a comment in PacketLineIn "must not string pool"... why not?

The StringBuilder construction comes from JGit commit 08a9682e3 "Revert "[findBugs] Silence DM_STRING_CTOR on PacketLineIn"" (2010-11-09). Before that it was actually new String("").
Comment 3 Andrey Loskutov CLA 2020-11-19 07:14:33 EST
(In reply to Thomas Wolf from comment #1)
> So with Java 15, a basic assumption of JGit's PacketLineIn is violated.

The assumption was wrong.

> Unclear to me whether that's a bug in Java 15. Seems to me that two
> different allocations should _never_ return the same object. 

Yes, but

private static final String END = new StringBuilder(0).toString(); 
private static final String DELIM = new StringBuilder(0).toString();

are *not* two different allocations, but two calls to toString() method that never had any contract to return some specific object, just String representation of the buffer.

That *should* return two *different* objects:

private static final String END = new String("");
private static final String DELIM = new String("");

So I think this can be fixed in JGit.
Comment 4 Eclipse Genie CLA 2020-11-19 07:23:47 EST
New Gerrit change created: https://git.eclipse.org/r/c/jgit/jgit/+/172502
Comment 6 Eclipse Genie CLA 2020-11-19 15:34:42 EST
New Gerrit change created: https://git.eclipse.org/r/c/jgit/jgit/+/172535
Comment 8 Eclipse Genie CLA 2020-11-20 18:49:32 EST
New Gerrit change created: https://git.eclipse.org/r/c/jgit/jgit/+/172601
Comment 10 Jonah Graham CLA 2020-11-23 20:26:36 EST
Thanks folks for fixing this Java 15 related bug so quickly.
Did this bug exist and is it now fixed in simrel's version of jgit? The reason I ask is that the current plan is to ship EPP packages and Eclipse installer with Java 15 bundles for 2020-12. The 2020-12 M2 had Java 14, but M3 will have Java 15.
Comment 11 David Ostrovsky CLA 2020-11-24 02:44:00 EST
(In reply to Jonah Graham from comment #10)
> Thanks folks for fixing this Java 15 related bug so quickly.
> Did this bug exist and is it now fixed in simrel's version of jgit? The
> reason I ask is that the current plan is to ship EPP packages and Eclipse
> installer with Java 15 bundles for 2020-12. The 2020-12 M2 had Java 14, but
> M3 will have Java 15.

Yes. The bug was fixed in JGit in the main branch: [1] and the fix
was cherry-picked to stable-5.8: [2] and stable-5.9 branches: [3].

[1] https://git.eclipse.org/c/jgit/jgit.git/commit/?id=7da0f0a8f37e35e9c3108588f1e6f7a7381d8f77
[2] https://git.eclipse.org/c/jgit/jgit.git/commit/?id=9f3616dcb43246b883f9943af0de7cf67836c443
[3] https://git.eclipse.org/c/jgit/jgit.git/commit/?id=e2663a8b85cf92f6a84d72834257243a84066e9d
Comment 12 Thomas Wolf CLA 2020-11-24 03:17:24 EST
(In reply to David Ostrovsky from comment #11)
> (In reply to Jonah Graham from comment #10)
> > Did this bug exist and is it now fixed in simrel's version of jgit?
> > M3 will have Java 15.
> 
> Yes. The bug was fixed in JGit in the main branch

Matthias will need to create 5.10.0-m3 releases of JGit and EGit and update the simrel aggregator to pick those up, though. And this would need to happen today or tomorrow.

Don't know if he has the time for this. However, this bug surfaces only in protocol V2, and the JGit client side has no implementation for protocol V2 yet but always uses protocol V0, and Eclipse only uses the JGit client side. So even if we don't make it for M3 that shouldn't be a problem.

The final 2020-12 release will include a JGit containing this fix for sure.
Comment 13 Matthias Sohn CLA 2020-11-24 05:54:04 EST
(In reply to Thomas Wolf from comment #12)
> (In reply to David Ostrovsky from comment #11)
> > (In reply to Jonah Graham from comment #10)
> > > Did this bug exist and is it now fixed in simrel's version of jgit?
> > > M3 will have Java 15.
> > 
> > Yes. The bug was fixed in JGit in the main branch
> 
> Matthias will need to create 5.10.0-m3 releases of JGit and EGit and update
> the simrel aggregator to pick those up, though. And this would need to
> happen today or tomorrow.

I plan to contribute the latest version which includes the fix to m3 today or tomorrow

> Don't know if he has the time for this. However, this bug surfaces only in
> protocol V2, and the JGit client side has no implementation for protocol V2
> yet but always uses protocol V0, and Eclipse only uses the JGit client side.
> So even if we don't make it for M3 that shouldn't be a problem.
> 
> The final 2020-12 release will include a JGit containing this fix for sure.
Comment 14 Eclipse Genie CLA 2020-11-30 07:04:07 EST
New Gerrit change created: https://git.eclipse.org/r/c/jgit/jgit/+/173012