Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 317416

Summary: jgit hangs in IndexPack
Product: [Technology] JGit Reporter: Edwin Kempin <edwin.kempin>
Component: JGitAssignee: Shawn Pearce <sop>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: P3 CC: sop
Version: unspecified   
Target Milestone: 0.9.0   
Hardware: PC   
OS: Windows XP   
Whiteboard:

Description Edwin Kempin CLA 2010-06-21 07:15:34 EDT
Build Identifier: 20100218-1602

jgit version: org.eclipse.jgit-0.8.4

In my scenario I'm using Gerrit for code review.
The problem is that the MsysGit Console gets stuck when I'm pushing a change to Gerrit:

Edwin@WDFD00185753A /c/temp/newProjects/project2 (master)
$ git push ssh://Edwin@localhost:29418/new/project2.git HEAD:refs/for/master
Counting objects: 3, done.
Writing objects: 100% (3/3), 216 bytes, done.
Total 3 (delta 0), reused 0 (delta 0)
_

By debugging the Gerrit Server I could identify the hanging thread:

Thread [SSH git-receive-pack '/new/project2.git' (Edwin)] (Suspended)	
	IndexPack.inflateFromInput(long) line: 998	
	IndexPack.whole(int, long, long) line: 824	
	IndexPack.indexOneObject() line: 783	
	IndexPack.index(ProgressMonitor) line: 394	
	ReceivePack.receivePack() line: 782	
	ReceivePack.service() line: 628	
	ReceivePack.receive(InputStream, OutputStream, OutputStream) line: 576	
	Receive.runImpl() line: 78	
	Receive(AbstractGitCommand).service() line: 104	
	AbstractGitCommand.access$0(AbstractGitCommand) line: 93	
	AbstractGitCommand$1.run() line: 69	
	BaseCommand$TaskThunk.run() line: 391	
	Executors$RunnableAdapter<T>.call() line: 441	
	FutureTask$Sync.innerRun() line: 303	
	ScheduledThreadPoolExecutor$ScheduledFutureTask<V>(FutureTask<V>).run() line: 138	
	ScheduledThreadPoolExecutor$ScheduledFutureTask<V>.access$301(ScheduledThreadPoolExecutor$ScheduledFutureTask) line: 98	
	ScheduledThreadPoolExecutor$ScheduledFutureTask<V>.run() line: 207	
	WorkQueue$ProjectTask<V>(WorkQueue$Task<V>).run() line: 324	
	ThreadPoolExecutor$Worker.runTask(Runnable) line: 886	
	ThreadPoolExecutor$Worker.run() line: 908	
	Thread.run() line: 679	

It hangs because there is an endless loop in org.eclipse.jgit.transport.IndexPack#inflateFromInput(...) method.
For me the while loop in inflateFromInput(...) is never finishing:

private byte[] inflateFromInput(final long sz) throws IOException {
  final byte[] dst = new byte[(int) sz];
  final Inflater inf = inflater;
  try {
    int n = 0;
    int p = -1;
    while (!inf.finished()) {
      if (inf.needsInput()) {
        if (p >= 0) {
          crc.update(buf, p, bAvail);
          use(bAvail);
        }
        p = fillFromInput(1);
        inf.setInput(buf, p, bAvail);
      }

      n += inf.inflate(dst, n, dst.length - n);
  }
  if (n != sz)
    throw new DataFormatException(JGitText.get().wrongDecompressedLength);
    n = bAvail - inf.getRemaining();
    if (n > 0) {
      crc.update(buf, p, n);
      use(n);
    }
    return dst;
  } catch (DataFormatException dfe) {
    throw corrupt(dfe);
  } finally {
    inf.reset();
  }
}

In my case the while loop is never finishing since inf.finished() always returns false. I guess this return value is correct since after the dst buffer is filled inf.getRemaining() still returns 22, so in principle there is something more to inflate. However for the call of Inflater#inflate(...) now always a length of 0 is given (dst.length - n = 0) and so the status of the inflater is never changed again. As a result I end up with an endless loop.

Here are some more values which I retrieved from the debugger, maybe they can help to solve the problem:
inf.finished()
	 (boolean) false	 
inf.getRemaining()
	 (int) 22
inf.getTotalIn()
	 (int) 12	 
inf.getBytesRead()
	 (long) 12
inf.getTotalOut()
	 (int) 6	 
inf.getBytesWritten()
	 (long) 6	 	 
inf.getAdler()
	 (int) 124256764
dst
	 (byte[]) [99, 46, 116, 120, 116, 10]
dst.length
	 (int) 6
n
	 (int) 6
this.dstIdx
	 (java.io.File) C:\workspaces\projects\test_site\git\new\project2.git\objects\incoming_6507350032416264131.idx
this.dstPack
	 (java.io.File) C:\workspaces\projects\test_site\git\new\project2.git\objects\incoming_6507350032416264131.pack
this.outputVersion
	 (int) 2
this.objectCount
	 (long) 3

Reproducible: Sometimes

Steps to Reproduce:
1. create new project in Gerrit:
ssh -p 29418 Edwin@localhost gerrit create-project --name new/project2
2. clone the Git repository:
git clone ssh://Edwin@localhost:29418/new/project2.git
3. create one dummy file in the empty Git repository and commit it
4. push the change to Gerrit
git push ssh://Edwin@localhost:29418/new/project2.git HEAD:refs/for/master
The push is hanging as explained in the details.

As described I've create one local Git repository from which I always can reproduce the problem by trying to push the latest commit. If it can help to solve the problem I can share this Git repository.
For other Git repositories which I created in the same way the problem is not occuring.
Comment 1 Edwin Kempin CLA 2010-06-23 09:04:17 EDT
My MSysGit from which I'm doing the push has the following version: 1.6.3.2.1299.gee46c
Comment 2 Edwin Kempin CLA 2010-06-23 09:23:43 EDT
Upgraded my MSysGit to 1.7.0.2.msysgit.0 but the problem is still the same
Comment 3 Shawn Pearce CLA 2010-07-02 20:32:18 EDT
I've looked at this code, the code in IndexPack is incorrectly using the Inflater API, resulting in some room for errors.  I'll post a patch in a few minutes which tries to correct those bad usages.

In the meantime though, it sounds like there is another error here.  For this particular object the inflater isn't done, but the destination array is full.  That implies that there is data corruption in the stream because the inflated size in the object header disagrees with the inflated size coming from the inflater itself.  Which is really bogus.

I'd be really interested in seeing that repository, to try and understand how we have such a badly mangled pack coming out of a recent msysgit binary... or maybe how JGit's IndexPack is misreading that particular object header.
Comment 4 Shawn Pearce CLA 2010-07-02 21:24:08 EDT
I think the patch in http://egit.eclipse.org/r/1049 fixes this.  Can you try to patch locally and verify?
Comment 5 Edwin Kempin CLA 2010-07-05 04:31:51 EDT
The repository which allows me to reproduce the problem is rather simple. I've uploaded it to GitHub and verified that I can still reproduce the issue after cloning the repository from GitHub:
git clone git://github.com/EdwinKempin/RepoForJgitBug317416.git

I will let you know when I've tried out that fix.
Comment 6 Edwin Kempin CLA 2010-07-05 10:11:05 EDT
I only had time to try out the fix very shortly.

The fix solves the issue with the infinite loop and the push is now failing with a proper error message:
D044411@WDFD00185753A /c/temp/newProjects/RepoForJgitBug317416 (master)
$ git push ssh://Edwin@localhost:29418/new/project13 HEAD:refs/for/master
Counting objects: 3, done.
Writing objects: 100% (3/3), 219 bytes, done.
Total 3 (delta 0), reused 3 (delta 0)
error: unpack failed: error Packfile corruption detected: wrong decompressed length
To ssh://Edwin@localhost:29418/new/project13
 ! [remote rejected] HEAD -> refs/for/master (n/a (unpacker error))
error: failed to push some refs to 'ssh://Edwin@localhost:29418/new/project13'

Tomorrow I will once more debug the issue and see whether I can get any further details about this error.
Comment 7 Edwin Kempin CLA 2010-07-06 07:04:38 EDT
As additional test I was now pushing the same commit to a Gerrit running on another system. Since this simply worked without any problems I would exclude any issues with the source repository but rather think that it's indeed the inflater which is doing strange things on my system. I have no good idea how to continue the investigation here since in the end the inflater is doing a native call in which the strange things seem to happen. 
Anyway my main concern was the infinite loop that occured in jgit due to this error, as this is solved by the fix and now a proper error message is given I would agree to see this issue as resolved.
Comment 8 Shawn Pearce CLA 2010-07-16 18:12:39 EDT
Infinite loop fix committed in 0b46e70155e1a14edc77f32e030094fb499887cf.

It'll be in Gerrit Code Review 2.1.4, and JGit 0.9.x when it releases later this fall.