Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 347485

Summary: JGit commit problem on Android
Product: [Technology] JGit Reporter: Anders Ek <aekextra>
Component: JGitAssignee: Project Inbox <jgit.core-inbox>
Status: RESOLVED FIXED QA Contact:
Severity: major    
Priority: P3 CC: christian.halstrick, christian.halstrick, eric.ml.mail, palmer, robin.rosenberg, robin
Version: unspecified   
Target Milestone: 1.1   
Hardware: Other   
OS: other   
Whiteboard:

Description Anders Ek CLA 2011-05-27 13:20:09 EDT
Build Identifier: 20090920-1017

I am trying to use the JGit library to synch documents on my android phone to a server.

It almost works... but I can not get the commit to work. Whenever I try a commit I get an exception with the description "data error". I tracked down the problem to the following statements in org.eclipse.jgit.util.IO:

public static void readFully(final InputStream fd, final byte[] dst,
        int off, int len) throws IOException {
    while (len > 0) {
        final int r = fd.read(dst, off, len);
        if (r <= 0)
            throw new EOFException(JGitText.get().shortReadOfBlock);
        off += r;
        len -= r;
    }
}

The line fd.read(dst, off, len) will throw the execption. I think the fd parameter is an InflaterInputStream so this code tries to read from a compressed archive.

A am trying to run this on an HTC Desire Z with Android 2.2.

I am using org.eclipse.jgit-0.12.1.jar and com.jcraft.jsch_0.1.31.jar to access the jgit functionality.

I have done some test examples to try to understand the problem

The following code runs without problems on my Windows XP machine:

public class Test {
/**
 * @param args
 */
public static void main(String[] args) {
    try {
        // Create a new directory
        File dirF = new File("C:\\Test\\TestDir");
        dirF.mkdir();
        log(">>> Created directory.\n");

        // Initialize git repository
        InitCommand init = Git.init();
        File initFile = new File("C:\\Test\\TestDir");
        init.setDirectory(initFile);
        init.call();
        log(">>> Git Init done.\n");

        // Create a file
        File newfile = new File("C:\\Test\\TestDir\\myfile.txt");
        newfile.createNewFile();
        PrintStream os = new PrintStream(newfile);
        os.println("Some text");
        os.close();
        log(">>> File created.\n");

        // Add to git
        FileRepositoryBuilder builder = new FileRepositoryBuilder();
        File f = new File("C:\\Test\\TestDir\\.git");
        Repository db = builder.setGitDir(f)
        .findGitDir() // scan up the file system tree
        .build();
        Git git = new Git(db);
        AddCommand add = git.add();
        add.addFilepattern(".").call();
        log(">>> Git Add done.\n");

        // Commit the change
        CommitCommand commit = git.commit();
        commit.setAll(true);
        commit.setMessage("A JGit message");
        commit.call();
        log(">>> Git Commit done.\n");

        // Check the log
        for (RevCommit c : git.log().call()) {
            log(c.getId() + "/" + c.getAuthorIdent().getName() + "/"
                    + c.getShortMessage());
        }
    } catch (Exception e) {
        e.printStackTrace();
    }

}
private static void log(String s) {
    System.out.print(s);
}

}

The following code will throw an exception on my android phone:

public class JGitSimpleAndroidActivity extends Activity {
StringBuilder sb;
TextView tv;
/** Called when the activity is first created. */
@Override
public void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    sb = new StringBuilder();
    tv = new TextView(this);
    try {
        // Create a new directory
        File sdCard;
        if (Environment.getExternalStorageState().equals(Environment.MEDIA_MOUNTED)) {
            sdCard = Environment.getExternalStorageDirectory();
        } else {
            log(">>> SD card is not available.");
            return;
        }
        File dirF = new File(sdCard,"TestDir");
        dirF.mkdir();
        log(">>> Created directory.\n");

        // Initialize git repository
        InitCommand init = Git.init();
        init.setDirectory(dirF);
        init.call();
        log(">>> Git Init done.\n");

        // Create a file
        File newfile = new File(dirF,"myfile.txt");
        newfile.createNewFile();
        PrintStream os = new PrintStream(newfile);
        os.println("Some text");
        os.close();
        log(">>> File created.\n");

        // Add to git
        FileRepositoryBuilder builder = new FileRepositoryBuilder();
        File f = new File(dirF,".git");
        Repository db = builder.setGitDir(f)
        .findGitDir() // scan up the file system tree
        .build();
        Git git = new Git(db);
        AddCommand add = git.add();
        add.addFilepattern(".").call();
        log(">>> Git Add done.\n");

        // Commit the change
        CommitCommand commit = git.commit();
        commit.setAll(true);
        commit.setMessage("A JGit message");
        commit.call(); // >>>>>>>> EXCEPTION THROWN HERE <<<<<<<
        log(">>> Git Commit done.\n");

        // Check the log
        for (RevCommit c : git.log().call()) {
            log(c.getId() + "/" + c.getAuthorIdent().getName() + "/"
                    + c.getShortMessage());
        }
    } catch (Exception e) {
        log(e.getMessage());
    }
}
private void log(String s) {
    sb.append(s);
    sb.append('\n');
    tv.setText(sb);
    setContentView(tv);
}
}



Reproducible: Always

Steps to Reproduce:
1. Create an Android app according with the class above as the activity
2. Run it on an Android phone
-> Exception
Comment 1 Anders Ek CLA 2011-05-28 03:23:49 EDT
Some more information.

I debugged what happens a bit more comparing what happens on Windows with what happens on the Android phone.

The problem appears as stated above when running the method "call" on a CommitCommand object. When executing this it seems like a commit file is created in the objects part of the git repository.

The problem occurs when trying to open this file again.

More specifically the execution differs in the method "static ObjectLoader open(InputStream in, File path, AnyObjectId id, WindowCursor wc)" in class org.eclipse.jgit.storage.file.UnpackedObject.

The test isStandardFormat(hdr) returns false on Android and true on Windows.

This causes the program on Androd to try to open the file using a different algorithm, and this fails.

On windows the program opens the file as a standard compressed file and this works nicely.

I've tried to open the problematic file on Android as a standard zip file and this works nicely.
Comment 2 Anders Ek CLA 2011-05-28 03:41:23 EDT
To test the hypothesis that the problem is in the isStandardFormat test I modified this method.

Old version;

	private static boolean isStandardFormat(final byte[] hdr) {
		// Try to determine if this is a standard format loose object or
		// a pack style loose object. The standard format is completely
		// compressed with zlib so the first byte must be 0x78 (15-bit
		// window size, deflated) and the first 16 bit word must be
		// evenly divisible by 31. Otherwise its a pack style object.
		//
		final int fb = hdr[0] & 0xff;
		return fb == 0x78 && (((fb << 8) | hdr[1] & 0xff) % 31) == 0;
	}

New version for test purpose:

	private static boolean isStandardFormat(final byte[] hdr) {
		// Try to determine if this is a standard format loose object or
		// a pack style loose object. The standard format is completely
		// compressed with zlib so the first byte must be 0x78 (15-bit
		// window size, deflated) and the first 16 bit word must be
		// evenly divisible by 31. Otherwise its a pack style object.
		//
		final int fb = hdr[0] & 0xff;
		boolean res1 = (fb == 0x78) && (((fb << 8) | hdr[1] & 0xff) % 31) == 0;
		boolean res2 = (fb == 72);
		return res1 || res2;
	}

With this modifed version of isStandardFormat the example given in the description of this case works also on my android phone (HTC Desire Z, Android 2.2).

Seems a bit dangerous as a general change though...
Comment 3 Robin Rosenberg CLA 2011-08-03 02:23:04 EDT
Is this still a problem?

Have you tried running your code using a non-sun^Horacle version of
the JVM, e.g. Harmony on your PC.

See http://code.google.com/p/android/issues/detail?id=11755#c6

Presumably you'd see the bug in an older version of Harmony on a PC, but
not in a new version.
Comment 4 Eric Faccer CLA 2011-08-06 09:52:32 EDT
This is still an issue in the version of harmony bundled with the android 3.2 sdk rom. If that got fixed in January it seems like it should be in there.

(In reply to comment #3)
> Is this still a problem?
> 
> Have you tried running your code using a non-sun^Horacle version of
> the JVM, e.g. Harmony on your PC.
> 
> See http://code.google.com/p/android/issues/detail?id=11755#c6
> 
> Presumably you'd see the bug in an older version of Harmony on a PC, but
> not in a new version.
Comment 5 Nick Palmer CLA 2011-08-31 07:54:06 EDT
This is actually caused by the fact that zip on android uses a different window size.

See this bug and a patch for it here:

https://bugs.eclipse.org/bugs/show_bug.cgi?id=346524

Note however, that repos written on Android with this patch applied are not readable with git version 1.7.6 as Git reports:

error: inflate: data stream error (incorrect header check)

I have not had time to dive into the Git source code but I assume that the implementation is actually checking the window size as well.
Comment 6 Robin Rosenberg CLA 2013-03-30 19:32:35 EDT
Isn't this fixed by https://git.eclipse.org/r/#/c/4071/ ?
Comment 7 Robin Stocker CLA 2014-04-25 18:30:07 EDT
Marking as fixed then.