Bug 323754

Comment 1 Andrew Eisenberg 2012-03-01 19:25:07 EST

Finally getting a chance to look at this more deeply.  I do see major jumps in memory usage when there are lots of jars to be copied for AJ projects, but not much of a jump for Java projects.

Comment 2 Andrew Eisenberg 2012-03-02 11:52:20 EST

I dug a bit deeper here.  It seems that when there is a full build, AspectJ controls file copying, but when there is an incremental build AJDT takes over.

The copy mechanism for AJDT uses the Eclipse workspace APIs (IResource, IFile, etc) which are very efficient.  On the other hand, ApsectJ uses this chunk of code, which can easily use significant amounts of memory for large files:


	private void copyResourcesFromFile(File f, String filename, File src) throws IOException {
		if (!acceptResource(filename, true)) {
			return;
		}
		FileInputStream fis = null;
		try {
			fis = new FileInputStream(f);
			byte[] bytes = FileUtil.readAsByteArray(fis);
			// String relativePath = files[i].getPath();

			writeResource(filename, bytes, src);
		} catch (FileNotFoundException fnfe) {
			// pr359332: looks like the file moved (refactoring?) just as this copy was starting
			// that is OK
		} finally {
			if (fis != null) {
				fis.close();
			}
		}
	}

Three options here:

1. Make the method above more efficient by not reading the entire file in at once, but rather chunking it up.
2. Avoid calling this method from inside of AJDT, rather use a callback so that AJDT handles the copy.
3. Let AJDT manage all file copying on full builds.

Comment 3 Andrew Eisenberg 2012-03-02 12:12:06 EST

Doing some research on the web, I found a comparison of different mechanisms to copy files:
http://www.baptiste-wicht.com/2010/08/file-copy-in-java-benchmark/

For files under 5Mb and on the same disk, it looks like the NIO Buffer and the NIO transfer benchmarks do the best.  This article is looking at speed, not memory so the recommendations may not be entirely appropriate.  However, just about anything would be better than reading in the entire file to disk before copying.

Comment 4 Andrew Eisenberg 2012-03-02 14:23:54 EST

Simpler solution, or at least a solution that doesn't require changes to aspectj:

It looks like there is some dead code in AJDT that copies all files in a project to the output directory.  I know in the past, we went back and forth over whether or not AspectJ or AJDT should be in charge of resource copying.  And currently, we have AJDT copy resources on incremental builds and AspectJ on full builds.

It would be easy enough to stop AspectJ from doing any copying and let AJDT handle it.  I have looked at the unused code and it appears to be doing (mostly) the right thing, but I need to verify it through a few tests.  I did need to make some changes to ensure that copied files were appropriately marked as derived.

It is also simple to disable AspectJ from doing the copying.  In CoreCompilerConfiguration.getSourcePathResources() we simply need to return null and AspectJ will not do the copying.

Comment 5 Andrew Eisenberg 2012-03-02 14:26:47 EST

I am now happy with this fix.  Memory usage is way down and not spiking during a full build of a project with lots of jars.

Commit is here:
http://git.eclipse.org/c/ajdt/org.eclipse.ajdt.git/commit/?id=9ecd669b72dd02fb36a07ab672152708805bfaab