Description Jens Von Pilgrim 2010-10-11 10:52:22 EDT

Created attachment 180601 [details]
Demo grammar with 11 expression groups

Xtext runs into memory problems under certain circumstances. I'm using a pattern for left-factoring expressions:

----------------------------------
// overall set up:
Expression:	Expression_1;
PrimaryExpression returns Expression: '(' Expression ')' | TerminalExpression;

// expression groups:
Expression_1 returns Expression: Expression_2
  	(
		{Op1aExpression.lhs=current} "op1a" |
		{Op1bExpression.lhs=current} "op1b" |
		{Op1cExpression.lhs=current} "op1c" 
		rhs=Expression_2
	)*;
...
Expression_n returns Expression: PrimaryExpression
  	(
		{OpnaExpression.lhs=current} "opna" |
		{OpnbExpression.lhs=current} "opnb" |
		{OpncExpression.lhs=current} "opnc" 
		rhs=PrimaryExpression
	)*;
----------------------------------
	
That is, there are n expression "groups", each group defines expressions with operators having the same precedence. There may be 1 to k operators in one group, and the expressions may look a little bit differently in case of unary expressions, for example. However, the rules all look more or less similar.

Depending on the number of expression groups, Xtext runs into memory problems. E.g., on my machine, up to 10 groups are handled correctly. 

11 expression groups lead to the following exception (is is an OutOfMemoryException in the end):

----------------------------------
63900 [main] ERROR enerator.CompositeGeneratorFragment  - java.lang.reflect.InvocationTargetException
org.eclipse.emf.common.util.WrappedException: java.lang.reflect.InvocationTargetException
	at org.eclipse.xtext.generator.parser.antlr.AntlrToolFacade.runWithParams(AntlrToolFacade.java:127)
    ...
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	...
	at org.eclipse.xtext.generator.parser.antlr.AntlrToolFacade.runWithParams(AntlrToolFacade.java:125)
	... 14 more
Caused by: java.lang.OutOfMemoryError: Java heap space
	at java.lang.Class.getDeclaredMethods0(Native Method)
	at java.lang.Class.privateGetDeclaredMethods(Class.java:2395)
	...
	at org.antlr.stringtemplate.language.ASTExpr.writeAttribute(ASTExpr.java:659)
	at org.antlr.stringtemplate.language.ActionEvaluator.action(ActionEvaluator.java:81)
64231 [main] INFO  .emf.mwe2.runtime.workflow.Workflow  - Done.
----------------------------------

12 expression let Xtext fail earlier in the process:

----------------------------------
21725 [main] ERROR mf.mwe2.launch.runtime.Mwe2Launcher  - Java heap space
java.lang.OutOfMemoryError: Java heap space
	at java.util.zip.InflaterInputStream.<init>(InflaterInputStream.java:71)
	...
	at org.apache.log4j.Category.error(Category.java:322)
	at org.eclipse.xtext.generator.CompositeGeneratorFragment.generate(CompositeGeneratorFragment.java:85)
	at org.eclipse.xtext.generator.LanguageConfig.generate(LanguageConfig.java:69)
----------------------------------

I have attached a test file with 11 expressions for testing purposes. The problem is, that under some circumstances, Xtext will run with a given number of groups, but will fail if only a new operator is added to one group. This makes the whole issue rather hard to solve, Xtext "feels" rather fragile because of that. And it is very difficult for a grammar developer finding that problem, as you usually expect an error in your grammar. One may argue that too many expressions are used, and instead operator enumerations (enum opi: opi1 | opi2 .. | opik) could be used. However, this is not always possible, and the example is only for testing purposes.

Note that Xtext 1.0.1 seems to run into problems with less expressions defined than Xtext 1.0 (this is why I selected version 1.0.1). At least, my original grammar (using that pattern) worked fine with Xtext 1.0 and didn't worked with Xtext 1.0.1. I had to merge some groups, loosing some precedence levels (this is how I identified that problem).

Cheers,

Jens