Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 331531

Summary:	Timeout/heap space issue results in "ANTLR could not analyze this decision in rule Tokens" error
Product:	[Modeling] TMF	Reporter:	Jens Von Pilgrim <developer>
Component:	Xtext	Assignee:	Project Inbox <tmf.xtext-inbox>
Status:	CLOSED NOT_ECLIPSE	QA Contact:
Severity:	normal
Priority:	P3	CC:	henrik.lindberg, sebastian.zarnekow, sven.efftinge
Version:	1.0.1
Target Milestone:	---
Hardware:	All
OS:	All
Whiteboard:

Description Jens Von Pilgrim

2010-12-01 07:13:13 EST

Apparently, ANTLR uses a timout in order to stop generation of code in certain cases (see http://www.antlr.org/pipermail/antlr-interest/2009-May/034563.html ). Thus, ANTLR might produces weird error message, as the cause of the error cannot be located in the grammar (as it is a timeout). This issue has already been posted in the TMF newsgroup (see http://www.eclipse.org/forums/index.php?t=msg&th=165624&start=0& ), and one possible solution is to increase the heap space in the MWE configuration (i.e. add VM argument "-Xmx2000m").

From my google searches, I assume a typical error message produced by ANTLR looks like

warning(205): ../.../ui/contentassist/antlr/internal/Internal....g:1:8: ANTLR could not analyze this decision in rule Tokens; often this is because of recursive rule references visible from the left edge of alternatives.  ANTLR will re-analyze the decision with a fixed lookahead of k=1.  Consider using "options {k=1;}" for that decision and possibly adding a syntactic predicate.

and a whole bunch of more error messages following this initial one. Actually, as the timeout only occurs on larger grammars, one can spent hours in order to find a problem in the grammar. And it can occur everywhere, e.g., I was able to produce that error simply by adding some (otherwise irrelevant) brackets in the grammar (making me wonder wether brackets can cause the problem). And the issue can be solved by weird fixes, i.e. by changing keywords (in order to reduce some overhead when parsing the tokens).

The whole system (i.e. Xtext) becomes very fragile due to that issue. As this error message is completely misleading, I would suggest to at least generate some kind of hint (to increase heap space) and to make the ANTLR timeout configurable (and add a hint about that as well). Heap space and timeout could easily be retrieved in order to print out the hint, couldn't they?

Cheers,
Jens

Comment 1 Sebastian Zarnekow

2010-12-01 07:16:07 EST

The timeout is configurable.
See here: http://20000frames.blogspot.com/2010/09/dealing-with-could-not-even-do-k1-for.html

I'm inclined to close this as not eclipse. Any objections?

Comment 2 Jens Von Pilgrim

2010-12-01 07:58:29 EST

Sebastian, thanks for the link. So, this solves part of this problem, however, one still has to know about this timeout problem in the first place.

I'm well aware of this issue being an ANTLR issue rather then an Xtext one. However, from the Xtext user's point of view, it is Xtext (Bad enough that ANTLR error messages pop up with some weird locations pointing to some weird *.g-files). As this problem is really hard to find, it would be extremely helpful to at least add a hint to the console output "fixing" the wrong ANTLR hint.

Cheers,
Jens

Comment 3 Sven Efftinge

2010-12-01 10:11:11 EST

We don't aim to make the usage of Antlr transparent. We also use EMF, Google Guice, Google Collection and most of the Eclipse Workbench. If we tried to abstract over any exception and error messages thrown by those libraries, you wouldn't be able to discuss the real issue within the corresponding projects, because you would still think it can be solved by us. But it can't we could just patch thy symptom, but even for that Antlr is the better place.

So you really should spent your energy fixing the real problem, i.e. raise the issue over at Antlr.org. 
Thanks :-)

Comment 4 Henrik Lindberg

2010-12-01 22:08:49 EST

(In reply to comment #1)
> The timeout is configurable.
> See here:
> http://20000frames.blogspot.com/2010/09/dealing-with-could-not-even-do-k1-for.html
> 
> I'm inclined to close this as not eclipse. Any objections?

Maybe add the tip to the documentation ?