Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 363894

Summary: BacktrackingSemanticSequencer can take forvever.
Product: [Modeling] TMF Reporter: Ed Willink <ed>
Component: XtextAssignee: Project Inbox <tmf.xtext-inbox>
Status: CLOSED FIXED QA Contact:
Severity: major    
Priority: P3 CC: moritz.eysholdt, sebastian.zarnekow
Version: 2.0.0   
Target Milestone: ---   
Hardware: PC   
OS: Windows Vista   
Whiteboard:
Bug Depends on:    
Bug Blocks: 362517    
Attachments:
Description Flags
Exatr debug none

Description Ed Willink CLA 2011-11-16 01:21:14 EST
Using the M3 release an attempt to open Ecore.ecore with the OCLinEcore editor (from the MDT/OCL Examples feature) takes forever. This now uses the new serializer. With the old serializer, it worked.

If Ecore.ecore is simplified, variously by removing all EStructuralFeatures, or all EAnnotations, or all EOperations or all inheritance the trimmed files can be opened.

On one occasion while instrumenting with Visual VM, Ecore.ecore did successfully open, eventually.

Instrumenting BacktrackingSemanticSequencer, I see the following suspicious behaviour.

a) approximately 50% of calls to sortFollowers have a zero entry list (well worth a bypass test)

b) approximately 50% of calls to sortFollowers are unsortable, because FollowerSorter.compare returns 0, sometimes because (o1.getAssignedGrammarElement() == null && o1.getAssignedGrammarElement() == null)
but more often because !o1Opt && !o2Opt.

This makes me suspect that the intermittent behaviour is dependent on a fortuitous memory ordering and that the failures are due to an inappropriate dependence on sorting.
Comment 1 Ed Willink CLA 2011-11-24 12:01:00 EST
Ping. Are there any plans to fix this.

Not being able to handle large models is a problem for me. I'm considering reverting to the old serializer, which will be painful because I'll have to undo the cleanup that the new architecture enabled me to do.
Comment 2 Moritz Eysholdt CLA 2011-11-24 12:10:30 EST
I'll take a look at it. Are these the sources you're working with? http://git.eclipse.org/c/mdt/org.eclipse.ocl.git/tree/examples
Comment 3 Ed Willink CLA 2011-11-24 12:43:23 EST
(In reply to comment #2)
> I'll take a look at it. Are these the sources you're working with?
> http://git.eclipse.org/c/mdt/org.eclipse.ocl.git/tree/examples

Yes. I appreciate working with a large body of someone else's code is not much fun which is why I looked to try to provide a more modest repro, but found that I could make a variety of different 25% trims to Ecore.ecore and the problem went away. Further investigation revealed suspicious problems with ordering, which at least offer optimisations even if they aren't actually the bug.
Comment 4 Ed Willink CLA 2012-01-03 12:38:04 EST
Any news? Bug 361649 is causing me trouble too.
Comment 5 Ed Willink CLA 2012-03-17 10:45:57 EDT
Any news. Problem is still present in a-just-before M6 build. It's really disappointing that the ability to load Ecore.ecore or UML.ecore was present in Indigo and not in Juno.
Comment 6 Ed Willink CLA 2012-03-28 07:20:41 EDT
Moritz, I have an example looping predictably. It keeps creating a new FollowerSorter that is identical to the previous one because TraceItem.getNextGrammarElement() returns null. I'll look for you on the itemis stand at about 8:45 so that you can step through it.
Comment 7 Ed Willink CLA 2012-03-28 09:27:32 EDT
Created attachment 213290 [details]
Exatr debug

To reproduce the problem:

Install 4.2M6 modeling EPP.
Install Xtext, OCL Examples and Editors [and MWE2, Acceleo, Subversive, EMF Compare 1.3M5, EGIT 2.0.0 nightly]
Import org.eclipse.xtext as a Plugin with Sources and use the instrumented BacktrackingSemanticSequencer.java attached.
Checkout the http://git.eclipse.org/c/mdt/org.eclipse.ocl.git repo and the XtextSerializationInvestigation branch.
Run the org.eclipse.ocl.examples.xtext.tests (standalone) launch config
Kill it once a test has started (all tests preceding testSerialize_Ecore should pass).
Browse to org.eclipse.ocl.examples.test.xtext.SerializeTests (about seven from the bottom). Open it and select testSerialize_Ecore.

Running testSerialize_Ecore shows no progress with the Console showing...
    FollowerSorter 65177 EObject: RootPackageCS.ownedNestedPackage[0]->PackageCS'ecore'.ownedType[2]->ClassCS'EClass'
Values: name(1), ownedConstraint(8), ownedSuperType(1), ownedOperation(9), ownedProperty(16)
 null
    FollowerSorter 65178 EObject: RootPackageCS.ownedNestedPackage[0]->PackageCS'ecore'.ownedType[2]->ClassCS'EClass'
Values: name(1), ownedConstraint(8), ownedSuperType(1), ownedOperation(9), ownedProperty(16)
 null

To debug set a breakpoint in BacktrackingSemanticSequencer.FollowerSorter in the counter > 310 branch.
Comment 8 Ed Willink CLA 2012-03-28 09:32:20 EDT
(In reply to comment #7)
> Checkout the http://git.eclipse.org/c/mdt/org.eclipse.ocl.git repo and the
> XtextSerializationInvestigation branch.
Import the following plugins from the GIT workspace

examples/org.eclipse.ocl.examples.codegen
examples/org.eclipse.ocl.examples.domain
examples/org.eclipse.ocl.examples.library
examples/org.eclipse.ocl.examples.pivot
examples/org.eclipse.ocl.examples.ui
examples/org.eclipse.ocl.examples.xtext.*
tests/org.eclipse.ocl.examples.xtext.tests
Comment 9 Sebastian Zarnekow CLA 2012-04-02 17:01:56 EDT
Moritz, is this the one that you fixed, recently?
Comment 10 Ed Willink CLA 2012-04-03 05:16:30 EDT
Moritz, following out partial investigation at EclipseCon, have you got everything you need to investigate this?
Comment 11 Moritz Eysholdt CLA 2012-04-03 08:30:35 EDT
(In reply to comment #9)
> Moritz, is this the one that you fixed, recently?

no, that fix did not yet solve the problem reported in this ticket.

(In reply to comment #10)
> Moritz, following out partial investigation at EclipseCon, have you got
> everything you need to investigate this?

yes, Ed, I can reproduce this now on my machine.
Comment 12 Moritz Eysholdt CLA 2012-04-03 18:22:05 EDT
Hi Ed,

i've implemented a fix for this issues. See 
http://git.eclipse.org/c/tmf/org.eclipse.xtext.git/commit/?id=ba26c7a89e55a1f2a4a517d1493c04679b5db155

Now the test
org.eclipse.ocl.examples.test.xtext.SerializeTests.testSerialize_Ecore()
fails with a model comparison error in
org.eclipse.ocl.examples.xtext.tests.XtextTestCase.assertSameModel(Resource, Resource, Map<String, Object>)

To compare models, you might want to use 
String expected = EmfFormatter.objToStr( expectedResource.getContents().get(0));
String actual = EmfFormatter.objToStr( actualResource.getContents().get(0));
Assert.assertEquals(expected, actual);
because in case of a failure, you can use JDT-Junit's excellent text comparison dialog to inspect the differences.

The problem was caused by rule 
ClassCS returns base::ClassCS:
	qualifier+='abstract'?
	'class' name=UnrestrictedName
	(ownedTemplateSignature=TemplateSignatureCS)?
	('extends' ownedSuperType+=TypedRefCS (',' ownedSuperType+=TypedRefCS)*)?
	(':' instanceClassName=SINGLE_QUOTED_STRING)?
	('{' qualifier+='interface'
	 '}')?
	(	('{' (ownedAnnotation+=AnnotationElementCS
	        | ownedOperation+=OperationCS
	        | ownedProperty+=StructuralFeatureCS
	        | ownedConstraint+=InvariantConstraintCS)* '}')
	|	';'
	)
;
in your grammar. The optional elements at the beginning and the multiple-alternative at the end led to many possible paths the backtracking algorithm could take. It spent too much time with trying the wrong paths. I assume the algorithm would have terminated at some point, but neuter of use was willing to wait long enough ;)

My fix prevents the backtracking algorithm to try paths which can not lead to success. Such paths are paths that have no possibility of consuming at least one of the remaining values from an EObject. They can not consume a value because the path does not include a grammatical assignment for the EStructuralFeature the value resides in.

Ed, could you verify if the fix works for you?
Comment 13 Ed Willink CLA 2012-04-04 02:41:39 EDT
(In reply to comment #12)
> Ed, could you verify if the fix works for you?

Thanks. What an easy fix! The N-build works for me; now I just have a little bit of my own bit rot to repair.
Comment 14 Moritz Eysholdt CLA 2012-04-04 04:57:54 EDT
ok, then I consider this fixed. Please reopen if I missed something.
Comment 15 Ed Willink CLA 2012-04-04 06:17:55 EDT
(In reply to comment #12)
> To compare models, you might want to use 
> String expected = EmfFormatter.objToStr(
...
> because in case of a failure, you can use JDT-Junit's excellent text comparison
> dialog to inspect the differences.

Nice. listToStr() is even better.