Bug 282957

Comment 1 Sven Efftinge 2009-07-09 04:39:17 EDT

That would be a great addition! :-)

Comment 2 Achim Demelt 2009-07-13 09:29:48 EDT

Here's a summary of what is necessary to implement this feature. Review and comments appreciated.

GENERAL OVERVIEW
Incremental generation relies on two additional artifacts that need to be produced during the generation process: 1) a DIFF model, and 2) a TRACE model. The DIFF model is computed using EMF Compare and, well, describes the differences of the input model between the previous generation run and the current generation run. The TRACE model is computed during the generation itself and contains information about which model element was involved (through property access) in the generation of each particular file (i.e. scoped by the «FILE» statement and all its invoked sub templates.)

Now it's obvious that if we know about the input model elements for a particular file, we don't need to regenerate that file if none of its input model elements has changed since the last generation run.

In order to use incremental generation, the typical "read-validate-clean-generate" MWE workflow must be altered. In pseudo-workflow code, it will look like this:

<!-- read and validate model file -->
<XmiReader "currentModel.file" => "currentModelSlot" />
<Validate "currentModelSlot" />

<!-- read old model and compare with current model -->
<XmiReader "oldModel.file" => "oldModelSlot" />
<EmfCompare "oldModelSlot", "currentModelSlot" => "diffModelSlot" />

<!-- read old trace model -->
<XmiReader "oldTraceModel.file" => "oldTraceModelSlot" />

<!-- DON'T clean ouput folder! -->

<!-- generate. register trace callback that -->
<!--  a) computes the new trace model -->
<!--  b) prevents generation of files that don't need to be generated -->
<Generator "currentModelSlot", ...>
	<TraceCallback "diffModelSlot", "oldTraceModelSlot" => "newTraceModelSlot" />
</Generator>
<!-- clean single files that would not have been generated with the current model -->
<FileCleaner "oldTraceModelSlot", "newTraceModelSlot" />

<!-- write trace model and copy model file so that we can use them for the next run -->
<XmiWriter "newTraceModelSlot" => "oldTraceModel.file" />
<FileCopier "currentModel.file" => "oldModel.file" />

NEW AND ADAPTED WORKFLOW COMPONENTS:
1) The XmiReader needs to be fault tolerant since we may not always have a file to read from. For example, the first generator run will _never_ have "oldModel" to read from. Moreover, since the "oldModel.file" will be stored in a temporary location, it may be (accidentally or deliberately) deleted. This must not cause the XmiRead to fail. Instead, the slot should remain empty, and the generator will then perform a full non-incremental generation instead. I suggest a new optional parameter for the XmiReader component to control this fault-tolerant behavior.
2) We obviously need a new wokflow component for EMF compare. Not very spectactular... Input: Old and new model slot(s). Output: The DIFF model in the specified output slot.
3) Since we don't delete the output directory anymore, we need to clean up any individual files that have been generated by the previous generator run, but are no longer generated with the current model. We can infer this information from the old and new trace model, and a specific FileCleaner component can clean up these files after the generator run.
4) We need to perform a file copy to save a copy of the (current) model file. Simply writing the slot with an XmiWriter might change the XMI IDs of the elements, which will make it impossible to perform a proper model diff later. A special FileCopier component will have to do that.

THE GENERATOR CALLBACK
Apart from the workflow components described above, the major part of the work is done in a special generator callback. It has two responsibilities:
1) It captures all invocations of «FILE» statements as well as FeatureCall expressions. It keeps a list of all FeatureCalls per File and captures this in a TRACE model. The TRACE model itself is an EMF model, which makes working with it very easy.
2) If an old trace model as well as a diff model are  present, it also intercepts «FILE» statements _before_ their execution (using the pre() callback). If there is no diff that would justify writing the file, it instructs the generator to skip this statement.

IMPORTANT NOTE: This is an API change, since the pre() callback must now return a boolean value, which indicates if the generator should continue with this statement or not. Please comment on this.

TECHNICAL ISSUE: The callback requires access to the WorkflowContext in order to retrieve the diff and trace models from their workflow slots. I therefore suggest a new initialize(WorkflowContext ctx, Issues issues) method for the Callback interface. Again, this is a breaking API change. Please comment.

Well, I guess that's it. I hope this comment was concise enough so you had the motivation to read through it, and at the same time precise enough so that you got an idea how this thing is supposed to work.

Looking forward to your feedback! :-)

Comment 3 Sven Efftinge 2009-07-13 09:38:19 EDT

Sounds good. AFAIR you had some very promising measurements. Am I right? Can you provide them?

A technical question:
Doesn't the callback have to update the old trace with the new information?

Comment 4 Achim Demelt 2009-07-13 09:50:12 EDT

We have a diploma thesis in German which contains performance measurements. I think I'm not supposed not attach non-English content to Bugzilla, so I'll post it via email to you.

> Doesn't the callback have to update the old trace with the new information?

Basically yes, but we're creating a new trace model during generation. For each file we decide the following:
1) File needs to be written because of a changed model element => compute new trace information for that file.
2) File _would_ be written, but does not need to be because no input element changed => keep old trace information.

We then have a _new_ trace model, in addition to the (unchanged) old trace model. The file cleaner component then deletes each file that is contained in the old trace model, but not in the new one. Finally, we can just write the trace model to disk.

Last question before I start implementing tomorrow: Do you think the API changes are ok? How many clients do you suspect will we break with that?

Comment 5 Sven Efftinge 2009-07-13 11:03:59 EDT

> Last question before I start implementing tomorrow: Do you think the API
> changes are ok? How many clients do you suspect will we break with that?

I think as long as we do not introduce the change in 0.7.x (that is some maintenance) it's ok. 
The change makes sense and given that I only know two users using Callback (which both are with itemis :-)) it shouldn't be a problem.

Comment 6 ekkehard gentz 2009-07-14 03:32:13 EDT

great ideas, Achim,

have to re-think my workflows and cartridges, but I think it's worth.
fast responses and generation - lifecycle is the key to user-success.

ekke

Comment 7 Achim Demelt 2009-07-15 10:36:20 EDT

I've linked a couple of new issues to have easily digestable patches.

The implementation for the incremental generation feature is basically finished now. Tomorrow, I'll go over the files and add some copyright notices. Then I'll post the patch. It's surprisingly small :-)

Comment 8 Achim Demelt 2009-07-16 03:30:13 EDT

Created attachment 141737 [details]
Projects for incremental generation

This archive contains the code for incremental generation. Tests are included as well. I could only run the tests on my linux box with UTF-8 encoding. I hope they're fine on Mac and Windows as well.

These are the limits of the current implementation:

1) We can only track access to features within Xpand/Xtend. If such a feature is a derived feature that in turn access other features to compute its value, we cannot establish a trace. If the user changes a value in that input feature, we cannot determine that the file has to be generated. For example, if your template only iterates over eAttributes, adding a new EAttribute will not be recognized as a relevant change since the new EAttribute changes the value of the eStructuralFeatures feature, and not that of eAttributes. Bug #247130 will solve this problem, since we can then attach a read adapter and capture feature access directly at the EMF level.

2) The EmfCompare workflow component currently compares two (root) EObjects with each other. My tests currently involve simple models that consist of exactly one model file. I have not tested cases where models are split over several files. I think that EmfCompare is technically capable of detecting changes in referenced models, too. So we just have to try it out...

Review and feedback appreciated.

PS: We should handle additional changes to this feature as separate enhancement requests since this big contribution will have to be approved by the EMO. If we have the "rough cut" of the feature in CVS, early adopters can try it out at soon as possible.

Comment 9 Achim Demelt 2009-08-20 08:11:46 EDT

Committed to HEAD as "org.eclipse.xpand.incremental" and "org.eclipse.xpand.incremental.tests".

The bundle is not yet included in any feature. We should probably define a new one since the incremental generation stuff introduces a dependency to EMFCompare.

I'd appreciate a review and some feedback. Please take a look at the tests. The run fine on my Linux box, but I'd be interested if I screwed up some encoding or path things that prevent the tests from running on Windows/Mac.

In addition to the concepts outlined in earlier comments, I also included an "IncrementalGenerationFacade" that greatly simplifies incremental generation for the standard case. Please take a look at "incrementalcallback.mwe" and "incrementalfacade.mwe" in the test project.

Comment 10 Sven Efftinge 2009-09-29 05:43:42 EDT

The tests run fine on my mac. Achim also already added the mentioned feature.
Still some documentation and the example workflow we discussed would be very nice :-).

Comment 11 Achim Demelt 2009-09-29 11:39:38 EDT

> Still some documentation and the example workflow we discussed would be very
> nice :-).

I'm pretty much overloaded with work in a customer project at the moment. I have some office days on the next couple of Fridays. I plan to use those to write up the documentation. Sorry for the delay.

Comment 12 Achim Demelt 2010-02-10 13:52:43 EST

I finally came around to writing and committing the documentation. Consider case closed :-)

Comment 13 Karsten Thoms 2013-02-21 08:13:35 EST

Bug resolved before Xpand 1.2 release date => Closing