Community
Participate
Working Groups
Since MDT-OCL is migrating grammars and parsing infrastructure to LPGv2, QVTo project will have to do the same to make its parser work with MDT-OCL 3.0.0 I've decided to contribute the migration from QVTo side. If I have any doubts, I'll post them here. First one: I have noticed that you have some modified OCL grammars for the QVTo ones. This is not good, since Ill have to do double efforts. In any case, I hope that they are aligned with the current MDT-OCL grammars, aren't they ? I'm not doing that work, just migrating the current grammars you have. Cheers, Adolfo.
Radek, Doing a simple compare between MDT-OCL/EssentialOCL.g grammar and QVTo/EssentialOCL.g one reveals me that they are identical. Why are the OCL grammars cloned in the QVTo project ? Cheers, Adolfo
(In reply to comment #1) Hi Adolfo, thanks! I guess it's so mostly for historical reasons. Sergey has agreed to take over the LPG2 migration task on QVTO side, so he will be the right person to discuss things.
(In reply to comment #2) > (In reply to comment #1) > > Hi Adolfo, thanks! > > I guess it's so mostly for historical reasons. > Sergey has agreed to take over the LPG2 migration task > on QVTO side, so he will be the right person to > discuss things. Historical reason is the main thing why there's a copy of OCL grammar. There were some modifications (mostly in predefined tags) that helps us reusing grammar. Possibly one or two modifications still exists. However I'd prefer to stay with grammar copy since in that case we're not immediately affected with changes in OCL project. About migration itself. In case there's some guidelines for migration I can do it myself (I believe that wouldn't be more difficult then recent migration to OCL 3.0.0 grammar ;). In case Adolfo decide to provide patch only modifications to QVTO *.g files are required.
Hi Sergey, I'm afraid that they are not only grammars changes. LPGv2 migration implies changes in the LPG runtime. For example, the namespace of the packages is changed (lpg.runtime instead of lpg.lpgjavaruntime). Fortunately, regarding API changes, there are not too much changes (at least, for the first step I want to do in the migration process). Yesterday, I was able to do the migration from the QVTo side (to acomplish the first step). However, a lot of test cases were failing. I would need to investigate what is happening. Let's see if I get some results during this afternoon. Cheers, Adolfo.
Created attachment 154960 [details] proposed patch: First attempt
The attached patch shows the current status of my workspace against the current QVTo CVS, so that QVTo is able to work using LPG runtime v 2.0.17 To make all work you must follow the these instructions (assuming you are working on windows). - Apply the last patch in the the Bug 242153 to make MDT-OCL work with LPGv2. - Apply the provided patch in this bug to make QVTo work with LPGv2. - Download lpg.runtime.java project into your workspace (from the CVS dev.eclipse.org/tools/org.eclipse.orbit/lpg.runtime.java project, BRANCH v2_0_17). - Download lpg.generator.win32_86 into your workspace (from the CVS SourceForge LPG project, tag/version v2_0_17). - Place the lpg.generator.win32_86 project into the org.eclipse.m2m.qvt.oml.cst.parser/.externalTool folder - Rename lpg.generator.win32_86/lpgexe/lpg-win32_x86 file so that it has the ".exe" extension (This is to make the run-lpg.cmd work). - If not already placed, place a perl installation into the org.eclipse.m2m.qvt.oml.cst.parser/.externalTool folder - Execute "OperationalQVT CST Parser" external tool launch configuration. Apart from the expected changes of the migration, I have to mention some unexpected one: - The run-plg.cmd command has been slightly modified to make the LPG generating process success (At least in my machine and workspace). - During the migration of the OCL grammars, I had some minor problems when the $id$ of the OCLLexer, which I was not able to solve, due to LPG problems (generator bugs). I have also had some similar problems in the migration of the QVTo regarding the $NON-NLS-1$ comments in some actions blocks. To solve this, I have removed the NON-NLS comments from the grammars, and changed the run-lpg command so that they are again included by the script. I hope this is not an inconvenient. I haven't been able to properly run the QVTo test cases, I have also reverted to QVTo CVS Head with no luck in the launching. On the other hand, I have noticed a lot of changes in the CVS during these weeks, so I have decided to upload the patch so you may run the tests by yourself. Any advise, about QVTo tests launching will be welcomed. At least, there is no compilation errors after LPG generation, although we must ensure the parser behaves as it's expected. There was no problems from MDT-OCL side. LPG output noticed a lot of warnings (uses non terminals) when generating Lighweight(Type)Parser.lpg grammars. I think this is an expected output since its "lightweight" ;) For QVTo grammars, I would recommend extending the Essential grammar (EssentialKWLexer.gi, EssentialLexer.gi, EssentialOCL.gi). You could probably remove some Drop symbols and/or informative warnings about useless symbols. This is a temporal patch so you can figure out the impact of migration to LPGv2. Regarding the inclusion of LPGv 2.0.17 into Orbit. The final bundle will be named in the same fashion it's named in the sourceforge CVS. If you want to follow the progress: Bug 288666 No action should be taken until MDT-OCL side migration is complete: Bug 242153 Any feedback about the contribution will be welcomed. P.S: Sergey, If you have skype and/or gmail account, please let me know it by email so we both can solve any doubt in an agile way. I would specifically like to make the QVTo tests run :). Regards, Adolfo.
These installation procedures for LPG usage are far too complicated. I have created org.eclipse.mdt/org.eclipse.ocl/releng/psfs/parser-tools.psf to automate the LPG fetches (and fix some typos). It would be nice to reduce the residual activities to a single requirement to map the Run/Debug->String Substitution lpg2.exe => ${resource_loc:/lpg.generator.win32_x86/lpgexe/lpg-win32_x86} to select the user's favourite LPG2. I'm not at all clear why perl is needed.
Ed, I have just adapted myself to QVTo's developers tools (after understanding the mechanism :P). I'm not very enthusiastic about it, because it can only be used in windows, but it's very useful since it does several tasks at once: several LPGs runnings, externalizes the generated files, moves the files in the correct place,... I think that configuring a psf to obtain the generator could be good for us. Moreover, since psf are thought for developers, and the LPG generator is actually needed for them, I think that it should even be included in the official psf. However, I'm not sure if we are allowed to refer external projects. In any case, I'll probably deal with LPGv2.0.18 (runtime and generator) inclusion into Orbit in january/february , for IMP's project. If no changes from MDT-OCL side are required, we could then consider moving to the new version and point the our official psf to the Orbit's LPGv2.0.18 generator. About perl. A perl's script is used to externalize the strings (include the $NON-NLS$ when required - well only one per line is supported-) of the generated-by-lpg files. Sergey, MDT-OCL will migrate today so QVTo won't work when done. I'm also having some holidays (I'll back the 4th in January) today. so it could be interesting having some feedback before "disappearing". I think I could respond emails during the weekend. P.D: Merry Christmas.
(In reply to comment #6) Hi, Adolfo, Great thanks for that amazing patch and detailed instructions! According to steps you provided I've smoothly applied the patch. > The attached patch shows the current status of my workspace against the current > QVTo CVS, so that QVTo is able to work using LPG runtime v 2.0.17 > > To make all work you must follow the these instructions (assuming you are > working on windows). > > - Apply the last patch in the the Bug 242153 to make MDT-OCL work with LPGv2. > - Apply the provided patch in this bug to make QVTo work with LPGv2. > - Download lpg.runtime.java project into your workspace (from the CVS > dev.eclipse.org/tools/org.eclipse.orbit/lpg.runtime.java project, BRANCH > v2_0_17). > - Download lpg.generator.win32_86 into your workspace (from the CVS SourceForge > LPG project, tag/version v2_0_17). > - Place the lpg.generator.win32_86 project into the > org.eclipse.m2m.qvt.oml.cst.parser/.externalTool folder > - Rename lpg.generator.win32_86/lpgexe/lpg-win32_x86 file so that it has the > ".exe" extension (This is to make the run-lpg.cmd work). > - If not already placed, place a perl installation into the > org.eclipse.m2m.qvt.oml.cst.parser/.externalTool folder > - Execute "OperationalQVT CST Parser" external tool launch configuration. > > Apart from the expected changes of the migration, I have to mention some > unexpected one: > - The run-plg.cmd command has been slightly modified to make the LPG generating > process success (At least in my machine and workspace). Thank you for introducing %LPG_EXE% and fixing problem with files generated for completion grammar (reference to 'completion' folder was missed). > - During the migration of the OCL grammars, I had some minor problems when the > $id$ of the OCLLexer, which I was not able to solve, due to LPG problems > (generator bugs). I have also had some similar problems in the migration of the > QVTo regarding the $NON-NLS-1$ comments in some actions blocks. To solve this, > I have removed the NON-NLS comments from the grammars, and changed the run-lpg > command so that they are again included by the script. I hope this is not an > inconvenient. Not at all :) Thank you again for the modifications. > > I haven't been able to properly run the QVTo test cases, I have also reverted > to QVTo CVS Head with no luck in the launching. On the other hand, I have > noticed a lot of changes in the CVS during these weeks, so I have decided to > upload the patch so you may run the tests by yourself. Any advise, about QVTo > tests launching will be welcomed. At least, there is no compilation errors > after LPG generation, although we must ensure the parser behaves as it's > expected. There was no problems from MDT-OCL side. There's ONE minor deviation in OCLLexer.g(OCLLexer.gi) in our local copy that affects tests. I have to made it since I can't find how to overcome the problem without grammar modification. Possibly it's due to some bug in LPG generator. This deviation is about '$kw_lexer_class' definition in 'OCLLexer.gi'. Our modification is of one character: line [50] in OCL $kw_lexer_class /.OCLKWLexer./ line [50] in our local copy $kw_lexer_class /.$OCLKWLexer./ Trying to redefine $kw_lexer_class /.QVTOKWLexer./ in QVTOLexer grammar file doesn't help. So addition indirection was introduced and QVTOLexer grammar file now defines $OCLKWLexer /.QVTOKWLexer./ Once providing that modification all QVTo tests (except 5 tests that are about error recovery during parsing and should be revised) become green. > > LPG output noticed a lot of warnings (uses non terminals) when generating > Lighweight(Type)Parser.lpg grammars. I think this is an expected output since > its "lightweight" ;) Right ;) > > For QVTo grammars, I would recommend extending the Essential grammar > (EssentialKWLexer.gi, EssentialLexer.gi, EssentialOCL.gi). You could probably > remove some Drop symbols and/or informative warnings about useless symbols. I didn't quite understand what do you mean. I can't find those files in 'org.eclipse.ocl' plug-in. > > This is a temporal patch so you can figure out the impact of migration to > LPGv2. > > Regarding the inclusion of LPGv 2.0.17 into Orbit. The final bundle will be > named in the same fashion it's named in the sourceforge CVS. If you want to > follow the progress: Bug 288666 > > No action should be taken until MDT-OCL side migration is complete: Bug 242153 > > Any feedback about the contribution will be welcomed. So far only splendid impression. Excellent patch! I'll take a look at those tests (5 in total) that fail now and come back with comments. > > P.S: Sergey, If you have skype and/or gmail account, please let me know it by > email so we both can solve any doubt in an agile way. I would specifically like > to make the QVTo tests run :). > > Regards, > Adolfo.
(In reply to comment #8) Intent in using perl (and command shell as well) is to 100% automatize parser generation. So that none hand modification would be required. Any suggestions how to archive that using other (simpler) solutions are welcome. > Ed, > > I have just adapted myself to QVTo's developers tools (after understanding the > mechanism :P). I'm not very enthusiastic about it, because it can only be used > in windows, but it's very useful since it does several tasks at once: several > LPGs runnings, externalizes the generated files, moves the files in the correct > place,... > > I think that configuring a psf to obtain the generator could be good for us. > Moreover, since psf are thought for developers, and the LPG generator is > actually needed for them, I think that it should even be included in the > official psf. However, I'm not sure if we are allowed to refer external > projects. > > In any case, I'll probably deal with LPGv2.0.18 (runtime and generator) > inclusion into Orbit in january/february , for IMP's project. If no changes > from MDT-OCL side are required, we could then consider moving to the new > version and point the our official psf to the Orbit's LPGv2.0.18 generator. > > About perl. A perl's script is used to externalize the strings (include the > $NON-NLS$ when required - well only one per line is supported-) of the > generated-by-lpg files. > > Sergey, MDT-OCL will migrate today so QVTo won't work when done. I'm also > having some holidays (I'll back the 4th in January) today. so it could be > interesting having some feedback before "disappearing". I think I could respond > emails during the weekend. Ok, we're ready for migration. > > P.D: Merry Christmas.
Sergey thank you very much for A quick response. Today (after running all the OCL tests again ) I have also noticed some failures in some OCL backtracking-related tests. It seems that the error message has changed, I'm looking into it to check if its grammar's failure, or what's the reason of this error message change. Response to your comments, later. Cheers, Adolfo.
(In reply to comment #11) > Sergey thank you very much for Your comments :) > A quick response. Today (after running all the OCL tests again ) I have also > noticed some failures in some OCL backtracking-related tests. It seems that the > error message has changed, I'm looking into it to check if its grammar's > failure, or what's the reason of this error message change. Oh !!! I have discovered a funny message for the backtracking parser: OCLParserErrors.gi:35:60:35:70:970:980: Warning: The terminal symbol "ERROR_TOKEN" was not imported from OCLBacktrackingLexer.gi It seems that I have just focused on the deterministic, and I haven't payed enough attention to the backtracking on. Let's see If i can find out why the generator can't import or identy the error_token. This must be a difference between LPGv1 vs LPGv2 ^^ Cheers, Adolfo
Well, The OCL tests cases fail because an different error code is generated. Now it receives an INVALID_TOKEN code when it previously received an SUBSTITUTION_SECONDARY one. I guess that the warning (about the ERROR_TOKEN mentioned before) could be changing the generated parser behaviour, and hence a different error code. However: 1. I'm not sure about that, 2. I haven't discovered how to fix it. I'll go on investigation after lunch. Cheers, Adolfo.
> > Hi, Adolfo, > > Great thanks for that amazing patch and detailed instructions! > According to steps you provided I've smoothly applied the patch. > Great !!! > > There's ONE minor deviation in OCLLexer.g(OCLLexer.gi) in our local copy that > affects tests. I have to made it since I can't find how to overcome the problem > without grammar modification. Possibly it's due to some bug in LPG generator. > > This deviation is about '$kw_lexer_class' definition in 'OCLLexer.gi'. > Our modification is of one character: > line [50] in OCL > $kw_lexer_class /.OCLKWLexer./ > line [50] in our local copy > $kw_lexer_class /.$OCLKWLexer./ > > Trying to redefine > $kw_lexer_class /.QVTOKWLexer./ > in QVTOLexer grammar file doesn't help. > So addition indirection was introduced and QVTOLexer grammar file now defines > $OCLKWLexer /.QVTOKWLexer./ > > Once providing that modification all QVTo tests (except 5 tests that are about > error recovery during parsing and should be revised) become green. Sergey, try to do the following: - Remove the unnecessary %import LexerBasicMap.gi (It's already imported in the EssentialOCLLexer.gi) - Interchange %define and %Import OCLLexer.gi sections (So that firstly the import is done, and then the macro definitions are overriden). %Import OCLLexer.gi %DropRules Token ::= '"' SLNotDQ '"' %End %Define -- -- Definition of macro used in the included file OCLLexer.g -- $kw_lexer_class /.QVTOKWLexer./ %End This should let you generate the appropiate QVTOLexer without that indirection (and then no need to clone the OCLLexer). > > > > > LPG output noticed a lot of warnings (uses non terminals) when generating > > Lighweight(Type)Parser.lpg grammars. I think this is an expected output since > > its "lightweight" ;) > > Right ;) > > > > > For QVTo grammars, I would recommend extending the Essential grammar > > (EssentialKWLexer.gi, EssentialLexer.gi, EssentialOCL.gi). You could probably > > remove some Drop symbols and/or informative warnings about useless symbols. > > I didn't quite understand what do you mean. I can't find those files in > 'org.eclipse.ocl' plug-in. In OCL 1.3 OCLParser grammar was split into two files: OCLParser.g and EssentialOCL.g so that unnecessary CompleteOCL-concerns are not included or used by EssentialOCL extenders languages such as QVTo or QVTd. Some milestones ago, Ed also split the OCLLexer and OCLKWLexer into two grammars files for the same purpose. These files already reside in the same place other .g (.gi) files are. Actually QVTo has also cloned those files (look at org.eclipse.m2m.qvt.oml.cst.parser\cst). I basically suggestes changing the QVTo grammars so they extend EssentialKWLexer, EssentialLexer and EssentialOCL instead of OCLLexer, OCLKWLexer and OCLParser respectively. > > > > > This is a temporal patch so you can figure out the impact of migration to > > LPGv2. > > > > Regarding the inclusion of LPGv 2.0.17 into Orbit. The final bundle will be > > named in the same fashion it's named in the sourceforge CVS. If you want to > > follow the progress: Bug 288666 > > > > No action should be taken until MDT-OCL side migration is complete: Bug 242153 > > > > Any feedback about the contribution will be welcomed. > > So far only splendid impression. Excellent patch! > I'll take a look at those tests (5 in total) that fail now and come back with > comments. Thank you very much. I guess that those errors are caused by the same problem which violates the OCL test cases. Let's see if I we can solve the problem during today. Cheers, Adolfo. > > > > P.S: Sergey, If you have skype and/or gmail account, please let me know it by > > email so we both can solve any doubt in an agile way. I would specifically > like > > to make the QVTo tests run :). > > > > Regards, > > Adolfo.
(In reply to comment #14) > > > > > There's ONE minor deviation in OCLLexer.g(OCLLexer.gi) in our local copy that > > affects tests. I have to made it since I can't find how to overcome the problem > > without grammar modification. Possibly it's due to some bug in LPG generator. > > > > This deviation is about '$kw_lexer_class' definition in 'OCLLexer.gi'. > > Our modification is of one character: > > line [50] in OCL > > $kw_lexer_class /.OCLKWLexer./ > > line [50] in our local copy > > $kw_lexer_class /.$OCLKWLexer./ > > > > Trying to redefine > > $kw_lexer_class /.QVTOKWLexer./ > > in QVTOLexer grammar file doesn't help. > > So addition indirection was introduced and QVTOLexer grammar file now defines > > $OCLKWLexer /.QVTOKWLexer./ > > > > Once providing that modification all QVTo tests (except 5 tests that are about > > error recovery during parsing and should be revised) become green. > > Sergey, try to do the following: > - Remove the unnecessary %import LexerBasicMap.gi (It's already imported in the > EssentialOCLLexer.gi) > - Interchange %define and %Import OCLLexer.gi sections (So that firstly the > import is done, and then the macro definitions are overriden). > > %Import > OCLLexer.gi > > %DropRules > > Token ::= '"' SLNotDQ '"' > > %End > > %Define > > -- > -- Definition of macro used in the included file OCLLexer.g > -- > $kw_lexer_class /.QVTOKWLexer./ > > %End > > This should let you generate the appropiate QVTOLexer without that indirection > (and then no need to clone the OCLLexer). Just tried that but unfortunately that doesn't help. Result is that QVTOLexer.java is not compilable. I remember that I spent quite some time to resolve that problem and tried similar approaches (import reordering, etc.). But the only solution I found is an additional indirection for macro definition. > > > > > > > > > LPG output noticed a lot of warnings (uses non terminals) when generating > > > Lighweight(Type)Parser.lpg grammars. I think this is an expected output since > > > its "lightweight" ;) > > > > Right ;) > > > > > > > > For QVTo grammars, I would recommend extending the Essential grammar > > > (EssentialKWLexer.gi, EssentialLexer.gi, EssentialOCL.gi). You could probably > > > remove some Drop symbols and/or informative warnings about useless symbols. > > > > I didn't quite understand what do you mean. I can't find those files in > > 'org.eclipse.ocl' plug-in. > > In OCL 1.3 OCLParser grammar was split into two files: OCLParser.g and > EssentialOCL.g so that unnecessary CompleteOCL-concerns are not included or > used by EssentialOCL extenders languages such as QVTo or QVTd. Some milestones > ago, Ed also split the OCLLexer and OCLKWLexer into two grammars files for the > same purpose. > > These files already reside in the same place other .g (.gi) files are. Actually > QVTo has also cloned those files (look at > org.eclipse.m2m.qvt.oml.cst.parser\cst). I basically suggestes changing the > QVTo grammars so they extend EssentialKWLexer, EssentialLexer and EssentialOCL > instead of OCLLexer, OCLKWLexer and OCLParser respectively. Now I see. Sounds very reasonable. I'll check and migrate onto 'essential' grammar after aligning with LPG2. > > > > > > > > > This is a temporal patch so you can figure out the impact of migration to > > > LPGv2. > > > > > > Regarding the inclusion of LPGv 2.0.17 into Orbit. The final bundle will be > > > named in the same fashion it's named in the sourceforge CVS. If you want to > > > follow the progress: Bug 288666 > > > > > > No action should be taken until MDT-OCL side migration is complete: Bug 242153 > > > > > > Any feedback about the contribution will be welcomed. > > > > So far only splendid impression. Excellent patch! > > I'll take a look at those tests (5 in total) that fail now and come back with > > comments. > > Thank you very much. I guess that those errors are caused by the same problem > which violates the OCL test cases. Let's see if I we can solve the problem > during today. > > Cheers, > Adolfo. > > > > > > > P.S: Sergey, If you have skype and/or gmail account, please let me know it by > > > email so we both can solve any doubt in an agile way. I would specifically > > like > > > to make the QVTo tests run :). > > > > > > Regards, > > > Adolfo.
(In reply to comment #14) > > > > So far only splendid impression. Excellent patch! > > I'll take a look at those tests (5 in total) that fail now and come back with > > comments. > > Thank you very much. I guess that those errors are caused by the same problem > which violates the OCL test cases. Let's see if I we can solve the problem > during today. > Looks like definition of ERROR_TOKEN should be moved from EssentialOCL.gi somewhere to lexer part (to EssentialOCLLexer.gi for example). Otherwise it can't be recognized equally by lexer and parser parts. For my cases that helps.
> > Just tried that but unfortunately that doesn't help. Result is that > QVTOLexer.java is not compilable. > I thought I had checked my suggestion, and all worked as expected . I'm afraid that we will need to wait until tomorrow, to complete the migration, I can't do it right now. > Looks like definition of ERROR_TOKEN should be moved from EssentialOCL.gi > somewhere to lexer part (to EssentialOCLLexer.gi for example). Otherwise it > can't be recognized equally by lexer and parser parts. > For my cases that helps. Hopefully, your proposal solves the OCLBactrackingParser as well. I'll face on this tomorrow. Merry Christmas !!! Adolfo.
> > I thought I had checked my suggestion, and all worked as expected . I'm afraid > that we will need to wait until tomorrow, to complete the migration, I can't do > it right now. Checked, Test cases run for me. So the solution I proposed should work. I'll send a new patch including this changes when the backtracking-related issue is solved. > > > Looks like definition of ERROR_TOKEN should be moved from EssentialOCL.gi > > somewhere to lexer part (to EssentialOCLLexer.gi for example). Otherwise it > > can't be recognized equally by lexer and parser parts. > > For my cases that helps. > > Hopefully, your proposal solves the OCLBactrackingParser as well. I'll face on > this tomorrow. > Sergey, Did I'm wrongly understand that you fixed your test cases, suppressing the said warning as you have suggested. I have tried a lot of things with no luck from MDT-OCL side. Ed, I don't know what to do with this. I shouldn't commit anything until test cases properly run again. I'm not sure if this changes in the error codes generated by backtracking parser in the test cases, is what the generated parse should do or it shouldn't. In anycase, it seems that warning is a clue that something is not going well. That said, I have required some help from LPG team to try to discover the cause of that warning. Regards, Adolfo.
Hi, sorry for the delayed response; Xmas... The changes to make QVTd work with LPG2 were straightforward, and QVTd uses the backtracking parser; no reordering was needed, so I'm not clear whether QVTd happens to have a good order, or has test cases that fail to identify the problem. With OCL and QVTd seemingly ok, and QVTo within reach of a fix, perhaps we can commit.
Committed.
Comment on attachment 154960 [details] proposed patch: First attempt Adding iplog+
