Bugzilla – Full Text Bug Listing
| Summary: | EclipseCorner Article on Functional Test Automation using TPTP | ||
|---|---|---|---|
| Product: | Community | Reporter: | Govin Varadarajan <gvaradar> |
| Component: | Articles | Assignee: | community.articles-inbox <community.articles-inbox> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | P3 | CC: | amehrega, elias, gvaradar, paulslau, popescu, scott.schneider, slavescu |
| Version: | unspecified | ||
| Target Milestone: | --- | ||
| Hardware: | PC | ||
| OS: | Windows XP | ||
| Whiteboard: | |||
| Attachments: | |||
|
Description
Govin Varadarajan
I believe that Eclipse Article contributions are supposed to be opened against the "Community" project's "Articles" component. I'll move this over. That said, I'm adding the proposed new Project Lead of the TPTP Test project (Paul Slauenwhite) to the cc field both as a TPTP contact for you and so that someone from TPTP knows when your article is approved and we can add a link to it from the TPTP web site. Thank you for your interest in TPTP. +1 +1 +1 Anything I can do to help make this article happen? Govin, can you provide an update on your progress? We have the major portions of the paper written but still have to put it in the formal "EclipseCorner article" format and also update the introduction section(s) of the article to include the philosophy behind our solution design. But, we can have the paper ready for review by EclipseCorner article editors in two weeks" I'm hardly one to pass judgment on missed self-imposed deadlines, but how's that two week thing working for you? Remember that--in terms of formatting--simpler is better... The paper has been written and is under management review. Created attachment 59682 [details]
Article on how used TPTP Technologies in designing our functional test automation solution
(In reply to comment #10) > Created an attachment (id=59682) [details] > Article on how used TPTP Technologies in designing our functional test > automation solution > I see that you created your own query model service, is this based on the existing TPTP model query mechanism which is currently used in several places like Log Viewer, Log Interaction Viewer, Trace related viewers, including the Test Log Viewer -> Search (to search in the current view although it's fairly easy to do it also across resources like we use it in the other cases). In TPTP 4.4 the model access and query mechanism will be extended and partially replaced by a new (provisional) API (especially in the test execution results area), if you'll like to target this article also for TPTP 4.4 would be good to mention the new mechanisms, I can provide more information if you are interested. In reply to comment # 11 We created our own model query service that runs on TPTP ASF because at the time we built our solution (around May/June 2006) the model interrogation service that was available from TPTP ASF was very rudimentary and did not meet all our needs (that is we were not able to gather all the information that we needed to gather from the executionhistory and testsuite files with the interrogation service that was available then). I understand that TPTP4.4 GA is around June/July 2007 and I am also aware of some other AGR enhancement work (such as GEF and Drag/Drop to diagram surface support) that is presently in progress. Perhaps I could write another update article that includes these enhancements. In any case, I would certainly be interested in understanding more about the new mechanisms that you mention. Thanks. (In reply to comment #12) > In reply to comment # 11 > > We created our own model query service that runs on TPTP ASF because at the > time we built our solution (around May/June 2006) the model interrogation > service that was available from TPTP ASF was very rudimentary and did not meet > all our needs (that is we were not able to gather all the information that we > needed to gather from the executionhistory and testsuite files with the > interrogation service that was available then). I suspect the TPTP ASF is not exposing the mechanism used in the Test Log viewer. > I understand that TPTP4.4 GA is around June/July 2007 and I am also aware of > some other AGR enhancement work (such as GEF and Drag/Drop to diagram surface > support) that is presently in progress. Perhaps I could write another update > article that includes these enhancements. In any case, I would certainly be > interested in understanding more about the new mechanisms that you mention. > Thanks. > Sure just let me know when you need more info regarding this subject. TPTP Test Project would like to review this article before publishing. Can we take the next week to review? (In reply to comment #14) > TPTP Test Project would like to review this article before publishing. Can we > take the next week to review? > Please do. Hi Govin, Great article. I have a few comments: 1) In the Technology Section, you state 'The AGR's verification support via Java Hooks mechanism supports our approach'. As an introduction to this type of verification, it might be good to explain it more clearly. On this note, you should double check for consistency of the terms (Java hook/Java hooks mechanism) 2) You may want to qualify your opening statements about the testing system that is not sensitive to minor UI changes. Later you explain that great care should be taken to avoid any differences in behaviour between your two environments. Also, you don't want to give the impression that AGR will not be sensitive to UI changes in the sense that it will pass a test case in error. 3) It would be a good idea to mention the version of TPTP used earlier in the article. 4) Perhaps mention that the Object Mine capability is optional (therefore developers wishing to copy and paste Macros can still do so when not using the Object Mine). 5) Maybe Paul can comment more on this but I am not aware of the AGR moving to GA in the 4.4 release. 6) The Verification classes are an extension of JUnits but it might be worth pointing out this extension (rather then referring to them as Junit classes). 7) The items in a tree are resolved differently based on a number of factors. In some instances they are resolved using their index. Maybe you could explain the use of the threshold value and the WidgetReg file to clarify. Created attachment 61468 [details]
Article updated based on Liz's comments and some additional information
Thank you Liz for the comments. We have updated the article based on your comments and also provided some additional information. All our updates are in blue for the reviewers convenience.
Created attachment 61474 [details]
Figure 1
Govin,
Here's another version of figure 1 that I created. You may want to use this instead.
Thanks,
Ali Mehregani
Created attachment 61544 [details]
Paper updated with Figure from Ali Mehregani
When the editors find no other changes to be made to the paper, I shall change the text that is blue back to black. Thanks for the figure, Ali.
Thanks,
Govin
Please assume that there are no comments from reviewers if no one replies to this defect by end of this week. Let's get this rolling by Monday March 26th. Thanks Govin for your commitment. I will plan to push this out next week then. I'd still like a comment to the effect of "it's ready to publish". Also, I need a comment from the author that states that the article is all original content and that the author agrees to release it under the EPL. I will post my comments by EOW. Govin, As we discussed on the phone, would be good to make more clear what you capture in the database. In the Introduction, where you say "a database to store the test execution results" and in the other places where the custom log and database is mentioned. Govin and Kim, This is a great article! TPTP and the Test Tools Project appreciate the visibility that your article provides for the project and the AGR. Our hope is that type of case study will motivate readers to use TPTP/AGR to experience the usefulness and extensibility of the TPTP Test framework and exemplary tooling. Apologies for the lateness, but here are few comments based on the original article (https://bugs.eclipse.org/bugs/attachment.cgi?id=59682): 1) We do not have concrete plans to move the AGR to GA so please remove the reference to the AGR moving to GA in the 4.4 release (e.g. last paragraph in [Appendix A: Technology in the Functional Test Automation Solution --> TPTP Automated GUI Recorder]). 2) [Testing Solution Design --> Overview] Consider changing "TPTP captures the test suites containing these types of tests in instances of EMF models implementing test data model specifications such as UML2 Testing Profile (UML2TP)." to "TPTP models test suites containing these types of tests in instances of EMF models implementing open-source test data model specifications such as the OMG's UML2 Testing Profile (UML2TP).". 3) [Testing Solution Design --> Overview] Consider changing "These execution history files may be queried for test execution information and results." to "These execution history files may be viewed using the Tset Log view, reported using the integrated and extensible BIRT reports and queried for test execution information and results.". 4) [Test Execution Service] Consider changing "TPTP captures the developed test suites in instances of the TPTP Testing Profile model (*.testsuite)." to "TPTP captures the developed test suites in instances of the TPTP Testing Profile model (serialized as compressed XMI in *.testsuite files).". 5) [Model Query Service] Consider changing "The execution of the test suites result in the creation of instances of TPTP Execution History model." to "The execution of the test suites result in the creation of instances of TPTP Execution History model (serialized as compressed XMI in *.execution files).". 6) [Test Execution in RMPTAS] Consider changing "although an IAC will suffice" to "although the IAC will suffice". 7) [Test Execution in RMPTAS] Consider changing "Start standalone Agent Controller." to "Start the stand-alone Agent Controller.". 8) We commonly refer to Technology Previews using title case. 9) [Some Known Issues with AGR --> Known Problems with AGR] Note which ones have since been or will be (e.g. targeted to 4.4 or an iteration) resolved in 4.4. Also provide links to the defects. 10) [Some Known Issues with AGR --> Known Problems with AGR] In bullet #5, title case Windows. 11) [Some Known Issues with AGR --> Known Problems with AGR] In bullet #5, title case Eclipse. 12) [Some Known Issues with AGR --> AGR Feature Enhancements] Provide links to the engagements. 13) [Concluding Remarks] Mention how users/extenders can get involved in the project by providing links to the project's Building on TPTP (http://www.eclipse.org/tptp/home/project_info/general/buildon.html) and Contributing to TPTP (http://www.eclipse.org/tptp/home/project_info/general/contribute_or_feedback.html) sections. 14) [Concluding Remarks] Remove "complete the instrumentation of drag and drop and". 15) [Appendix A: Technology in the Functional Test Automation Solution --> TPTP] Change "2005" to "2004". 16) [Appendix A: Technology in the Functional Test Automation Solution --> TPTP] Consider changing "The platform is designed to facilitate integration with tools used in the different processes of a software lifecycle under Eclipse environment." to "The platform is designed to facilitate integration with tools used in the different processes of a software lifecycle under an Eclipse environment.". 17) [Appendix A: Technology in the Functional Test Automation Solution --> TPTP] Consider changing "TPTP defines four sub projects – Platform, Testing tools, Trace & Profiling and Monitoring." to "TPTP defines four sub projects – Platform, Testing Tools, Trace & Profiling and Monitoring.". 18) [Appendix A: Technology in the Functional Test Automation Solution --> TPTP Communication Framework - Agents and Agent Controller] Consider changing "The TPTP communication framework is specifically designed to support testing and performance analysis engine, and provides a streamlined connection to the rest of TPTP." to "The TPTP communication framework is specifically designed to support testing and performance tools, and provides a streamlined connection to the rest of TPTP.". 19) [Appendix A: Technology in the Functional Test Automation Solution --> TPTP Communication Framework - Agents and Agent Controller] Consider changing "The Agent Controller contains a server that has the capability to launch and manage local or remote applications from a local TPTP workbench." to "The Agent Controller provides a server that has the capability to launch and manage local or remote applications from a local TPTP workbench.". 20) [Appendix A: Technology in the Functional Test Automation Solution --> TPTP Communication Framework - Agents and Agent Controller] Title case Agent Controller. 21) [Appendix A: Technology in the Functional Test Automation Solution --> TPTP Communication Framework - Agents and Agent Controller] Consider changing "TPTP workbench also provides a feature called the Integrated Agent Controller (IAC) that could be used if the tests are to be deployed and executed only locally." to "TPTP workbench also provides a feature called the Integrated Agent Controller (IAC) that is used to launch tests locally from the TPTP workbench.". 22) [Appendix A: Technology in the Functional Test Automation Solution --> TPTP Communication Framework - Agents and Agent Controller] Consider changing "For example: there is no need to install and configure the local Agent Controller when using the Test perspective or Profiling and Logging perspective locally." to "For example: there is no need to install and configure the local Agent Controller when launching tests locally from the TPTP workbench.". 23) Note the version of TPTP you are using early in the article and update reference [3] is not 4.2.x. 24) [Appendix A: Technology in the Functional Test Automation Solution --> TPTP Automated GUI Recorder] Consider changing "AGR captures the user gestures through XML based scripts." to "AGR captures the user gestures as XML based scripts.". 25) [Appendix A: Technology in the Functional Test Automation Solution --> TPTP Automated GUI Recorder] Consider changing "AGR provides a simple “VCR-type” interface to record and play back test cases." to "AGR provides a simple “VCR-type” interface to record test cases.". 26) [Appendix A: Technology in the Functional Test Automation Solution --> TPTP Automated GUI Recorder] Consider changing "In the Standard mode the test cases are run on a run-time workbench using a run configuration." to "In the Standard mode the test cases are run on a run-time workbench using a launch configuration.". 27) [Appendix A: Technology in the Functional Test Automation Solution --> TPTP Automated GUI Recorder] Consider changing "Defining a context for test suite execution typically requires the user to create a deployment specifying what needs to be executed (the test suite) and where it needs to be executed (the location)." to "Defining a context for test suite execution typically requires the user to create a deployment specifying what needs to be executed (the artifact containing the test suite and possible datapool) and where it needs to be executed (the location).". 28) [Appendix A: Technology in the Functional Test Automation Solution --> TPTP Automatable Services Framework] Title case Automatable Service Framework. Created attachment 62019 [details]
Updated with comments from Paul
Paul, Thanks for the comments. Updated article has been attached.
Created attachment 62414 [details]
Title case modifications
A few minor modifications in the spirit of some "Title case" comments made earlier by Paul. Also, the authors state that the article is original content and agree to release it under the EPL.
Article Reviewers: Any further comments?
(In reply to comment #26) No. Thanks to all the article reviewers for the comments. Article Publisher(s): Any thing else I need to do? Based on positive feedback from the reviewers, I will start the publishing process. I've posted the article. http://www.eclipse.org/articles/article.php?file=Article-TPTPAutomateFunctionalTesting/index.html Please give it a once-over and let me know if everything is okay. When I get your go-ahead, I'll add it to the database so that others can find it. Wayne: Thank you for your efforts for bringing the article so close to publishing. However, I just realised a couple of things that need to be done before we publish it to the world. 1. I just realised that I forgot to propagate some of the changes that the co-author (Kim) made to the "Technology" section in the later versions of article submissions. 2. Also, I would like to modify one "choppy" sentence in Item #1 of section entitled "Known Problems with TPTP AGR4.2.1 and 4.3". What is the best way to get these changes in? Would it be easier for you if I just provide you with the modified parts or the whole article as a zip file ? On the other hand, a couple of other minor comments on the formatting of the article: 1. Title for Figure 3 should come after the figure (to be consistent with the other figure titles). 2. In the "Known Problems with AGR4.2.1 and 4.3" section, the spacing between the items seems inconsistent. Also, some paragraphs seem to have a different font size compared to the rest. Thanks, Govin Created attachment 63657 [details]
Some article change before publishing
Wayne:
I have created a notepad file attachment entitled "AGRArticleChanges_20070412_1453.txt" for your convenience. In this document I have provided sufficient explanation to help you locate the right sections in the article where the changes need to be made, and the changes that need to be made.
Please let me know if you need further information.
Thanks,
Govin
(In reply to comment #32) > Created an attachment (id=63657) [details] > Some article change before publishing Sorry for the delay. I'll try to work in these changes over the next few days. Any updates Wayne ? Posted. Created attachment 65476 [details]
mylar/context/zip
Thanks everyone (Wayne, Scott, Joe, Paul, Ali, Liz, Marius and Ruth) for all your help, efforts and encouragement in writing this paper and getting it published. The ride was really awesome!! I believe comment https://bugs.eclipse.org/bugs/show_bug.cgi?id=165526#c15 was intended for this defect. Created attachment 67387 [details]
Figures 1
Author: Ali M.
I believe Ali attached a newer/updated version of the figure on March 20 as Comment #18 to this bug report. This newer/updated version of the figure was used in the article. Re: Comment#38: I was not aware of this comment from Ali that he by mistake had posted to https://bugs.eclipse.org/bugs/show_bug.cgi?id=165526#c15 . I shall work on incorporating the comments in the next update to the article, soon after the TPTP4.4 release. Thanks Ali & Paul Govin |