Bugzilla – Full Text Bug Listing
| Summary: | Add Ant task for generating Eclipse Help from a Confluence wiki similar to "mediawiki-to-eclipse-help" | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | z_Archived | Reporter: | Torkild Resheim <torkildr> | ||||||||
| Component: | Mylyn | Assignee: | Torkild Resheim <torkildr> | ||||||||
| Status: | RESOLVED WONTFIX | QA Contact: | David Green <greensopinion> | ||||||||
| Severity: | enhancement | ||||||||||
| Priority: | P3 | CC: | caniszczyk, contact, dsciamma, d_a_carver, gmorice, greensopinion | ||||||||
| Version: | unspecified | Keywords: | noteworthy | ||||||||
| Target Milestone: | --- | ||||||||||
| Hardware: | All | ||||||||||
| OS: | All | ||||||||||
| Whiteboard: | |||||||||||
| Attachments: |
|
||||||||||
|
Description
Torkild Resheim
Created attachment 187593 [details]
mylyn/context/zip
We have something working as part of the Turmeric project at www.ebayopensource.org. Example usage can be seen here: https://www.ebayopensource.org/viewsvn/turmeric/soa-platform/trunk/tools/eclipse/docs/org.ebayopensource.turmeric.eclipse.doc/buildHelp.xml?view=markup Binary JARs and dependency: https://www.ebayopensource.org/viewsvn/turmeric/soa-platform/trunk/tools/eclipse/docs/org.ebayopensource.turmeric.eclipse.doc/lib/ Source here: https://www.ebayopensource.org/viewsvn/turmeric/soa-platform/trunk/tools/eclipse/utilities/confluence-ant-task/ This is licensed under the APL 2.0, and we use it to generate our documentation maintained on the Turmeric Wiki into eclipse help format: https://www.ebayopensource.org/wiki/display/TURMERICDOC/Turmeric+Documentation+Overview It includes similar functionality to the existing mediawiki ant task. If you're willing to re-license as EPL maybe we can toss the code on GitHub while it's massaged for a code donation to eclipse.org? (In reply to comment #2) > We have something working as part of the Turmeric project at > www.ebayopensource.org. Doh... Wish I knew. Anyway I should have a EPL licensed patch ready tomorrow. I'm working on cleaning up the code and writing a user guide. From what I can see the implementations are similar in features so it's up to you guys what you would like to do with it. (In reply to comment #4) > (In reply to comment #2) > > We have something working as part of the Turmeric project at > > www.ebayopensource.org. > Doh... Wish I knew. Anyway I should have a EPL licensed patch ready tomorrow. > I'm working on cleaning up the code and writing a user guide. From what I can > see the implementations are similar in features so it's up to you guys what you > would like to do with it. If you have something that already works Torkild that's great. It would save Dave time from refactoring source and then he could just use what you have (need to make sure it meets his needs though) Fantastic. Are either of you planning on bringing this to Eclipse.org? (In reply to comment #5) > If you have something that already works Torkild that's great. It would save > Dave time from refactoring source and then he could just use what you have (need > to make sure it meets his needs though) OK, I'll have a closer look at the feature set in the Turmeric project and see if I'm missing something vital. Anything else should be straightforward to add. (In reply to comment #6) > Fantastic. Are either of you planning on bringing this to Eclipse.org? Yes. That's the plan. Marintek (http://www.marintek.com) was needing a feature like this and the tool they were testing out did not quite do it. So they decided to implement it themselves (which I've been doing this week). When I brought up the question about donating they were quite happy about it as long as they got some credit. As soon as you have something decent, feel free to throw up a patch here or on GitHub. We can start the CQ process once we have something decent :D I've pushed my repository clone to GitHub (https://github.com/turesheim/org.eclipse.mylyn.docs/commit/de173c742fc5fc27b252c559f53226ac2fff082f) so you can pull the changes from there. Please let me know if something needs to be fixed (although I think it's pretty decent :-). The documentation (user guide) has been updated so you should be able to get an idea of it's features. Looks good. Can you attach a patch to this bug with the code changes? Also, can you state that you: a. authored 100% of the code b. has the rights to donate the content to Eclipse c. contributes the content under the EPL Once you do this, David can create the CQ and we can move this contribution forward. (In reply to comment #10) > Looks good. > > Can you attach a patch to this bug with the code changes? Sure, need to figure out how. > > Also, can you state that you: > > a. authored 100% of the code Yes, except from the parts generated by Axis. > b. has the rights to donate the content to Eclipse Yes. > c. contributes the content under the EPL Yes. > > Once you do this, David can create the CQ and we can move this contribution > forward. Good... I'll start looking at the patch thing. Created attachment 187999 [details]
Ant task for generating Eclipse Help from a Confluence wiki
Great patch, thanks! A few things that I would like to see addressed: * All classes should be in the @org.eclipse.mylyn.internal.wikitext.confluence@ package namespace -- I noticed org.apache and com.atlassian packages, those should be renamed * Please consider refactoring @org.eclipse.mylyn.internal.wikitext.mediawiki.core.PageMapping@ into @org.eclipse.mylyn.wikitext.core.parser.markup@ rather than copying the class * This patch adds several new compile-time dependencies. It looks like these are due to the use of Apache Axis? Please ensure that these are a minimal set and that they're added as dependencies to the bundle manifest (they should not appear in the .classpath file) ** org.apache.axis_1.4.0.v201005080400 ** org.apache.commons.discovery_0.2.0.v201004190315 ** javax.xml.rpc_1.1.0.v201005080400 ** javax.xml.soap_1.2.0.v201005080501 ** javax.wsdl_1.6.2.v201005080631.jar ** org.apache.commons.logging_1.1.1.v201005080502 * With these additional dependencies, please move contribution to take the form of a new bundle, named @org.eclipse.mylyn.wikitext.mediawiki.confluence.core@ * Please don't use the @src_ant@ source folder. Use the @src@ folder instead. bug 302616 has details on why we're not doing this in WikiText. Overall the contribution looks great! Note that this contribution relates to our "Project Charter":http://wiki.eclipse.org/Mylyn/Charter since it involves integration with a 3rd party commercial system. I've put this on the agenda for this week's "meeting":http://wiki.eclipse.org/Mylyn/Meetings. Feel free to attend to be part of the discussion. (In reply to comment #13) > Great patch, thanks! A few things that I would like to see addressed: Thanks for the feedback. Will look into this. I have some questions/comments though. > * All classes should be in the @org.eclipse.mylyn.internal.wikitext.confluence@ > package namespace -- I noticed org.apache and com.atlassian packages, those > should be renamed Please note that the reason for this was that this code was automatically generated from the Confluence WSDL using Axis. If a new version of the web service description comes out the process should be repeated. > * Please consider refactoring > @org.eclipse.mylyn.internal.wikitext.mediawiki.core.PageMapping@ into > @org.eclipse.mylyn.wikitext.core.parser.markup@ rather than copying the class Yes. That makes sense. > * This patch adds several new compile-time dependencies. It looks like these > are due to the use of Apache Axis? Please ensure that these are a minimal set Yes. We also need a few of these for runtime because of the remote procedure calls. > and that they're added as dependencies to the bundle manifest (they should not > appear in the .classpath file) > ** org.apache.axis_1.4.0.v201005080400 > ** org.apache.commons.discovery_0.2.0.v201004190315 > ** javax.xml.rpc_1.1.0.v201005080400 > ** javax.xml.soap_1.2.0.v201005080501 > ** javax.wsdl_1.6.2.v201005080631.jar > ** org.apache.commons.logging_1.1.1.v201005080502 OK. > * With these additional dependencies, please move contribution to take the form > of a new bundle, named @org.eclipse.mylyn.wikitext.mediawiki.confluence.core@ Why mediawiki.confluence as mediawiki is not really a part of this? Maybe @org.eclipse.mylyn.wikitext.confluence.ant@? > * Please don't use the @src_ant@ source folder. Use the @src@ folder instead. > bug 302616 has details on why we're not doing this in WikiText. Ok, thanks. > > Overall the contribution looks great! Thanks, not much too it though. The Eclipse webtools did all the hard work :-) > > Note that this contribution relates to our "Project > Charter":http://wiki.eclipse.org/Mylyn/Charter since it involves integration > with a 3rd party commercial system. I've put this on the agenda for this week's > "meeting":http://wiki.eclipse.org/Mylyn/Meetings. Feel free to attend to be > part of the discussion. Ok, good. I won't be able to attend though — another day of double booked meetings. David some comments just from my experience working with Confluence. (In reply to comment #13) > Great patch, thanks! A few things that I would like to see addressed: > * All classes should be in the @org.eclipse.mylyn.internal.wikitext.confluence@ > package namespace -- I noticed org.apache and com.atlassian packages, those > should be renamed Axis2 generated classes will have the Axis namespace. Ideally, generated code should never be checked in, and should always be generated during the build. So the only thing in this case that may need to be checked in is the WSDL, but from an eclipse IP standpoint that may be a grey area for IP as it is created by confluence web services. On a side note, doesn't confluence have a lighter weight API? If not, the Axis stubs are fine... (In reply to comment #16) > On a side note, doesn't confluence have a lighter weight API? > > If not, the Axis stubs are fine... It has a REST API but it isn't complete yet. It doesn't provide enough functionality that is needed to correctly get not only the SPACE information, but the Attachements as well. In the implementation I did for ebayopensource.org, I originally went the REST API route to avoid the Axis stubs, but it just isn't where it needs to be yet. Thanks for the info Dave! (In reply to comment #16) > On a side note, doesn't confluence have a lighter weight API? I have the same experience as Dave. Started out with the REST API and did manage (trough some trickery) to get the attachments. But it was very slow. My tests showed on average 1m20s using the REST API and around 20s using the RPC API. And as Dave says the REST API is close to useless. I could look into generating the stubs though if there is an interest for that. Also I guess the WSDL file could be pulled from a remote site during building? (In reply to comment #19) > (In reply to comment #16) > > On a side note, doesn't confluence have a lighter weight API? > I have the same experience as Dave. Started out with the REST API and did > manage (trough some trickery) to get the attachments. But it was very slow. My > tests showed on average 1m20s using the REST API and around 20s using the RPC > API. And as Dave says the REST API is close to useless. > > I could look into generating the stubs though if there is an interest for that. > Also I guess the WSDL file could be pulled from a remote site during building? I could live without the stubs being generated as part of the build for now, just to get something through the IP process and ready for consumption. We can always work on improvements iteratively. (In reply to comment #14) > > * All classes should be in the > @org.eclipse.mylyn.internal.wikitext.confluence@ > > package namespace -- I noticed org.apache and com.atlassian packages, those > > should be renamed > Please note that the reason for this was that this code was automatically > generated from the Confluence WSDL using Axis. If a new version of the web > service description comes out the process should be repeated. I understand -- however package naming is important. It should be relatively trivial to use Eclipse's refactoring to rename the package should the classes need generating again. > > * With these additional dependencies, please move contribution to take the > form > > of a new bundle, named @org.eclipse.mylyn.wikitext.mediawiki.confluence.core@ > Why mediawiki.confluence as mediawiki is not really a part of this? Maybe > @org.eclipse.mylyn.wikitext.confluence.ant@? Oops, a copy/paste error. @org.eclipse.mylyn.wikitext.confluence.core@ is probably the way to go. I hesitate to put 'ant' into the name, since it could also be used for maven targets, or IDE integration for example. > > Note that this contribution relates to our "Project > > Charter":http://wiki.eclipse.org/Mylyn/Charter since it involves integration > > with a 3rd party commercial system. I've put this on the agenda for this > week's > > "meeting":http://wiki.eclipse.org/Mylyn/Meetings. Feel free to attend to be > > part of the discussion. > Ok, good. I won't be able to attend though — another day of double booked > meetings. that's a shame. I'll bring it up at the meeting anyways and let you know the outcome. (In reply to comment #15) > Axis2 generated classes will have the Axis namespace. Ideally, generated code > should never be checked in, and should always be generated during the build. I agree in principle however there are some benefits to checking in generated code. For example, the build process remains simple, and it's easy to see how the code changes if it's generated again. > I understand -- however package naming is important. It should be relatively > trivial to use Eclipse's refactoring to rename the package should the classes > need generating again. I would be careful to refactor the generated classes, you might end up breaking namespaces encoded in strings. You can provide a namespace mapping at generation time though to generate classes in desired packages. We used to do this for the JIRA connector: http://dev.eclipse.org/viewsvn/viewvc.cgi/org.eclipse.mylyn/zArchive/org.eclipse.mylyn.jira.core/?root=Tools_Project&sortdir=down (In reply to comment #22) > > I understand -- however package naming is important. It should be relatively > > trivial to use Eclipse's refactoring to rename the package should the classes > > need generating again. > > I would be careful to refactor the generated classes, you might end up breaking > namespaces encoded in strings. You can provide a namespace mapping at > generation time though to generate classes in desired packages. > > We used to do this for the JIRA connector: > http://dev.eclipse.org/viewsvn/viewvc.cgi/org.eclipse.mylyn/zArchive/org.eclipse.mylyn.jira.core/?root=Tools_Project&sortdir=down Agreed which is why I would recommend generating the code during build time and not checking in the changes. The only thing you would probably need to get approved then in the WSDL. (In reply to comment #23) > Agreed which is why I would recommend generating the code during build time and > not checking in the changes. The only thing you would probably need to get > approved then in the WSDL. Ok. I think that's a good idea. Will get started on that then. (In reply to comment #21) > > > * With these additional dependencies, please move contribution to take the > > form > > > of a new bundle, named > @org.eclipse.mylyn.wikitext.mediawiki.confluence.core@ > > Why mediawiki.confluence as mediawiki is not really a part of this? Maybe > > @org.eclipse.mylyn.wikitext.confluence.ant@? > > Oops, a copy/paste error. @org.eclipse.mylyn.wikitext.confluence.core@ is > probably the way to go. I hesitate to put 'ant' into the name, since it could > also be used for maven targets, or IDE integration for example. > I don't quite understand what you mean David. The contribution is already in a bundle named @org.eclipse.mylyn.wikitext.confluence.core@ albeit in a source folder named src_ant to separate it from the "proper" source. Created attachment 188151 [details]
Ant for generating Eclipse Help from a Confluence wiki (with improvements)
* Moved PageMapping to org.eclipse.mylyn.wikitext.core.
* Moved ANT code to "src" from "src_ant". confluence-ant.jar will no
longer be built. Users of the ANT task should reference the plug-in
JAR instead.
* Dependencies are in bundle manifest instead of in .classpath.
* The Confluence web service description (WSDL) is now downloaded from
the web when required and the source code for this service is
automatically generated at build time.
* Corrected namespace for generated code.
Contribution has not been moved to a new bundle as requested. Awaiting more input.
Thank you for this contribution Torkild. I am struggling to figure out what the best approach is here, as integrations with commercial systems are outside of the project charter. They tend to require a whole different level of maintenance from the committers, and experience has shown that hosting open source reference implementations is the best way for us to focus on Mylyn's frameworks and core tools. Also notably, Atlassian requested that we move the JIRA integrations out of Eclipse. Your contribution looks very useful and has garnered some interest. What if you put the contribution in its own bundle (as discussed) and host it somewhere like EclipseLabs. Once that's done we could also consider listing it on Connector Discovery. We could also make it easier to find by referencing it under Extensions on the project wiki. Hi David. I've had a chat with the project manager and it looks like we're going to pull the plug on this one. The fork may continue to be available on GitHub though. We'll see what happens. Resolving this as WONTFIX. ANT tasks for generating XSL:FO and Eclipse help has been separated from Mylyn Docs proper and moved to a new GitHub project (https://github.com/turesheim/no.marintek.mylyn.wikitext). I'm currently working on cleaning up the code and making it work properly again. However there are a few Eclipse issues that should be resolved. These relate to rendering of markup in HTML or XSL:FO: Bugs #336592, #336683, #336905 and #336813. I have provided patches for all of these and I don't thing the implementation is problematic so it should be fairly straightforward to apply. Torkild, thanks for the patches on the bugs that you mentioned. I haven't had a chance to get to them but will take a look when I return from holidays, early August. Thanks David. Enjoy your vacation! Per conversation with David Green via twitter, I'm re-opening this particular bug. I have developed an ant task for mylyn-wikitext for confluence that does similar things as the MediaWiki task, so just re-openning to start the discussion again. Current code: https://github.com/kingargyle/turmeric-eclipse/tree/master/utilities/confluence-ant-task Dave, thanks for raising this to our attention again. Thinking over the project charter and support constraints again, I don't see that anything has changed since comment #27. That said, Mylyn WikiText is about providing framework and API for working with wiki markup -- perhaps a good way to move forward here would be to create an abstraction for remote sources of wiki content. As you point out, we have two prime examples of this, Confluence and MediaWiki. If there are enough commonalities we could make it easy to plug in new wikis by creating pluggable API in the form of an extension point (and make it resolve using Java services, such as we have for markup languages). Apropos... I've recently done some work on the MARINTEK implementation. EPUB support has been added along with some cleaning up. See https://github.com/turesheim/no.marintek.mylyn.wikitext With Atlassian removing the Confluence markup format in their 4.0 release I think this if far less interesting. Proposing to resolve as WONTFIX. http://blogs.atlassian.com/2011/11/why-we-removed-wiki-markup-editor-in-confluence-4/ resolving as WONTFIX per comment #36 |