Bugzilla – Full Text Bug Listing
| Summary: | CBI should be able to generate the API tools description | ||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Technology] CBI | Reporter: | Paul Webster <pwebster> | ||||||||||||||||||||||
| Component: | build help | Assignee: | Krzysztof Daniel <krzysztof.daniel> | ||||||||||||||||||||||
| Status: | VERIFIED FIXED | QA Contact: | |||||||||||||||||||||||
| Severity: | major | ||||||||||||||||||||||||
| Priority: | P3 | CC: | akurtakov, andrea.ross, curtis.windatt.public, daniel_megert, david_williams, d_a_carver, igor, IngedevTeam.fr, jan.sievers, john.arthorne, krzysztof.daniel, Lars.Vogel, manderse, Michael_Rennie, mistria, Silenio_Quarti, stryker, thanh.ha | ||||||||||||||||||||||
| Version: | 1.0 | Flags: | thanh.ha:
review+
|
||||||||||||||||||||||
| Target Milestone: | --- | ||||||||||||||||||||||||
| Hardware: | All | ||||||||||||||||||||||||
| OS: | All | ||||||||||||||||||||||||
| Whiteboard: | |||||||||||||||||||||||||
| Bug Depends on: | 395281, 402095 | ||||||||||||||||||||||||
| Bug Blocks: | 372792 | ||||||||||||||||||||||||
| Attachments: |
|
||||||||||||||||||||||||
|
Description
Paul Webster
*** Bug 267673 has been marked as a duplicate of this bug. *** I had a quick look at exiting org.eclipse.pde.api.tools and from what I can tell it requires running eclipse workspace instance. This makes it pretty much impossible to embed in Tycho build. It should be relatively straightforward to implement .api_description file generation using ASM, but this runs the risk of Tycho and PDE disagree about content of the file in some cases. Another approach would be to refactor existing pde.api.tools to eliminate dependency on running eclipse workspace, but this almost certainly means heavy refactoring of jdt.core and pde.api.tools bundle, something only the Platform Team has the expertise (and commit priveleges) to do. Yet another approach would be to figure out how to start workspace as part of the build, but this goes against some fundamental design decisions we made it Tycho, will never work well and, generally, something I do not believe is a viable long term solution. Curtis, when API tools runs during a headless build, does it fake having a workspace? PW The API Tools ant tasks were designed to be run headless without a running Eclipse instance. http://wiki.eclipse.org/PDE/API_Tools/Tasks (In reply to comment #4) > The API Tools ant tasks were designed to be run headless without a running > Eclipse instance. > > http://wiki.eclipse.org/PDE/API_Tools/Tasks That page does not explain how to run the tasks really standalone, from command line ant. Can you point me at any documentation that explains minimal list of jars needed to generate .api_description file? (In reply to comment #5) > That page does not explain how to run the tasks really standalone, from > command line ant. Can you point me at any documentation that explains > minimal list of jars needed to generate .api_description file? Right... the tasks were designed to run as part of a headless (PDE) build, so no OSGi runtime but all of the Eclipse bundles are available on the classpath. The tasks use code from OSGi, PDE, JDT, Runtime etc. Unfortunately, there isn't a compact list of the required jars. This is the dependency list of the api tools core bundle. The tasks won't use every dependency in the list, but it gives a rough idea. org.eclipse.osgi;bundle-version="[3.4.0,4.0.0)", org.eclipse.core.runtime;bundle-version="[3.4.0,4.0.0)", org.eclipse.pde.core;bundle-version="[3.3.102,4.0.0)", org.eclipse.jdt.core;bundle-version="[3.8.0,4.0.0)", org.eclipse.jdt.launching;bundle-version="[3.6.100,4.0.0)", org.eclipse.core.resources;bundle-version="[3.4.0,4.0.0)", org.eclipse.text;bundle-version="[3.4.0,4.0.0)", org.eclipse.core.filebuffers;bundle-version="[3.4.0,4.0.0)", org.eclipse.ant.core;bundle-version="[3.1.300,4.0.0)", org.objectweb.asm;bundle-version="[3.2.0,4.0.0)", org.eclipse.team.core;bundle-version="[3.4.0,4.0.0)", org.eclipse.core.filesystem;bundle-version="[1.2.0,2.0.0)", org.eclipse.equinox.frameworkadmin;bundle-version="[2.0.0,3.0.0)", org.eclipse.core.variables;bundle-version="[3.2.0,4.0.0)" I was under impression that PDE/Build actually creates and starts headless workspace. I also noticed that pde apitools bundles were using jdt ICompilationUnit among many other JDT core classes, for example, and from my experience that requires running workspace instance. I can be wrong about this, of course, but I will need exact list of dependencies and specific instructions how to run pde apitools ant tasks outside of PDE/Build with plain ant if you want me to try to embed existing pde apitools in Tycho. (In reply to comment #7) > I was under impression that PDE/Build actually creates and starts headless > workspace. I also noticed that pde apitools bundles were using jdt > ICompilationUnit among many other JDT core classes, for example, and from my > experience that requires running workspace instance. I can be wrong about > this, of course, but I will need exact list of dependencies and specific > instructions how to run pde apitools ant tasks outside of PDE/Build with > plain ant if you want me to try to embed existing pde apitools in Tycho. Mike pointed me to the scripts folder in api tools. The api-tasks.xml is runnable from the command line and supports running apifile_generation. As for dependencies to put on the ant classpath, I don't have a list. Typically we would point at an Eclipse installation plug-ins folder. I am guessing the build scripts would point at whatever install PDE Build was running on. Is there a folder of jars that Tycho could point to? No, Tycho does not have "a folder of jars", at least nothing that includes enough jars to run pde/build. (In reply to comment #8) > Mike pointed me to the scripts folder in api tools. The api-tasks.xml is > runnable from the command line and supports running apifile_generation. As > for dependencies to put on the ant classpath, I don't have a list. > Typically we would point at an Eclipse installation plug-ins folder. I am > guessing the build scripts would point at whatever install PDE Build was > running on. Is there a folder of jars that Tycho could point to? Curtis, can you try assembling the minimal # of jars to run generate the .api_description file without running an eclipse instance? We need the list of jars that plain ant needs in order to generate the description. PW (In reply to comment #10) > (In reply to comment #8) > > Mike pointed me to the scripts folder in api tools. The api-tasks.xml is > > runnable from the command line and supports running apifile_generation. As > > for dependencies to put on the ant classpath, I don't have a list. > > Typically we would point at an Eclipse installation plug-ins folder. I am > > guessing the build scripts would point at whatever install PDE Build was > > running on. Is there a folder of jars that Tycho could point to? > > Curtis, can you try assembling the minimal # of jars to run generate the > .api_description file without running an eclipse instance? > As I mentioned to Curtis, you do not need a running version of Eclipse if you use the api-tasks.xml buildfile, all you have to do is edit the 'eclipse.install.dir' property to point to the folder that contains plugins. For example when we use the task we point that property to the /plugins folder of an extracted/built Eclipse and the build file takes care of the rest. > We need the list of jars that plain ant needs in order to generate the > description. If you really want to be specific, the following is an Ant fileset that will allow the task to run: <fileset dir="${eclipse.install.dir}"> <include name="org.eclipse.ant.core_*.jar"/> <include name="org.eclipse.core.contenttype_*.jar"/> <include name="org.eclipse.core.filebuffers_*.jar"/> <include name="org.eclipse.core.jobs_*.jar"/> <include name="org.eclipse.core.resources_*.jar"/> <include name="org.eclipse.core.runtime_*.jar"/> <include name="org.eclipse.core.variables_*.jar"/> <include name="org.eclipse.equinox.common_*.jar"/> <include name="org.eclipse.equinox.frameworkadmin_*.jar"/> <include name="org.eclipse.equinox.preferences_*.jar"/> <include name="org.eclipse.jdt.core_*.jar"/> <include name="org.eclipse.jdt.launching_*.jar"/> <include name="org.eclipse.pde.api.tools_*.jar"/> <include name="org.eclipse.osgi_*.jar"/> <include name="org.eclipse.pde.core_*.jar"/> <include name="org.eclipse.text_*.jar"/> <include name="org.objectweb.asm_*.jar"/> </fileset> (In reply to comment #11) > If you really want to be specific, the following is an Ant fileset that will > allow the task to run: forgot to mention that you will also need the 'apitooling-ant.jar' file from the pde.api.tools bundle on the Ant classpath, which the api-tasks.xml buildfile auto-extracts for you. (In reply to comment #12) > (In reply to comment #11) > > > If you really want to be specific, the following is an Ant fileset that will > > allow the task to run: > > forgot to mention that you will also need the 'apitooling-ant.jar' file from > the pde.api.tools bundle on the Ant classpath, which the api-tasks.xml > buildfile auto-extracts for you. Did some further tweaking and you can remove jdt.launching, pde.core, core variables, equinox.frameworkadmin and ant.core from the fileset - those deps were because of bundle deps on api.tools, but in reality the task will run without them. Created attachment 221047 [details]
api-tools.properties tweaked to only load the API description task
Created attachment 221048 [details]
Running the task outside of eclipse
The minimal set of jars needed to run the apifile_generation ant task is:
apitooling-ant.jar, which comes from inside org.eclipse.pde.api.tools
Also:
<include name="org.eclipse.equinox.common_*.jar"/>
<include name="org.eclipse.equinox.preferences_*.jar"/>
<include name="org.eclipse.osgi_*.jar"/>
<include name="org.eclipse.core.jobs_*.jar"/>
<include name="org.eclipse.core.resources_*.jar"/>
<include name="org.eclipse.core.contenttype_*.jar"/>
<include name="org.eclipse.jdt.core_*.jar"/>
<include name="org.eclipse.text_*.jar"/>
<include name="org.eclipse.core.runtime_*.jar"/>
<include name="org.eclipse.pde.api.tools_*.jar"/>
<include name="org.objectweb.asm_*.jar"/>
I ran this inside my o.e.ui.workbench project and it generated me a .api_description file
PW
(In reply to comment #14) > Created attachment 221047 [details] > api-tools.properties tweaked to only load the API description task The name of the ant task is apigeneration, if you use that properties file it will fail from the commandline. I fixed this in: http://git.eclipse.org/c/pde/eclipse.pde.ui.git/commit/?id=b50ab0ce5f729e41545b734b4ce4703ad08823de The api tools *can* run on bundles that lack the .api_description file, in general. The current installations and current apitooling-ant.jar cannot. But that restriction seems more of a fault of the ant task and not the underlying apitools features.
For example, if in APIToolsAnalysisTask, you comment out the following:
if (!Util.isApiToolsComponent(apiComponent)) {
allNonApiBundles.add(name);
continue;
}
You can get reports on bundles that do not have the .api_description file. Again, this would require modifications to the ant tasks.
The ant tasks also suffer from some other issues, most of which could be fixed, but not without a severe cleanup of the code. One thing I came across is that if you provide an includes list, listing bundles you'd like analysis to be run on, the ant task actually runs an analysis on ALL bundles, but only generates the reports in the end for whichever ones were in your list.
This results in a very long (12+ minutes) process for what could be done in about 30 seconds if you only want to analyze one bundle.
Again, much of these issues could be fixed, but there'd need to be a strong cleanup of the code.
(In reply to comment #17) > One thing I came across > is that if you provide an includes list, listing bundles you'd like analysis > to be run on, the ant task actually runs an analysis on ALL bundles, but > only generates the reports in the end for whichever ones were in your list. > We already have an open issue for that: bug 327036 If you have found any other issues (suggestions) with the API tools tasks, please open bugs for them. Ideally, that would be a Maven plugin with 2 goals: * a "generate-api-description" that would happen during "compile" phase, and would create the .api_description by reading sources, and would put it into target/classes folder so it gets part of the output jar * a "report" goal which would take as input a baseline (format to be specified: a folder, a list of artifacts, a list of IUs, a target definition...?) and would generate API reports comparing the current Maven project and the baseline. Rob (Stryker) has extracted part of PDE Ant tasks so we could successfully get some reports from Tycho using the antRunner application. But this is still Ant-based stuff. In term of implementation, it looks like we would need the Maven plugin to start an Eclipse instance containing the PDE API tools jars. That's more or less what Tycho does with an embedded target-platform if I understand correctly. @Tycho folks: Does it seem possible to make a Maven plugin depend on some PDE jars and start an Eclipse instance made of these jars? The requirement of having the .api_description file is actually enforced only inside the ant tasks. If the lines demanding / checking for that was removed from the ant task only, apitools would actually still be able to still run api reports against it. I've written some custom ant tasks that depend on pde which demonstrate this behavior and are able to generate reports without a problem, without the need for inserting a .api_description file. Of course if the jars don't have a .api_description file, odds are any filters and customizations will not be included and you'll only be able to run against some global set of settings on a per-plugin level. (In reply to comment #19) > Ideally, that would be a Maven plugin with 2 goals: this came up on yesterday's CBI conf call. while it's possible to develop a maven plugin for this, a simpler approach that may not require tool development would be to use the tycho-eclipserun-plugin [1] to execute the existing ant tasks via the eclipse antrunner application during the tycho build. I think that Krzysztof was interested to try this. > @Tycho folks: Does it seem possible to make a Maven plugin depend on some > PDE jars and start an Eclipse instance made of these jars? yes probably possible but may not be worth the effort just to get something up and running now. [1] http://wiki.eclipse.org/Tycho/Additional_Tools The first problem that I have found is that the maven antrunner requires configuration in each of the bundles that will require api. This is a maintenance hell. I'll try to implement the api generation once in the eclipse-parent/pom.xml in a profile and make it activated by a marker file in the plugin that wants api generated. (In reply to comment #22) > The first problem that I have found is that the maven antrunner requires > configuration in each of the bundles that will require api. This is a > maintenance hell. Wouldn't we be running tycho-eclipserun-plugin to run org.eclipse.ant.core.antRunner instead of plain maven antrunner, to easily pick up the PDE ant tasks? > > I'll try to implement the api generation once in the eclipse-parent/pom.xml > in a profile and make it activated by a marker file in the plugin that wants > api generated. Bundles that need a .api_description file have entries in their .project file, like:<nature>org.eclipse.pde.api.tools.apiAnalysisNature</nature> PW (In reply to comment #23) > Wouldn't we be running tycho-eclipserun-plugin to run > org.eclipse.ant.core.antRunner instead of plain maven antrunner, to easily > pick up the PDE ant tasks? That's right. > > > > I'll try to implement the api generation once in the eclipse-parent/pom.xml > > in a profile and make it activated by a marker file in the plugin that wants > > api generated. > > Bundles that need a .api_description file have entries in their .project > file, like:<nature>org.eclipse.pde.api.tools.apiAnalysisNature</nature> The only problem with this approach is that maven can't read the .project. So, in the best scenario I'd need to write an ant script that processes .project to decide whether or not to run the api generation. This would add an additional Eclipse launch for every bundle (kind of expensive). > PW $ find . -name .project -exec grep -H "org.eclipse.pde.api.tools.apiAnalysisNature" {} \; | wc -l
280
$ find . -name .project | wc -l
804
(In reply to comment #24) > The only problem with this approach is that maven can't read the .project. > So, in the best scenario I'd need to write an ant script that processes > .project to decide whether or not to run the api generation. This would add > an additional Eclipse launch for every bundle (kind of expensive). kinda hack, but groovy maven plugin may be used for determining whether the project has an API nature [1] however in order for this to be useful, eclipserun-mojo would need a boolean mojo parameter "skip" based on a maven project property which could then be set accordingly in the inline groovy script. Otherwise the antrunner process would always be forked even if there is nothing to do. Let me know if you want the skip parameter in eclipserun-mojo, this would be no big deal. ... OTOH considering the hoops we are jumping through here it may be better to just write a custom mojo for this. I guess you could do this in CBI similar to maven-cbi-plugin and if it's general-purpose, we can put it into tycho-extras. [1] <plugin> <groupId>org.codehaus.gmaven</groupId> <artifactId>gmaven-plugin</artifactId> <version>1.4</version> <executions> <execution> <phase>generate-resources</phase> <goals> <goal>execute</goal> </goals> <configuration> <source><![CDATA[ def dotProject = new File(project.basedir,'.project') boolean hasApiNature = false if (dotProject.file) { def xml = new XmlSlurper().parse(dotProject) def apiNature = xml.natures.children().find( {it.text() == 'org.eclipse.pde.api.tools.apiAnalysisNature'}) hasApiNature = apiNature?.name() == 'nature' } println '[DEBUG] hasAPINature: ' + hasApiNature // TODO set eclipserun skip project property here instead if no API nature found project.properties.setProperty('hasAPINature', String.valueOf(hasApiNature)) ]]></source> </configuration> </execution> </executions> </plugin> (In reply to comment #26) > > kinda hack, but groovy maven plugin may be used for determining whether the > project has an API nature [1] > however in order for this to be useful, eclipserun-mojo would need a boolean > mojo parameter "skip" based on a maven project property > which could then be set accordingly in the inline groovy script. Otherwise > the antrunner process would always be forked even if there is nothing to do. > Let me know if you want the skip parameter in eclipserun-mojo, this would be > no big deal. > General notice: Eclipse build system is way complicated and getting groovy into the game would not help anyone but will make the life of people that build from source way more harder. (In reply to comment #27) > (In reply to comment #26) > > > > kinda hack, but groovy maven plugin may be used for determining whether the > > project has an API nature [1] > > however in order for this to be useful, eclipserun-mojo would need a boolean > > mojo parameter "skip" based on a maven project property > > which could then be set accordingly in the inline groovy script. Otherwise > > the antrunner process would always be forked even if there is nothing to do. > > Let me know if you want the skip parameter in eclipserun-mojo, this would be > > no big deal. > > > > General notice: Eclipse build system is way complicated and getting groovy > into the game would not help anyone but will make the life of people that > build from source way more harder. The same thing (reading .project and setting a "hasAPITooling" property the maven project) could be done in a Mojo, the same way the Jacoco-Maven plugin has a "prepare-agent" goal to pre-set surefire configuration properties. Having a maven plugin rather than an Ant script would make it easier to share with all the other projects interested (at Eclipse.org and out of Eclipse.org). (In reply to comment #27) > General notice: Eclipse build system is way complicated and getting groovy > into the game would not help anyone but will make the life of people that > build from source way more harder. agreed. I was overstretching it trying to find ways to spare the CBI team having to develop their own maven plugins. let's go for a solution based on custom maven plugin(s) then. Whatever it's going to be, let me know if you need changes in tycho to make this work. Actually I'm writing a simple mojo to generate xml file that could be consumed by api tools ant runner. Currently it looks like this: http://fpaste.org/15fr/ Created attachment 225435 [details]
A patch for maven-cbi-plugin
So, as I have written before:
the process of api generation will have two phases:
* phase I: - .apigen.xml generation
A custom mojo added to the maven-cbi-plugin checks for the existence of api nature in the .project, and if it finds it, it writes an ant build file which will be used later to build api.
* phase II: - api generation
A new profile is added to the eclipse-parent which is activated by the presence of .apigen.xml file. eclipse-runner is used to invoke Eclipse ant runner and invoke api generation defined in .apigen.xml.
Created attachment 225436 [details]
Patch for eclipse-parent
Patch for Eclipse parent that actually makes the use of the previously generated xml
Both patches are work in progress.
Ok, I'm seeing: [ERROR] Failed to execute goal org.eclipse.tycho.extras:tycho-eclipserun-plugin:0.17.0-SNAPSHOT:eclipse-run (default) on project org.eclipse.osgi: Eclipse runtime includes artifact from the reactor which has not yet been built: eclipse-plugin:org.eclipse.ant.core:3.2.401.qualifier: MavenProject: org.eclipse.platform:org.eclipse.ant.core:3.2.401-SNAPSHOT Jan, Tobias, how should I deal with it? (In reply to comment #33) > how should I deal with it? we probably need to fix bug 395281 to make this work. bundles needed by the api tools need to available right from the start of the build so using what has just been built is no longer an option here will try to find some time for investigating this tomorrow BTW maven profile activation based on files generated later during build won't work AFAIK because profiles are evaluated before the actual build starts. (In reply to comment #34) > BTW maven profile activation based on files generated later during build > won't work AFAIK because profiles are evaluated before the actual build > starts. It looks like this approach works (I'd not get the error message otherwise). Created attachment 225884 [details]
Part I.
Created attachment 225885 [details]
Part II
Andrew, Thanh, please verify (both patches are needed). It is also necessary to build maven-cbi-eclipse before eclipse-parent. (In reply to comment #38) > Andrew, Thanh, > > please verify (both patches are needed). It is also necessary to build > maven-cbi-eclipse before eclipse-parent. Is this a new build step? or do you mean the existing maven-cbi-plugin needs to be build first? The existing maven-cbi-plugin needs to be built first. Based on Rob's tweaks of current API Tools, I provided some Mojos to run API description generation and API reports generation: https://github.com/mickaelistria/apitooling . Most of what you'd be interested to know can be found in the README. Any feedback is welcome. Rob and I would be very happy to see this contributed into CBI or anywhere at Eclipse.org where we could consume it also for JBoss Tools. (In reply to comment #41) > Based on Rob's tweaks of current API Tools, I provided some Mojos to run API > description generation and API reports generation: > https://github.com/mickaelistria/apitooling . Most of what you'd be > interested to know can be found in the README. > Any feedback is welcome. Rob and I would be very happy to see this > contributed into CBI or anywhere at Eclipse.org where we could consume it > also for JBoss Tools. To me this definetely belongs to tycho-extras. Created attachment 225994 [details]
Part III
Attached patch Part III bumps the maven-cbi-plugin version in eclipse-junit-tests bundle.
(In reply to comment #40) > The existing maven-cbi-plugin needs to be built first. Thanks Krzysztof, I was running into a build issue yesterday. Which is fixed by the Part III patch I attached. I'll see if I can successfully run a full build now. With the 3 patches I was able to successfully complete a build. I can see in the source bundles there now exists a api_description.xsd as well. inflating: xml/api_description.xsd inflating: xml/api_filters.xsd inflating: xml/api_profile.xsd Krzysztof, is there anything else I should check for to confirm the patch is working? If not I can push these patches soon. I believe this is what was expected. (In reply to comment #46) > I believe this is what was expected. Pushed to master, R4_2_maintenance, and R3_8_maintenance. master: http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/commit/?id=e20c0830d1da052b286fe2206ee818c6073dfae1 R4_2_maintenance: http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/commit/?h=R4_2_maintenance&id=1c02a4a2d06ee410d1e48bd1bcbc3cf49d375a26 R3_8_maintenance: http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/commit/?h=R3_8_maintenance&id=967f2e94b0fa0cdd9453f55f29468a974f7e9426 When is this supposed to be run? I'm in the process of a build in master, and while it generated the org.eclipse.osgi/.apibuild.xml file, it doesn't appear to have done anything to the jars: bash-4.1$ jar tf org.eclipse.osgi-3.9.0-SNAPSHOT.jar | grep -i api bash-4.1$ jar tf org.eclipse.osgi-3.9.0-SNAPSHOT-sources.jar | grep -i api at this point, it says it's already installed my jars in my local maven repo. PW Is the problem that by the time it's generated the .apibuild.xml maven has already processed the profile activation? That would explain why it would work on a second run but not the first (but this is our main usecase) PW I guess it doesn't work because my concerns in comment 34 (In reply to comment #50) alternatively, you could use the skip parameter I introduced [1] MavenProject.getProperties().setProperty("eclipserun.skip", "true") in attachment Part I to skip api description generation if the project does not have the API nature. [1] http://git.eclipse.org/c/tycho/org.eclipse.tycho.extras.git/tree/tycho-eclipserun-plugin/src/main/java/org/eclipse/tycho/extras/eclipserun/EclipseRunMojo.java#n137 (In reply to comment #51) > (In reply to comment #50) > alternatively, you could use the skip parameter I introduced [1] > > MavenProject.getProperties().setProperty("eclipserun.skip", "true") > But... will it disable all eclipseruns, won't it? Created attachment 227124 [details]
Fix incorporating Jan's suggestions
the profile is enabled always, but the eclipse runner is suppressed when there is no api nature.
So, I have prepared a patch, but I'm really afraid that no eclipse will be run as a part of the build process for bundles that don't have api nature.
A theoretical solution (of a really poor performance) would be to generate apibuild.xml always, but create meaningful content only if the api should be generated...
(In reply to comment #52) > But... will it disable all eclipseruns, won't it? it will disable all eclipserun executions for the module for which the property eclipserun.skip=true was set. If you happen to use several eclipserun invocations (for different things) within the same module I think there is still a way to control which one is skipped: apart from using a property, you can also configure mojo parameters _per execution_ [1] and this has precedence over configuration via property. Something like: <build> <plugins> <plugin> <groupId>org.eclipse.tycho.extras</groupId> <artifactId>tycho-eclipserun-plugin</artifactId> <version>${tycho-version}</version> <executions> <execution> <id>dont_skip_me</id> <phase>compile</phase> <configuration> <skip>false</skip> </configuration> <goals> <goal>eclipse-run</goal> </goals> </execution> </executions> </plugin> </plugins> </build> Not elegant but works. [1] http://maven.apache.org/guides/mini/guide-configuring-plugins.html#Using_the_executions_Tag If we want to go forward with that, then it looks like we'll have to include the skip false in the doc projects: eclipse.platform.common/bundles/org.eclipse.platform.doc.isv/pom.xml eclipse.platform.common/bundles/org.eclipse.jdt.doc.user/pom.xml eclipse.platform.common/bundles/org.eclipse.pde.doc.user/pom.xml eclipse.platform.common/bundles/org.eclipse.jdt.doc.isv/pom.xml eclipse.platform.common/bundles/org.eclipse.platform.doc.user/pom.xml Those were the bundles that I could find that included tycho-eclipserun-plugin that weren't actually the eclipse-parent/pom.xml PW Released fix as http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/commit/?id=18467502f7c7b200ee11b5eff7ad1c014061507c and http://git.eclipse.org/c/platform/eclipse.platform.common.git/commit/?id=f8a2a7df27033d6809624ec7226e796b3f4d242d I see the doc being run and I see .api_description files in some bundles. They now need to be verified. PW (In reply to comment #56) > Released fix as > http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/ > commit/?id=18467502f7c7b200ee11b5eff7ad1c014061507c and > http://git.eclipse.org/c/platform/eclipse.platform.common.git/commit/ > ?id=f8a2a7df27033d6809624ec7226e796b3f4d242d > > I see the doc being run and I see .api_description files in some bundles. > > They now need to be verified. > > PW In the build I tested yesterday, there were many .api_description files missing. (In reply to comment #57) > In the build I tested yesterday, there were many .api_description files > missing. OK, got the pattern: they are present in the plug-ins that are packaged as JARs, but missing in those packaged/exploded in a directory. The API description files are not correct: 1) types are not fully qualified 2) only the first entry of a type is listed, all others are lost 3) the bundle qualifier is not replaced with the correct/actual value As a test case, compare the API descriptions files for jdt.ui. In the directory plugins, org.eclipse.pde.build_3.8.100.v20130225-1821 doesn't have an .api_description at all. org.eclipse.ui.intro.universal_3.2.600.v20130221-0120 doesn't have a .api_description at its root, but has one in the inner universal.jar PW sigh. fun with nested jars, episode 27. opened tycho bug 402095 (In reply to comment #59) > The API description files are not correct: > > 1) types are not fully qualified > 2) only the first entry of a type is listed, all others are lost > 3) the bundle qualifier is not replaced with the correct/actual value > > As a test case, compare the API descriptions files for jdt.ui. From a first glance comparing attachment Part I and api ant task docs [1], I assume the following bugs: - binary attribute of the ant task should be a _path_ containing all classes output dirs. Instead it's hardcoded to target/classes - it's not clear from the ant task docs how the source locations are determined from the project ant task attribute. Does it scan build.properties for source.* entries? If not, we probably also have to set extraSourceLocations attribute Implementation hint: you can use getEclipsePluginProject().getOutputJars() [2] to get source dirs and corresponding ouptput folders. [1] http://help.eclipse.org/juno/index.jsp?topic=/org.eclipse.pde.doc.user/reference/api-tooling/ant-tasks/filegeneration-ant-task.htm [2] http://git.eclipse.org/c/tycho/org.eclipse.tycho.git/tree/tycho-compiler-plugin/src/main/java/org/eclipse/tycho/compiler/AbstractOsgiCompilerMojo.java#n236 (In reply to comment #62) > - binary attribute of the ant task should be a _path_ containing all classes > output dirs. Instead it's hardcoded to target/classes "binary=\"" + project.getBuild().getDirectory() + "\"\n"); "target=\"" + project.getBuild().getDirectory() + "/classes\"\n"); Looking at the commit the project.getBuild() directory is supplied, not sure if this would be accurate or not. The attribute takes a list of folders or archives. I'm looking more into what this attribute was set to during the PDE Build. The target attribute looks odd though. Not sure why the api_description file would be output to /classes? > - it's not clear from the ant task docs how the source locations are > determined from the project ant task attribute. Does it scan > build.properties for source.* entries? If not, we probably also have to set > extraSourceLocations attribute Source locations are not collected. The project root is searched for .java files which are later scanned. The extrasourcelocations attribute is only needed for SWT which needed to supply additional java files outside of the project. (In reply to comment #63) > Looking at the commit the project.getBuild() directory is supplied, not sure > if this would be accurate or not. The attribute takes a list of folders or > archives. I'm looking more into what this attribute was set to during the > PDE Build. If PDE Build had a set of workspace locations, the workspace output directories were passed in. Otherwise the compiled jars were added. This was only possible because the task was being run after all the bundles were built. It allows the javadoc tag scanner to resolve method signatures. (In reply to comment #64) > If PDE Build had a set of workspace locations, the workspace output > directories were passed in. Otherwise the compiled jars were added. This > was only possible because the task was being run after all the bundles were > built. It allows the javadoc tag scanner to resolve method signatures. apidescription generator is run before jar packaging, so I assume passing in all classes ouput dirs of the bundle being built (including those for nested jars if any) will be fine? (In reply to comment #63) > The target attribute looks odd though. Not sure why the api_description > file would be output to /classes? that's because api_description should end up in the (outer) bundle jar being packaged afterwards. This kind of works, but not in case of nested jars (hence tycho bug 402095). I intend to fix bug 402095 soon and will then provide instructions how to make sure api_description always ends up in the outer bundle jar. (In reply to comment #65) > apidescription generator is run before jar packaging, so I assume passing in > all classes ouput dirs of the bundle being built (including those for nested > jars if any) will be fine? While not the perfect setup, this will be fine. Not having the binary output of all the jars does affect the description files (types in method restrictions would not be fully qualified). However, the cases where not having fully qualified api descriptions would affect api analysis are exceedingly rare. (In reply to comment #67) > (In reply to comment #65) > > apidescription generator is run before jar packaging, so I assume passing in > > all classes ouput dirs of the bundle being built (including those for nested > > jars if any) will be fine? > > While not the perfect setup, this will be fine. > > Not having the binary output of all the jars does affect the description > files (types in method restrictions would not be fully qualified). just to make sure I get this right: is this similar to a compilation classpath, i.e. would it help to put all jars/classes dirs on compilation classpath to the "binary" task attribute? (In reply to comment #68) > (In reply to comment #67) > > While not the perfect setup, this will be fine. > > > > Not having the binary output of all the jars does affect the description > > files (types in method restrictions would not be fully qualified). > > just to make sure I get this right: is this similar to a compilation > classpath, i.e. would it help to put all jars/classes dirs on compilation > classpath to the "binary" task attribute? After doing some testing I retract my previous statement (comment #67). The PDE Build releng scripts did include all of the output folders on the binary path. However, doing so is not necessary. Just the binary output from the project will be enough to resolve the method signatures. No binary: <type name="Activator" restrictions="0"> <field name="context" restrictions="8"/> <method name="getContext" restrictions="8" signature="()QBundleContext;"/></type> Project Binary: <type name="Activator" restrictions="0"> <field name="context" restrictions="8"/> <method name="getContext" restrictions="8" signature="()Lorg/osgi/framework/BundleContext;"/> <method name="start" restrictions="8" signature="(Lorg/osgi/framework/BundleContext;)V"/> <method name="stop" restrictions="8" signature="(Lorg/osgi/framework/BundleContext;)V"/> </type> You can also see another bug, which is that if an unresolved method is encountered the task continues but all additional methods in that compilation unit are skipped. I will open a new bug for this. Created attachment 227935 [details]
git patch for GenerateAPIBuildXMLMojo
see commit message for details
Created attachment 227936 [details]
corresponding patch for eclipse-platform-parent
see commit message for details. can only be applied after previous patch 227935 is applied and new version 1.0.1-SNAPSHOT of GenerateAPIBuildXMLMojo is available
last two patches should fix the bugs mentioned in comment 59 I did some local testing with a small sample project that has nested jars and multiple output folders. (In reply to comment #70) > Created attachment 227935 [details] > git patch for GenerateAPIBuildXMLMojo > > see commit message for details I pushed this into org.eclipse.cbi.maven.plugins. http://git.eclipse.org/c/cbi/org.eclipse.cbi.maven.plugins.git/commit/?id=734d4039a7fb0c814a6ef189badcf02fbe56b397 I also deployed it into repo.eclipse.org. (In reply to comment #71) > Created attachment 227936 [details] > corresponding patch for eclipse-platform-parent > > see commit message for details. can only be applied after previous patch > 227935 is applied and new version 1.0.1-SNAPSHOT of GenerateAPIBuildXMLMojo > is available For this patch we'll also need to include the cbi-snapshots repository for it to work since at the moment we only have the global releases repo configured. (In reply to comment #74) > (In reply to comment #71) > > Created attachment 227936 [details] > > corresponding patch for eclipse-platform-parent > > > > see commit message for details. can only be applied after previous patch > > 227935 is applied and new version 1.0.1-SNAPSHOT of GenerateAPIBuildXMLMojo > > is available > > For this patch we'll also need to include the cbi-snapshots repository for > it to work since at the moment we only have the global releases repo > configured. I pushed this patch into master: http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/commit/?id=5d724baf041f535d61d4ac241ac2663cca840eaa The next Platform build should now include the changes by Jan. I can see the issues from comment 59 fixed in the latest I-build from http://download.eclipse.org/eclipse/downloads/drops4/I20130311-0800/ @Thanh, Paul can you confirm? Out of curiosity, why does PDE need api_description file? The file itself and classes it is generated from are included in the same bundle, so the information in the file can be derived from the classes, no? I am not suggesting CBI build should not produce the file, but trying to understand the reason the file was introduced originally. (In reply to comment #78) > Out of curiosity, why does PDE need api_description file? The file itself > and classes it is generated from are included in the same bundle, so the > information in the file can be derived from the classes, no? I am not > suggesting CBI build should not produce the file, but trying to understand > the reason the file was introduced originally. Parts of the API description are contained in the javadoc, and so not in the binary file by default. The .api_description file contains those processed avadoc tags. http://wiki.eclipse.org/PDE/API_Tools/Use_Cases#Javadoc_Tags.2C_API_Visibilities_.26_Restrictions PW works for me now Verified in I20130430-2000, that unpacked plug-ins have the API description, and sanity checked several API descriptions. |