Community
Participate
Working Groups
Please "Buckminsterize" the Mylyn build.
Here is a rough description of the current ant-based build: * Based on PDE build using the Eclipse basebuilder * It runs against different Eclipse versions: 3.5, 3.6, 3.7 (latest milestone) * Some plug-ins are branched so the map files may vary slightly based on the Eclipse version * It builds, runs tests, creates update site/repositories, signs, reports (junit etc.), publishes (copies files to download directories, updates composite sites) * It is generally run on build.eclipse.org but can by run on any machine from the org.eclipse.mylyn.releng plug-in without any additional setup * There are four different types of tests driven by different JUnit3 test suites: Plain Java Headless Tests, Eclipse Tests, SWTbot UI Tests, Performance Tests The targets that build and test generally do the following: * Download and extract a base target environment (Eclipse SDK) * Install a dependencies feature into target from a list of update sites specified through a properties file * Invoke the basebuilder to run PDE build or JUnit tests Let me know if you need any other input or more details.
Current build dependencies in addition to what is specified in map files: org.eclipse.swtbot.eclipse org.eclipse.cdt.platform Kenn, who would be the best person to assign this task to?
Created attachment 174453 [details] mylyn/context/zip
Created attachment 178652 [details] org.eclipse.mylyn.java.tests-build_properties.patch patch dropping a duplicate entry in the build.properties file of the org.eclipse.mylyn.java.tests project preventing buckminster from resolving the project
Created attachment 178655 [details] org.eclipse.mylyn.wikitext-ant_builders.patch patch modifying ant scripts for building mylyn contributed ant tasks and javadoc to be usable in a Buckminster driven build
I have created two new projects to drive the Buckminster build of Mylyn, they are available at: http://testsvn.cloudsmith.com/eclipse/tools/trunk/org.eclipse.mylyn However the build won't run without applying the attached patches: org.eclipse.mylyn.java.tests-build_properties.patch org.eclipse.mylyn.wikitext-ant_builders.patch The first one - org.eclipse.mylyn.java.tests-build_properties.patch - just drops a duplicate entry in the build.properties file of the org.eclipse.mylyn.java.tests project, so it should be safe to apply. The other one - org.eclipse.mylyn.wikitext-ant_builders.patch - changes the ant scripts used to build mylyn contributed ant tasks and javadoc. The intent was not to change the behavior of the scripts when invoked the original way, but to allow to use them in the Buckminster build when invoked with extra properties set. The use of the scripts in the original way has not been tested though, so please review carefully. My next step is to create a Hudson job running the build at our Hudson instance.
Thanks Michal! I have fixed the build.properties. David, can you review and apply the other patch? It affects the WikiText build.
Just for reference, it would be nice to have the following build jobs eventually: mylyn--heartbeat - run limited set of tests against Eclipse 3.6 on each commit mylyn--heartbeat--connectors - run limited set of connector tests against Eclipse 3.6 on each commit mylyn-3.4.x - build and test m_3_4_x stream against Eclipse 3.4, 3.5 and 3.6 mylyn-e3.5 - build and test against Eclipse 3.5 mylyn-e3.6 - build and test against Eclipse 3.6 mylyn-e3.7 - build and test against Eclipse 3.7 (latest milestone) mylyn-e4.0 - build and test against Eclipse 4.0 (latest milestone) mylyn-nightly - run mylyn-e3.5, mylyn-e3.6, mylyn-e3.7, mylyn-e4.0 nightly
(In reply to comment #7) > Thanks Michal! I have fixed the build.properties. > > David, can you review and apply the other patch? It affects the WikiText build. Michal, it looks like the build will change the WikiText bundle jars in that the Ant task classes are not in the bundle jar. The WikiText bundles must be usable as simple jar files as described in the WikiText User Guide under "Conversion using Ant build scripts":http://help.eclipse.org/helios/index.jsp?topic=/org.eclipse.mylyn.wikitext.help.ui/help/Mylyn%20WikiText%20User%20Guide.html -- that is, the jar files must be usable outside of an Eclipse runtime as additions to a java classpath.
Michal, does Buckminster support creating a Maven repository for the headless bundles in addition to a P2 repository?
(In reply to comment #10) > Michal, does Buckminster support creating a Maven repository for the headless > bundles in addition to a P2 repository? No Buckminster does not support that. But b3 aggregator does. The functionality is not well tested and maybe limited though. Adding Filip to the CC list as he's implemented the functionality.
(In reply to comment #9) > Michal, it looks like the build will change the WikiText bundle jars in that > the Ant task classes are not in the bundle jar. This was a misunderstanding on my part, I thought that the ant tasks sources were put on the bundle classpath just so that all the Eclipse java tools works with them. I'm looking into it and going to release a fixed patch shortly.
Created attachment 180943 [details] org.eclipse.mylyn.wikitext-ant_builders_v2.patch Update version of the patch which does not modify the .classpath and build.properties files..
(In reply to comment #12) I've updated the patch so that it does not modify .classpath and build properties files so the resulting bundles should be same as before. Note the the classpath setup of the bundles is not ideal, the classes implementing the ant tasks are available through two class loaders to ant which causes problems described in: Bug #273325 and Bug #302616. I spent time looking into the issue and realized that a solution would be to put the classes implementing the ant tasks into the root of the bundle file (unpacked) ant the rest of the bundle classes into an embedded jar. I.e. kind of reverse the situation with respect to the current state. The only problem is that Eclipse currently require that the contributed ant tasks be located in an embedded jar. It turns out that the change required to support an tasks in a directory structure (i.e. not in a jar file) is quite simple so I went ahead and provided a path for that in Bug #327769. You may want to set that bug as a dependency for the Bug #302616.
Thanks. David, could you take a look at the patch and apply it if the changes look good to you?
I'm sorry but I discovered a problem with my previous patch - Buckminster got confused by the fact that a single source folder (src_ant/) was supposed to be compiled into two different target folders (bin/, bin_ant/) and disregarded the former completely. I'm attaching another version of the patch which removes the definition of the task library jar from the build.properties. This should be ok as the library jar file is created by a support ant build anyway. In any case this combination produces the desired result when built with Buckminster (and I'm convinced that it should not change things for the original build either).
Created attachment 181092 [details] org.eclipse.mylyn.wikitext-ant_builders_v3.patch Updated version of the patch to finally get the Buckminster build to produce bundles with the desired content.
(In reply to comment #15) > Thanks. David, could you take a look at the patch and apply it if the changes > look good to you? The patch looks good. A couple of questions: it looks like the build.properties is being modified such that the PDE build will no longer work. (build order is removed, for example). Is that necessary? It would be good to have both build systems available to allow for a transition period. Ultimately the build artifacts must be compatible with those produced by the current build system. Due to some complexity related to how the Eclipse platform handles Ant classpath, I'd like to be able to verify correctness of the build output. I'm not setup to run the Buckminster build here. Would it be possible to create a web-based location that has the build artifacts, including update site and stand-alone zips, resulting from the Buckminster build? Ideally a CI build could be set up to run the Buckminster build. This would make it much easier to test the build output and would confirm that we can run unit tests etc. in our CI environment.
(In reply to comment #18) > The patch looks good. A couple of questions: it looks like the > build.properties is being modified such that the PDE build will no longer work. > (build order is removed, for example). Is that necessary? I'm not an expert on the PDE build but I think it should still work, since: - the java sources implementing the ant tasks are compiled by the support build file (/org.eclipse.mylyn.wikitext.core/scripts/buildWikiTextAntSupportJar.xml lines 31-54) and this support build is AFAIK executed during the PDE build - the compile order should also remain the desired one (i.e. first the bundle classes, then the ant tasks classes) only now it is governed by the order of the <buildCommand> tags in the /org.eclipse.mylyn.wikitext.core/.project file (where the support build - org.eclipse.ui.externaltools.ExternalToolBuilder - comes after the build of the bundle classes - org.eclipse.jdt.core.javabuilder) > It would be good > to have both build systems available to allow for a transition period. > > Ultimately the build artifacts must be compatible with those produced by the > current build system. Due to some complexity related to how the Eclipse > platform handles Ant classpath, I'd like to be able to verify correctness of > the build output. I'm not setup to run the Buckminster build here. Would it > be possible to create a web-based location that has the build artifacts, > including update site and stand-alone zips, resulting from the Buckminster > build? Ideally a CI build could be set up to run the Buckminster build. This > would make it much easier to test the build output and would confirm that we > can run unit tests etc. in our CI environment. This has always been the plan (cf. with comment #6) and now it became a reality: http://dev.cloudsmith.com/testhudson/job/Mylyn/ Few notes regarding the build: - The build picks up sources of several bundles from our SVN repository instead of from the Eclipse CVS. Namely those bundles I had to modify with patches attached to this bugzilla. Once the patches are applied we can remove the copies of the bundles from our SVN and the build will use the upstream versions. - There are two more projects hosted in our SVN needed for the build, these projects are used to actually drive the build and would have the moved to eclipse CVS in order to be able to run the build at eclipse. - Most of the test failures are caused by missing credentials file.
> I'm not an expert on the PDE build but I think it should still work, since: > - the java sources implementing the ant tasks are compiled by the support build > file (/org.eclipse.mylyn.wikitext.core/scripts/buildWikiTextAntSupportJar.xml > lines 31-54) and this support build is AFAIK executed during the PDE build No, the support build is not executed during the PDE build :-( I'm attaching yet another version of the patch in which I revert the changes to build.properties once again and instead hint Buckminster by means of a buckminster.cspex file to make it produce the desired result despite the quirky setup in build.properties.
Created attachment 181783 [details] org.eclipse.mylyn.wikitext-ant_builders_v4.patch Yet another version of the patch reverting changes which broke the original PDE build and instead hinting the Buckminster based build to still produce the desired results.
That looks good to me. Are the changes to /cvsroot/tools/org.eclipse.mylyn/org.eclipse.mylyn.wikitext.help.sdk/.project actually needed?
(In reply to comment #22) > That looks good to me. Are the changes to > /cvsroot/tools/org.eclipse.mylyn/org.eclipse.mylyn.wikitext.help.sdk/.project > actually needed? Well, yes and no. Yes, because otherwise the javadoc invocation in /cvsroot/tools/org.eclipse.mylyn/org.eclipse.mylyn.wikitext.help.sdk/build-helper.xml may run before the projects referenced from the .project file are compiled. And since the classes to be processed by the javadoc import many of the classes from those projects this results in a number of errors reported by javadoc. And no, because the javadoc probably produces the correct result even though it reports those errors. So in short these changes (to the .project and build-helper.xml files) can be dropped at the expense of having output of the javadoc invocation cluttered with a number of errors.
Thanks for the clarification. I have applied and committed the patch.
Thanks Steffen! I have removed the patched copies of the projects from our SVN repo so the next build should pick up the upstream versions. This shrinks the projects, copies of which I need to keep in our SVN, to: org.eclipse.mylyn.java.tests As unfortunately it appears that the build.properties file in this project still lists the about.html file twice in the bin.includes property. Could you please fix this too? (Please see the org.eclipse.mylyn.java.tests-build_properties.patch)
Sorry, I thought the build.properties were fixed already. Fix should be in head now.
(In reply to comment #26) Thanks, I've removed the copy of the last outstanding project from our SVN too.
Steffen let me know when a build is available based on these changes and I'll check the WikiText bundles to ensure that they're correctly built.
A new build is now available from the weekly update site.
We have decided not to pursue moving to a Buckminster based build further at this point and to use an alternative build system for the time being (bug 306191). Nevertheless, thanks for providing support here, Michal!
