Bugzilla – Full Text Bug Listing
| Summary: | [plan item] Improve PDE build | ||
|---|---|---|---|
| Product: | [Eclipse Project] PDE | Reporter: | Mike Wilson <Mike_Wilson> |
| Component: | Build | Assignee: | pde-build-inbox <pde-build-inbox> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | enhancement | ||
| Priority: | P4 | CC: | Darin_Swanson, David.Olsen, francis, gheorghe, gunnar, igor, kim.moir, Michael.Valenta, mpcarl, pombredanne, sgambino, tjchiass, tlroche |
| Version: | 3.1 | Keywords: | plan |
| Target Milestone: | 3.2 | ||
| Hardware: | All | ||
| OS: | All | ||
| Whiteboard: | |||
|
Description
Mike Wilson
Improved debuggability would also be appreciated. E.g. at present error messages are often quite inscrutable. In addition to being in violent agreement with Tom about better error messages, I have been thinking about the last two things in Mike's list myself recently. "increased integration with the workspace model": This is something that has bugged me and some of my coworkers about PDE Build, that builds done through PDE Build are done differently than builds developers do within their workspaces. Because they use different mechanisms, the Java code may be compiled with different options, and other aspects of the build (such as running custom Ant scripts) are done differently in the two environments. I think PDE Build should use the exact same mechanism as the Eclipse IDE to do the bulk of the build. PDE Build would just perform the extra task of setting up the workspace before invoking the Eclipse IDE build. "support for additional repository providers": I spent a while trying to get PDE Build to work with ClearCase and came to the conclusion that it is just not possible without a major overhaul of PDE Build. PDE Build's view of source control is very CVS-centric and can't be mapped to ClearCase or other repositories that don't behave like CVS. The features that tie it most tightly to CVS are (1) that each directory in the repository can be fetched using a different tag/branch/label, and (2) the directory structure of the fetched source can be completely different from the directory structure in the repository. This does not map to ClearCase, where using different branch/label rules for different parts of a VOB is non-trivial, and where the directory structure of a view needs to match the directory structure in the repository. There is also the fact that PDE Build expects to fetch the source fresh from the repository with every build, while ClearCase is designed to keep a view around for a while and just update the view when doing a build. (This isn't a fatal flaw, since creating a new ClearCase view with each build isn't too hard. But there is a record of the view on the server, so you have to remember to remove the view when the build is done, unlike CVS where the server has no record of checkouts. And creating a new view with every build runs into ClearCase's significant networking issues.)
McQ 2005-08-05 13:01
> Potential improvements to PDE build include parallel cross-building,
What is meant by "parallel cross-building"? Inquiring minds ...
McQ 2005-08-05 13:01 >> Potential improvements to PDE build include [...] support for >> additional repository providers David Olsen 2005-08-08 12:20 > In addition to being in violent agreement (Careful, you'll scare our Canadian pals--they already think we're all imperialist gun nuts !-) > "support for additional repository providers": I spent a while > trying to get PDE Build to work with ClearCase and came to the > conclusion that it is just not possible without a major overhaul of > PDE Build. I believe (I Could Be Wrong, Your Mileage May Vary, I certainly don't know as much about PDEB as I would like) that the main differences between PDEB and its users are that the latter are more likely to * use SCMs with non-CVS semantics * bring in a larger proportion of build inputs as binaries (e.g. prebuilt components) If so, then I suspect (ICBW, YMMV) both PDEB and its users can be well accomodated by a PDEB that * can start up "pointing to" some workspace (e.g. a CC MVFS) * has something like a custom ant task for fetching from CVS * has something like a custom ant task for fetching binaries Then, e.g., * a PDEB user like R*D could start up in its dynamic view, and in its fetch phase get · whatever additional sources they required (from contributors who are not using CC) · its binary dependencies * PDEB could just call its CVS task as usual in its fetch phase This may be possible with today's PDEB (I certainly haven't done it yet), but likely requires more hoop-jumping than users would like. (In reply to comment #2) > "support for additional repository providers": I spent a while trying > to get PDE Build to work with ClearCase and came to the conclusion > that it is just not possible without a major overhaul of PDE Build. This is covered in bug 34757. I've also written a patch, which works well here in our ClearCase environment. Basically, it enhances the map file syntax to fetch from different repositories. Someone might want to write a specific ClearCase provider that creates a view with a specific config spec, fetches the source and destroys the view afterwards. However, you can't use the tag+release concept that Eclipse uses with CVS with ClearCase. IMHO it's too much overhand if you create and destroy a view for every plugin. I have estahblished another approach in our environment. PDE Build is invoked throuh CruiseControl. Before building any Eclipse stuff, the view is updated with a specified config spec (e.g., latest or milestone builds). CruiseControl -> bootstrap builder -> start builder Builder -> get config spec & update view -> start patched PDE Build (In reply to comment #5) > IMHO it's too much overhand ... I meant to say "overhead". Comment #5 From Gunnar Wagenknecht 2005-08-09 00:14 > [Support for additional repository providers] is covered in bug 34757. So perhaps discussion of that particular PDEB improvement should move there? And perhaps separate bugzillas should be spawned for each of the other mentioned enhancements: * parallel cross-building * incremental building of plug-ins * increased integration with the workspace model * improved debuggability This bugzilla could then be reserved for new enhancement proposals. Ted Chiasson (from AST) has submitted several PDEB-related bugzillas: 84351: Can't package a feature when patch is present, may fail silently 93299: PDE packager deletes tree at the end of packaging step 97346: Ability to specify default build.properties 100460: root property in packaging.properties does not seem to be used 106490: generated unzipper.xml does not work correctly (-x /features) 106505: build.properties in feature dir causes feature to not be packaged I would to precise some of the items that have been listed in the initial comment, scope some of the work that is being planned, and list other things we'd like to do. However nothing guarantee that all of these items will be covered in the 3.2 timeframe. It just constitutes a list of improvements / directions that are being considered. Moreover it does not include the obvious bugs that must be fixed. I like to classify things, so there are really 4 themes of work that I can see happening for PDE Build. Usability ======= This theme encompasses various aspects of PDE Build usability such as error reporting, flexibility in the input, complexity / modularity of the infrastructure, etc... . - 68048 Better handling of errors on state build - 75285 Footprint required for upgrade to 3.0.1 not known - 86116 Useless error message for failure Exporting Feature - 97835 Simplification of application export - 97929 Provide an option to sign feature root files - 106123 UI feature export crashes when version qualifier contains underscores - 35371 Ways to hook in an existing build.xml - 105282 Improve support for JNLP export - More flexible layout for input to the build - Improve support for product export - Ensure composibility of the various tasks Runtime fidelity ============ Like any other plugin in the PDE space, PDE build sole purpose is to tool the runtime. Therefore it should be as faithful as possible to the runtime. - 106173 The classpath computation does not take friends and private export into consideration - 98757, 105408 Restrict the set of bundles considered by PDE Build (based on feature) Scalability ======== As eclipse and eclipse based products become bigger, the build takes longer and it takes longer to deliver the drops. In order to cut this linear relationship between the size and the time, the provided infrastructure needs to provide ways to make build faster. - incremental fetch - delta drops (only contain the plugins and features that have changed) - build in parallel - review the scripts that are being generated Workspace integration ================== PDE build design point has been to be decoupled from the workspace / IDE. However, as the workspace provides richer and richer functions, the decoupling is more and more problematic. With the priority and timing/resource constraints known as I'm writing this, only the "preference" section can be considered by 3.2. For a longer discussion on the full problem of workspace integration please go to the bug 106594 Preferences - 81918 encoding used in projects is ignored - 55353, 76218 project or workspace compiler settings are ignored - 82414, 103645 source filters are ignored IResource awareness - 101966 "Export as deployable plugins" fails when prerequisite plugin is linked binary project What happens here with respect to the remote resource work going on Builder awareness (35371) - 74880 PDE build does not use project class libraries - 101241 Plugin exporter should support exporting plugins from their output folder - 104156 PDE build.xml should use all project builders Build input =============== From the packager and the builder, PDE provides a set of tasks that help fetching binary drops from URLs and source code from CVS. However, theses tasks have restrictions and should be opened. - 34757 Multiple repository: the fetch task provides a way to automate the fetching of code from a CVS repository. It is nice this it guarantee that all the right depends are fetched and it is minimal for the things being built. However if the code comes from multiple repository and repository that do not offer the same flexibility than CVS things get harder. Note that the fetch step is optional. - build from update site https://bugs.eclipse.org/bugs/show_bug.cgi?id=106214#c9 From Pascal Rapicault 2005-08-10 00:44 > I like to classify things, so there are really 4 themes of work that > I can see happening for PDE Build[: usability, runtime fidelity, > scalability, workspace integration, build input.] Umm, seems like 5 to me :-) Could we add a sixth? My impression is that folks have problems with the packager. In addition to comments in Constellation Build calls (will try to get more input on constellation-build), there are Chiasson's bugzillas 84351: Can't package a feature when patch is present, may fail silently 93299: PDE packager deletes tree at the end of packaging step 100460: root property in packaging.properties does not seem to be used 106490: generated unzipper.xml does not work correctly (-x /features) 106505: build.properties in feature dir causes feature to not be packaged So perhaps theme=Packaging is in order? Hi. I opened a number of packager defects, because fundamentally I believe that packaging must be part of a complete solution. I can compile code, but at the end of the day, what I need to produce is a product. (collection of plugins, features and "other") Other than that, my main concern is ease of use and diagnostics. If you are only compiling a few plugins against a given base, then it is usually easy to determine what the problem is when something breaks. But when we dump in a new binary prereq, such as a new IES to build against, then hundreds of plugins and features may have changed, and it is very hard to isolate the problem. Based on the following suggestions, I made Jie Chen 08/10/2005 03:40:49 PM: > * After PDE build finishes, could we choose not to delete the tmp > directory (where all the files get zipped from) if we want?. I > noticed that in the assemble.*.xml/package*.xml files, they always > have a <delete dir="${assemblyTempDir}" /> task in the main target. > Could we disable it if we like? https://bugs.eclipse.org/bugs/show_bug.cgi?id=106685 (for theme=usability) and > * For the PDE build, is there a way to split English and NL feature > packaging? The current PDE process seems always have the NL feature > packaged within the English feature (it probably because the English > feature includes the NL feature in the feature.xml file). Is there a > plan to make it happen in the future? https://bugs.eclipse.org/bugs/show_bug.cgi?id=106687 (for proposed theme=packaging). https://bugs.eclipse.org/bugs/show_bug.cgi?id=106214#c9 > Scalability > ======== > As eclipse and eclipse based products become bigger, the build takes > longer and it takes longer to deliver the drops. In order to cut > this linear relationship between the size and the time, the provided > infrastructure needs to provide ways to make build faster. > - incremental fetch Perhaps also incremental compile? Why I ask: As the concept of continuous integration gains increasing mindshare, CruiseControl is increasingly adopted (even in IBM :-) for its implementation. IIUC if PDEB did incremental fetch/compile it would be much more useful to CruiseControl users. But perhaps incremental compilation is more germane to https://bugs.eclipse.org/bugs/show_bug.cgi?id=106594 > Make PDE fully workspace aware ? Closing as fixed as most of the item from the list given in comment #9 have been addressed. usability ant friendly logging complete end to end support for product export documentation on how to use feature and product export more flexible input with the plugins property that supports ways to hook in build.xml at plugins and features level support for versioning runtime fidelity filter what is being put on the classpath based on the import / export possibility to compile a bundle against a particular class library Workspace integration encoding preferences compiler preferences build input pluggable support for repository |