Bugzilla – Full Text Bug Listing
| Summary: | Build Equinox separately as part of the CBI | ||
|---|---|---|---|
| Product: | [Eclipse Project] Equinox | Reporter: | DJ Houghton <dj.houghton> |
| Component: | Compendium | Assignee: | equinox.compendium-inbox <equinox.compendium-inbox> |
| Status: | RESOLVED WONTFIX | QA Contact: | |
| Severity: | normal | ||
| Priority: | P3 | CC: | Ed.Merks, gunnar, jeffmcaffer, kim.moir, overholt |
| Version: | 3.5 | ||
| Target Milestone: | --- | ||
| Hardware: | All | ||
| OS: | All | ||
| Whiteboard: | |||
| Bug Depends on: | 238626, 251918, 253286 | ||
| Bug Blocks: | |||
|
Description
DJ Houghton
Some comments: It is a large piece of work to decouple the two builds. I see this work going on in parallel to allow the Eclipse build to proceed until the Equinox feature and build time features can be built separately. Also, I'm not sure when the CBI will be ready for new consumers to adopt in the manner that DJ describes it with a signed feature at the end. Andrew Overholt, do you have any dates available? Is the Equinox project in a state today where it can contribute bundles to the Eclipse Project once a week? Otherwise, you'll be spinning a build once to create your equinox bundles, then updating the eclipse build's maps with your contribution since you will no longer be contributing directly to the nightly build. One thing to consider with the current CBI is that currently the tests and build run on the same machine. So there aren't different platforms to run the tests on. This enhancement is covered in bug 251933. (In reply to comment #1) > Andrew Overholt, do you have any dates available? Unfortunately nothing hard at this point. Nick may have a better idea. just catching up on this... was something actually done here? It is marked as FIXED... It is only marked as fixed if you are reading bug reports at 1:30am. On my machine it is still open and NEW. ;-) (In reply to comment #1) > Is the Equinox project in a state today where it can contribute bundles to the > Eclipse Project once a week? Otherwise, you'll be spinning a build once to > create your equinox bundles, then updating the eclipse build's maps with your > contribution since you will no longer be contributing directly to the nightly > build. I think this story can be improved with p2. For example, the Equinox build publishes every build into p2 repository. The Eclipse map files just reference that repository (eg. an Equinox-I-builds rep) and p2 fetches the latest available (recommended) build from that repository. Thus, the map files would no longer reference a specific version but an allowed version range (or just a minimum version). I think this bug can be closed as wontfix. I talked to John about this a few months ago and because all the Equinox bundles have so many dependencies on the Eclipse project bundles, there really isn't a way to separate the two builds at this point. Hmmm, CBI or otherwise it seems that we should be able to build Equinox independently. If you play this out then we have to build ECF, Equinox, EMF (in e4), ... all at the same time cause they all depend on one another. There may well be times when some new thing in say the workbench is needed by the p2 UI code so there will be a build order contention, that is extremely rare. Most of the rest of the time all we need is the API from the UI etc and the most recent I build of that will be suitable. Put another way, if we cant figure out a way to build our modules in a modular way, how can others with their systems. I understand it may not be the absolute highest priority but it is a pretty important topic for Eclipse adopters. Perhaps b3 will handle this? Certainly would be a good scenario for that work. (In reply to comment #7) > Put another way, if we cant figure out a way to build our modules in a modular > way, how can others with their systems. I understand it may not be the > absolute highest priority but it is a pretty important topic for Eclipse > adopters. I agree. I saw other projects doing something with dependencies in Hudson, i.e. once project X was built successfully, built project Y and consume the latest available stuff produced by project Z. yeah, the challenge really is to start looking at how to split up the "Equinox build" into parts that are at the bottom (built first) and parts that are on top (built later) and getting all the prereqs for the latter. We likely would need to step away from the "build features" approach and build individual parts in proper order. I'm aware that you can have chained builds in hudson and build upon artifacts from previous builds. As Jeff states, this isn't the issue. Rather the problem is that the Equinox bundles have dependencies on Eclipse bundles and vice versa. It would be interesting to use the pde visualization tool to map the dependencies. However, since our current build is based on features, this may change the feature structure as well. (In reply to comment #2) > (In reply to comment #1) > > Andrew Overholt, do you have any dates available? > Unfortunately nothing hard at this point. Nick may have a better idea. Athena work (like in all Open Source) is prioritized this way: (a) benefits me personally (ie., work for JBoss) (b) benefits all of Eclipse, patch(es) contributed (c) benefits all of Eclipse (common framework improvements) (d) everything else (corner cases) Since I have no personal need to build on Linux and run tests elsewhere, that request (bug 251933) isn't getting much love ATM. In fact, other bugs related to running tests on Win or Mac are similarly collecting dust. Submit a patch and see them get more attention! :) (In reply to comment #10) > Since our current build is based on features, this may > change the feature structure as well. Athena assumes ONE master feature and ONE tests feature. Wouldn't take much to allow the system to build a list of features instead of a single ALL or Master feature (PDE/Build has no qualms about doing this), but it sounds like to accomplish the functional splitting in Equinox, you just need more features, so you have more entry points from which to start a build. Then collect those features into an uberfeature, and build that. Of course if you have plugin A depends on plugin B and plugin B depends on plugin A, then you'll probably have to collect them both in the same feature and build them concurrently. Or perhaps allowBinaryCycles=true [1] will solve this problem? [1] http://code9.com/2008/10/28/tip-pde-build-and-binary-cycles/ Nick, I wasn't suggesting that you undertake this work. I'm the release engineer for Equinox so I would do this testing if the Equinox features needed to be reworked. As for the tests on multiple platforms, we don't have the actual hardware at the Eclipse Foundation to run the tests on multiple win32, linux and mac platforms. Today, the Eclipse Project's tests take 6-8 hours each x 9 machines. Would other teams appreciate if we started consuming all build.eclipse.org's results to run our Linux tests? Probably not. We could use the Amazon EC2 plugin to provision machines in the cloud. However, they don't support Mac and only Windows Server 2003. Yes, they could possibly be run on vmware or other virtualized services. Does this provide the same testing coverage as the real OS? I don't know, I have to test this to mitigate the risk that we release software that doesn't behave properly on one of the platforms we support. We don't have any binary cycles. The features just need to be analyzed. In addition, the build would need to be reworked so that we would reconsume bundles from the previous build but yet generate source for all the new bundles. See bug 154083 for issues surrounding an incremental build. My comments weren't a complaint on the understandable lack of progress on this bug but rather on technical hurdles that I face to solve this issue. (In reply to comment #12) > Nick, I wasn't suggesting that you undertake this work. I know, I was just explaining how/when things happen in Athena. :) > My comments weren't a complaint on the understandable lack of progress on this > bug but rather on technical hurdles that I face to solve this issue. If you have no binary cycles, creation of new features should be trivial (if time consuming). You can then set up an equinox job @ build.eclipse.org (I'll help!) to verify the build can be done... ... but the larger issue of cross-platform testing and a lack of windows/mac/other slaves attached to build.eclipse.org persists. That one needs time, staff, and of course funding. I suspect vbox or vmware images would be sufficient for running tests, but it's not my head on a lance should problems occur that are missed when run virtually vs. on real hardware, so I'll just bow out of this conversation for now. :) I have hudson admin rights at eclipse so I'm well aware on how to set up hudson jobs. We use hudson@build.eclipse.org today to run test builds. thanks for the additional input here. I'm a little "concerned" about the discussion around reworking features etc to accomodate build requirements. I mean, sure, we could define a mess of features that help us organize the build but these should be treated as just lists of bundles rather than something that we ship. We don't want to ship our development structure. Moreover, an interesting goal here would be to build Equinox without having to build anything else. We should simply be able to consume our prereqs like everyone else does. As mentioned in comment 7, it is rare that there are interlocking changes between the interlocking pieces. So dependencies outside Equinox should be treated as dependencies and taken from the target/base for the purposes of compilation. <meta-question> is there any interest in reopening this bug?</meta-question> So the steps would be 1) assemble Eclipse project bits in a baselocation/repo for the Equinox build. 2) run Equinox build. 3) contribute Equinox bits to Eclipse build via baselocation/repo etc. This means that the Equinox team would have to contribute to two builds each time. They would have to run an Equinox build and build the dependancies before contributing to an Eclipse build. I mentioned this to a few Equinox committers before and they said they weren't ready to do this but if this has changed, please feel free to open the bug. (Yes, I know builds can be chained in Hudson, but as I mentioned earlier, we can't run our entire build in Hudson yet, because I have to test migrating our test infrastructure to the cloud) good/clarifying summary Kim. Thanks. I have a slight variation on step #3 however. The contribution or consumption of a lower level piece to a higher level piece is a bi-partisan endeavor. We could also cast #3 as "the Eclipse team elects which version of Equinox to include in its SDK build". In reality it ends up being a negotiation but the consuming team has to say what they are want to consume. This is much like the Equinox team does with ECF and (I assume) the e4 team does with EMF. Actually, along the e4 lines, does the e4 build rebuild Equinox? Do we have an Equinox build that builds or includes e4? My hope on both counts is NO. These should be done independently and the resultant binaries consumed and composed as needed. Perhaps we can use the same approach on the Eclipse SDK builds? Similarly, we will have to consider what version of the platform we are interested in consuming when building Equinox (e.g., the p2 ui). I understand and support the reluctance to do extra work here. I'm hoping to get to an low-effort position. The benefits are - freedom of action in Equionx builds - eating our own build dog food - ensuring that others who are in similar positions can build their interwoven stacks These are all good outcomes on our quest for highly modular systems (In reply to comment #17) > Actually, along the e4 lines, does the e4 build rebuild Equinox? Do we have an > Equinox build that builds or includes e4? My hope on both counts is NO. These > should be done independently and the resultant binaries consumed and composed > as needed. +1. That's how I do it in our builds and thanks to the p2IU fetch factory in map files it works great. Once you have a repository it's easy to consume it in your builds. I think that the story could also be enhanced to always fetch the latest available. Thus, teams won't have to update their map files just for consuming a higher Equinox build. Instead they would just point to a repository of all the 3.6 Equinox I-Builds and p2 would fetch the latest for their build. The e4 teams consumes platform integration builds from our p2 repo. They don't rebuild our bundles from source. No, there isn't an equinox build that includes e4. Jeff, I talked to John and Tom regarding this issue. The expressed concern that this change would cause the team even more pain than they do today because they would have to prepare two build inputs. They noted that we already consume ECF bundles and this is often a problematic process. In all honesty, I would rather concentrate my time in the 3.6 cycle in improving the test infrastructure so the tests don't take eight hours to complete. This is a serious problem in our build. Splitting out the Equinox and Eclipse builds is an interesting use case but I don't see how it will benefit the teams if the actual build process still takes forever to produce the test results. As an aside, since the Equinox build is a relatively small build that encompasses a lot of functionality (publisher, launchers, product builds, repos etc), it would be interesting to use it as a test case for b3. I completely understand your desire to focus on getting the tests faster etc. You should do what you think is the right thing. I'm selfishly pointing out that fixing this bug does help the Equinox team in that they would no long have to wait for the 7.5 hours of testing on JDT, PDE and all manner of other stuff that has little to do with Equinox :-) More generally I'm striving for a place where all teams can run "a build" quickly and easily with or without tests. This should in turn reduce the number of failed I builds we do and reduce the need to rerun tests. As for adopting b3, sure, when the time comes, it would be good to look at it. |