Bugzilla – Full Text Bug Listing
| Summary: | EMFT Build Stack Automation | ||
|---|---|---|---|
| Product: | Community | Reporter: | Nick Boldt <nboldt> |
| Component: | Cross-Project | Assignee: | Nick Boldt <nboldt> |
| Status: | RESOLVED WONTFIX | QA Contact: | |
| Severity: | normal | ||
| Priority: | P3 | CC: | davidms, richard.gronback, steven.wasleski, webmaster |
| Version: | unspecified | ||
| Target Milestone: | --- | ||
| Hardware: | PC | ||
| OS: | Windows XP | ||
| Whiteboard: | |||
| Bug Depends on: | 85485, 116912 | ||
| Bug Blocks: | 141152 | ||
(From Dave Steinberg) Hi Nick, There's much that you're asking that's not for me to comment on, but there are a couple of points I do have thoughts on... I thought we were being asked us all to use "Calisto" milestone numbers, which are the same as the Eclipse Project's. I think using the same build type and M/RC number as the upstream dependency makes sense. Also, I don't think it makes sense to do an extra N build, in advance, for smoke testing or whatever. It seems like a waste of time and CPU, and personally I'd rather smoke test the actual build that will be released, rather than one that we hope will be identical to it. It seems especially pointless if it had to be based on an older platform build because of timing. Perhaps a better idea would be a two-stage promote. A cascading build could be automatically promoted when ready, with just the RSS announce. At this point, there would be no announce in the newsgroup or on What's New, no Release Notes, and the build would not show up on the Downloads page (I realize the download page is automatically generated from the contents of the drops directory, but maybe a dotfile could be used to list builds to ignore?) or in Update Manager. Then, after manual smoke testing or whenever the team is comfortable, a manual reveal/release/announce (or whatever you want to call it) step could be initiated to unhide and announce the build. This would even make sense for promoting the build before the tests are complete (as the platform does). You could do: build promote (with RSS announce only, tests pending) update RSS announce (when tests complete) reveal. Obviously, if the tests failed, you'd never get to 4. I think this approach would also enable the minimum lag time possible for the cascade. Cheers, Dave (from Kenn Hussey) Dave, Yes, all projects on the "release train" are supposed to be on the same milestone numbering scheme. But of course, UML2 isn't officially "on" the train. :( I agree that the extra N build seems like bit of a waste, but personally I like the idea of using a successful N build on the server (of the right timestamp) as an indication of whether that night's I build should take place. There are some weeks when I don't do a UML2 build, in which case the absence of an N build on the server could indicate that the last known good build should be used instead... I also like the idea of a "reveal" step so that project leads (or whoever) can have the final say when/whether the build is announced. (from Dave Steinberg) I don't really follow what you're saying about the N build being used to determine whether the I build should take place. I think Nick's proposal was to automatically do the N build, so it would always be there (unless, I suppose, it failed). I think it's reasonable that you should be able to tell the system that you don't want to build that week, and then others will use your previous build (I'm not sure how they'd know...maybe we'd need another kind of feed entry for this case?), but I don't see how the N build does that. I'm glad to see some discussion starting on this. Thanks, Dave, for getting the ball rolling. ;-) Four items are discussed below: 1. Why Wed N build stack? 2. Enable/Disable membership in the build stack 3. UML2 as dependent project, but not part of Release Train 4. Build "Reveal" step As always, comments/feedback/suggestions welcome. Thx, Nick -------- 1. Why Wed N build stack? Some background... there are a few reasons behind the Wed noon N build idea: a) sanity check (EMFT has decided? requested? that their components need to ensure that everything is "compiler error free / JUnit test failure free" prior to Wed night when the Thu build cascade starts.) This is optional, of course, but recommended, and not that far off what we do Wednesday for EMF, in order to make sure the past week's commits haven't wrecked havoc on the JUnit (or other) tests. b) having an automated N build means that everyone in the stack will be on the latest (n' greatest?) Eclipse driver, plus all the downstream other dependencies. This means that that elusive, hitherto unseen Build Convergence idea that ISS has for M5 and beyond is (in concept, anyway) feasible, since everyone will be automatically building using the latest upstream component w/o having to worry about grabbing the wrong SDK accidentally or being a week off, etc. While the process in EMF land is to manually grab the latest Eclipse driver about once a week (depending on how many JUnit failues it's reporting), and similar for UML2 (adding the latest EMF driver to the mix) this is much more cumbersome when you have to grab 3 SDKs (Eclipse, EMF, UML2) from three different webpages, then build OCL, and grab that URL to build Query and Validation, and finally grab the Validation URL to build Transaction. I've done it a few times, and frankly, it's a royal pain. And that's not just the cough syrup talking. ;-) I can imagine that as the guy that built the UI (http://emf.torolab.ibm.com/emft/build.php), I have *way* more patience than Christian & crew will after the first few iterations thru the process, so I'd like to make it easier now rather than wait for their headaches to be reported. c) doing an EMF-to-EMFT-Transaction stack build at noon acts as a dry run for the I build stack to follow 12 hours later (at midnight), giving us two useful pieces of data: i. how long does each build step take, and have certain parts gotten longer due to more code to compile, more zip packages, or more tests (requiring adjustment of the crontab scheduled starting times)? ii. did everything work along the way, and if not, which builds failed? ---- 2. Enable/Disable membership in the build stack As to the idea of enabling/disabling one's membership in the stack on a given week, with the assumption that if absent the stack will build around you and use your latest good available driver instead of the fresh-from-the-oven one, this can be accomplished in a number of ways, depending on how the members are coupled together. For the moment, I'm going to assume we have no desire to build a "I'm done, now you start" PUSH mechanism. Therefore the PULL mechanism would work like this: a) each project in the stack would have its own crontab, which would be watching for a specific event to happen (within a specific timed window) in order to automagically kick a build. This "event" is most likely to be the presence of an updated datestamp in an XML file indicating a new build is available. (This is how the UML2 N builds currently fire.) For example, the Transaction crontab would listen for a new Validation build around 2pm on Wed afternoon, then if not found, sleep for a minute before trying again; after, say 30 mins, it would stop check/sleep cycling and fall into a coma until reawakened by the crontab's magical spell at about 2am, for the I build cycle. (The sleeping/cycling aspect is new; I currently have the UML2 N builds set to simply check at a precribed time.) b) if a project wanted to remove itself from the stack, a web UI (or shell script) would essentially disable that entry in the crontab, and it would stop "listening" for that file. Reenablement would be the reverse, turning the listener back on to wait for the next cron cycle to spring to life. The idea of opting out then opting in a week or two later introduces another reason for the need for the Wed N build stack... if the project has been "out of the loop" for the past couple weeks, it may not have implemented whatever fix/change the platform has forced on us, and may therefore break when compiled against the newest SDK. Similar issue if EMF or UML2 add some sort of breaking change and OCL hasn't had time to react due to illness, vacation, or the more likely event, other more urgent bugzillas. The N build should allow the once-absent project to "catch up" to the latest stack dependencies, and verify that nothing unexpected has happened, getting it ready for the Thu I build stack. This further begs the question - if you've been away from the stack for a couple weeks, and you rejoin to discover you've broken... should the stack go on without you, as it did the previous week, just using your latest good driver? Or would this NOW be considered a stack-breaking problem that would cause the downstream projects to all fail, and hold off building until their upstream project(s) can resolve the failure(s)? I can argue either case, so I'll leave this up to ya'll to decide it. ---- 3. UML2 as dependent project, but not part of Release Train UML2 may not be on the train, but it's a dependency of Query and Validation, and Validation is a dependency of Transaction. So without being "in the train", three EMFT projects are dependent on it nonetheless. And then GMF is of course dependent on all 6 (EMF + UML2 + 4 EMFT projs). But if we can make the stack work, and build a nice workflow and set of guidelines around that, we'll have a very compelling case to get the other further downstream projects on board the train, too. ---- 4. Build "Reveal" step I also like the idea of a manual "reveal" step. If no one wants to see autopromoted builds, then that's less work for me. ;-) We currently have this system in place for EMF - I do weekly automated I builds every Thu at 2am. Thu morning I then manually verify if the build's tests all passed, re-run any that might have experienced what I'll call "the gravitational anomaly effect", and if everything's hunky-dory, promote it. The last manual step is to update the affected Assigned bugzilla to Fixed, which we might be able to some day automate as well, with a little help from Denis & Matt (if they want to give us SQL write access to bugzilla). Similar process for UML2 and EMFT - builds are run and tested on emf.torolab, but not "revealed" until manually uploaded to download1.eclipse.org via promoteToEclipse.sh script. If we all want to maintain this process, so be it. I don't see value in splitting the identification of a build being "okay to promote" from the process of doing the promote itself - if you like a build, you promote it. Once UML2 and EMFT have web UI for doing these promotes, it'll be easy for them to effectively mark a build "okay to promote" and promote it automatically at the same time, too. (from Dave Steinberg) Some clarifications... 1. My point on the Wednesday N build is this: what's the difference between a successful N test build and the following I build? The name? So, why not just make it an I build on Wendesday in the first place, and respin Thursday only if it doesn't work? Either way, if you want it to contain the latest and greatest, you can't start until everyone's done all their changes. So, you're not really doing anything "in advance." You're just guaranteeing that you'll have to do another build (or, at least, a rename), even if everything's fine. 2. I thought that part of the point of this build cascade is that if someone discovers a problem in their component, they can just kick off their build, and everyone downstream automatically builds. That would require a push mechanism, which I think would demand an active opt-out (probably a different kind of feed entry) from any component that's not in a good state to build. Perhaps the push vs. pull question will be better resolved with the input from the rest of IES (if that ever happens). 4. The point of the two-step promote is that other projects are going to need to get our builds to participate in the cascade. We'd like them to appear as quickly as possible on eclipse.org so that they can start building (the whole world doesn't have access to emf.torolab). We don't want them to have to wait until we've manually tested and promoted. If we find a problem, and we can cause them to rebuild (there's that push model again), there's no problem with that. But we certainly don't want the public to see our builds until we've checked them out and manually ok'ed a "reveal." 1. The only difference between an N and an I build is that N builds do not tag CVS, and thus are not considered to be supported. I/M/S/R builds all tag CVS and thus can be reproduced easily, so are considered to be supported. But you're right, we could resolve the need for pre-flight testing by running the I build at noon on Wed instead of that night, and only respinning if necessary. Further, I can untag CVS for builds that fail or are never published, using purgeCVSTag.sh, so I would be okay with a mid-day Wed I build instead of a midnight or crack-of-dawn build, as it adds a lot more wiggle room for possible problems & their speedy resolution. Of course if the post-M5 plan is that Eclipse and ISS will all release the same day, then this of course won't work; in that case we'd have to watch for a new SDK on fullmoon.torolab and immediately kick the stack then and there, and hope for the best. <shrug> 2. That mechanism can be accomplished two ways, depending on how we think the best way to implement it is. To cause downstream builds to automatically fire off their builds, we could: a) have the new build turn on its downstream builds' crons (which implies that we know who all the downstream projects are, which may not be the case for projects outside our stack - consider if we wanted to have a new EMF driver kick off a new WTP build. How could we tell that server to start building?), or b) have crons checking 24/7 (unless manually disabled) on any/all servers "subscribed" to this notification service I'm all for (b), as long as we're clear what that means. Does that mean any time I kick and EMF N build to test something, the whole stack would follow suit? Or would this listening only apply to I/M/S/R builds? Should the stack itself listen for new Eclipse SDKs, and kick itself off, starting with EMF, every time a new "good'ish" driver is available? And it occurs to me... let's say project C depends on B which depends on A (Transaction -> Validation -> OCL). When there's a project that's opted out from the stack, how would project C know that it was time to build if A rebuild and B opted out? I'm thinking that B would have to listen for the event from A and pass it on to C so that the chain could continue; otherwise, the stack would stop at the first opt-out point. Yikes. Or, what if D depends on A, B and C (Query -> OCL+EMF+UML2), and B opts out? I suppose in that case D's cron would need to be written such it was listening for three messages before responding with an automatic build, and then would get the message from A and C that it had rebuilt, and from B that it would not be rebuilding. This just got hairy. So not only do inert projects have to pass messages, they have to also pass their own "I'm not building right now" messages in case anyone is listening and would otherwise be waiting on them. Conversely, builds have to wait for up to N dependents to report that they've either built, haven't built (waiting for a dependent), or will not be building (project is dormant / opted-out). Okay, my brain hurts... Dave, does the above logic make sense? Am I oversimplifying/overcomplicating things? 4. You're right, I missed that point... a "hidden promote" is certainly possible, and in that context totally makes sense; in fact, it's very easy in that it only means uploading to a folder not indexed by the downloads page. Minor tweak here and there in promoteToEclipse.sh to simplify the split between [scp'ing the download folder to eclipse.org] and [everything else - release notes, ISS map file, etc.] and an extra if condition when displaying builds on the downloads.php page and we'd be good to go. There's already a -droponly flag to only do the scp step and ignore everything else; I'd just need to have a new option for -drophidden or something like that to allow the drop to be uploaded in a non-visible way, and a -reveal flag to (instead of uploading) move the hidden folder to a visible one. Would downloads/drops/.hidden/2.2.0/I200601120200/ work for you? The only wierdness here is that it's likely that the mirrors will mirror these "hidden" drops even if they're not linked from the downloads page, since I'm pretty sure rsync is just a folder-by-folder compare. Are you cool with that feature-not-a-bug? ;-P So, to sum up, we need to track and traffic the following status messages: 0. waiting for upstream build (I'm having a nap, wake me in 5 mins and I'll check again for any updated dependencies), 1. building (I will stop responding to events until my build is done), 2. ready/waiting - new build available (you need to start a new build too, unless you're a dormant project; 5 min nap time), 3. dormant (please skip me and use my last known good build - no build for me this time 'round), 4. promoting (don't promote another build from my project or your project at the same time) 5. promoted/waiting (I'm ready to reveal this build, once the tests are done; 5 min nap time), 6. revealing (I'm creating all the web artifacts (release notes, news, etc.) and revealing this build to the public), & 7. revealed/waiting (I'm having another nap, wake me in 5 mins and I'll check for any updated dependencies so I can build again). Actually, it's worse than that... since the whole idea here is to promote ASAP - before the tests are done - we're looking at one of two things: a) conceding that build = (build + JUnit), but build != (build + JUnit + JDK tests + FVT Tests + Perf Tests + Manual Sniff / Sanity Tests, etc.), or b) splitting the PDE process into two parts, and somehow kicking off a promote while the build folder is still in flux I don't like (b) here - seems like that would violate some fundamental PDE rule about testing what you build or would otherwise be a royal pain. I suppose it's possible to copy the in-progress build parts to its final destination folder before starting the tests, and running them from somewhere else so that the scp process doesn't get confused about which files it's supposed to be sending; or to scp immediately after the build is done, and prevent the tests from running in parallel. Are we sure this is what we want to do? I'm all for promoting while the 45-min FVT tests are still running, but I'm not so sure about messing about with the PDE build-and-JUnit harness, or delaying test results by 20-30 mins to allow the build to be promoted, THEN tested. On the other hand, doing a promote and running memory-intensive tests concurrently might be a bad idea too, especially if other builds are also starting at the same time since they just got notification that there's a new EMF driver available. Thoughts? Thanks, Nick, for posting this e-mail discussion to the bug report. One point I meant to make, is I think it would be better to hide and reveal the build without moving it. Generally, it's considered bad etiquette to move things on the Web, no? I could even imagine that a downstream build might see a release announcement, waiting for another dependency, and finally, right when it's ready to do its build, the first dependency build gets moved before it can send its GET. Oops. My idea was to just add a file (.hidden, for example) to the drops directory. Its content could just be one build per line, e.g. # Builds identified in this file should not be shown on the download page 2.2.0/I200601120200/ The script that produces the builds page could read this file first, and just exclude any builds listed in it. Then, I200601120200 wouldn't show up. Revealing the build would just mean removing that line from the .hidden file. Does that sound reasonable? Certainly a good idea, and it gets around the performance issue w/ moving a build (ie., mirrors would have to delete and then copy it over from download again), since in this case only one file would change. One wrinkle here is FTP mirrors... they don't use PHP to display folder contents, nor do they have 403 errors when you browse directories directly, or even do they display a default index.html instead of a dir listing if one's there. All you get is dir browsing, and a .hidden file wouldn't change anything. So for the case of FTP mirrors, "hidden" builds would not be hidden at all, unless we have some way of configuring the ftp servers remotely like with an .htaccess file or some other dotfile. Given there's no homogeneity enforced on mirrors, this isn't bound to be an easy task, even if permission was granted to create some sort of hiding rule and if the servers themself supported such a technique. Do we care about FTP mirrors not conforming to a .hidden rule? I don't think FTP mirrors matter. I don't mind if people can find it, I just don't want it to look like we're announcing it. I wonder if the mirrors would behave better (in terms of hiding/unhiding) if there was one .hidden file per build (in that build's folder - emft/transaction/downloads/drops/1.0.0/I200601020000/.hidden) or if it would be better to have a single file above that in a central place, eg. emft/.hidden. Does anyone know if rsync works better/faster in response to changes to lastmod date on a file or to creation/deletion of a file in a folder that's otherwise unchanged? Moving to Callisto I didn't really get a chance to read through all the discussion, but Nick asked for my feedback. If your files are placed on download.eclipse.org, I don't really see a way to "hide" them from mirrors, unless you create a directory called "hidden" and I exclude the hidden directory from our RSYNC process. When you want to un-hide your build, you simply move the files from hidden/ to R-2375761212-whatever. If you decide to go this route, I'd like for the "hidden" thing to be something that can be used by all projects; i.e. the standard hidden directory be called, say, "z-hidden" where I can configure an rsync option that will patten-match "z-hidden" globally. For this, I suggest opening a bug against Eclipse Foundation/Website to gather feedback from everyone. D. There is a lot of discussion here, that does not really seem like a callisto issue. (only some project are interested?) I think some of this is a dup of 128556. There is, now, an "exclusion" list that is not mirrored ... if that's most of what's desired in this train of discussion. (and, if you want to be consistent, we in webtools use /webtools/committers/ for our non-mirrored continuous builds. Then we we are satisfied a build is ready for others (end users and adopters) we copy (not move) it to /webtools/downloads/ [we leave the copy then till the following cycle ... just in case someone was in the process of getting it, but then delete all the other "temporary" builds. Is this still an outstanding issue? Or can this be closed? (In reply to comment #13) > Is this still an outstanding issue? Or can this be closed? I haven't heard anyone recently mention the need for promoting but hiding builds or synching release numbers (on the contrary, we've moving in the opposite direction), so I'm going to close this as WONTFIX. If any of the ideas in this bug are important to anyone, please reopen with comments, or better yet, open a fresh bug w/ a simple enhancement request. |
Anyway, please take a few mins to peruse this, and provide me your comments to what I'm proposing and to the issues/questions at the bottom. Your feedback will form the basis of how Callisto will be implemented in the coming months, where they plan to have the entire product stack, from the Eclipse SDK all the way down, releasing milestones and RC builds on the same day. There's a summary of questions at the bottom, to make it easier to comment on the various debateable aspects. Thanks! -- -- -- Many things need to come together to make this work, and it probably won't be for a month or more, but if the stars align properly, I'd like to have a process in place by M5 so that every Wed night, perhaps at midnight, the EMF-UML2-EMFT(OCL)-EMFT(Query+Validation)-EMFT(Transaction) stack can automatically build itself and be ready for promotion Thursday morning (or sooner, if I set up automatic promotion, too!) Since these builds are all on the same server (emf.torolab), we can take advantage of that to speed up the process between them; however, this means I'll need two levels of dependency verification ... check local THEN check remote. Or better yet, local then ssh to download.eclipse.org. That would at least bypass the mirror problem, since ssh doesn't have the same mirror/redirection problem that http does. The reason for this is that in our case ("Stack"), I can have crons checking for files locally, whereas in the "Stack+1" case (where +1 is some other IBM or non-IBM eclipse project, like GMF), I have to check fullmoon.torolab for uploaded builds, which introduces at least an hour's lag for mirror delay (often 2 hrs). But with ssh to check download.eclipse, the lag is less as I'm only worried about internode synching, not interserver synching. Anyway, the point here is that I've got a plan to make the process of getting EMF, UML2, and at least the first 4 IBM EMFT projects out the door before the sun comes up on Thursday morning. What I need from the various team leads is buy-in to this plan and a commitment that whatever's in CVS by a certain cutoff time on Wed night (let's say midnight) is stable enough that it can be used for an I build, which (second part of the buy-in) I can automatically promote if everyone's cool with that idea. This implies that if necessary, a sniff/sanity test has been done before EOD on Wed, ideally by (manually? automatically?) kicking off an N build which includes the latest Eclipse EMF, UML2, OCL and/or Validation drivers in its build dependencies so that there aren't any surprises Thursday morning. Actually, having an automatic N build cascade set up to run on Wed at noon might be a wise idea too, which could be something like: EMF 12:00-12:50pm, UML2 1pm-1:30, OCL 1:45-1:55pm, Query/Validation 2-2:10pm, Transaction 2:15-2:25pm. Then, in 2-1/2hrs, we'd have a full stack; any problems could ideally be found, corrected, and respun, leaving lots of time to verify. One problem with this is that we might not be able to use the latest Eclipse I or S driver, since they move around a little (Tue/Wed/Thu/Fri), and take ages to complete testing. This isn't something we can fix, however. That's *their* process. ;-) In the event that a build is promoted which is imperfect (for whatever reason) I'd like to set up a side door to this process so that if anyone needs a respin, their downstream components can autobuild as well. A new Validation build would spark a new Transaction build, a new OCL build would spark the other 3 EMFT builds, UML2 would spark 4 new EMFT builds, and a respun EMF would kick off the whole stack anew. Perhaps this side door would be a checkbox on each project's build page, for "[ ] Kick Off Build Cascade When Build Completed?" ... thoughts? Is that too much power to be assigned to a single mouseclick? Should it be an option on the first build's promote script, so that is the respin fails testing, the rest of the stack won't bother to rebuild? Or do I have to make sure that the build itself, when determining if the build "passed", verifies itself before automatically kicking the rest of the stack? (How much automation is too much? ;-)) The other option, instead of this "push" model of each project telling its downstream project that it's time to build, is the "pull" model where projects need to have their "listener" turned on or off, and if they hear about a new upstream project, they can choose how to respond ("New upstream project found. Abort, rebuild, ignore?") ;-) This would allow, say, Query to opt out of the stack on a given week if its developers were all on vacation or hadn't had time to complete a fix and thus the code was in an unusable state in CVS. <shrug> Is that useful? The "pull" model is the RSS notification scheme that I'm currently piloting for UML2 N builds based on EMF N builds, and waiting for feedback from the other ISS teams as to viability/usefulness. The "push" model would be something else on top of this which could speed up the listening/waiting cron cycle so that builds could fire IMMEDIATELY after each other without the cron delay. Again, this only works if the projects are on the same server, but could chop hours out of the stack's elapsed time. This is my vision for M5 and beyond. The two parts I need leads to accept (and for ya'll to voice any concerns/thoughts/suggestions you might have) are: * automated build cascade (ideally, starting from any point in the stack downward, not just from EMF) * automated build promotion during cascade (after each build+tests are completed, promotion should start immediately and the next one in the stack should be notified that it can start. For local builds, this would be immediate; for remote builds, this would start a cron to check every n minutes for the specified build being available on the nearest fullmoon mirror). In the event that sufficient testing has NOT been done on the Wednesday the day before (people on vacation, other work items more urgent, power failure, etc.), what system should be in place as a workaround/stopgap? Should the cascade omit any builds for which there's no N available dated that Wednesday? Should it use the latest dependencies in the build.php page (eg., EMF 2.2.0M4 (2.2.0.S200512220106)) instead of whatever's most current on eclipse.org + latest in the stack (eg., EMF 2.2.0.I20060105)? Should downstream projects which are okay be able to rebuild even if something in between fails? eg., EMF rebuilds and is good, kicks UML2, UML2 fails, but UML2 kicks EMFT anyway, then OCL builds using latest EMF + previous known good UML2, etc. Or should the cascade stop at the first failed build in the stack (in the above scenario, UML2)? Any/all of these flows are possible. I just need direction from the projects leads (and everyone else who wants to chime in) as to what makes the most sense for all of you, and the assumptions that I therefore have to encode into this engine. How should build types be handled? Should the first build in the stack determine the build type for all subsequent builds? Eg., if EMF does an S build for its M5, should all the UML2 and EMFT builds autobuilt based on that build also be S drivers? How should the milestone number be determined? Same as the stack-starter? Or hardcoded via some config webpage or property file? For I & R builds, this is easy, but for S builds there's an extra piece of information which isn't placed anywhere (yet). -- -- -- Here's a quick summary of the big questions: 1. [IF build in stack {OCL 20060309} fails, continue downstream {Query 20060309}, using previous known good dependency instead of failed upstream build {OCL 20060302}? stop building stack?] 2. [IF testing {Validation 20060309} not performed night before, continue anyway {Transaction 20060309} using untested new build {Validation 20060309}? omit that build {Validation 20060309} from the stack as if it failed (above), and use previous {Validation 20060302}? stop building stack?] 3. [IF builds must determine their own success/fail, what checks must be done, if any, other than JUnit results and presence of required zips?] 4. [OK to automatically build N driver stack Wed at 12:00pm to 2:30pm?] 5. [OK to automatically build I driver stack Thu at 12:00am to 2:30am?] 6. [OK to have web UI to enable/disable automatically promoting YOUR good I build immediately after build?] 7. [OK to have web UI to automatically kick off (PUSH) downstream project builds (accelerated stack build)?] 8. [OK to have web UI to enable/disable listening (PULL) to upstream builds in stack for PUSH notifications (opt-in/opt-out of normal or accelerated stack build)?] 9. [OK to have web UI to enable/disable different types of PUSH notifications: Wed noon N builds, weekly midnight N builds, Thu midnight I builds, etc.] 10. [OK to have first build in stack (which could be EMF, UML2, OCL, etc.) determine build type (S = S, I = I, M = M, R = R)? What about Milestone numbering - same as first build?*] 11. [OK to have web UI where upcoming Thu build's details can be specified (rather than assumed) for all builds in the stack, ie., build type + milestone #s if type is S?] Thanks for reading all the way to the bottom! * Note that UML2 is on M2, but EMF is on M4. I'm not sure if the EMFT projects are planning to release an M1 or synch with Eclipse/EMF and go right to M5. Also, I doubt Kenn wants to skip from M2 to M5 without the intervening milestone releases. We can simplify this next time around, but for this release, we clearly have a disconnect which needs a solution.