Community
Participate
Working Groups
The simultaneous release is currently divided in several steps +0, +1, +2 ,+3 and EPP packages day. Unfortunately, this sometimes gives not enough time for high level projects to integrate and to publish their release. for example, the MDT Papyrus project relies on components that are produced on +3 day, like MoDisco project. Thus, for the release days, I can only wait for the end of the promoting action of Modisco project before I can start building and publishing my own repository. I am on a kind of +3.5 step, but I must finish before the EPP packages start to be build. This does not give me enough time to publish and test my own repository. Last indigo M4 (the first time we get into the release train), we were desactivated from the aggregation build because of mistakes on the update site, that I had no time to fix. Is this possible to reorganize the releases, so more +x days are proposed? I believe there is currently no tool that computes dependencies towards project, to have the depth of dependencies. This would give the real number of needed +x days, before EPP package builds. (This bug is coming from a talk on cross-project mailing list: http://dev.eclipse.org/mhonarc/lists/cross-project-issues-dev/msg05164.html)
I personnally think that we should not have more +x days but we should encourage people to move away from +3. For Example Riena (that I am the project lead of) is +3. But I can start building once +0 is finished. I have no dependency to +1 or +2 projects. But its comfortable to be +3, you have more time etc. Maybe the same is true for Modisco. They could also move to +2 or +1 and allow others to consume them more easily. So I believe we should stay away from +4, +5 and +6 etc.
Modisco is at least +3 ;-)
Created attachment 185537 [details] Project dependencies Note that I can not guarantee the correctness of all dependencies shown in this diagram!
Created attachment 185542 [details] Project dependencies Better diagram
Created attachment 185544 [details] Project dependencies All arrows pointing downwards now. Seems we'd need +0 til +12 :P
I blogged about this very issue last year when I had troubles producing a +3 component when other components were late. http://ahuntereclipse.blogspot.com/2010/05/helios-rc1-1-2-3-go.html . Modeling in general does not fit (and never has) into the +0, +1, +2 ,+3. We have just done what we have needed to before the deadline.
(In reply to comment #5) > Created an attachment (id=185544) [details] > Project dependencies > > All arrows pointing downwards now. Seems we'd need +0 til +12 :P And in this diagram, is missing our Papyrus contribution, which is again on top of Modisco and several other projects: Xtext, OCL, GMF ;) Anthony, according to your blog, you were often late using this +x strategy. What was people reaction of GMF contributions at this time?
(In reply to comment #7) > Anthony, according to your blog, you were often late using this +x strategy. > What was people reaction of GMF contributions at this time? I have tried to make GMF Runtime run as a 2+ component the best I can. That is the top of my mountain :-). GMF Tooling was not in Indigo M4 for various reasons (it could be now, GMF Tooling 2.4 M4 was done over the weekend).
(In reply to comment #5) > Created an attachment (id=185544) [details] > Project dependencies > > All arrows pointing downwards now. Seems we'd need +0 til +12 :P Cool diagram! You are missing Platform and Equinox, I assume this is because almost everyone needs them? However even at that level we have interesting dependencies. Platform requires Equinox, Equinox requires ECF, and soon Platform requires EMF base. I agree with the general point that we cannot actually have a full day for every tier in the dependency chain, or we will have +n growing large. In the end each project has to just do what it takes to ship on time. Here are some ideas to reduce the pain though: - If projects contributed a "test candidate" several days early, it gives downstream projects something to build and test against earlier. The delta between the candidate and the final contribution will be small, minimizing the chance of impact. - Projects should be encouraged to declare as soon as they are ready and not wait until the deadline. For Platform and Equinox we have traditionally waited until end of day to declare our milestone, just in case something came up at the last second. However this rarely happens, and we could likely declare much earlier in the day to give downstream projects a head start. Now that parts of EMF have moved to +0 we will be making a conscious effort to do this. - Perhaps leaf projects that are *not* part of EPP could contribute on +4 day (same day EPP packages are being built)? This is definitely a known problem and we have had several discussions about this in the planning council in the past. The +0/+1/+2 is certainly a huge simplification of the actual project dependencies. The diagram is a cool way to demonstrate what we are up against here, and gives each project a way to see exactly what is upstream so they know if they will be impacted when a given project is late.
Eike, how did you produce the diagram? By hand? I would be interested to complete it with Papyrus. For the notion of leaf project not being integrated in EPP packages and being build on +4 day, this would mean for them that they would not be accessible as easily as the others: in fact, their repository would not be accessed from the indigo composite repository. Or perhaps am i missing something? (In reply to comment #9) > (In reply to comment #5) > > Created an attachment (id=185544) [details] [details] > > Project dependencies > > > > All arrows pointing downwards now. Seems we'd need +0 til +12 :P > > Cool diagram! You are missing Platform and Equinox, I assume this is because > almost everyone needs them? However even at that level we have interesting > dependencies. Platform requires Equinox, Equinox requires ECF, and soon > Platform requires EMF base. > > I agree with the general point that we cannot actually have a full day for > every tier in the dependency chain, or we will have +n growing large. In the > end each project has to just do what it takes to ship on time. Here are some > ideas to reduce the pain though: > > - If projects contributed a "test candidate" several days early, it gives > downstream projects something to build and test against earlier. The delta > between the candidate and the final contribution will be small, minimizing the > chance of impact. > - Projects should be encouraged to declare as soon as they are ready and not > wait until the deadline. For Platform and Equinox we have traditionally waited > until end of day to declare our milestone, just in case something came up at > the last second. However this rarely happens, and we could likely declare much > earlier in the day to give downstream projects a head start. Now that parts of > EMF have moved to +0 we will be making a conscious effort to do this. > - Perhaps leaf projects that are *not* part of EPP could contribute on +4 day > (same day EPP packages are being built)? > > This is definitely a known problem and we have had several discussions about > this in the planning council in the past. The +0/+1/+2 is certainly a huge > simplification of the actual project dependencies. The diagram is a cool way to > demonstrate what we are up against here, and gives each project a way to see > exactly what is upstream so they know if they will be impacted when a given > project is late.
(In reply to comment #10) > For the notion of leaf project not being integrated in EPP packages and being > build on +4 day, this would mean for them that they would not be accessible as > easily as the others: in fact, their repository would not be accessed from the > indigo composite repository. Or perhaps am i missing something? No, this is probably me not understanding the sequence of events. Is the composite repository locked prior to building the EPP packages? I was (probably naively) thinking projects could continue contributing to the common repository while EPP was being built. Maybe this is not safe or practical.
(In reply to comment #9) > Cool diagram! You are missing Platform and Equinox, I assume this is because > almost everyone needs them? Yes, the algorithm was: 1) Parse Indigo's content.xml for all "<unit-id> requires <unit-id>" 2) Narrow the unit-ids to projects according to a mapping list that was originally created from the feature ids found in our .b3aggrcon files and then manually adjusted (error-prone!) 3) Everything not in that list (incl. Platform, Equinox and Orbit) is ecluded from the graph > However even at that level we have interesting > dependencies. Platform requires Equinox, Equinox requires ECF, and soon > Platform requires EMF base. I have 2 Java files that create the .dot (Graphviz) content and they could easily be changed to produce other output. Would you like to see them?
(In reply to comment #10) > Eike, how did you produce the diagram? By hand? See my comment #12 > I would be interested to complete it with Papyrus. I'll attach the code in a minute. There is some JavaDoc but feel free to ask about details ;-)
Created attachment 185591 [details] Java Project for the graph creation
> > No, this is probably me not understanding the sequence of events. Is the > composite repository locked prior to building the EPP packages? I was (probably > naively) thinking projects could continue contributing to the common repository > while EPP was being built. Maybe this is not safe or practical. Correct. The common repository is locked prior to building EPP packages. That is only good releng practice. We have, on occasion, for special circumstances, done a rebuild of the common repository and not EPP packages, but I'd hate to "build in" such variability and lose that tiny 1 day buffer. As for the general problem, I can make several points, some of which have already been said, but I'll repeat (to save me from having to read too closely :) The +n categories are not meant to be the trigger for a consuming project's final build. Maybe they have to be in some circumstances, but in general, for most of us, they are supposed to be merely the last day the projects in that category can provide their final bits. "final" not "trigger" is kind of a subtle distinction, but if everyone was well behaved ... consumers used API, and producers know enough to not make any big changes the last week or two ... then most projects can build their own final bits against the near-final bits of their pre-req'd projects. And, yes, those projects may still want to do re-builds and final-final testing based on the final-final builds ... but normally would not require bits to be re-contributed. I know there are some special cases where "runtime targets" must specify some exact versions of pre-reqs. I'll admit I don't completely understand that technical issue ... but that's the only case I know if where project really truly depend on the final-final bits of some pre-reqs. RAP is the one case I know about. Are there others? Was this bug opened to accommodate another of those runtime-target cases? To pull off our general aggregation, it does depend, usually, on people building against their project's pre-reqs download site deliveries ... not building against the common repository itself. Is that the source of the problems here? Where there are tight or complicated dependencies between projects, there often has to be communication or agreements between projects for specific pieces to stabilize or be provided "early" or on a different schedule than the +n categories ... but (so far) that's always just been worked out project-to-project. Sometimes, this even has led to requests, as an example, of one project providing their final builds earlier in the day, or similar. It is unlikely that merely adding more +n days would solve all dependency problems, even if we could add enough. There are some (nearly) circular dependencies in some of the projects that depend on using "near final" bits (for example, parts of WTP depend on DTP and parts of DTP depend on WTP). True, we could do more work to make everything perfectly linear dependencies, but seems little reason to. Another key aspect of success is to be able to automate and produce repos reliably, quickly, and without error. As a Remi said in original comment, "Last indigo M4 (the first time we get into the release train), we were deactivated from the aggregation build because of mistakes on the update site, that I had no time to fix." ... hopefully this was simply a "first time" growing pain, and can be improved in future. If there's any issue or common problems producing those repos for aggregation, I'm sure we'd all be glad to help solve or explain those. Also, keep in mind, the degree of change and the smoothness of our automation will improve milestone to milestone, so even if something is done wrong for an early milestone, the main goal is the final release and each milestone should get us closer and closer to that, in a gradual way. (part of the reason to join early). So, given all that ... from what I can tell, we do not need more +n categories so far ... but, given my long winded explanation, perhaps others can now better explain to me what I'm missing, if anything, or what other options we might have to improving the aggregation process? Thanks,
(In reply to comment #15) > So, given all that ... from what I can tell, we do not need more +n categories > so far ... but, given my long winded explanation, perhaps others can now better > explain to me what I'm missing, if anything, or what other options we might > have to improving the aggregation process? > Since this discussion originated with my proposal on the cross-project mailing list, I'll try to give my view of what's missing. Today, all participating projects must define their own source of input for their builds. All projects will thus set up some kind of maps or similar to retrieve data from Orbit and other suitable sites from the projects that they consume. There are basically no rules around this. It causes a lot of grief where different versions of Orbit bundles is just a small part of it. As one of the participants in Helios, I know that finding the "correct" sites for to use as input can be both time consuming and error prone. Some projects can be volatile and do large changes at the last minute. I think that's a good thing. Indigo permits incubation projects and an agile but well defined process is far better than a stringent one that impedes development. At least for the first four milestones. If we had the build chain clearly defined with intermediate repositories between all +n steps then that chain, or selected parts of it, could be built with some determined frequency. The "final" Indigo milestone build would in essence just be a matter of promoting a successful chain build by cleaning it up using the b3 aggregator. The advantage would be that all projects would know exactly what to consume and from where. It would perhaps require up to 12 build steps, but hey, if that's what the dependency chain stipulates, then that's actually what it takes to provide an environment where all participants can rely on their respective input. With 12 steps, and if we want to give each participating project 2 days to get stuff right, then +0 must be built 24 days before +12. Why is that a problem if it's a well known and planned fact?
> > With 12 steps, and if we want to give each participating project 2 days to get > stuff right, then +0 must be built 24 days before +12. Why is that a problem if > it's a well known and planned fact? And don't forget to exclude weekends! :) The main reason is that it shortens the "feedback" time. If there were errors or bugs in the components near +0, they'd often be reported too late to fix for the next milestone, so then those that depended on that bug being fixed might be delayed even longer in implementing their fixes or functions. I am sympathetic to the general problem but I think using the common repository as a build target would bring a whole new set of unanticipated problems, and (so far) has not been the purpose of that repository. Its purpose is to provide an easy place for Eclipse users to find new things to install. In Planning Council meetings, in past years, when this topic has come up, no projects represented were willing to "sign up" for this "change in purpose" ... so, I'd say the first step in making this sort of change would be to get an advocate on the planning council to convince the rest of the council it should be something that each project would supported. But, I do not think we could ever literally chain each and every dependency in a strict order ... we'd be better off just building everything from scratch, IMHO. That's not to say grass-roots efforts could not improve things. There's bug 331385 that's been open recently, for example. Perhaps improve some wiki pages or tools? And, I don't mean to poo-poo any of these ideas or concerns. These are some difficult issues and am glad to hear everyone's ideas on how to improve things.
Hello, I think that Papyrus only uses the facet and customization features of MoDisco. Those features are being migrated to EMF Facet and EMF Facet has joined the release train. I think that Papyrus should start to use EMF Facet in place of MoDisco. This should resolve the problem for the Papyrus case, but not for the general case. Regards, Gregoire Dupe
One more information: EMF Facet is built on +2. Regards, Gregoire Dupe
(In reply to comment #19) > One more information: EMF Facet is built on +2. > > Regards, > Gregoire Dupe We haved moved Papyrus to EMF Facet, this should effectively make the release process easier. Thanks for your precision, Gregoire.
I would like to close this as "won't fix", since there's been no movement in changing our funny, and short +x system for Juno. I don't mean to say there are no issues, or that we couldn't improve, but I don't see as making the "build window" larger would help ... that is, without creating problems of its own. If I've missed something, or anyone disagrees, please re-open and/or work with your planning council representative to have a concrete proposal brought up. Thanks for opening ... and thanks for all the discussion. Important issues, and again, I'd glad to consider improvements in how we do things.
