Bugzilla – Full Text Bug Listing
| Summary: | Galileo delivery | ||
|---|---|---|---|
| Product: | Community | Reporter: | Richard Gronback <richard.gronback> |
| Component: | Cross-Project | Assignee: | Cross-Project issues <cross-project.inbox> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | normal | ||
| Priority: | P3 | CC: | ahunter.eclipse, aniefer, contact, david_williams, denis.roy, dsciamma, Ed.Merks, give.a.damus, greg, kim.moir, mober.at+eclipse, pascal, pwebster, slewis, tjwatson, wgp010 |
| Version: | unspecified | ||
| Target Milestone: | --- | ||
| Hardware: | All | ||
| OS: | All | ||
| Whiteboard: | |||
|
Description
Richard Gronback
I like the idea of a single integrated build (espacially assuming that this give potential for running unittests on the integrated build). It looks like this might take quite a bit of burden off individual project build enginners. There's a few points which I do not understand: * Will this scale? How long do you anticipate a Galileo build to run? Especially with signing + packing turned on? * How are projects going to specify what tag/milestone of their project is going to be contributed to Galileo? - Having a single file/model for such specification might become a bottleneck at delivery time (multiple changes required at the same time). * Will the file to be edited by projects be a human readable text file, or will modeling tooling be required to edit that file? There's probably going to be more questions as this idea evolves... (In reply to comment #1) > * Will this scale? How long do you anticipate a Galileo build to run? > Especially > with signing + packing turned on? The only thing that's "built" are the galileo feature and branding plugin. The rest are packaging steps that simply install the included features using p2, which is similar to the old update manager installation. Each project will continue to sign/pack their own jars, and since the galileo repo will redirect to each site's repository, I don't think there's a need to sign/pack at the galileo level. I could be wrong. > * How are projects going to specify what tag/milestone of their project is > going to be contributed to Galileo? - Having a single file/model for such > specification might become a bottleneck at delivery time (multiple changes > required at the same time). It's just like in Ganymede... each project specifies their included feature version numbers, which are specified precisely in the p2.director install. I don't think it's much of a bottleneck to have projects editing a single .product file, but we'll see. > * Will the file to be edited by projects be a human readable text file, or > will modeling tooling be required to edit that file? The .product file can be opened with its own editor in PDE. The .build model is just an XML file, not unlike what we edited in .sc files for Ganymede. Alternatively, it can be opened with the EMF reflective editor, which also provides validation for the model after editing, unlike the .sc file approach which had no editor. I'm sure we could get fancy and provide even a diagram editor for the file using GMF ;) > There's probably going to be more questions as this idea evolves... No doubt. Thanks for these. I think this is a good idea. This would be useful for assembling all the Galileo bits in one place. However, as you point out, I don't know that many consumers would be interested in consuming this giant all-in-one Galileo download. It would be interesting to take this one step further and have the EPP packages define products that would be useful to consumers. See bug 191378 and bug 239611. One item to think about might be error reporting. It seems like Bjorn & Team put quite some effort into the "auto blame-parsing system" for Ganymatic, and still there have been cases where the builds were stalled due to some missing or inconsistent bits on some project's site -- leading to lots of manual investigation, disabling etc (Thanks Bjorn and David!). I don't know what P2 can offer in that respect. But an error reporting facility that is able to identify what's going wrong and inform the right parties seems to be very important. The direction that you have taken to create the downloads for the Amalgam project is very interesting and is exactly in alignment with what we wanted to see happening with teams creating their own p2-enabled downloads. In fact this is what I was hoping the EPP packager would provide. This needs to be pushed further and maybe some of it could be folded back into PDE Build. > 1. A p2 content.xml and artifacts.xml repository files are generated during the process, which can be published as a common repository for all of Galileo. Source repository references are included, making the copying of each project jar files unnecessary (requires confirmation... Anyone?). This would likely be possible, however this will make the mirroring (in the foundation sense or mirror) much harder since several locations would have to be mirrored. > 2 [..] I haven’t yet looked into how to categorize the content, so pointers would be appreciated. Currently this can be done by writing a site.xml and generating metadata over it. Since this is not very convenient, I have opened bug #250987. Now as for creating an all-in-one package for Galileo, I don't think this brings much value to the community because users typically have a center of interest (modeling, jee, etc.) and probably prefer having something "tailored" rather than the kitchen sink. Also this package would probably expose some weaknesses in our integration and a few badly behaving pieces could turn the whole product in a disaster, where we have enough of that to handle in each package. One thing that could really ease the consumption of Galileo is to properly categorize the content to break down the silos we created around projects. Also defining a set of well-known categories could even help non eclipse foundation hosted plug-ins to be presented nicely. What other means do you want to deliver Galileo with? I do think that there is interest in an all-in-one download, especially for companies doing their own private mirroring/install/configure story. See some comments near bug 224729 comment 87 for instance. I proposed an archived update site at that time, integrated with an installer, with all contents packed (pack200) in order to reduce size. Given that distributed repositories make it even harder for adopters to get all-of-galileo, I think that there is a need for something that allows adopters to get everything -- either with a giant ZIP download or with a p2 downloading application that sucks everything from the distributed repositories. Another important point is that companies who create products on top of Eclipse need a reproducable build system with a clear bill of materials. ZIP archives with clear names and download URLs provide that, while our update sites have been kind of a "moving target" at least up to now. Arguably, such companies could also download ZIPs from individual participating projects (and will likely do so), but an all-in-one galileo package might also help. (In reply to comment #5) > The direction that you have taken to create the downloads for the Amalgam > project is very interesting and is exactly in alignment with what we wanted to > see happening with teams creating their own p2-enabled downloads. In fact this > is what I was hoping the EPP packager would provide. This needs to be pushed > further and maybe some of it could be folded back into PDE Build. Agreed. As I recall from conversations in the past, Markus wasn't familiar with product-based builds when EPP began, but I agree it should be the basis of each package. Source repository references are included, making the copying of each project jar files unnecessary (requires confirmation... Anyone?). > This would likely be possible, however this will make the mirroring (in the > foundation sense or mirror) much harder since several locations would have to > be mirrored. Aren't they already being mirrored? > > 2 [..] I haven’t yet looked into how to categorize the content, so pointers would be appreciated. > Currently this can be done by writing a site.xml and generating metadata over > it. Since this is not very convenient, I have opened bug #250987. OK, I'll take a look. Are you sure it's bug 250987? > What other means do you want to deliver Galileo with? A continuing topic for the next PC meeting. For sure a common repository and EPP packages, possibly with the all-in-one, though I share your concerns about it. And then there's the custom installer option, though I've not looked into it yet (as a non-friend of Eclipse ;). I can see the value for member companies to get the whole content of a release in one shot esp for the scenarios we are talking about. Now, when the term "galileo package" was mentioned, I interpreted "package" in the EPP way of things, whereas what we are really after is a way to get a complete copy of what is on the final galileo repository either by having a big zip or a p2 tool. Note that such a p2 tool exists under the form of our mirroring applications (plz use the most recent version of it). > while our update sites have been kind of a "moving target" at least up to now. You are pointing here at a real issue. Repositories need to be treated more carefully (for example I think it is unfortunate that the 3.4.0 content got removed from the Ganymede repo) and we need to come up with a set of guidances for the community. Since 3.5, the platform has started producing several sites with a clear expectations on their content (http://wiki.eclipse.org/Eclipse_Project_Update_Sites). The bug number related to improvements around categorization is #250974. From these discussions here is a quick recap of how I could see things fitting together. * From the provider angle - Projects provide p2 repositories (maybe several) - For teams producing only plug-ins and features this is already integrated into PDE Build. - Projects contribute to galileo by specifying the repositories in which their content is published (probably similar to what the ganymatic was doing). This is done by the galileo-matic. - The galileo-matic either aggregates all these sites in one site by reference, or actually get all the jars and all the metadata. This is done by the p2 mirroring apps. - Package maintainers author a .product file (maybe with some extensions) and publish it to a repo. This repo should also be mirrored to the galileo repo. This is done by some extended version of what is PDE Build around product build. - Galileo repo owner validates the repo. p2 has some level of that but it is not fully functional. - Package maintainers build the packages from the repo. This is done by the p2 director app. * From the user angle - Users can get an EPP package or the SDK by pointing at a single IU in the galileo repo using p2 simple installer. - Users can dl pre-built EPP packages, probably still a method of choice given the popularity - Users can use the installer wizard from EPP. Whatever the mean by which they got things on their machine, if two users installed the same product IU, they will get the same thing. * From the extender point of view - Download the galileo repo on build machine. This could be done using p2 mirror app. - I'm myself a "provider" (I'm authoring plug-ins for my company) and I build my repo. This way I can use the same mechanism than eclipse to produce my product. - I can rely on the galileo update sites such that I can even leverage the various mirrors to help with the load of distributing my product. (In reply to comment #9) > (for example I think it is unfortunate that the 3.4.0 content got > removed from the Ganymede repo) Me to. As I was the last person to see the 3.4.0 content, I'll relate some of the background in case it helps identify weak points in our practice and procedures. About 10 PM the night before we were due to announce and make the SR1 available, I noticed no one had moved the 'staging' code to 'releases'. I sent Bjorn a note, and got back an "out of office" message. So, there I was ... code was suppose to have been mirroring all day ... I did the best I could on short notice. I think to have a merged repository would take some work, and some testing, well ahead of time. And, it seems that no one actually had that job or responsibility. BTW, I still have a copy of the content if someone wants to volunteer. :) But, I'm not sure how it could be done ... what should the merged site.xml file would look like? Separate categories for each release or just rely on "show latests version only". I've tried to do the later (in WTP) but found it incomprehensible to users, since we have so many features, and such disjointed and hard to read version numbers. Seems that's an a use-case to plan ahead for with Galileo. The tooling necessary for mirroring of repo has been reviewed / improved in 3.5 M3 to prepare for this scenario. When 3.5 M3 comes out we can give a try at merging the Ganymede repo as a real test case. Btw, the platform team also encountered the same pb that you had but a few hours earlier :-) I think an "everything in Ganymede" is also interesting from the point of view of testing the integration of all these technologies. Problems like the popup menu on a project being "littered with noise" will be much more evident. It might help more projects realize how their contributions will look relative to hundreds of other contributions. I hope there will be some focus on ensuring that the sum total of everything isn't a mess, because we can't ever be sure which two things will be used together... I think this bug has lived it's life. See also bug 270058 and bug 272857 for related discussion. |