Community
Participate
Working Groups
There's a few issues with the current build which makes it take too long, and makes it "fragil". I'm hoping this bugzilla entry can be used to discuss general ideas a approaches to improve it, and as specific actionable items are identified, then new bugs can be branched off to handle those specific items.
For example, one issue is that each project provides a .build file, with features, versions and a pointer to their update site. Even if correct at one point in time, it seems that fairly often that .build file and the update site it points to gets "out of sync" somehow. I know I've done this myself, sometimes, moving something from a "tempsite" to a "milestone" site, and forgetting to update the .build file. One possible thing to help might be to provide a "push" utility, where each project can "push" their content to some location on build.eclipse.org which is to be used only for this purpose, so less likely to accidentally get out of sync. Plus, the "push" would fail if the .build file and site were not quite right, so the project releng team would know right away if there was an error. Just an idea.
Another issue is that the build takes a long time, and must be perfectly correct, or else everything "fails". It would be better to be more incremental, so projects that "work" get to "staging" and projects which don't work can continue to fix their stuff, but in the mean time, the rest of the projects are not held up. This is especially important to get the "package projects" up to date, while the "leaf" projects are not on any critical path.
Another thing is that there's no way of knowing if a particular feature version is for a particular milestone. Having a signoff page on the wiki that you have to reconcile is not the best approach here. Perhaps it would be better for the build model to have milestone entries, with each contribution providing input for the milestone by reference. This would allow for improved status reporting (no need for the wiki page) and preserve each milestone config. Perhaps there could be an entry for the latest I-build, to allow for interim galileo builds?
(In reply to comment #3) > Perhaps it would be better for the build model to have milestone entries, with > each contribution providing input for the milestone by reference. +1 The practice of update sites as "moving targets" where the previous milestone gets overridden when the next one is produced is bad. The build model should be cumulative, listing all milestones. Can't see why we didn't think about this earlier... of course PDE Build and friends need to be equipped to produce cumulative update sites / p2 repos easily.
If everyone starts producing zips of their contributions (an archived p2 repo) instead of simply pointing at their unpacked update site / p2 repo, there'd be less danger of the unpacked site being out of synch, or being updated to no longer include some specified feature. And if those zips were properly versioned (2.5.0/S200905011234/emf-Update-2.5.0M7.zip) you'd know immediately from the URL what milestone it was, when it was done, and for which branch it was for. Digging into the metadata in the zip would then allow you to map between feature version and update site version, but this detail could still be placed in the .build file for convenience. For Modeling we generate that file from the information available during a build, then check it into CVS automatically so there's no danger of doing a build and forgetting to update the .build file. I believe PDE/p2 now supports the notion of building an update site from archived update sites, so you wouldn't need to do much more than use the p2 director to assemble the aggregate galileo site from the individual zips. As a first step, could the Galileo build be tweaked to understand what to do with: <repositories location="/home/data/httpd/download.eclipse.org/modeling/emf/emf/downloads/drops/2.5.0/S200905011234/emf-Update-2.5.0M7.zip" instead of <repositories location="http://download.eclipse.org/modeling/emf/updates/milestones/" ? (I still need to add the metadata to our generated update site zip.)
Here are some notes from Andrew and I looking at the Galileo build yesterday to try and determine if there is anything that can be done to make it faster. Note there are some comments which overlap with Nick's suggestions about using repos, etc but I am including the whole email here because... well... its easier. :-) Richard, please feel free to comment on anything which I have which is incorrect. ----------- How does the build work? - There is a galileo product which includes all the features (and specific versions) for all the components on the Galileo release train. - Components contribute a ".build" file which includes the id of their feature(s), the version(s) and the repository to get it from. - These files are run through a generator which builds a customTargets.xml file and a bunch of calls to the director. (one for each component) - The build runs a product build on the Galileo product file. It uses (currently) an Eclipse SDK 3.5 M6 as the baseBuilder. - Early in the build the postSetup target in the customTargets is called and it calls getBaseComponents. This extracts an SDK into "base" and then iterates over all the Galileo components and installs them one by one into "base" using the director. It does this to build up something to compile against. - Then later in the product build the builder looks at "base" and sees it has plugins and features and generates metadata for them and uses it in the build. We believe this step is repeated for each config. (which explains why it is 5x slower when building for 5 platforms) So where is the extra time going? - Some of the update sites that we are connecting to aren't p2 sites (point directly to a site.xml) so we are downloading JARs and creating metadata for them. I believe this will be corrected for the Galileo release as generating p2 metadata is a "must-have". - A new version of the profile is created each time the director is called. In my local build this resulted in over 2G (at last check... build is still running) of profile files in the profile registry. Note that the individual calls to the director don't seem to be taking too long (relatively speaking) since this also includes the time to go to the server and download the metadata and artifacts. In my case, about 95% of the calls to the director took less than 1 minute. But when you are downloading these things X times for X configurations, the time will add up. - The director installs things into the profile and puts the artifacts (features and plug-ins) in the features/ and plugins/ directories. Later on the builder sees these directories and runs the metadata generator on them and creates new metadata. It is believed that this is where the majority of the extra time is spent. (no, we don't have exact numbers) What can we do about it? Note that one added bonus about either of these options is that we will be using the p2 metadata that is produced by the component teams since we will not be regenerating it again locally. So if there are any special instructions they won't get lost like they do now. 1). Use the profile as a repository. This is the easiest thing to try and will result in the least number of changes to the current build process. Since we are installing into "base", the latest profile contains all the metadata for everything that is installed. So we will set the latest profile in "base" to be a "context repo" for PDE/Build so it won't regenerate the metadata for the things it finds in base/plugins and base/features. I have run a simple build (with only EMF) and it works. I am in the process of running a full Galileo build. The changes that I had to make in order for this to work are: - remove the run.director call from the build.xml. This is the call which installs the galileo product but PDE/Build already takes care of this. - add the following properties to the build.properties file p2.gathering=true transformedRepoLocation=file://${baseLocation}/p2/org.eclipse.equinox.p2.engine/profileRegistry/SDKProfile.profile/ p2.director.profile=SDKProfile 2). Mirror the repos locally. This would be the better solution as it would: - remove the multiple calls to the director and huge profile registry - get a copy of all repositories locally before installing which would be quicker for multiple installs as well as reduce the chances of errors happening when remote repos get modified while doing a build If the current calls to the director for each component were instead changed to call the mirror application using the slicing option, they (the metadata and artifacts) would be adding to a repo which would eventually contain everything. Then we would tell PDE/Build about this repository and it would convert it into "runnable" form if necessary (e.g. run the repo2runnable task on it to make sure it can compile against everything) and use it to build the product. One interesting thing about this approach is that if you kept that repository around across builds, then a rebuild would be MUCH quicker because the mirror action would only go and fetch things which had changed since the last build. Andrew put together a great example of a product build using this approach for EclipseCon. It can be found at: dev.eclipse.org /cvsroot/eclipse pde-build-home/examples/org.eclipse.pde.build.examples.rcp.cloud.releng
Could you shed some light on the need to create a "galileo" product?
Why not change the process so that: 1. Each project provides an update site that contains exactly the bits they want included in the build. No more, no less. Let this p2 repository define the build rather then the .build file. 2. Do the galileo build by just creating a composite repo of all project repos and then just mirror that. If all projects provide pack.gz files then there should be no need to post process the result.
(In reply to comment #6) > The changes that I had to make in order for this to work are: > - remove the run.director call from the build.xml. This is the call which > installs the galileo product but PDE/Build already takes care of this. > - add the following properties to the build.properties file > p2.gathering=true > transformedRepoLocation=file://${baseLocation}/p2/ > org.eclipse.equinox.p2.engine/profileRegistry/SDKProfile.profile/ > p2.director.profile=SDKProfile I tried this on my local build test machine, which does just one config, and takes a little over an hour and a half, and it saved about 9 minutes (with just one measurement) ... is this the kind of improvement you were expecting, or that you see? If so, that 8% is normally nothing to sneeze at, but we need something much bigger to make this realistic. Something that's an order of magnitude faster. At least on a per-day basis ... currently each build on build.eclipse.org takes 8 or 9 hours ... and if the tiniest thing fails, it's another 8/9 hours for next build. And by then, that's long enough for someone to have broken something else :) Perhaps someone will volunteer to try one of the mirroring type procedures? Remember ... we don't just need ideas ... we need "bakers to bake the bread". :) Maybe we should have a "build that train" contest :)
(In reply to comment #7) > Could you shed some light on the need to create a "galileo" product? > Rich can provide more light, but I know part of the reason was to make sure everything really was compatible (that is, installable). In principle, this makes things easier for the EPP project to build specific packages (as well as adopters and end-users). But, ... if that's the only reason, I think that could be re-conceived as a huge "junit test" and leave the "build step" more incremental, and allow it to "partially succeed". What approach would you suggest, Pascal? (That is, what are the alternatives?)
(In reply to comment #9) > (In reply to comment #6) > >... but we need something much > bigger to make this realistic. Something that's an order of magnitude faster. > At least on a per-day basis ... currently each build on build.eclipse.org takes > 8 or 9 hours ... > One approach that might also help is to first only build one "configuration", and if that succeeds, then do the rest. From doing my "local" builds with one config (1.5 hours) compared to build.eclipse.org builds with 5 configs (8 hours) it seems to be a pretty linear function. First, is that expected? I wouldn't think there'd be that much different from one platform to another ... just a few items). And second, I realize this might not be as "pure" since the way things are now, I assume, something might on an update site from one run to the next ... but, for an 80% speed up in routine builds, it might be worth it.
(In reply to comment #10) > Rich can provide more light, but I know part of the reason was to make sure > everything really was compatible (that is, installable). > I think this could be achieved by running the planner multiple times, one per configuration, on the complete mirror I suggested earlier.
To clarify, the planner would then plan the install of the galileo product but not actually perform it.
Wouldn't it be hella simpler if the update site creation was: 1. for (each project); do unpack project's input ziparchived p2 repo (includes jar.pack.gz, jars, and site.xml files) into some target directory [or, copy from project's update site if no zip provided] done 2. merge site.xml files into one large file so that categorized features would retain their categories (could use xslt, EMF, DOM, whatever). (This could also be manually done by the project leads so that only the features they want categorized are included. If updated manually in CVS, then the feature versions would have to be xslt'd/merged in at build time to ensure the correct versions. IIRC the Ganymede or Europa build did this.) 3. generate p2 metadata for whole site using generator app, target dir, and aggregate site.xml (from CVS + feature # update or dynamically created from input site.xmls) How is the .build file providing more information that couldn't simply be applied as a filter (/.*\.sdk.*/ only) when xslt aggregating the site.xml's into a single set of categorized features? And why does it take 8 hrs? Are you re-signing/re-packing everything, and if so, wtf? I like Thomas' idea of providing ONLY the bits that will be on the update site, but that's not practical IMHO, unless we use p2 to mirror a subset of our update site to another location for a single feature (eg., gmf.sdk) or set of features (emf*.sdk). Then we're essentially duplicating the disk usage and Galileo will triple it. Sure, space is no longer an issue on build.eclipse.org, but it becomes a management issue. It's one more step projects will be required to do, and until Athena can do it for everyone, it's one more thing that may get done wrong, be forgotten, etc. Automation = good, human processes = bad. :)
(In reply to comment #14) > merge site.xml files... > What site.xml files? We don't have them anymore. Regarding duplicating (or tripple) disc storage. How about this: Every project provide their bits on a P2 site and promise not to change those bits between milestones. Then make Galileo a composite repository of those update sites. Don't mirror it. The composite *is* the Galileo repo. Elegant, simple, space saving, and fast. A special repo for the Galileo categories need to be created of course. That too is then included in the composite. This repo must also contain the IU grouping that represents all features in Galileo. Verification is made by creating a provisioning plan to the install this feature, once for each configuration. It should all be automatic of course.
(In reply to comment #15) > What site.xml files? We don't have them anymore. How do you categorize your features w/o a site.xml? There is no other way atm, afaik. I've tried generating p2 meta w/o a site.xml, and everything ends up in the "uncategorized" bucket (which doesn't really exist anymore, despite the usability issues associated w/ that). https://bugs.eclipse.org/bugs/show_bug.cgi?id=269199 https://bugs.eclipse.org/bugs/show_bug.cgi?id=269226 > Regarding duplicating (or tripple) disc storage. How about this: > > Every project provide their bits on a P2 site and promise not to change those > bits between milestones. Then make Galileo a composite repository of those > update sites. Don't mirror it. The composite *is* the Galileo repo. Elegant, > simple, space saving, and fast. So you mean that Galileo simply points at the other sites? Is the notion of "composite p2 repos" final, or is that still being worked on in p2land? (I only heard rumours of this new option at EclipseCon, but saw no code for how to do so.) > A special repo for the Galileo categories need to be created of course. That > too is then included in the composite. This repo must also contain the IU > grouping that represents all features in Galileo. Verification is made by > creating a provisioning plan to the install this feature, once for each > configuration. How are the categories defined? > It should all be automatic of course. :)
(In reply to comment #16) > (In reply to comment #15) > > What site.xml files? We don't have them anymore. > > How do you categorize your features w/o a site.xml? There is no other way atm, > afaik. I've tried generating p2 meta w/o a site.xml, and everything ends up in > the "uncategorized" bucket (which doesn't really exist anymore, despite the > usability issues associated w/ that). > > https://bugs.eclipse.org/bugs/show_bug.cgi?id=269199 > https://bugs.eclipse.org/bugs/show_bug.cgi?id=269226 > Does it matter? The Galileo builder will do the categorization anyway and will not reuse the categories provided by the included sites, right? To answer your question, We use the p2 publisher to create our repo and Buckminster to orchestrate the whole thing. More reading here if you're interested: http://wiki.eclipse.org/Building_an_Update_Site_using_Buckminster > So you mean that Galileo simply points at the other sites? Is the notion of > "composite p2 repos" final, or is that still being worked on in p2land? > Yes, that's what I meant. Composite repos has been used for some time now. The P2 team uses it to make nightly- and integration builds available. No special code is needed. You just provide compositeContent.[xml|jar] and compositeArtifacts.[xml|jar] and it's immediately recognized. It is as final as anything else in p2-land I guess. We're passed M6. > How are the categories defined? > The definition of the Galileo categories would remain in the .build files. The scenario I have in mind would not require each project to change anything unless what they publish is more then what they want included in Galileo.
(In reply to comment #7) > Could you shed some light on the need to create a "galileo" product? Because I had created the model-driven product-based build script generator for Amalgam and volunteered to use the same approach for Galileo, which works, though with some obvious enhancement requests. I required product-based builds for Amalgam downloads, but there's definitely no need for product bundles in Galileo, just a repository. Given the changes in p2 that would facilitate creating the required repository, I'm all for ditching the current approach in favor of something simpler, quicker, and with less opportunity for error-prone human interaction (not to mention letting someone else deal with the inevitable headaches that are sure to follow).
I'm working on a prototype that acts according to what I proposed earlier. It uses the generated galileo.build file as input. If it performs well, and if you're interested, I can make it available as an ant-task that you can use in the Galileo builder.
(In reply to comment #19) > I'm working on a prototype that acts according to what I proposed earlier. It > uses the generated galileo.build file as input. From the discussions above, it sounds like we should be able to "assemble" the Galileo repo without requiring a build model, i.e. if each contributing project simply push up a repo that represents their current Galileo contribution. And, if bug 250974 is resolved we can eliminate the current workaround to provide categories using the generated site.xml file and metadata generator.
(In reply to comment #20) > (In reply to comment #19) > > I'm working on a prototype that acts according to what I proposed earlier. It > > uses the generated galileo.build file as input. > > From the discussions above, it sounds like we should be able to "assemble" the > Galileo repo without requiring a build model, i.e. if each contributing project > simply push up a repo that represents their current Galileo contribution. > Fair enough. That would be easy to do. We still need the list of contributions with contact information. > if bug 250974 is resolved we can eliminate the current workaround to provide > categories using the generated site.xml file and metadata generator. > That would require all projects to contribute a site.xml file. I think it would be much nicer if the categories were assembled directly from IU's residing in their p2 repositories. As I mentioned earlier, our project no longer provide a site.xml since it's not a requirement in Galileo. We do provide categorized P2 metadata nevertheless. We just skip the detour with the site.xml.
Regarding comment #16 There are p2 ant tasks to implement composite repos. We been using them since February See http://wiki.eclipse.org/Equinox/p2_Ant_Tasks http://relengofthenerds.blogspot.com/2009_02_01_archive.html
I would like to step back from the actual implementation details for a sec and just try to understand what it is the Galileo build is trying to produce and the criterions that need to be met. Richard, David?
(In reply to comment #23) A p2 repository from which consumers can obtain Galileo release train projects, categorized as the Planning Council dictates. That's it, unless things have changed since I was on the PC.
(In reply to comment #23) I think the goal is still as Richard stated, but I might clarify with some of the terms we used to use: It was to be a common discovery site. A discovery site that could be included in the lowest "platform" someone could install, thereby giving them as easy way to discover what else what available to install, rather than having to sort that out from the scores of Eclipse Projects and add a lot of individual update sites themselves. Secondarily, it's become a means to make EPP packages easier to build, though that wasn't it's original purpose, and, IMHO, is not a critical "must have" but more a "nice to have". Our goals for this common discovery site also relates to the need to have copies of the jars at the common site, or if it suffices to "point back" to the original sites. Originally, the goal was to "point back" to original site, but that turned out to be nearly impossible with update manager. In the course of doing this it was felt that some projects would frequently break their own update sites, so it was thought to be advantageous to make a copy, so at least that particular snapshot would continue to work from the common site. I'm sort of 60/40 on this sub-goal. I'm 60% in favor of making a smooth experience for end-users, but 40% of me says we should work through the pain of having projects be responsible for their own stability, and if they do not accomplish it, then we need a way that "the blame" is obviously theirs, not the central site, and it's their reputation that would suffer and/or perhaps be removed from the central site if there were repeated violations. Perhaps, as an alternative, some "unit tests" could be put in place that could be ran once a day or once a week. Like I said ... I'm 60/40 on this one and open to suggestions.
Thanks for the clear statements of the goals. Given that most projects are still new to repository management (how long should they stay up, how to avoid republication, etc), the approach where Galileo only refers to other sites (or composite repository) will not work. Also the categorization would not be the one from Galileo but from each site which is not necessarily what you want. IMHO for this release we should stick with the approach of Galileo being an uber repo containing all the jars since it will provide more control over what we make available and preserve us from badly managed repos. On the implementation side, we "simply" need to mirror each individual repository in our Galileo repo (without the categories), add the new categories, et voila. This assumes that everyone has p2 metadata to start with, which is fine since this is a must have anyway. The thing we don't want to do is generate metadata for other projects. After that we could run a sanity check tool over the repo (to be developped, but most the code is there) to verify that all the artifacts are here and that everything is individually installable. Thomas given that you are already familiar with the Galileo build system, and with p2, would you be able to implement this?
(In reply to comment #26) > Thomas given that you are already familiar with the Galileo build system, and > with p2, would you be able to implement this? > I already have. Right now I'm working on the verification phase. I.e running the planner to verify that everything can be installed without actually installing.
This is very cool. How fast is the new build without the verification? Is the verification validating that everything is installable altogether or individually? If you are looking for some code sample to validate each IU you may want to take a look at the RepoValidator class in the p2 tests plug-in.
(In reply to comment #28) > This is very cool. How fast is the new build without the verification? > Without mirroring, and running from home, which means that all meta-data is copied over the big pool in the first phase, I get the following numbers with the current Galileo maps: Starting generation of composite repository Done. Took 205578 ms Starting generation of categories Done. Took 315 ms Starting planner verification Done. Took 23915 ms So, all in all, below 4 minutes. The planner only runs for one configuration so the end result might be slightly more then that (24 secs per config or so). > Is the verification validating that everything is installable altogether or > individually? > I pass one huge group that contains all top-level features to the planner. > If you are looking for some code sample to validate each IU you may want to > take a look at the RepoValidator class in the p2 tests plug-in. > I don't find a class by that name. What package is it in?
The RepoValidator leaves in the package org.eclipse.equinox.p2.tests.full, however as I said it does thing one by one.
I now have the code to build the Galileo update site. I made some decisions in this implementation that I think needs to be discussed so I would very much appreciate some input before I continue. The build uses the generated galileo.build as input. I chose this because I feel that the included projects are familiar with this already. The build executes the following steps: 1. Create one composite meta-data and artifact repository consisting of all contributed repositories. 2. Create an extra meta-data repository that contains everything from the appointed target plaform, including a JRE definition. This repository is added to the composite. 3. Create an extra meta-data repository that contains all generated categories and a feature that includes all top-level features. This repository is added to the composite. 4. Create a provisioning plan for installing the generated 'includes-all' feature, once for each configuration, using the composite repository as input. This verifies that all of galileo can be installed on all platforms. 5. Given that everything above succeeded, perform the time and resource consuming task of creating a mirror of all contributed repositories. Filter out all categories. 6. Add the generated categories (but not the generated feature) to the mirror. The reason I do the mirroring last is that step 1-4 is all about meta-data. It completes within minutes and most errors should be trapped during that time. Questions that I have: 1. The current flow assumes that all sites include only the bits intended for Galileo. There is no way that I can determine the transitive scope needed to make exclusions so everything (except the categories) is mirrored from the provided repositories. Is this OK? 2. What is the best way to figure out where to send emails when things go wrong? I know that in the current build, emails are sent out based on failure to install a feature. That approach is not entirely correct since Galileo is hierarchical in the sense that some projects consume others. Unless the leafs of such a hierarchy are installed first, the blame might many times be incorrect due to implicit installs. I would like a way to discover the leaf dependency problem, and the traverse backwards until I find a feature included in a contribution. I'm sure this information can be obtained using the new explanation support in P2. Can someone from the P2 team please shed some light on this? 3. How do I interface with the current Galileo build? I think this calls for a discussion (chat or call, I'm happy to do both). Who should participate and when can we set this up?
I can easily modify the generator output for whatever you'd like, or you can use as set of modified templates that override/augment what's there now. Or, you can use your own scripts/generator entirely. Just let me know if you need anything from me, preferably with new bugzillas.
It seems to me that using the build model might be overkill, in this case. If each project is to supply a repo that will be included in the overall Galileo repo, there's no real need to specify each feature and version in a separate model, afaiu. That is, if the repo provided contains only those bits for the current milestone. A simple property file could provide the mapping for feature and responsible individual(s), and a site.xml maintained in the galileo.build project could provide the categorizations until bug 250974 is resolved.
(In reply to comment #31) > Questions that I have: > 1. The current flow assumes that all sites include only the bits intended for > Galileo. There is no way that I can determine the transitive scope needed to > make exclusions so everything (except the categories) is mirrored from the > provided repositories. Is this OK? From my perspective this works, since STP publishes a specific Galileo site that just contains items designated for the release. > 2. What is the best way to figure out where to send emails when things go > wrong? I almost feel that at this stage sending an email to cross-project would be acceptable, and we could work on a more detailed approach after the release... > 3. How do I interface with the current Galileo build? I think this calls for a > discussion (chat or call, I'm happy to do both). Who should participate and > when can we set this up? Certainly happy to participate. Is there a way that your current build approach could be reproduced by others, Thomas? That would help in the discussion...
(In reply to comment #33) > It seems to me that using the build model might be overkill, in this case. If > each project is to supply a repo that will be included in the overall Galileo > repo, there's no real need to specify each feature and version in a separate > model, afaiu. That is, if the repo provided contains only those bits for the > current milestone. > Unfortunately, this does not seem to be the case. The emf update site for instance, contains several versions of each feature so a lot of copying will be redundant. I'm testing a new approach that looks promising. During the planning phase, I collect all IU's to be installed. I then prune that list so that it does not include any of the IU's in the target platform repository. Finally, I only mirror those IU's and their respective artifacts. I'm not at all unhappy about using the build model b.t.w. It's easy enough to work with, having annotated Java interfaces and all. Emf rocks!
(In reply to comment #35) > (In reply to comment #33) >> That is, if the repo provided contains only those bits for the > > current milestone. > > > Unfortunately, this does not seem to be the case. The emf update site for > instance, contains several versions of each feature so a lot of copying will be > redundant. I think a lot of problems can be avoided if each project would provide repositories that contain ONLY the bits from the build that is being contributed. The mixing of frequent build versions in a single repo is always a problem, imho.
(In reply to comment #34) > Is there a way that your current build approach could be reproduced > by others, Thomas? That would help in the discussion... > I have plans to make that happen. Ideally, I think the builder should be published as an installable feature. Richard, can you please provide a list of what features/bundles that must be present? It should be possible to: 1. Generate the galileo.build form the contributions 2. Load the model to be used in Java. and preferably there should be no UI dependencies.
(In reply to comment #37) > Richard, can you please provide a list of what features/bundles that must be > present? The Amalgam DSL Toolkit includes everything you need. If you'd rather install it using p2 rather than downloading and extracting the package provided, use the URL of the download as the repo location, e.g. http://download.eclipse.org/modeling/amalgam/dsltk/downloads/drops/I20090320-0729/
(In reply to comment #36) > >> That is, if the repo provided contains only those bits for the > > > current milestone. > > Unfortunately, this does not seem to be the case. The emf update site for > > instance, contains several versions of each feature so a lot of copying will > I think a lot of problems can be avoided if each project would provide > repositories that contain ONLY the bits from the build that is being > contributed. The mixing of frequent build versions in a single repo is always > a problem, imho. See bug 271486 for sidebar discussion about how Modeling can fix this problem for our update sites. I need all the Modeling PMCs to buy in that it's a problem and agree to the fix before I can proceed.
(In reply to comment #38) > The Amalgam DSL Toolkit includes everything you need. > Yes, that's what I'm currently using. What I'm asking for now is the minimum configuration so that I can compile a headless feature that is installable into any platform with as little footprint as possible. > If you'd rather install it using p2 rather than downloading and extracting > the package provided, use the URL of the download > I find a lot of features there. "Amalgam DSLTK Examples" is listed twice. One really is examples, the other one contains the builder but it also contains editors and hence, have UI dependencies.
(In reply to comment #40) > (In reply to comment #38) > I find a lot of features there. "Amalgam DSLTK Examples" is listed twice. One > really is examples, the other one contains the builder but it also contains > editors and hence, have UI dependencies. Oops. I renamed the feature and am rebuilding now so it's more obvious which one is the builder ;) Does it really make that much of a difference to download some UI plug-ins in the builder? I doubt it's worthwhile for me to create a core builder feature, particularly as I've already indicated it's likely overkill to use the Amalgam builder for this use case anyway.
(In reply to comment #41) > Does it really make that much of a difference to download some UI plug-ins in > the builder? I doubt it's worthwhile for me to create a core builder feature, > particularly as I've already indicated it's likely overkill to use the Amalgam > builder for this use case anyway. > But I'm not asking you to create a feature :-). I merely want you to give me a hint on what the minimum requirements are so that save some time when I do it myself. I think it's worth while. I don't think we can roll back the current state of things. The build model is there, and it will last for the reminder of the Galileo life cycle. That is, unless we want to introduce yet another way of doing things this late in the game. I didn't think that was an option. Regarding overhead; surely a parenthesis in the case but yes, it's a significant overhead. A headless configuration can typically be around 20 - 30MB in size. A full Dsltk is almost 200MB. Depending on use-case and network speed that can matter a whole lot.
Obviously, you'll need EMF. The templates are Xpand, and require the GMF Xpand variant, which requires M2M QVTO, which requires MDT OCL. So, you're looking at most of what's in the DSL Toolkit, less their UI plug-ins. If the generator doesn't change much, and it's included in the DSL Toolkit, where's the overhead in having this on the build machine? It's not like it needs to be downloaded and installed on each build.
(In reply to comment #43) > If the generator doesn't change much, and it's included in the DSL Toolkit, > where's the overhead in having this on the build machine? It's not like it > needs to be downloaded and installed on each build. > Fully agree. If it was just the build machine, it wouldn't matter at all. My objective is to make it really easy to build locally and to make the last mirroring steps optional. That way, any project that so wish, can add the full Galileo build (short of mirroring) as a last verification step in their own builds.
In answer to comment #31 >I would like a way to discover the leaf dependency problem, and the traverse >backwards until I find a feature included in a contribution. I'm sure this >information can be obtained using the new explanation support in P2. Can >someone from the P2 team please shed some light on this? In case of failure, the mutistatus returned by the getProvisioningPlan should contain enough information since it goes all the way from the leaf to the root(s). See for example the test called ExplanationLargeConflict.
In reply to comment #35 that is testing another approach to only obtain the necessary bits out of repos, could you please explain how you are "pruning the list" of IUs? Depending on how this is implemented this could be put back into p2 as part of the p2.mirror / slicing operation. We can take that off-line. Also are we validating that all the IUs of the repo are installable together? If so then we may want to have some control over what is verified with what, as it may be possible that the runtimes of the RT project (e.g. Riena) may rely on things that could not necessarily be installed in an IDE context which would be fine since it is not there purpose. That said I don't think this is a must have for now.
(In reply to comment #46) > Also are we validating that all the IUs of the repo are installable together? > If so then we may want to have some control over what is verified with what, > ... I'd like to see this validation step(s) to be conceptualized as "unit tests" to follow the "mirror and build meta data" portion of the build. I could imagine we might want several "standard" ones (e.g. for known EPP packages) and may, in time, think of new sorts of install scenarios to test. For example, even if not installed in an IDE, surely runtime targets would want to be installed in a target with some other things. In general, though, the goal should be to allow for "partial success" so some core components or configurations can make progress, even if some "leaf" components are not installable (yet).
I just uploaded my first cut on the builder. Feel free to give it a try. More information here: http://wiki.eclipse.org/Buckminster_Galileo_Builder
Thanks Thomas. I gave it a quick try, which leads to following comment and question. I made it through the -verifyOnly path ok (worked ok with current .build files, and then I deliberately broke one (in local copy) to make sure it failed there, which is did). I noticed if I just tried to run again, without deleting previous run output, I'd get an error that said: Adding a profile with duplicate id: GalileoTest. (and then it'd immediately terminate). Seemed it ought to be smart enough to write over itself? Or is that a feature and just means "there's nothing new to do"? Then I removed the -verifyOnly, worked as before, but then got a "URI is not hierarchical" error ... last few lines on console output were: Done. Took 91119 ms Starting generation of categories URI is not hierarchical If I had to guess, I'd suspect I don't have the -targetPlatformRepository correct. Is that a URL to a zip file? Or, a true update site repo? Can you give an example?
(In reply to comment #49) > I noticed if I just tried to run again, without deleting previous run output, > I'd get an error that said: > > Adding a profile with duplicate id: GalileoTest. (and then it'd immediately > terminate). > Not good. I added a bugzilla for this, see bug #272263. The intended behavior is that unless -update is specified, all results of a previous run should be wiped out. > Then I removed the -verifyOnly, worked as before, but then got a "URI is not > hierarchical" error ... last few lines on console output were: > > Done. Took 91119 ms > Starting generation of categories > URI is not hierarchical > I was not able to reproduce this. Added bug #272266 to track. > Can you give an example? > I'll add some examples to the wiki page.
I just uploaded a new version of the builder where I believe Davids problems have been fixed. One thing I discovered was that the choice of -update was a bad one. Apparently the Eclipse launcher swallows it. I renamed it to -updateOnly. More info, such as new options, and how to get the source, can be found on the updated wiki page http://wiki.eclipse.org/Buckminster_Galileo_Builder
(In reply to comment #51) > I just uploaded a new version of the builder where I believe Davids problems > have been fixed. One thing I discovered was that the choice of -update was a > bad one. Apparently the Eclipse launcher swallows it. I renamed it to > -updateOnly. > > More info, such as new options, and how to get the source, can be found on the > updated wiki page http://wiki.eclipse.org/Buckminster_Galileo_Builder > After using a work-around for bug 272353 (i.e. removing the ep.build contents!?) the process seems to finish, but nothing is mirrored. Is that expected to work at this point? The end of the log is ... Adding child meta-data repository http://download.eclipse.org/rt/riena/1.1.0.M6/update/ Adding child artifact repository http://download.eclipse.org/rt/riena/1.1.0.M6/update/ Done. Took 42097 ms Starting generation of categories Done. Took 77222 ms Starting planner verification Verifying config: win32.win32.x86 Is that the expected output? (Oh, and that was removing the other configs ... apparently it verifies just one, the first in the list? Prior to this run it just did the mac cocoa config). Thanks again.
(In reply to comment #52) > After using a work-around for bug 272353 (i.e. removing the ep.build > contents!?) > the process seems to finish, but nothing is mirrored. Is that expected to work > at this point? > > The end of the log is > ... > Adding child meta-data repository > http://download.eclipse.org/rt/riena/1.1.0.M6/update/ > Adding child artifact repository > http://download.eclipse.org/rt/riena/1.1.0.M6/update/ > Done. Took 42097 ms > Starting generation of categories > Done. Took 77222 ms > Starting planner verification > Verifying config: win32.win32.x86 > > Is that the expected output? > No, it fails right there and sends out emails. You didn't get them? :-) Seriously, I forgot to log that fact. Currently it will *only* make an attempt to send emails and that will be ignored since I guess you run with emailing disabled. > (Oh, and that was removing the other configs ... apparently it verifies just > one, the first in the list? Prior to this run it just did the mac cocoa > config). > It stops after a failed verification on a config.
Forgot to mention; try running the build again and provide the option: -mockEmailTo david_williams@us.ibm.com but then again, perhaps it will fail. Currently I think it requires the host to run listen to SMTP on localhost port 25 (the old builder did this too). What's the best solution for this? Provide options to the command for setting SMTP host/port/credentials ?
(In reply to comment #54) > Forgot to mention; try running the build again and provide the option: > > -mockEmailTo david_williams@us.ibm.com > I tried, but just failed right away, tried -mockEmailCC too but no luck. Checked to make sure I was current (as you know from bug 272414 :) and do seem to be. > but then again, perhaps it will fail. Currently I think it requires the host to > run listen to SMTP on localhost port 25 (the old builder did this too). > > What's the best solution for this? Provide options to the command for setting > SMTP host/port/credentials ? > Honestly, I think on any machine where mail is desired to be sent from, it will have SMTP on localhost. In general, though, all messages should be logged ... not rely on email notification alone -- always nice to have a record. I think in this case I might have ran out of heap memory so trying again with 1G.
This was cool ... As an attempt to work around bug 272353, I used a target repo of /releases/galileo, just to see. The builder did make it through all the motions, and ... the cool part ... eventually said Found 2610 units to install. Now pruning using target platform 87 units remain after pruning And sure enough, mirrored those few things not yet put on /releases/galileo (e.g. riena, and a few others). Finished pretty quick, only requiring 87 bundles. I didn't see any features mirrored though ... just a /mirror/plugins directory was created ... we'll need those features to have an update site. (right?)
(In reply to comment #56) > This was cool ... As an attempt to work around bug 272353, I used a target repo > of /releases/galileo, just to see. > > The builder did make it through all the motions, and ... the cool part ... > eventually said > > Found 2610 units to install. Now pruning using target platform > 87 units remain after pruning > That suggests that the /releases/galileo repository contains everything, including a target platform. I didn't think that was the case. If it is, I need to remove the code that prevents the mirroring of the TP. > And sure enough, mirrored those few things not yet put on /releases/galileo > (e.g. riena, and a few others). > > Finished pretty quick, only requiring 87 bundles. > > I didn't see any features mirrored though ... just a /mirror/plugins directory > was created ... we'll need those features to have an update site. (right?) > Yes. When I run it, I get a full mirror with the following layout: artifacts.jar content.jar features/ <all features, packed and unpacked> plugins/ <all plugins, packed and unpacked> Anything less then that indicates a problem. Could this too be a heap-space limit that kicks in? You really should see an OutOfMemoryException somewhere if that's the case.
I have a question regarding the capability plugins. The current way of getting them into Galileo strikes me as a bit odd. 1. You (a contributing committer) create the plug-in. 2. You add it to someone elses feature, separate from the contribution 3. You provide the source for it in a map file, separate from the contribution I can make the new builder support this scheme, but before I do, I would like to understand why it's done this way. Am I correct in assuming that the objective is to provide one single "Galileo capability feature" that the user optionally installs? If so, I would like to propose a change so that: 1. You create a feature that includes your capability bundle(s) 2. You make the feature available in the contributed repository 3. You add the feature to the build contribution using the special category labeled 'org.eclipse.galileo' Behind the scene, the builder transforms the generated category into the 'org.eclipse.galileo' feature by extracting the bundles and discarding the features that provided them. This way, the Galileo builder can remain a pure assembler/verifier type of utility. It does not need to build things from source. The project maintainer can keep things in one place. No map files, and no foreign feature that needs to be updated.
(In reply to comment #57) > That suggests that the /releases/galileo repository contains everything, > including a target platform. I didn't think that was the case. If it is, I need > to remove the code that prevents the mirroring of the TP. > Or wait, now I find that the EP contribution indeed has a repository declared. That's fine by me. But if it has, the -targetPlatformRepo should really be left empty. Otherwise, there will be two target platforms to consider.
(In reply to comment #58) > I would like to propose a change so that: > > 1. You create a feature that includes your capability bundle(s) > 2. You make the feature available in the contributed repository > 3. You add the feature to the build contribution using the special category > labeled 'org.eclipse.galileo' > > Behind the scene, the builder transforms the generated category into the > 'org.eclipse.galileo' feature by extracting the bundles and discarding the > features that provided them. > I'd be in favor of this. There's currently several issues (e.g. signing, versioning, qualifiers) with current scheme that would have to be fixed, so either way takes work and your proposal makes conceptual sense to me. I would suggest adding "capabilities" to the end of the category and feature name(s). We should post this question wider to cross-project list, since there _might_ be someone not following this bug :) I'd like for you to, but if you'd prefer not, let me know and I will.
I'd like to count this general bug as 'fixed' and then open more specific ones for specific work that is still needed. Such as bug 272857 and bug 272853. The Buckminster based build is documented in http://wiki.eclipse.org/Buckminster_Galileo_Builder To summarize the outcome or results that indicate this is the way to go is as follows: 1. When ran on build.eclipse.org, the total build time is less than 30 minutes. (It's not a whole lot longer when ran remotely ... but maybe an hour or so). This is an order of magnitude faster than the 8 hours of the previous build. 2. Many problems, such as typos in feature names or versions, or update site URLs, will show up in about 5 minutes. (If a bundle does not exist, that won't show up till the "mirroring" step, but still within the 30 minutes). 3. While good and bad, the bucky builder can create a site that might still have an error. The bad part of course is that it has an error (even though it is logged, and email notices sent out). The good part is that if it is in a leaf component, it will be possible for others to use or test the "good" part of the site, pending the fix. 4. The old system only ended up with jar files, so a "pack" step would have been needed eventually, which would probably take about 4 additional hours. The new system actually mirrors the existing pack.gz files, if they exist (and they should!) so that's an additional savings. The bottom line is that this new system can allow about 10 iterations per day, when needed, where as the other could only do one per day. I'm sure there's advantages and disadvantages that others could list, but I think that rapid iteration is required for us to be able to deliver.
