Bugzilla – Full Text Bug Listing
| Summary: | Recommend / enforce p2 repository structure for projects | ||
|---|---|---|---|
| Product: | Community | Reporter: | Pascal Rapicault <pascal> |
| Component: | Architecture Council | Assignee: | eclipse.org-architecture-council |
| Status: | CLOSED MOVED | QA Contact: | |
| Severity: | normal | ||
| Priority: | P3 | CC: | Achim.Loerke, andrea.ross, bbrodt, cvgaviao, david_williams, eclipse, Ed.Merks, gunnar, igor, irbull, jacek.pospychala, jeffmcaffer, kim.moir, loskutov, matthias.sohn, mikael.barbero, milesparker, mistria, mn, mober.at+eclipse, mrrussell, pwebster, sbouchet, sewe, steffen.pingel, stepper, vincent.zurczak, wayne.beaton, wim.jongman |
| Version: | unspecified | ||
| Target Milestone: | --- | ||
| Hardware: | PC | ||
| OS: | Mac OS X - Carbon (unsup.) | ||
| Whiteboard: | stalebug | ||
|
Description
Pascal Rapicault
I think it is a good idea. This has been discussed some in bug 291637. I doubt we could come up with anything to "enforce" since some would want to stick to whatever they have established, but a guideline would be great ... I'm sure new projects, or those looking to improve or change what they have would like something to go by. Perhaps, too, some better way to document the repository URLs, in some central site, such as in Eclipse Foundation metadata, would be most helpful to consumers? Or, maybe start off with a simple wiki page, to more easily see what patterns are in use, and then, eventually, see what would make sense to codify in Foundation metadata? +1 I agree that this is a good idea. We have a wiki page for our p2 repos, it would be a good idea to aggregate them for other teams. http://wiki.eclipse.org/Eclipse_Project_Update_Sites Great idea. There is an "updatesiteurl" field in the foundation project metadata. it only supports one URL and there is no space for description etc. but it might be a starting point. Perhaps we standardize on this being a pointer to a project maintained page that list repo using some common terminology/layout. If we need to make additional entries in the portal, we can do that. +1 to establish a set of best practices. Enforcement is hard. We can probably script something to identify problems and I've had pretty good success helping projects understand that conforming to a standard format is in their selfish best interest :-) +1. Having a "template" (a wiki page or something) that shows a recommended repository structure would definitely be helpful. Current proposal: - Having the information stored in the project metadata - Provide a web page where all repos at eclipse are listed (repos.eclipse.org). This page would be generated automatically from the project metadata. Things to discuss: - What is the recommended layout for the repos? Could ppl mention what they are currently doing? - Should the creation of a project automatically create the folders on download.eclipse.org following the recommended structure +1 for having a common template for update sites. Currently, for EEF project, i am using this structure : updates/ |-interim | |- 0.8 | \- 0.9 |-milestones | |- 0.8 | \- 0.9 | |- S201011090521 | \- S201012140442 |-nightly | |- 0.8 | \- 0.9 \-releases |- 0.8 \- 0.9 NOTE : 0.8 is for helios and is a composite p2 0.9 is for indigo and is a composite p2 I am also using this structure with different version for Acceleo, ATL and EMF Compare. So i can have a clear separate update site for each release train, for each kind of builds , and using composite repo, always present the last update site to consumers. Thx for the feedback Stephane, what do other projects do? I was wondering if we needed to have the name of the release train showing up somewhere in the path to help the user indentify things. Maintaining an Eclipse project that depends on quite a bit other Eclipse projects is a mess. Basically every projects currently comes with its own structure. Some projects even have a hidden repo for contributing to Indigo. Frankly, I think a template is a good start but won't help because people may also need some process help. I could imagine that there needs to be tooling which is able to deploy and remove builds from such a template structure. This becomes even more important when dependencies are maintained using a target platform definition. If repos go away a build will never be reproducible. I would prefer a simple structure. It seems that at least a separation of stable vs. release builds is common. I don't think the template should cover nightly builds. Sometimes, however, it might be desirable to offer maintenance as well as integration builds. There should be a clear pattern that could be used by mirrors to determine non-durable content that should not be mirrored. I'm wondering if the following structure is sufficient: /project/major.minor/ - composite for releases as well as stable builds /project/major.minor/testing - composite for I or M builds - s/testing/<whatever-expresses-nondurable> Those are nice and easy to remember urls. Maybe the key really is tooling which helps projects with publishing and archiving builds from a common repo structure. The CDO project lists its repositories here: http://www.eclipse.org/cdo/downloads/updates.php Eventually I'll add a script that lists the single builds/contributions. The disk layout is: updates/3.0/3.0-SR1 updates/3.0/3.0-GA updates/3.0-releases (composite repo) updates/4.0/4.0-M3-S201011081102 updates/4.0/4.0-M4-S201012140905 updates/4.0/4.0-I201012030411 updates/4.0-milestones (composite repo) updates/4.0-integration (composite repo) I'd be happy to follow a common strategy. *** Bug 377111 has been marked as a duplicate of this bug. *** Hi, +1 for a template. We started a similar discussion some weeks ago for the BPEL Designer. The version approach is good. Then, it should be up to every project to define the limits of a specific version. As an example, the next release of the BPEL Designer must work with Juno, Indigo and Helios. In some projects, a given version may only work with a specific Eclipse version. Stephane Bouchet's proposal looks good. But I don't know if there is an interest in having a "nightly" and an "interim". What is the difference? Could we say it is just a "snapshot"? There should be some rules to follow, beyong the template, for archives.eclipse.org and cleaning "milestones" directories after a release. Otherwise, Eclipse servers will need additional space. So +1 with Gunnar too: a template, some rules, and some tools to follow these rules. (In reply to comment #12) > The version approach is good. Copying my comments re: versions from bug 37111 comment 6: (In reply to bug 37111 comment 3) > I personally like URL containing explicitly the version of the project, and that > tell whether it is a release of a snapshot. Funny, this is one of my pet peeves. As a consumer, I don't care what version I want. I just want the latest. Having seperate version numbers means that as soon as a new version is released, I have to go and update my site location, with all of the P2 maintenance that entails. Same with target platform, etc.. My sense of the basic semantics would be that all of the basic sites would be for the latest version(s) (plus any archives projects deemed appropriate). So that the release location would always be referring to the latest release, milestone would be for latest version etc. Projects (like Mylyn for example) that maintain multiple sites for compatibility w/ prior Eclipse versions might use the latest released Eclipse for releases and the current release train for milestones and nightlies. But luckily this doesn't have to be an either/or situation. With Symlinks we could provide the versions an just point the release, milestone and nightlies to the latest release version url. > So +1 with Gunnar too: a template, some rules, and some tools to follow these > rules. I don't see why we couldn't have the rules without yet having the tools. The tools and template are an unnecessary block for providing guidance on a standard. (In reply to comment #13) > (In reply to bug 37111 comment 3) > > I personally like URL containing explicitly the version of the project, and that > > tell whether it is a release of a snapshot. > > Funny, this is one of my pet peeves. As a consumer, I don't care what version I > want. I just want the latest. Having seperate version numbers means that as > soon as a new version is released, I have to go and update my site location, > with all of the P2 maintenance that entails. Same with target platform, etc.. > > My sense of the basic semantics would be that all of the basic sites would be > for the latest version(s) (plus any archives projects deemed appropriate). So > that the release location would always be referring to the latest release, > milestone would be for latest version etc. Projects (like Mylyn for example) > that maintain multiple sites for compatibility w/ prior Eclipse versions might > use the latest released Eclipse for releases and the current release train for > milestones and nightlies. > > But luckily this doesn't have to be an either/or situation. With Symlinks we > could provide the versions an just point the release, milestone and nightlies > to the latest release version url. I agree with you: versionned repositories are not the best for consumers. However it is a very good way ensure new repositories will be deployed to a predictable URL and it's easy to know where to find older versions. Also it is very easy to set up and understand this convention. For end-users, I think symlinks are the good thing to use. Symlink would act like aliases. Hi, (In reply to comment #12) <snip> > > Stephane Bouchet's proposal looks good. > But I don't know if there is an interest in having a "nightly" and an > "interim". What is the difference? Could we say it is just a "snapshot"? > well, since last year, i agree that interim is not used often, so a 'snapshot/nightly' should be enough. > There should be some rules to follow, beyong the template, for > archives.eclipse.org and cleaning "milestones" directories after a release. it is up to projet to decides the retention policies of their repos (IMO) see our http://wiki.eclipse.org/EEF/Releng_Guide#Retention_Policy > Otherwise, Eclipse servers will need additional space. of course, webmasters are always trying to lower the bits on the servers, but i think the most important is to lower the bits to sent to mirrors. > > So +1 with Gunnar too: a template, some rules, and some tools to follow these > rules. +1 to for having a clear layout for p2 repo. maybe EMO/EF should investigate and add this to the EDP ? (In reply to comment #14) > However it is a very good way ensure new repositories will be deployed to a > predictable URL and it's easy to know where to find older versions. > Also it is very easy to set up and understand this convention. > For end-users, I think symlinks are the good thing to use. Symlink would act > like aliases. +1. It seems like both should be there, with versions required for any official releases. I think that would take care of Ian's concerns on the other bug re: archives of old versions. Someone building an aggregate site for a legacy install could use those. But I reiterate that going to a format that only mandated urls with versions and did not also require sites to maintain non-versioned links would be about the most consumer un-friendly thing we could do. (In reply to comment #16) > > But I reiterate that going to a format that only mandated urls with versions > and did not also require sites to maintain non-versioned links would be about > the most consumer un-friendly thing we could do. I believe format suggested in comment #7 with composite repositories at each non-leaf level provides both user-friendly "updates/releases" or "updates/milestones/0.8" urls. And it also provides separate urls for individual builds. This is what we are using for m2e [1] and it worked well for us so far. [1] http://wiki.eclipse.org/M2E_updatesite_and_gittags (In reply to comment #17) > (In reply to comment #16) > > > > But I reiterate that going to a format that only mandated urls with versions > > and did not also require sites to maintain non-versioned links would be about > > the most consumer un-friendly thing we could do. > > I believe format suggested in comment #7 with composite repositories at each > non-leaf level provides both user-friendly "updates/releases" or > "updates/milestones/0.8" urls. And it also provides separate urls for > individual builds. This is what we are using for m2e [1] and it worked well for > us so far. > > [1] http://wiki.eclipse.org/M2E_updatesite_and_gittags Yes, you're right. I had misread Stephane's that to be only supporting the leaf nodes of the tree. I'm arguing against the other way around that I think Eike and Gunnar had proposed, where the version information comes before the release type (R,M,I..). I can see the reasoning behind it as AFA the build system is concerned, types are sub-categories of versions. But seeing it form a consumer point of view, I think that's backward. BTW, totally coincidentally, I'm literally updating M2E as we speak to diagnose another bug. So you've saved me one google already. :) But that actually brings up the one other issue that came up on the duplicate bug: "Should the URLs follow the project / sub-project hierarchy, or the package hierarchy?" i.e. the question is whether to use: http://download.eclipse.org/technology/m2e/releases/ http://download.eclipse.org/modeling/gmf/tooling/snapshot or http://download.eclipse.org/m2e/releases/ http://download.eclipse.org/gmf.tooling/snapshot My feeling is that the latter is better, mostly for reasons of automated and consumer discovery. If I'm interested in gmf or m2e, I might not know or care what the super project is. And if I'm an automated tool that is looking based on say eclipse plugin namespace, I wouldn't have any way to discover that information. Since the project level names have to be unique anyway, what value does adding the super-project retain? (Except that it might involve permissions issues on download server, etc..) Finally, IIRC there actually is a justification for the difference between "interim" and "nightly" builds, and it is one that has come up quite a bit in the past for me when testing against other modelling builds. I can't find the page where this is described right now, but the basic idea is.. R, M: We know what these are. I: Interim/Integration Not a milestone, but you can rely on it being pretty stable, and it is what you should build against if you are trying to keep up with our rapidly moving API. N: Whatever our committers felt like pushing today. Each individual build should be stored in a way that the location never needs to be changed later. That forbids the use of versions because in the master branch the version can still change if code modifications require a minor or major version jump. We (the CDO project) are using a flat list, i.e., a single folder with one subfolder per build. Next, we should focus on the way builds are commonly being consumed. These "update channels" should be provided as composite repositories: 1) Many users want to consume the latest releases of a major/minor stream. 2) Some users want to consume unreleased builds from the active streams. We (the CDO project) only ever have two active streams at a time, a maintenance stream and an integration stream (no version needed for one-stream-maintenance). In each stream we compose two repositories: 2.1) Stable builds (S-builds, i.e., milestones and release candidates) 2.2) Weekly builds (M-builds or I-builds, depending on the stream) (Nightly builds could be promoted as children of a third category per stream.) We've developed a cron-based promotion service that monitors our Hudson jobs and produces this downloads page: http://www.eclipse.org/cdo/downloads . For multi-stream-maintenance projects (e.g. EMF) the stream version would have to be part of the maintenance composites. I'd be happy to hear feedback from others ;-) (In reply to comment #19) > Each individual build should be stored in a way that the location never needs > to be changed later. That forbids the use of versions because in the master > branch the version can still change if code modifications require a minor or > major version jump. Can you elaborate on this? Do you want to completely abstract versions behind release names? (In reply to comment #18) > [...] I'm arguing against the other way around that I think Eike > and Gunnar had proposed, where the version information comes before the release > type (R,M,I..). I can see the reasoning behind it as AFA the build system is > concerned, types are sub-categories of versions. But seeing it form a consumer > point of view, I think that's backward. My point actually was about the consumer site. As a consumer, I want clean, concise and easy to remember URLs. IMHO the "/release" or "/update" or "/whatever" part for official released and offered downloads is not necessary. URLs are like API. /project/ gives your the latest greatest official release available (like a giant composite of all release builds) (can get huge so performance my be a concern) /project/major the latest release of a particular major version (composite of all major.xyz; can get huge so performance my be a concern) /project/major.minor the latest release of a particular major.minor version (composite of all major.minor.xyz) /project/major.minor.service the software site for a particular release (qualifier is not necessary IMHO; there can only be one) (should not be a composite for better performance on direct access) /project/releasetrainname can be just a symlink to that "major.minor" that is intended to be contributed to a particular release train /project/.../testing (or /project/testing/...) - composite for I or M builds of that particular "..." - s/testing/<whatever-expresses-nondurable> One can argue if just "major" is necessary. Frankly, I don't care if the build type (for anything that's not an official release) is used as a prefix or suffix or somewhere in between. Oh and BTW, we have a PHP script running that generates a nice index.html. One just POSTs a compositeContent.xml to the PHP and gets back an index.html. The template should offer something similar. I'm happy to contribute that to wherever the templates are maintained. :) (In reply to comment #21) > My point actually was about the consumer site. As a consumer, I want clean, > concise and easy to remember URLs. IMHO the "/release" or "/update" or > "/whatever" part for official released and offered downloads is not necessary. > > URLs are like API. +10, and to use an API you don't need to know the version of that API you're using. I don't need to do: import org.eclipse.m2e.Editor.3.2.0; > /project/releasetrainname > can be just a symlink to that "major.minor" that is intended > to be contributed to a particular release train > > /project/.../testing (or /project/testing/...) > - composite for I or M builds of that particular "..." > - s/testing/<whatever-expresses-nondurable> > > One can argue if just "major" is necessary. Frankly, I don't care if the build > type (for anything that's not an official release) is used as a prefix or > suffix or somewhere in between. All I'm saying is that you need one more set here: /project/release <- Latest release. /project/milestone <- "" milestone. /project/nightly <- "" nightly Even if I'm consuming the milestone and interim releases I shouldn't have to know or care what the version is. I just want say the very latest build. That's the semantics of the "API". :) Beside the *easy of use* one criteria should be also *easy to reproduce*. The update sites are used for automated builds, and so must be "predictable", especially if the build scripts are pointing to the specific, older release. In enterprise world we can't rely on latest greatest BUT we still must be able to rely on the build system producing same artifacts. "/project/latest" or something without numbers (user - friendly) should always point to /project/major.minor.service (build script / enterprise friendly). This way everyone could be happy. (In reply to comment #20) > (In reply to comment #19) > > Each individual build should be stored in a way that the location never needs > > to be changed later. That forbids the use of versions because in the master > > branch the version can still change if code modifications require a minor or > > major version jump. > > Can you elaborate on this? Do you want to completely abstract versions behind > release names? Yes, but only in the download URLs. The downloads web page, which is automatically generated during each promotion run, of course always lists the builds in a properly versioned stream. (In reply to comment #23) > All I'm saying is that you need one more set here: > > /project/release <- Latest release. > /project/milestone <- "" milestone. > /project/nightly <- "" nightly I think it's fine to setup links or composites at each one of those. I actually proposed something for Orbit a while back. > Even if I'm consuming the milestone and interim releases I shouldn't have to > know or care what the version is. I just want say the very latest build. That's > the semantics of the "API". :) You know that API and versioning is a big selling point of OSGi, don't you? Now how will LTS work without nailing down dependencies to a particular version/version range? (In reply to comment #26) > You know that API and versioning is a big selling point of OSGi, don't you? Now > how will LTS work without nailing down dependencies to a particular > version/version range? I don't understand this problem/question. A build to be reproducible needs to remember its upstream repositories (URLs). So the URLs and the contents must never change. Versions in the URLs are not needed, though they *may* not harm in some cases (especially when the consumed build is a release or maintenance build). But a reproducible build should not consume upstream builds through "update channels" (composite repositories with changing content). (In reply to comment #27) > But a reproducible build should not consume upstream builds through "update > channels" (composite repositories with changing content). That's exactly my point. Repo URLs like "/project/milestone" aren't suitable for reproducible builds. Having a repo url with a version number (eg. "/equinox/3.7") will allow to express a dependency to a particular project *and* a particular version range *and* a particular build type (eg. official releases or integration/maintenance builds or snapshots or whatever). One can argue that the version is expressed in the target definition. However, I find that not very intuitive. Someone just opens a target definition, clicks "update" and whoops your version is bumped up to the latest and greatest unwanted build. Besides, such a common URL structure may also be used for non-p2 content that typically accompanies a release such as documentation, archives, etc. (In reply to comment #28) > That's exactly my point. Repo URLs like "/project/milestone" aren't suitable > for reproducible builds. Having a repo url with a version number (eg. > "/equinox/3.7") will allow to express a dependency to a particular project > *and* a particular version range *and* a particular build type (eg. official > releases or integration/maintenance builds or snapshots or whatever). +1 (In reply to comment #29) > (In reply to comment #28) > > That's exactly my point. Repo URLs like "/project/milestone" aren't suitable > > for reproducible builds. Having a repo url with a version number (eg. > > "/equinox/3.7") will allow to express a dependency to a particular project > > *and* a particular version range *and* a particular build type (eg. official > > releases or integration/maintenance builds or snapshots or whatever). > > +1 +1. As of having full project hierarchy in URL or swapping version and build type in URL, I don't care. The important is having something used by everyone, and containing enough info and that I can guess. URL require project, version and build-type, and the order must always be the same. We now all agree on that Technical criteria for choice: Assuming we want to provide URLs containing /latest instead of /version, and we want that to be a symlink to a specific version. Then that means that /version must be at the end (leaf) of the URL, because symlinks are "leaves". If we have /version/snapshot and /version/release, and want to alias version to /latest, it will affect both snapshots and release. We don't want it, since we want streams to remain separated. (In reply to comment #30) > Then that means that /version must be at the end (leaf) of the URL, because symlinks are "leaves". This argument is invalid, having aliases does not introduce any constraint on URL for this use-case. Please ignore that part. So project/buildType/version or project/version/buildType are totally the same for me and both will work with symlinks. It sounds like what little controversy here is resolved by an understanding that we're simply trying to serve two different kinds of consumers: 1. Tool consumers (aka "Users"), who should be given a simple URL and don't care what version they have as long as it is the latest and greatest % how far they want to be on the bleeding edge. 2. Build consumers, who others have noted should know what exactly they are getting and how that might effect their downstream consumers. If we support both (with I think an emphasis on at least providing 1) so that we have at least a clear public facing standard) then we're all good, right? For Mylyn we currently consider four types of consumers: * End users: Need easy access to latest release compatible with their Eclipse instance * Early adopters: Need easy access to latest snapshot * Integrators: Need stable URLs for reproduce builds and be able to consume the latest snapshot of a stream for integration builds * Developers: Need nightly builds for validating changes We have several aliases to find releases based on maturity (build type), target (i.e. Eclipse release train) and stream (i.e. Mylyn version): pre. /releases/latest -- latest release /releases/3.6 -- latest 3.6.x release /releases/indigo -- latest release that is compatible with Indigo pre. /snapshots/weekly -- latest snapshot /snapshots/3.7 -- latest 3.7.x snapshot /snapshots/indigo -- latest snapshot that is compatible with Indigo /snapshots/nightly -- latest nightly pre. /drops/3.6.0/v20120101 -- release build (never removed) /drops/3.7.0/I20120202 -- integration build (removed once release build is available) All sites under /releases and /snapshots are composites that link to actual build artifacts under /drops. The composites are automatically generated based on filters that specify which releases to aggregate. As a consumer I always want the latest version of the project in the release that I am working in: .../nebula/juno/latest .../nebula/latest does not guarantee that it fits my version. My version is most of the times older than the current release. I also agree to skip the top level project when naming, where possible. This bug hasn't had any activity in quite some time. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. If you have further information on the current state of the bug, please add it. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. -- The automated Eclipse Genie. This bug hasn't had any activity in quite some time. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. If you have further information on the current state of the bug, please add it. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. -- The automated Eclipse Genie. This bug hasn't had any activity in quite some time. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. If you have further information on the current state of the bug, please add it. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. -- The automated Eclipse Genie. This issue has been migrated to https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/139. |