Community
Participate
Working Groups
There seems to be some momentum for migrating Eclipse project CVS repositories from CVS to Git. (See bug 345479 for the Eclipse Platform/Equinox one) I'm opening this bug report so we can capture discussion about moving the Orbit "source" repository. From a consumer point of view, the 90% case is clients will not notice a change. Most people either download bundles/zips from the downloads page or their builds consume Orbit bundles via map files. (which can be done via p2 or from CVS, http, etc.) The clients who will be effected the most will be developers who check Orbit projects out of CVS to test new versions of Orbit bundles, and those of us who are the maintainers of Orbit bundles. A change from CVS to Git will have the biggest effect on the build process, as it will have to change to handle the new system. Also the Eclipse-SourceReferences bundle header is being discussed in bug 345670. Thoughts?
how's the current build work?
Do we want one big repo or repo per bundle?
We currently use the concept of a "branched project" to store all the different versions of particular bundles. So, if you looked a just branches, we'd have scores of long term branches, though most would have only one (sometimes a few) bundles in that branch. I get the impression this idea of lots of "branched projects" is kind of foreign in Git ... more common to "branch a repository". True? I'm not sure it is essential we branch each module/project as we have been ... there could be other schemes, I'm sure ... but, it'd be nice to understand the implications. For example, currently we can pull in two branches of a module into the workspace (with tweaked workspace project names) in order to work on them or compare them. Can you pull in two branched projects from Git ... using something with a "name as" funciton? (I don't think so ... but, admit, I've only scratched the surface of Git). It goes against my intuition to have "one repo per bundle" ... we'd have about 300!? Seems odd and hard to manage, somehow ... but, again, admit my "intuition" is based on CVS (and a few other) SCMs. The current build works based on a PDE build, driven by one feature, with tagged versions of bundles specified in map files. I know we can do all that with Git (using the Git fetch factory) ... but, I'm sure we'd run into issues. We do have some "custom" parts of the build that go beyond PDE build, such as the way we store, fetch, and construct source bundles. I don't think too much of that is really coupled to SCM ... but, a little might be as some "fetch scripts" are generated by our own code (rather than PDE code). I should also add, I don't think there's much rush to migrate Orbit (other than as yet another test case) since it is not a "highly collaborative" repository ... that is, not much is really gained (from what I know) of having a distributed, easily forked repo, for these third party jars. Thanks in advance for the discussion (and educating me :)
Unfortunately in the Orbit use case CVS is vastly superior to Git (sacrilege!!) If you had one Git repository for Orbit, then all the Orbit bundles in your workspace would have to belong to the same branch. This makes no sense because each bundle in Orbit typically has different branch versions. Or, you would need a different clone of that repository for each bundle you want to work with (also untenable). In either case the repository would be huge and a pain for people to load (want one bundle from orbit? Sorry, you've got to load them all). One Git repository per bundle would unfortunately be the closest fit but would be a huge pain to work with. Since we aren't maintaining source code for Orbit for the most part, perhaps something like a p2 or Maven repository would be a better fit. Actually a plain old web-accessible file system on eclipse.org with directories per bundle and branch would be better than git (using HTTP get for builds and something like SFTP to import/export from Eclipse workspaces).
(In reply to comment #4) > ... Since we aren't maintaining source code for Orbit > for the most part, perhaps something like a p2 or Maven repository would be a > better fit. Actually a plain old web-accessible file system on eclipse.org with > directories per bundle and branch would be better than git (using HTTP get for > builds and something like SFTP to import/export from Eclipse workspaces). I hope you are not suggesting we not use any SCM (that is sacrilege!!) :) While we don't have Java source code (except for one exception) the manifest.mf files, and other various property files are also "source" and I'd hate to lose that history and "reproducibility" of previous builds. BTW, as I'm sure you know, we do provide p2 repositories and html access to jars ... those are the 'results' of our builds ... and we do (should) encourage most developers to pull those into their workspace or targets when necessary instead of directly from cvs ... but, unless I'm misunderstanding, or someone just hasn't thought of a Git solution yet, this seems like a good case to keep cvs around.
(In reply to comment #5) > I hope you are not suggesting we not use any SCM (that is sacrilege!!) :) I'm just saying I think no SCM at all would be *better* than using Git for Orbit. I am not directly suggesting that we not use any SCM ;)
(In reply to comment #5) > (In reply to comment #4) > > ... Since we aren't maintaining source code for Orbit > > for the most part, perhaps something like a p2 or Maven repository would be a > > better fit. Actually a plain old web-accessible file system on eclipse.org with > > directories per bundle and branch would be better than git (using HTTP get for > > builds and something like SFTP to import/export from Eclipse workspaces). > > I hope you are not suggesting we not use any SCM (that is sacrilege!!) :) > > While we don't have Java source code (except for one exception) the manifest.mf > files, and other various property files are also "source" and I'd hate to lose > that history and "reproducibility" of previous builds. BTW, as I'm sure you > know, we do provide p2 repositories and html access to jars ... those are the > 'results' of our builds ... and we do (should) encourage most developers to > pull those into their workspace or targets when necessary instead of directly > from cvs ... but, unless I'm misunderstanding, or someone just hasn't thought > of a Git solution yet, this seems like a good case to keep cvs around. Personally, I think we need to rethink orbit and the way it is done. I've never liked including binaries into the repository, and have always felt that the orbit items should be rebuilt from source. Orbit really should be built from source.
(In reply to comment #6) > (In reply to comment #5) > > I hope you are not suggesting we not use any SCM (that is sacrilege!!) :) > > I'm just saying I think no SCM at all would be *better* than using Git for > Orbit. I am not directly suggesting that we not use any SCM ;) No SCM vs Git might be a real discussion we'll have to have. It appears that CVS will be retired completely at some point. https://bugs.eclipse.org/bugs/show_bug.cgi?id=358980
(In reply to comment #8) > No SCM vs Git might be a real discussion we'll have to have. It appears that > CVS will be retired completely at some point. I respectfully submit that the No SCM option should be discussed after the build from source option has been exhausted.
(In reply to comment #9) > (In reply to comment #8) > > No SCM vs Git might be a real discussion we'll have to have. It appears that > > CVS will be retired completely at some point. > > I respectfully submit that the No SCM option should be discussed after the > build from source option has been exhausted. I don't see how building from source is related to repository choice?
(In reply to comment #10) > I don't see how building from source is related to repository choice? Let me rephrase: I'd like to see rethinking suggested by Dave Carver Comment 7.
(In reply to comment #11) > (In reply to comment #10) > > I don't see how building from source is related to repository choice? > > Let me rephrase: I'd like to see rethinking suggested by Dave Carver Comment 7. Well, I'll rephrase too: I don't think comment #7 is related to this bug, or at least I don't see it. Beyond that, I guess I'd suggest if someone suggest "you should do X" they might at least say why. (I can imagine the reasons ... but ... we might all imagine different reasons). And, I still think a different issue. Please explain the connection.
I agree that moving Orbit to Git in the current form would be a bad idea. Git is really an SCM where SC means Source Code. If you put binaries into a Git repository, it just grows huge, and you force each user to download the entire binary history of the project (or of the whole Orbit, if you go for the all-in-one approach). Git is nice for tracking source trees where changes between releases are relatively small and thus well compressible. > I don't see how building from source is related to repository choice? If all Orbit projects were built from source, then the Orbit repo would not have the issues connected to storing binaries in Git. For storing binaries, CVS is still a better match than Git, so I would prefer to keep CVS for Orbit. And compared to a pure filesystem-based approach, it still has all the advantages of a versioning system for the source files that are still necessary to turn a plain library into an OSGi bundle.
Git gets a bad rep for storing binary files, it's gotten a lot better recently. It would be less of an issue if we had a git repository per Orbit library anyway. I would like for us to do some minimal tests before we make a decision. Another alternative is to use something like git-annex (http://git-annex.branchable.com/) to help dealing with very large binary files.
Thanks for explaining the connection of repo choice to binaries. So be honest, I doubt we could successfully build everything in Orbit from source ... well, at least, not without a person year of effort, or something extreme. The reason (that I am aware of) is often these things are built by their originators in special ways, with many (different assumptions) and often with pre-reqs that we may not have easy access to. Occasionally, I've seen, some are even built with other libraries in their paths, and if not there at runtime, that functionality is simply not ran or not relevant (e.g. they detect class not found exceptions and continue on without it if not there ... but, they still have to compile with it). Even if someone were to volunteer that person-year worth of effort, I'm not sure it is a good idea for us, in Orbit, to be "release engineers for the world". I think our main mission is to provide OSGi bundles of third party code. Big difference. So ... to continue the repo discussion ... here's an idea, and I think it smacks at the binary issues you mentioned. I've been seeing the problem here as the one of "many branches". I think that could be solved if we stopped doing that in Orbit. Say instead we had each module named with its version in the name (are they still called modules in git? :) For example, we've have two module in "head" (master) named javax.xml.bind-2.1.9 javax.xml.bind-2.2.0 instead of one module, javax.xml.bind, with two branches 2.1.9 and 2.2.0. We'd still build (assemble) them with their right bundle names and versions, just select off the "tag" not the branch. We could, technically, do this now, with cvs, so I'm assuming that sort of build/assembly is possible in Git. Now, if I understand what'd being said, in this scheme, someone would still have to download a whole bundle of stuff (whole repo), even if they wanted just one bundle from Orbit? I'm not sure this is too bad, since we actually want most users/developers to get things from the p2 repository produced, not directly from "code repo", but ... I guess Orbit committers (at least) would need to sync up over good high speed connections, and not expect to "get" just the one little module the wanted to work on. Is this a good characterization of (some of) the issues of having one bit repo? Made worse by having binary files only? Are all those binaries "tagged" and synchronized with each 'change set'? Or is it just a matter that they are not compressed when gotten the first time? Thanks as usual for the education. So, far, I'm still left with the impression that technically we could move to Git, but that it would not be as good a solution as CVS (and, I think Orbit has less of a "need" for Git ... that is, it's not a "highly collaborative" environment, where many people make many of their own "local branches").
Yes I think the repository granularity is the main problem. I was thinking along similar lines that the branches could be represented in master using sub-directories, but you would still end up with a massive Git repository in the end. We could always run a straight test migration on the current repository and see how big it is. I currently frequently check things out of Orbit just to get them in my target platform when building something. I don't know how common that is, but people doing this could migrate to provisioning orbit content into their PDE target platform straight from the p2 repository rather than checking out of the SCM.
(In reply to comment #16) > Yes I think the repository granularity is the main problem. I was thinking > along similar lines that the branches could be represented in master using > sub-directories, but you would still end up with a massive Git repository in > the end. We could always run a straight test migration on the current > repository and see how big it is. I've done something similar with git sub-modules. Each bundle would be a sub-module and you can clone only a subset of the modules. However, a submodule is basically another git repository (linked together by pointers), so the maintenance of this structure would likely be an issue. From my (limited) experience, CVS is a perfectly reasonable solution for managing the revisions of a number of unrelated files. Unfortunately (for CVS), source code is not *a number of unrelated files*. SCMs like Git are much more suited for working with source. However, CVS might be "The right tool for the job"(tm) for Orbit, since most of the files (bundles) are unrelated, and merging / community contribution / etc... are not a concern. I understand that the foundation wants projects to move away from CVS for their SCM. But, if CVS is the 'right tool for the job' do we have to move? -- especially if can be argued that Orbit is not using CVS for its *Source* control management.
FYI, a link showing the breakdown of repositories at Eclipse.org and the countdown to the CVS repos being made read-only. http://eclipse.org/projects/scmcountdown.php
(In reply to comment #18) > FYI, a link showing the breakdown of repositories at Eclipse.org and the > countdown to the CVS repos being made read-only. > http://eclipse.org/projects/scmcountdown.php That's cute ... guess they'll add the hours, minutes and seconds later :) To cross reference, interesting discussion in bug 361019 about less technical (more legal) reasons for keeping Orbit in CVS ... there are some advantages to centralized repositories! :) [And, I'm not saying that bug will necessarily dictate what we do ... but .. may influence it.]
(In reply to comment #19) > That's cute ... guess they'll add the hours, minutes and seconds later :) No. I'll be spending those last hours and minutes with my family.
Via email I've initiated an "executive discussion" with Mike M., Wayne, Denis, and a few board members (including one of our committer reps, John A), asking for a "official position" if Orbit can continue to exist in CVS past December. I personally think that's the best thing to do since little to gain from being in Git (people shouldn’t need to "make their own branch" of an Eclipse Orbit project (If they need that, they should work with original project, not the eclipse orbit project). In addition to little need to move to Git it would require lots of work and fundamental changes to make build/assembly work in Git, as has been discussed previously in this bug; so the cost to benefit ratio is not great. Not even sure it could be done by December (but, volunteers welcome :) I thought it'd be an easy decision but some interesting questions came up that are worth documenting here for everyone (even though no official decision yet) just to share knowledge and so others can correct any mis-statements I might make. First, it was asked, why use VCM at all, why not just keep jars on file system? I explained that is the end result (in the form of a p2 repo) but we don't just store "plain jars" in VCM but original .class files (and sometimes source java files, which we do not build), along with our tweaked manifests files, build properties, localization files, etc., that have all been converted to work with OSGi. These do sometimes need fixes or changes over time, so there is a history, that occasionally is important. That is, someone might need to see what's changed from one version or date to another. But there has been similar suggestions before, for example, see bug 310183 comment 4: <quote> There's an interesting third possibility, say called "jarredbundle" where we'd explode a jar, run bnd on it to update manifest, etc., and thereby automatically make it into an OSGi bundle a little more automatically. No idea how much work that'd take ... but know it would be more than a trivial amount. </quote> So, in this sort of case, there could (in theory) be a few files kept in VCM (the manifest, build.properties, etc.), the build/assembly stage would "explode" the jar, and put it all back together in the right form with the few files from VCM and eventually into a p2 repository. In that bug 310183 the idea was to still have the jar in VCM but I'm sure other options are possible. The point is, I wanted to admit its a valid question, perhaps there could be cases of "transforming" a jar, instead of "assembling it", but ... that would be a new system that someone would have to build. But good question. Another point: some (most/all) people feel safer having "source artifacts" in a VCM system (even if they are binary), since VCMs "protect" the artifact from unknown or unseen changes, whereas on a file system, its sometimes hard if not impossible to tell for sure if the file this year is exactly the same as it was, say, 3 years ago. So, even with a "transformation" system, I think most would still want the protection of a VCM for the source material. I wanted to document the question/issue here since valid question, and certainly if someone knows of existing or wanted to create a new build system based on transformation of a jar, rather than assembly of .class files, I'm sure volunteers would not be ignored. = = = Second interesting question asked was "how is this related to maven efforts"? I think the answer is that it is not (or, not very). This entry is about how to store "source artifacts" of 3rd party jars which we want to tweak and turn into OSGi bundles, and store in p2 repository. I think conceptually, the way to view the maven relationship is to imagine another project ... say, Morbit :) , that took third party jars, and tweak them, as needed, to "mavenize" them, and then put them in a maven repository. Now, in practice, there might be some "sharing" of "source artifacts", but the tweaks would be different (obviously) and the end-result repositories would be different. Currently, the CVS version of Orbit repository takes 1.6G on the file system so IMHO this is not a big hog of resources in disk space: > /home/data/cvs/tools/org.eclipse.orbit $ du ./ -sh 1.6G ./ Others may know more about these issues ... feel free to share ... but sounded like there were some questions about what Orbit is and what Orbit does that are not very clear. Hope my wordy remarks and any discussion is spurs helps clarify things.
I would definitely vote against moving away from a VCM.
(In reply to comment #21) > Via email I've initiated an "executive discussion" with Mike M., Wayne, Denis, > and a few board members (including one of our committer reps, John A), asking > for a "official position" if Orbit can continue to exist in CVS past December. The "executive decision" is that Orbit should stay in CVS for the foreseeable future. My position is that we should focus our efforts on providing durable repositories of Orbit artefacts; with first-class support for Maven being our primary target. I agree that we need to keep Orbit artefacts under version control. It's a little weird to keep binaries in version control, but it makes sense in the context of Orbit. Any attempt to build from sources is doomed to fail owing to the variation in build technologies required for each of the libraries. If and when we do migrate Orbit off of CVS, the target environment really has to be Git. If we have to do something special or different, then that something will have to be maintained, thereby consuming webmaster resources. If it's something other than Git, it might as well be CVS.
> The "executive decision" is that Orbit should stay in CVS for the foreseeable > future. FWIW, using CVS (the binary application) to store Orbit is something I don't have any issues with. I do intend on terminating the "anonymous CVS" pserver service, as well as all the cvs sync mechanisms we use to ensure pserver is up to date with the "ssh" version. From inside the eclipse.org network, you'd still be able to access CVS "anonymously" using the filesystem -- /cvsroot and CVS over SSH will also work (since you're calling the CVS binary directly).
(In reply to comment #24) > ... I do intend on terminating the "anonymous CVS" pserver > service, as well as all the cvs sync mechanisms we use to ensure pserver is up > to date with the "ssh" version. > While off topic, I assume this just means the "faster" pserver anonymous access, right? That users could still get pserver anonymous access if they wanted to ... it'd just be a slower than it is currently.
No, all of pserver will go away. There's no reason to keep it.
> No, all of pserver will go away. There's no reason to keep it. OK, so you'll add an SSH user "anonymous" without write access, so that non-committers can still access the repo?
(In reply to comment #27) > > No, all of pserver will go away. There's no reason to keep it. > > OK, so you'll add an SSH user "anonymous" without write access, so that > non-committers can still access the repo? Yes, I hope I didn't miscommunicate in some of the comments here or in emails. I think I was agreeing with someone's point that MOST users access to Orbit is from the p2 repositories, not the CVS repositories, but that doesn't there is zero need for non-committer (anonymous) access. A typical use-case might be someone finds a bundle from Orbit isn't working right for them, and they can see it does not have ... lets say ... an Eclipse-BuddyPolicy:Registered directive they think it should have. They might want to check it out from cvs, and try the fix to confirm that really is the issue before opening a bug, or suggesting that fix. Plus, seems kind of anti-open source not to allow anonymous access to all source repositories. I can open another bug if this is a long item for discussion, but perhaps a simple miscommunication?
I just went to http://eclipse.org/orbit and read: "This project will provide a repository of bundled versions of third party libraries that are approved for use in one or more Eclipse projects." If it's used by Eclipse projects, then I don't see a need to access the repo anonymously from the outside world. > Plus, seems kind of anti-open source not to allow anonymous access to all > source repositories. Well, move it to Git. Sorry -- I just can't maintain an indefinite list of services. When I signed up for Git, I said CVS had to go.
I played a bit with migrating Orbit to Git (using git cvsimport) over the weekend. I decided to make a single Git repository for each library. I picked a sample subset and set about doing the migration, building Git repositories on my local workstation. Each Git repository is smaller (i.e. takes up less disk space) than the corresponding directory in the CVS repository. The Git bare git repository for org.apache.log4j is 1.156MB, compared against the CVS directory at 4.928MB. I chose org.apache.log4j because I knew that there would be a few different branches in there. Checking out a branch does add a few KB, but I believe that the impact of having a complete clone of a single library's Git repository is quite manageable. Loading the library into a workspace from the Git repository is a little more cumbersome. There is no easy mechanism for browsing the libraries when the are represented as separate Git repositories. You just have to know the URL of the repository (which is easy enough to obtain, but still more work than simply browsing the CVS repository and picking the directory you want to pull into the workspace). Is there ever a need to pull *all* of Orbit into an Eclipse workspace? Once the library's Git repository has been cloned locally, however, checking out alternative branches is dead easy. No worse than for CVS. Maybe a little easier than CVS by expanding the branches in the the Git Repository browser. The libraries that I've checked all seem to have .project files, so they're pretty much good-to-go once they're loaded as projects into the workspace. The actual process of migrating doesn't seem like all that big a deal. It'll be a little time consuming, but is easily scripted. Aside from the inability to easily browse for the libraries, consuming out of Git repositories doesn't seem particularly difficult to me. The inability to easily browse repositories is really more of a PITA than a real hindrance IMHO. What am I missing? I've only just started to look into the releng scripts. These can likely be lumped into a single Git repository. It's been a long time since I've played with PDE Build, map files, and such, so this will likely be slow-going.
(In reply to comment #30) I'm not sure what to say. I don't want to sound argumentative, or uncooperative, ... but, I probably will. > I played a bit with migrating Orbit to Git (using git cvsimport) over the > weekend. I decided to make a single Git repository for each library. I think this kind of approach (a repo for each library) was already discussed in this bug and rejected as unworkable. Besides the vast quantity, there are some libraries that "go together" (such as the lucene bundles). There are other cases where one bundle is "highly related" to many others but should not be grouped with them in same repo, since that logic would eventually "pull in" about everything into one repo, but committers/contributers would have to have all related ones loaded in workspace before using them would "make sense". (And, by "using them" I mean commmitters contributing new ones, making sure they are ok, etc., as well as the few people who do like to load into their workspace). And, yes, there are occasions when loading all or nearly all the Orbit bundles is important for committers, such as to do a global search for who uses something, who else might be specifying imports/exports, are they consistent with version use, etc. Not to mention the vast quantity. Additionally, if you can figure out Orbit's build, you can count yourself among the top 1 % of PDE Build experts and build masters in general. :) I say that since there are several "custom" things done, that are not normally done by PDE builds. Such as the way we store and create "source bundles" for the third party libraries. {And, to be blunt, I do not want to be responsible for someone else's prototype build and left to make it production quality and working on a long term basis ... so ... if you want to do and be responsible for doing the Orbit builds, fine. I'll start the committer election right away :) ] No one is saying it is impossible to migrate Orbit to Git [after all Git can solve any problem :) ], I am just saying that a) it will take more work than usual (and more time that we have), and b) we do not yet know the best way to do it (such as, how to handle the right level of granularity). [Keep in mind, in the beginning, it took a small team of very smart people months to figure out how to aggregate these third party classes, to be OSGi friendly, with dozens of changes and improvements since then.] If push comes to shove, I've prefer to leave in CVS and let the community complain to EMO about lack of pserver access ... if they want to ... who knows ... maybe no one cares. But, I think that is actually a larger topic than this bug. Many people are under the impression that all the CVS repos would be "left around" in read-only mode for an extended period of time ... if that is not true at all, or if that means left for only commiters to access without pserver protocol, then that news should be better communicated than it has been so far. Especially since it would break the "get source references" function used in many past releases. The last 3 releases, I think ... and there's about 700 bundles that use it. (many not from Orbit of course).
AFAICT the multiple repositories option was dismissed because it didn't "feel right". The limited energy that I've put into this is intended to explore the option a little more. Something that, apparently, you believe needs to be done. > b) we do not yet know the best way to do it (such as, how to handle the > right level of granularity). It is not my intention to impose any kind of "prototype build" on anybody. I am merely exploring the idea so that I can better understand the issues. In the spirit of open source, I am doing so transparently so that others can benefit from the effort (or join in). I'm probably going to do some bonehead things in the process, but just dismissing the idea because it doesn't "feel right" isn't good enough IMHO. If--at some point--making me a committer makes sense, we can discuss that option. > there are occasions when loading all or nearly all the Orbit bundles is > important for committers, such as to do a global search for who uses > something, who else might be specifying imports/exports, are they > consistent with version use, etc. I don't understand what you mean? Is "global search" restricted to the contents of a workspace, or is it broader than that? AFAIK, Maven is quite good at reporting who uses what in a very large sense. Maven seems far more useful in this regard. If I have to load everything, can't I achieve that with a team project set? I don't know; this is one of those things that needs to be sorted out. So far, AFAICT, the only real problem is that browsing from within the IDE is harder with multiple repositories. I don't buy the creeping granularity concern. But it's worth exploring. If we're not compiling anything, then dependencies shouldn't be a problem, so even dependent bundles can be represented in individual repositories. At least I think so; that's something that's worth exploring. We're going to keep read-only CVS around as long as we can. But it's important that everybody understand that there's more at work here than the webmaster wanting to retire a service to conserve resources. CVS is no longer maintained. There are many open (and unattended) bugs that may turn into serious vulnerabilities. There is a real risk that it may be dropped from our Linux distribution (this has already happened for some distros). This is a technology that has served us well, but we need to move on.
(In reply to comment #32) Thanks for your comments and help Wayne. I think there's one constructive thing I can answer in your questions/comments. > ... If we're not compiling anything, then dependencies shouldn't be a problem I lied a little, there is one bundle we compile from source. Long story, forget which one, right off (there is a bugzilla entry where its discussed), but one "third party" only provides java source, no binary versions. Doubt it has dependency issues. But, more importantly dependency resolution is still important in our builds/process. On the one hand, this is a "side effect" of PDE build. But, on the other hand, it has always been considered important since we would not want to offer something in Orbit that can not be used, because there was a missing dependency, so it useful to have the build fail if a dependency is missing. There have been real cases where someone was adding something to Orbit, and then discovered "oh, I need another CQ because that bundle requires some other package". Hope my comments are helpful.
Are there some news on this topic? It is getting more important now that CVS was shut down.
(In reply to comment #34) > Are there some news on this topic? It is getting more important now that CVS > was shut down. Orbit was give a short extension to still use CVS (i.e. committers can still use it). As far as "anonymous access", users should be getting what they need from Orbit's p2 repository, or download "one jar" from the DL page. Am I missing your point or real question? If you have a "use case" not covered by just getting the bundle, you might document that use case in bug 397237.
(In reply to comment #35) > Orbit was give a short extension to still use CVS (i.e. committers can still > use it). As far as "anonymous access", users should be getting what they > need from Orbit's p2 repository, or download "one jar" from the DL page. Ideally, I'd like to have a move-forward plan captured this quarter (i.e. the end of March 2013). EclipseCon might be a good timeframe for finishing touches in a BoF session.
The new Eclipse Bundle Recipes project might have some potential to help with an Orbit migration to Git. http://eclipse.org/proposals/rt.ebr/ We move the actual raw (JAR) IP-cleared artefacts into a Maven instance (maven.eclipse.org). Store recipes for building bundles out of the JARs in the Orbit Git repository. These recipes would know how to obtain the raw JAR artefact Generate a p2 repository from the result of running the recipes. Additionally push the result of the build into maven.eclipse.org Thoughts?
I dislike any solution that doesn't involve orbit goop making it into maven central. Going to maven.eclipse.org is fine if there is a plan to sync it out to a broader audience from there. I saw that EBR proposal and it would work well if the IP story was changed so that if something from maven central was earmarked as IP clean then it could be used by an eclipse project and referenced from an IP log that would really clear it all up. Then that could be used as input to EBR which could totally replace Orbit if it is flexible enough.
(In reply to comment #38) > I dislike any solution that doesn't involve orbit goop making it into maven > central. Pushing to Maven Central is definitely on my list. > > Going to maven.eclipse.org is fine if there is a plan to sync it out to a > broader audience from there. My understanding is that doing some kind of automagic push from maven.eclipse.org is relatively easy. > I saw that EBR proposal and it would work well if the IP story was changed > so that if something from maven central was earmarked as IP clean then it > could be used by an eclipse project and referenced from an IP log that > would really clear it all up. Then that could be used as input to EBR which > could totally replace Orbit if it is flexible enough. For EBR, the source of the raw materials isn't all that important. The project doesn't intend to actually distribute any of the generated artefacts, just the recipes for generating what needs generating. For Orbit, though, blessing something that's on Maven Central won't cut it. The IP team is uneasy about consuming artefacts in that manner (I'm understating here). Moreover, from an LTS point of view, we need to avoid depending on an external entity's continued existence.
> For Orbit, though, blessing something that's on Maven Central won't cut it. > The IP team is uneasy about consuming artefacts in that manner (I'm > understating here). Moreover, from an LTS point of view, we need to avoid > depending on an external entity's continued existence. I would be interested in knowing the reasons as when folks open up a CQ they mostly just attach what they pull from maven central anyway...since that is where most everything is published to anyway. As for relying on maven central and LTS, the world would be in a whole lot bigger trouble if maven central were to disappear...
(In reply to comment #40) > I would be interested in knowing the reasons as when folks open up a CQ they > mostly just attach what they pull from maven central anyway...since that is > where most everything is published to anyway. I think that's not true. I do pull all the bits I contribute to Orbit from the original project source. There are some projects which just publish to Maven central but most also provide downloads by themselves. > As for relying on maven central and LTS, the world would be in a whole lot > bigger trouble if maven central were to disappear... Well, that's another argument to *not* depend on Maven central then. Anyway, I think I understand your point. Jetty is one of the projects that do not depend on Equinox/p2 at all. Your absolutely happy with what you consume from Maven central. Unfortunately, that's not the case for a lot Eclipse projects. Some bits in Maven central don't have OSGi manifest all. Others have them but their quality sucks. I think having maven.eclipse.org with all blessed, IP-cleared artifacts is a good idea. It helps preventing mistakes. Must poms define "soft" versions in their poms without actually knowing what "soft" means. From one build to another the project may start consuming an un-blessed artifact without ever recognizing. BTW, what is the best way to publish our "OSGi-ified" artifacts? Do they need to published with a new group and artifact id? I think if we go with a group id of "org.eclipse.orbit.bundle" and an artifact id of "bsn" than they should be easily identifiable in Maven central and can be consumed from Eclipse projects that way. Jesse, does that sound like a stupid idea for Maven experts? How do we handle stuff that isn't consumable with Maven? For example, for the daemons package I currently download a ZIP with the binary launchers, unzip it and create several fragments for the various platforms. The Java part (daemons jar) is in Maven but the binaries aren't.
> I think having maven.eclipse.org with all blessed, IP-cleared artifacts is a > good idea. It helps preventing mistakes. Must poms define "soft" versions in > their poms without actually knowing what "soft" means. From one build to > another the project may start consuming an un-blessed artifact without ever > recognizing. Sure, but maven.eclipse.org is either the 'home' of this stuff, or it is merely a halfway home for things enroute to maven central. If it is a halfway home then it shouldn't be deployed to at all without support for staging releases which is available in Nexus Pro which has been nixed in being used. Better to just have projects make an account on oss.sonatype.org nexus instance or plot an alternative route to maven central then nexus...one that has checks and balances like staging repositories, etc. > BTW, what is the best way to publish our "OSGi-ified" artifacts? Do they > need to published with a new group and artifact id? I think if we go with a > group id of "org.eclipse.orbit.bundle" and an artifact id of "bsn" than they > should be easily identifiable in Maven central and can be consumed from > Eclipse projects that way. Jesse, does that sound like a stupid idea for > Maven experts? It is not a good solution, but it is less bad then others I have seen. The real issue with it is that Joe user could easily depend on apache wicket and eclipse modeling and get two different log4j dependencies with two different coordinates and then toss both of those things into a build environment as opposed to resolving the correct one. Hazards of meshing declarative vs manifest driven approaches. You don't need me to detail this though, there are long long bugs on this topic though that hash, rehash, lie, explain, obfuscate and rehash again the issues...I linked one such, the one from 2009 :) > How do we handle stuff that isn't consumable with Maven? For example, for > the daemons package I currently download a ZIP with the binary launchers, > unzip it and create several fragments for the various platforms. The Java > part (daemons jar) is in Maven but the binaries aren't. that is easy, they can go up as classifiers on artifacts or simply as a zip: jetty-distribution: http://repo2.maven.org/maven2/org/eclipse/jetty/jetty-distribution/9.0.0.M5/ (see -tests and -config items) classifier: http://repo2.maven.org/maven2/org/eclipse/jetty/jetty-server/9.0.0.M5/ My bigger personal concern is that stuff in maven should be generally useful and not a repository for goop related to binary installers and the like that are more suited to standard download pages and the like....but that is probably something to bring up with the maven community (the maven pmc?) directly. We put our jetty-distribution in there because quite a fair number of people consume it into their builds and then add in their extras directly into it.
Sorry for joining the discussion this late but I have a need to put something in Orbit but I won't until we get rid of CVS. I didn't see much discussion about having our own maven repository where we mirror the artifacts that are approved, and then using maven-bundle-plugin and tycho to create the p2 repo. Are there objections to that?
(In reply to Doug Schaefer from comment #43) > I didn't see much discussion about having our own maven repository where we > mirror the artifacts that are approved, and then using maven-bundle-plugin > and tycho to create the p2 repo. Are there objections to that? We do have an eclipse supported maven repo at https://repo.eclipse.org/ where projects can host their bundles as appropriate. It's already set up that deploying to it from an HIPP instances is pretty easy. Platform uses this to host our JDT compilers for use by tycho during our builds. PW
(In reply to Paul Webster from comment #44) > We do have an eclipse supported maven repo at https://repo.eclipse.org/ > where projects can host their bundles as appropriate. It's already set up > that deploying to it from an HIPP instances is pretty easy. It is just not worth using that repo for any project that intends for their artifacts to exist in maven central since dependencies on anything in there are unlikely to exist at those coordinates in maven central under the current plans (unless that changed since I last saw). Same for putting orbit stuff in there, fine for a cloistered build environment but not particularly beneficial or useful outside of the eclipse ecosystem, which is fine if that is all you are concerned about.
(In reply to Jesse McConnell from comment #45) > It is just not worth using that repo for any project that intends for their > artifacts to exist in maven central since dependencies on anything in there > are unlikely to exist at those coordinates in maven central under the > current plans (unless that changed since I last saw). Same for putting > orbit stuff in there, fine for a cloistered build environment but not > particularly beneficial or useful outside of the eclipse ecosystem, which is > fine if that is all you are concerned about. Yes, I agree. repo.eclipse.org solves the eclipse ecosystem problem, providing eclipse control over orbit bundles. I don't think it's the route for publishing our bundles to maven central, but it can have a similar level of responsibility to our CVS orbit repos. PW
(In reply to Jesse McConnell from comment #45) > (In reply to Paul Webster from comment #44) > > > We do have an eclipse supported maven repo at https://repo.eclipse.org/ > > where projects can host their bundles as appropriate. It's already set up > > that deploying to it from an HIPP instances is pretty easy. > > It is just not worth using that repo for any project that intends for their > artifacts to exist in maven central since dependencies on anything in there > are unlikely to exist at those coordinates in maven central under the > current plans (unless that changed since I last saw). Same for putting > orbit stuff in there, fine for a cloistered build environment but not > particularly beneficial or useful outside of the eclipse ecosystem, which is > fine if that is all you are concerned about. I'm OK with leaving the artifacts on maven central. The ones I want to use in CDT are there (ANTLR4 and Freemarker). I guess we'd need some confidence that the bits don't change. Maybe just checking the sha1 checksum would be sufficient. In that case, adding something to Orbit is as simple as providing the pom's to check that and to create bundles for them as necessary. That can easily fit in a git repo since there binaries live in other places. And that's pretty much what I do for our QNX things.
(In reply to Ian Bull from comment #17) > I've done something similar with git sub-modules. Each bundle would be a > sub-module and you can clone only a subset of the modules. As far as I know of Git and Orbit, submodules are what I believe would be the most efficient and natural approach > However, a > submodule is basically another git repository (linked together by pointers), > so the maintenance of this structure would likely be an issue. Maintaining git submodules is usually the exact same effort than maintaining map files, so it's also a valid replacement for map files. So, when using PDE Build, instead of populating your build workspace with map files, you first clone the composite repo, update submodules, and then populate the workspace with submodules that were fetched. After those steps all build process remains the same.
(In reply to Mickael Istria from comment #48) > As far as I know of Git and Orbit, submodules are what I believe would be > the most efficient and natural approach I have some vague recollection of having this discussion already. I don't recall the outcome. I do recall a lot of concern that submodules aren't a full-baked concept. Has that changed? All third-party code distributed by Eclipse projects should come from Eclipse sources. What we ship has to exactly match what's been approved. Further, we need to mitigate the risk of losing the ability to build should external sources become unreachable (we refer to this as the "freedom of action"). I like the idea of putting approved libraries into repo.eclipse.org in their original (non-OSGi-ified) form (i.e. exactly as they are approved by the IP Team); Maven will take care of versioning the binaries. The Orbit Git repository, then, contains only the build files (manifest, pom files, whatever) that know how to build the OSGi forms using the original-form JARs from repo.eclipse.org. I think that pushing to Maven Central may be a separate problem.
(In reply to Wayne Beaton from comment #49) > (In reply to Mickael Istria from comment #48) > > As far as I know of Git and Orbit, submodules are what I believe would be > > the most efficient and natural approach > > I have some vague recollection of having this discussion already. I don't > recall the outcome. I do recall a lot of concern that submodules aren't a > full-baked concept. Has that changed? Yes. I think they are accepted now as "solid" ... though, the proposed structure may not be (I'll reply next about that). > > All third-party code distributed by Eclipse projects should come from > Eclipse sources. What we ship has to exactly match what's been approved. > Further, we need to mitigate the risk of losing the ability to build should > external sources become unreachable (we refer to this as the "freedom of > action"). > > I like the idea of putting approved libraries into repo.eclipse.org in their > original (non-OSGi-ified) form (i.e. exactly as they are approved by the IP > Team); Maven will take care of versioning the binaries. The Orbit Git > repository, then, contains only the build files (manifest, pom files, > whatever) that know how to build the OSGi forms using the original-form JARs > from repo.eclipse.org. What is the advantage of that? They'd be unsigned, unconditioned for pack200, etc. Seems to make things more complicated to me, but perhaps I'm not sure what you mean. I believe one scenario that was mentioned, that some OSGi bundles already "live in" maven central, and we'd want those bundles "in Eclipse" (assuming approved "as is") without modification (except perhaps for conditioning and signing) ... so if those can go directory into repo.eclipse.org (after conditioning and signing) then that should be all that's required, right? No reason to get Orbit involved? There might be some sticky issues if "same bundle existed in both places" ... especially if slight differences ... but then we just say "can only be in one place, which ever comes first"? Though I'll add, we do have the concept in Orbit of a "pre-built" bundle ... one that is correct, as is, and we just store to make sure "we" always have it -- though that assume's it is "very correct" being conditioned, etc., so we can sign it. > I think that pushing to Maven Central may be a separate problem. Yep. Much different. (And, actually, in my mind, repo.eclipse.org is a separate issue too!)
(In reply to David Williams from comment #50) > > I like the idea of putting approved libraries into repo.eclipse.org in their > > original (non-OSGi-ified) form (i.e. exactly as they are approved by the IP > > Team); Maven will take care of versioning the binaries. The Orbit Git > > repository, then, contains only the build files (manifest, pom files, > > whatever) that know how to build the OSGi forms using the original-form JARs > > from repo.eclipse.org. > > What is the advantage of that? They'd be unsigned, unconditioned for > pack200, etc. Seems to make things more complicated to me, but perhaps I'm > not sure what you mean. I believe one scenario that was mentioned, that some > OSGi bundles already "live in" maven central, and we'd want those bundles > "in Eclipse" (assuming approved "as is") without modification (except > perhaps for conditioning and signing) ... so if those can go directory into > repo.eclipse.org (after conditioning and signing) then that should be all > that's required, right? No reason to get Orbit involved? There might be some > sticky issues if "same bundle existed in both places" ... especially if > slight differences ... but then we just say "can only be in one place, which > ever comes first"? Though I'll add, we do have the concept in Orbit of a > "pre-built" bundle ... one that is correct, as is, and we just store to make > sure "we" always have it -- though that assume's it is "very correct" being > conditioned, etc., so we can sign it. The advantage is that we have a local copy as close to the original as possible without having to check it into a source control system. And we have a lot of maven knowledge in the community at this point. Signing and conditioning would be done with the build scripts that turn these things into bundles for the p2 repository. In many cases, a lot of us are already doing that with out commercial products. Using the same systems would make it a lot easier to contribute these build scripts to Orbit.
And I think I sense your confusion David. We're talking about putting the jars into repo.eclipse.org. These jars are them turned in to bundles with the maven-bundle-plugin.
(In reply to Doug Schaefer from comment #51) > The advantage is that we have a local copy as close to the original as > possible without having to check it into a source control system. Exactly. Maven is quite good at managing library files. I think that this addresses the issue of Git not being particularly good for that purpose. repo.eclipse.org becomes the repository for the "raw" bundles which has the added benefit of making them available for consumption by non-OSGi projects. Orbit could optionally put the bundlized versions of the libraries back into repo.eclipse.org, but I don't see this as a priority (or necessarily desirable)
(In reply to Wayne Beaton from comment #53) > Orbit could optionally put the bundlized versions of the libraries back into > repo.eclipse.org, but I don't see this as a priority (or necessarily > desirable) Agreed, I don't think it's desirable. The current Orbit p2 repos are fine. Most of it ends up in the simrel repo anyway.
(In reply to Mickael Istria from comment #48) > (In reply to Ian Bull from comment #17) > > I've done something similar with git sub-modules. Each bundle would be a > > sub-module and you can clone only a subset of the modules. > > As far as I know of Git and Orbit, submodules are what I believe would be > the most efficient and natural approach > > > However, a > > submodule is basically another git repository (linked together by pointers), > > so the maintenance of this structure would likely be an issue. > > Maintaining git submodules is usually the exact same effort than maintaining > map files, so it's also a valid replacement for map files. So, when using > PDE Build, instead of populating your build workspace with map files, you > first clone the composite repo, update submodules, and then populate the > workspace with submodules that were fetched. > After those steps all build process remains the same. I think you are basically correct ... but ... a few things I don't like about your proposal, at least as you posted in the mailing list: https://dev.eclipse.org/mhonarc/lists/orbit-dev/msg03779.html Mostly that there's "no improvement", things just get more complicated ... and, I think, a whole lot larger for committers to "get the whole repo"? (Though, not sure, maybe committer work flow is the same for a "flat" repo vs one with submodules of submodules?) I was thinking more of "submodules of submodules" so committers could more easily get the "family" of packages they needed, some how. Plus, I'd prefer to design it from the beginning as being able to "auto tag" in some way, if a change had occurred, but otherwise avoid having to "rebuild everything" as we do now. While I know I do, I don't mean to sound negative, and appreciate you keeping the discussion going. Have you tried prototyping this? Are you volunteering to work on this? Or, did you just think no one has thought of this before and that the idea alone was all we needed?
(In reply to David Williams from comment #55) > [...] The idea of a single repo with all versions of all bundles doesn't sound bad neither and may indeed be easier. So what you're suggesting is something like: * orbit/ ** releng/ ** bundles/ *** javax.activation/ **** 1.1.0/... *** javax.annotaction **** 1.0.0/... **** 1.1.0/... Did I get it right? I'll try to find some time later to prototype it. However, I have to admit I'm busy with many things right now and moving Orbit to Git is far from being my #1 priority, so I don't promise anything for a near future ;)
(In reply to Mickael Istria from comment #56) > (In reply to David Williams from comment #55) > > [...] > > The idea of a single repo with all versions of all bundles doesn't sound bad > neither and may indeed be easier. > So what you're suggesting is something like: > * orbit/ > ** releng/ > ** bundles/ > *** javax.activation/ > **** 1.1.0/... > *** javax.annotaction > **** 1.0.0/... > **** 1.1.0/... > > Did I get it right? Yes, similar, I was thinking more like * orbit/ ** plugins/ *** javax.activation/ **** javax.activation_1.1.0 *** javax.annotation **** javax.annotation_1.0.0 **** javax.annotation_1.1.0 > I'll try to find some time later to prototype it. However, I have to admit > I'm busy with many things right now and moving Orbit to Git is far from > being my #1 priority, so I don't promise anything for a near future ;) Understood. Sounds familiar.
(In reply to Doug Schaefer from comment #52) > And I think I sense your confusion David. We're talking about putting the > jars into repo.eclipse.org. These jars are them turned in to bundles with > the maven-bundle-plugin. You're close ... my confusion was if you have a "maven jar" and then "made an OSGi" bundle from it ... then I was assuming you had it all, in one jar, in one repo.eclipse. But, now I recall those discussions where people said "oh, no, you can't ever do that" ... which, technically isn't true ... but, culturally might be.
(In reply to David Williams from comment #58) > (In reply to Doug Schaefer from comment #52) > > And I think I sense your confusion David. We're talking about putting the > > jars into repo.eclipse.org. These jars are them turned in to bundles with > > the maven-bundle-plugin. > > You're close ... my confusion was if you have a "maven jar" and then "made > an OSGi" bundle from it ... then I was assuming you had it all, in one jar, > in one repo.eclipse. But, now I recall those discussions where people said > "oh, no, you can't ever do that" ... which, technically isn't true ... but, > culturally might be. Agreed. Bundles in maven repos just seem weird. I'd rather have them in p2 repos. Not sure why :).
(In reply to Doug Schaefer from comment #59) > (In reply to David Williams from comment #58) > > (In reply to Doug Schaefer from comment #52) > > > And I think I sense your confusion David. We're talking about putting the > > > jars into repo.eclipse.org. These jars are them turned in to bundles with > > > the maven-bundle-plugin. > > > > You're close ... my confusion was if you have a "maven jar" and then "made > > an OSGi" bundle from it ... then I was assuming you had it all, in one jar, > > in one repo.eclipse. But, now I recall those discussions where people said > > "oh, no, you can't ever do that" ... which, technically isn't true ... but, > > culturally might be. > > Agreed. Bundles in maven repos just seem weird. I'd rather have them in p2 > repos. Not sure why :). Wait, maybe I do know why. A lot of us put Orbit into our PDE target files and it doesn't know how to pull bundles from maven repos.
At any rate, my suggestion is to not check the jar files into any Source control management system which would make the discussion about git submodules mute.
I'm late in the game. Sorry. I was away on vacation. May I suggest to re-use the technology built for the EBR project? Actually, I do think it's possible to even re-use the recipes. At least, I'm interested in making this happen. The process would be as follows: Bundle recipe (bnd and pom file) -> SCM -> CI SCM Trigger -> build -> build-site build: - pulls approved jar from Maven central - applies bnd file to create OSGi metadata - runs Tycho to create p2 metadata - copies/uploads artifact using a dedicated group id to a Maven repo (can be repo.eclipse.org, can be Maven central, can be on local disc only) (suggested group id: org.eclipse.orbit.bundles) build-site[1]: - publishes Orbit artifacts from Maven repo to a p2 update site (for consumption by p2) [1] - must be a separate reactor build
(In reply to Gunnar Wagenknecht from comment #62) > - applies bnd file to create OSGi metadata Any reason why not directly writing a MANIFEST.MF ?
(In reply to Mickael Istria from comment #63) > (In reply to Gunnar Wagenknecht from comment #62) > > - applies bnd file to create OSGi metadata > > Any reason why not directly writing a MANIFEST.MF ? There are a few. Writing MANIFEST.MF file is hard. Bnd does things like usage calculations, import calculations, etc. Those are easy to miss when updating a library (may have new or less imports in future versions). PDE does not calculate those correctly. You can actually specify patterns and rules (eg., version ranges for import) which Bnd will apply correctly. Basically, a lot things I had to do manually when update a library in Orbit are obsolete this way.
(In reply to Gunnar Wagenknecht from comment #62) > - pulls approved jar from Maven central Except for this part, +1 Pulling from anywhere but repo.eclipse.org will require buy-in from the IP Team. I'm game to help assemble our case.
(In reply to Gunnar Wagenknecht from comment #62) > I'm late in the game. Sorry. I was away on vacation. > > May I suggest to re-use the technology built for the EBR project? Actually, > I do think it's possible to even re-use the recipes. At least, I'm > interested in making this happen. Sorry Gunnar, but I haven't looked at the EBR project. I was just wondering how it's different than the two step build (maven-bundle-plugin, then tycho eclipse-repository) documented here: https://wiki.eclipse.org/Tycho/How_Tos/Dependency_on_pom-first_artifacts I was just able to set up a build for freemarker this way that we are using internally and would like to start using in CDT.
(In reply to Doug Schaefer from comment #66) > I was just able to set up a build for freemarker this way that we are using > internally and would like to start using in CDT. I would say it is basically the process described there but with added infrastructure around it. For example, the create-recipe mojo simplifies creation of recipes. It bootstraps a recipe pom.xml with information suitable for Eclipse. In addition to that it bootstraps bundle.properties, about.html and about_files with license information.
Hi Eclipse team, I'm sorry to enter in the conversation, but I wanted to tell my point of view on the subject. The first part will be a critisism of the actual way of doing at Eclipse, please don't be shocked by it, I'm not angry, I'm just trying to explain my feelings on this. First of all, please take a strong care on the way you will implement the source/binaries repo, this is the main reason for what I'm not an active Eclipse contributor/patcher/commiter (even if I work for the Eclipse ecosystem) an the same for what I will never use Eclipse Gemini/Eclipselink/ECF in my OSGI projects (and that's just some examples). Here's the why: * Concerning source code, Eclipse internal git is not so visible, and it's very hard to find the project you want to see code for. Forking it is a nightmare (I can't have a distant free empty git repo on Eclipse to fork/pull my patch... * Concerning the OSGI bundle distribution repo (orbit), it's just not google referenced and unusable as it is: for example, I tried to use Gemini some years ago so I referenced the Orbit repo then Gemini-blueprint as a dependency. Unfortunately, Gemini depends itself on equinox-v3.6whatever that were not present on the Orbit maven repo... I finally ended up trashing Gemini, Virgo, eclipselink and more generally all Eclipse stuff for their Apache counterparts (Aries, openjpa...) even if I would at the beginning become an Eclipse contributor instead an Apache PMC. The second part is a critisism on this discussion (please read until the end, the final part will be on the possible solutions to improve thing IMHO). For the second time, please don't take this personally, this is just my vision of things, I can be wrong, but I've a strong background as an applicative architect/DevOps/OSS dev. * Git is definitely the way to go, please drop CVS/Subversion whatever that's not a DCVS, this is just not appropriate for an OSS community. * Git is a SOURCE control that allows BRANCHES, so NEVER put any binary in it is not deserved to this: most of cvs persists version in term of diff between the N-1 and the Nth version and two binaries are totally different so you'll end up with Gigabytes of loss and a strong coupling between your sources and your binaries. * Don't create a source tree containing different versions of your sources: branches are made for this, and this is totally orthogonal with the concern of working on sources. * P2 is not supported everywhere so please use remote binary management (Nexus/Archiva): I can't reference a P2 repo with Karaf, JBOSS, WebSphere, Glassfish, Tomcat, Maven (Jetty,...) project, and even with a pure OSGI one (P2 is not an OBR), so please, never forget that P2 is only true for Eclipse, and if you want to wider your plugins (e.g. all plugins that do not contain a mandatory plugin.xml file) remember it! So now (finally), let's me explain the heaven that will makes me an Eclipse power user/early adopter/commiter/active contributor on MOST of the Eclipse project. * Every Eclipse project entrypoint should be on Github! So as an Eclipse user I could easily fork and patch that f***g GMF bug easily. I think we can admit that every OSS devs have a Github account (not an Eclipse CLA approved one). It's so simple to search, fork, patch then pull request a project. * * Don't forget that Github is a <bold>D</bold>CVS so you can have a super-remote repository hosted at Eclipse for your IP reviews/distribution management. If you're the Github PMC, just make a pull request from your master Github repo to the Eclipse one, your IP can then review ONLY the DIFF between the precedent version you sent to them. Concerning the source management: no binaries, no versions in the source tree, just a branch for each released versions (automatically generated by Eclipse Hudson) and each Bugzilla tickets in progress. An example for the preceding workflow: * As Charlie, I've an enhancement idea on GMF (for sure I've some). * I create a ticket on this Bugzilla discussing on the value of this improvement. * While accepted, I just fork the Github repo on my account, create a branch then activate Mylyn... Then push to my fork and finally make a pull request on the GMF main Github repo. * Once my pull request is accepted, I can attach my mylyn context on the ticket and reference the commit id on the now resolved ticket. * Once GMF guys want to prepare a release, they make a pull request from the Github remote to the 'central' eclipse Git repo so that IP can review the pull req for acceptation. Now, let's move on on the workflow. * Once the IP accepts the pull request, the code is merged on the main 'Eclipse central DCVS repo' so that Eclipse(tm) Hudson CI instance can trigger the change and start to build it. * * As a side note, please force your project contributors to use one automatable build chain: Maven, EGF-Build/buckminster, Ant+Ivy, Graddle... So that every sources can be built and distributed to any of your Eclipse binary repo. At this point, CI is a very important thing and Eclipse can have a nice workflow: * Build can be triggered automatically * Javadoc can be generated (see mvn-jvdoc-plugin) * License header can be added (mvn-license-pg) * Manifest gen can be automated (mvn-bnd-pg): as a sidenote, I don't see why you need to have some non-manifested plugin, pojar just doesn't take care of it * Utests and itests can be executed then covered * Quality reports can be generated and build can fail if some quality is not reached * Finally binaries can delivered to a binary remote repo or applications can be deployed in production. Concerning the remote distribution management: * Nexus and Artifactory are creating P2 metadata so everything you'll deploy on it will automatically generate an update site. * I would prefer every Eclipse release bundle deployed on Maven Central and snapshots on maven.eclipse.. but I'm not totally against a pure Eclipse release repo (release.maven.eclipse in opposition to snapshots.maven.eclipse). * Sonatype Maven allows repository grouping so you can easily group/mirror every Eclipse bundle on central and vice-versa. Finally, as a developer, even if my bundle are strong or tight coupled, I just want to reference remote deployed bundles as dependencies in my Maven conf (pom.xml), Ivy descriptor, Graddle build, EGF chain. As a side note, here are some real life implementation of what I said: * org.apache.servicemix bundles are just manifest wrapped pojars deployed on Central and community managed on Github (with the ASF git masters in the middle). * Osgiliath Enterprise Framework is a JEE server made with most of these libraries and hosted on Github: I made it on my out-work hours and it was not a pain to maintain, improved, contribute to underlying Apache bundles... I hope I helped and that it will makes things goes the right way. Best regards to that wonderfull Eclipse community. Charlie
Sorry for the precedent mail, I was a little off subject (not for all, but mainly). Here's a sample of bnd OSGIfying with Maven: https://github.com/OsgiliathEnterprise/net.osgiliath.parent/blob/master/net.osgiliath.framework/net.osgiliath.helpers.parent/net.osgiliath.wrappers/net.osgiliath.wrapper.derby/pom.xml Regards,
(In reply to Charlie Mordant from comment #69) > Sorry for the precedent mail, I was a little off subject (not for all, but > mainly). Don't apologize. We need that sort of feedback.
(In reply to Wayne Beaton from comment #70) > (In reply to Charlie Mordant from comment #69) > > Sorry for the precedent mail, I was a little off subject (not for all, but > > mainly). > > Don't apologize. We need that sort of feedback. Well, I was going to say, to Charlie, "thanks for saying that" (that it was off subject) since I was trying to find a polite way of saying that ... without sounding like we weren't interested in hearing your feedback. While I'm glad you found one place to give it, not sure where a more meaningful place might be. Of course, most of it we've heard before ... though honestly, it's so long, I'll have to re-read it several times to be sure I'm understanding all of it. But, thanks.
(In reply to Gunnar Wagenknecht from comment #62) > I'm late in the game. Sorry. I was away on vacation. > > May I suggest to re-use the technology built for the EBR project? Actually, > I do think it's possible to even re-use the recipes. At least, I'm > interested in making this happen. > > The process would be as follows: > > Bundle recipe (bnd and pom file) -> SCM -> CI SCM Trigger -> build -> > build-site > > build: > - pulls approved jar from Maven central > - applies bnd file to create OSGi metadata > - runs Tycho to create p2 metadata > - copies/uploads artifact using a dedicated group id to a Maven repo > (can be repo.eclipse.org, can be Maven central, can be on local disc only) > (suggested group id: org.eclipse.orbit.bundles) > > build-site[1]: > - publishes Orbit artifacts from Maven repo to a p2 update site > (for consumption by p2) > > [1] - must be a separate reactor build Sorry I've not replied sooner. I was hoping to find time to understand some of what you say better, and to give detailed questions on things I didn't understand .... but, since I've not found that time, I'll at least say I think there is merit to thinking through some of this and it may well "work" and be the future of Orbit. I am not sure it "depends on moving to Git" (the subject of this bug) but if it solves that problem as a side-effect, that's fine. One over-riding comment: especially since I don't understand several of the technologies you mention, for this to ever "be real", you, or someone else, will have to take over being responsible for "the Orbit builds". I'm not opposed to learning new things ... but ... I've done it long enough anyway, so seems an expert should be the one to educate us all, and drive change in Orbit. Not sure if this is what you meant by "I'm interested in making this happen" ... so just thought I'd be explicit that is what it means to me. Some other questions ... (some of which may seem silly, to those of you that understand bnd and maven better than I do) ... - you are still assuming we don't "rebuild from 3rd party source", right? Or ... am I wrong on that? - what is a jar we want is not in Maven central? That doesn't fundamentally prevent your approach, right? - does your approach (and/or bnd) assume that "all prereqs are available" to correctly resolve things, correct compute "uses", etc. I would think so, but just wanted to be sure we were not losing that. [Though, obviously, there's pros and cons to both ways, I guess.] - But, at same time, k I'm guessing we could (still) figure out a way to not necessarily "rebuild everything" as we currently do? Would be good to get away from that. - I assume that inserting any sort of "Eclipse-SourceReferences" becomes impossible or meaningless in your system? Or am I assuming too much? Seeing something wrong? - I assume we'd still have qualifiers that would only change if contents changed? Not every build, right? - I'm not sure about the group-id: org.eclipse.orbit.bundles. That doesn't seem to be the way I've seen "group id" defined or used ... but ... not sure I know enough to say much with any authority. It's just that I've always thought we want to do things in a way that "the original third party" could do it the same way if/when they wanted to ... and that'd be one thing that'd be "off" from that goal. Probably a minor issue ... but would want to be sure we all understand how that effect consumption, etc. - It's unclear in what you describe (though suspect you just dint' bother including it) how we achieve "iteration cycles". Seems fundamental that we need the ability to have a "temporary" version, that people can test, and confirm all is well, before we "release" a repository in a "released" project or product. So, I'd like to hear more about that. - I guess we could not convert what we have so in Orbit. That we'd just have to start it "its final p2 repo" and then build on that ... and if someone needed something fixed/changed, we've basically just "start over" on that older version, to put back in current repo? - Have you a prototype of this system? That's all the questions I have patience for right now ... but, am sure I will think of others. Thanks very much for your constructive suggestions.
(In reply to David Williams from comment #72) > One over-riding comment: especially since I don't understand several of the > technologies you mention, for this to ever "be real", you, or someone else, > will have to take over being responsible for "the Orbit builds". I'm not > opposed to learning new things ... but ... I've done it long enough anyway, > so seems an expert should be the one to educate us all, and drive change in > Orbit. Not sure if this is what you meant by "I'm interested in making this > happen" ... so just thought I'd be explicit that is what it means to me. Agreed and your interpreting is correct. > - you are still assuming we don't "rebuild from 3rd party source", right? Or > ... am I wrong on that? Right. It is re-packaging from 3rd party binary. > - what is a jar we want is not in Maven central? That doesn't fundamentally > prevent your approach, right? It has to be in a Maven repo. The way it could work is that an Orbit committer (or another authority) uploads it to repo.eclipse.org/ip-approved-whatever first. > - does your approach (and/or bnd) assume that "all prereqs are available" to > correctly resolve things, correct compute "uses", etc. I would think so, but > just wanted to be sure we were not losing that. [Though, obviously, there's > pros and cons to both ways, I guess.] It does not require to have all prereqs available. AFAIK bnd scans the class files. If prereqs are available _and_ already provide OSGi metadata, then bnd will use it for detecting package versions. Otherwise it relies on the information specified as rules in the bnd file. > - But, at same time, k I'm guessing we could (still) figure out a way to not > necessarily "rebuild everything" as we currently do? Would be good to get > away from that. One can build individual recipes at any time. But setting this up in Hudson is a different story. Building just the parent aggregator is easier. > - I assume that inserting any sort of "Eclipse-SourceReferences" becomes > impossible or meaningless in your system? Or am I assuming too much? Seeing > something wrong? We could still insert one. The question is: what should it point to? > - I assume we'd still have qualifiers that would only change if contents > changed? Not every build, right? Right. Qualifiers are computed by Tycho, which includes qualifiers based on stable Git timestamps. > - I'm not sure about the group-id: org.eclipse.orbit.bundles. That doesn't > seem to be the way I've seen "group id" defined or used ... but ... not sure > I know enough to say much with any authority. It's just that I've always > thought we want to do things in a way that "the original third party" could > do it the same way if/when they wanted to ... and that'd be one thing that'd > be "off" from that goal. Probably a minor issue ... but would want to be > sure we all understand how that effect consumption, etc. I see the Maven group id like the vendor attribute in OSGi. Publishing under the original group id won't work (because an artifact in the same version is already deployed there and also because I think it's wrong). But that's just my take. Would like to here other opinions. > - It's unclear in what you describe (though suspect you just dint' bother > including it) how we achieve "iteration cycles". Seems fundamental that we > need the ability to have a "temporary" version, that people can test, and > confirm all is well, before we "release" a repository in a "released" > project or product. So, I'd like to hear more about that. That part of the story isn't written yet. But the intention is to have it similar to what is in Orbit now. I think it's even possible to produce the same p2 output structure from an orbit point of view, i.e. we'll have I and R builds, each with a p2 repo published in its own directory. > - I guess we could not convert what we have so in Orbit. That we'd just have > to start it "its final p2 repo" and then build on that ... and if someone > needed something fixed/changed, we've basically just "start over" on that > older version, to put back in current repo? It's not automated at this point. Thus, a lazy-manual approach is our only option right now. > - Have you a prototype of this system? Yes. With the source being at Eclipse.org ([1]) projects but the actual build system being outside using free services ([2]) because we con't produce binary artifacts in EBR. It uses a Travis build ([4]). The produced Maven artifacts are pushed to JFrog [3]. My idea was to publish p2 repos on Bintray. But they recently put on a 500MB limit on the packages which makes it impossible for Orbit p2 repos. I also have a prototype running using Hudson and local Maven repos on my local machine. That's the setup I could imagine for Orbit. [1] http://git.eclipse.org/c/ebr/org.eclipse.ebr.recipes.git/ [2] https://github.com/runolarity/osgi-bundles [3] http://oss.jfrog.org/simple/oss-snapshot-local/com/runolarity/osgi/bundles/ [4] https://travis-ci.org/runolarity/osgi-bundles
(In reply to Gunnar Wagenknecht from comment #73) > > - Have you a prototype of this system? > > Yes. With the source being at Eclipse.org ([1]) projects but the actual > build system being outside using free services ([2]) because we con't > produce binary artifacts in EBR. It uses a Travis build ([4]). The produced > Maven artifacts are pushed to JFrog [3]. My idea was to publish p2 repos on > Bintray. But they recently put on a 500MB limit on the packages which makes > it impossible for Orbit p2 repos. > > I also have a prototype running using Hudson and local Maven repos on my > local machine. That's the setup I could imagine for Orbit. Just a suggestion ... but perhaps you could request a "HIPP instance for Orbit" ... and do some prototype work there? (Not sure what you'd use for the "maven prototype repos" ... but ... am sure Eclipse Foundation would be understanding and helpful if they could be). If things work out, and you get "buy in" the prototypes would eventually become "real builds" and if not for some reason, wouldn't hurt to use a HIPP instance for current build. I was thinking of doing that sometime, but sounds like you are pretty far along, so with luck, I won't need to. Just to highlight one concern, you say jar "has to be in a maven repo first". I'm not sure that fits that many of our current 3rd party bundles, does it? Maybe 50%? (If nothing else, as you know, often we provide the jar without one or two "pieces" which may not have the correct pedigree). So, end the end, you'd have to provide a way to do "builds" of binary artifacts that produce a maven artifact? Right? At least for some cases, even if other jars were already in a maven repo and more of a "copy"? [Plus, even if we get "from maven central", then we are not really "getting" directly from the 3rd party source ... I know of no reason that's not ok ... but ... somehow seems like something to stay aware of (and be sure others are aware of it)]. Thanks again!
(In reply to David Williams from comment #74) > (In reply to Gunnar Wagenknecht from comment #73) > Oh, and one other thought ... maybe you could educate me on. One reason we put "binary artifacts" in a CVS was for "version control" -- high degree of assurance that a revision would not change, could be retrieved for LTS, etc. Do we get that same degree of "version control" in this system you are talking about? I think we would if the maven artifacts used "4 part versioning" ... but ... not sure we would if we have 3 parts only, or 3 parts plus "SNAPSHOT", etc. ... so, maybe something else to define well? Put another way, make sure the p2's repository and maven repo artifacts were always one-one mappable, even for I-builds, not to mention milestones, and of course releases would be easy ... but not sure it's obvious how to do for I-builds.
Created attachment 245158 [details] two stage maven/tycho creation for org.freemarker As an example to show concretely what I'm suggesting, I've attached the scripts and poms and such that I use to create a p2 site for org.freemarker. It pulls it out of maven central, creates the bundle using the maven-bundle-plugin, and then creates the repo using tycho.
(In reply to David Williams from comment #75) > Oh, and one other thought ... maybe you could educate me on. One reason we > put "binary artifacts" in a CVS was for "version control" -- high degree of > assurance that a revision would not change, could be retrieved for LTS, etc. > Do we get that same degree of "version control" in this system you are > talking about? I think we would if the maven artifacts used "4 part > versioning" ... but ... not sure we would if we have 3 parts only, or 3 > parts plus "SNAPSHOT", etc. ... so, maybe something else to define well? Put > another way, make sure the p2's repository and maven repo artifacts were > always one-one mappable, even for I-builds, not to mention milestones, and > of course releases would be easy ... but not sure it's obvious how to do for > I-builds. If you look at a maven repository from a certain angle, it is a version control system. repo.eclipse.org should suite that fine.
(In reply to Doug Schaefer from comment #76) > Created attachment 245158 [details] > two stage maven/tycho creation for org.freemarker > > As an example to show concretely what I'm suggesting, I've attached the > scripts and poms and such that I use to create a p2 site for org.freemarker. > It pulls it out of maven central, creates the bundle using the > maven-bundle-plugin, and then creates the repo using tycho. Any reason you don't use p2-maven-plugin, which takes it one step further and generates the p2 repo for you, with out using tycho buggy consider pom dependencies? https://github.com/reficio/p2-maven-plugin
(In reply to David Carver from comment #78) > (In reply to Doug Schaefer from comment #76) > > Created attachment 245158 [details] > > two stage maven/tycho creation for org.freemarker > > > > As an example to show concretely what I'm suggesting, I've attached the > > scripts and poms and such that I use to create a p2 site for org.freemarker. > > It pulls it out of maven central, creates the bundle using the > > maven-bundle-plugin, and then creates the repo using tycho. > > Any reason you don't use p2-maven-plugin, which takes it one step further > and generates the p2 repo for you, with out using tycho buggy consider pom > dependencies? > > https://github.com/reficio/p2-maven-plugin Other than I'm not familiar with it? Does it create one repo per jar file? At the end of the day, we want one repo for all of Orbit. Although I guess we could add a mirror step to the build to merge all the repos.
(In reply to Doug Schaefer from comment #79) > (In reply to David Carver from comment #78) > > (In reply to Doug Schaefer from comment #76) > > > Created attachment 245158 [details] > > > two stage maven/tycho creation for org.freemarker > > > > > > As an example to show concretely what I'm suggesting, I've attached the > > > scripts and poms and such that I use to create a p2 site for org.freemarker. > > > It pulls it out of maven central, creates the bundle using the > > > maven-bundle-plugin, and then creates the repo using tycho. > > > > Any reason you don't use p2-maven-plugin, which takes it one step further > > and generates the p2 repo for you, with out using tycho buggy consider pom > > dependencies? > > > > https://github.com/reficio/p2-maven-plugin > > Other than I'm not familiar with it? Does it create one repo per jar file? > At the end of the day, we want one repo for all of Orbit. Although I guess > we could add a mirror step to the build to merge all the repos. Sigh...my talk a couple years ago, went unheard. :) No you can configure it to take all the maven central artificats you specify in its configuration, and it'll create one p2 repo with everything. No need to do a mirror. http://eclipsecon.org/2013/sessions/real-world-tycho The PDF there contains the presentation I did around p2-maven-plugin... You were even in the audience for that presentation. :)
There is now a prototype of the Orbit proposal on Git running at Eclipse.org. The Orbit Git repository is here: -> https://git.eclipse.org/c/orbit/orbit-recipes.git/ There is now also a HIPP setup for Orbit with a job that currently builds a p2 repository: -> https://hudson.eclipse.org/orbit/job/orbit-recipes/ The technology is based on EBR: -> https://github.com/eclipse/ebr/ There are a few changes from the current CVS setup. For example, the ip_log.xml file is now maintained within the recipe itself. There is still a feature.xml but we might be able to get rid of it. At this point, the p2 repository is the final build output. If the individual downloads are still needed, we could generate the download html from the p2 repository linking directly to artifacts (jar and source).
(In reply to Gunnar Wagenknecht from comment #81) Thanks for your work in this area, Gunnar. Orbit committers and interested parties ... we'll be having more discussion, if not "phone meetings", over the weeks or months ahead, but in the mean time, wanted to be explicit we are not proposing a change before Mars release, this close to the end of Mars dev. cycle. If everything can be worked out, though, I'd anticipate a change "right after" the Mars release. I know I need to study and catch up with Gunnar's work, and suggest others do the same so you can voice any concerns or lend your support. Also, if we do change processes after Mars release, I think that makes it all the more important we need to have the Mars release p2 repo in pristine condition, which might include some wide spread "touches" for bugs, such as bug 450222 and bug 464100. Thanks,
(In reply to Gunnar Wagenknecht from comment #81) > At this point, the p2 repository is the final build output. If the > individual downloads are still needed, we could generate the download html > from the p2 repository linking directly to artifacts (jar and source). I do not think we need the individual downloads ... but, the community might tell us differently? I was thinking it would be important to have a listing of CQ's and packages, since projects had to enter a "piggy back" CQ to say they were using a CQ from Orbit. BUT, I THINK that is one of the things that is changing about IP process. Wayne, can you verify/confirm what is needed from Orbit, in terms of CQs?
(In reply to Gunnar Wagenknecht from comment #81) > There is now a prototype of the Orbit proposal on Git running at Eclipse.org. > > The Orbit Git repository is here: > -> https://git.eclipse.org/c/orbit/orbit-recipes.git/ > > There is now also a HIPP setup for Orbit with a job that currently builds a > p2 repository: > -> https://hudson.eclipse.org/orbit/job/orbit-recipes/ > From just glancing at the prototype, one thing I don't like, is having "orbit" in the bundle name. There's several reasons I don't like it, but thought I'd just say that much to begin with, and ask if that can be changed? Also, I've not literally gone through all the steps, but I think the "original jars" are not under version control (i.e. stored somewhere reliable) at eclipse.org. Can anyone confirm? And, again, can that be changed? I know both of these issues my have pros and cons, if not be outright controversial, so wanted to begin discussion.
(In reply to Gunnar Wagenknecht from comment #81) > There is now a prototype of the Orbit proposal on Git running at Eclipse.org. A frequent question is "do all the existing bundles in Orbit have to be converted" to the new system. And, the answer is "no". The current thinking is that we'd provide something like a composite repo, consisting of "last release" of "old method" Orbit repository, combined with new repository produced by new method. Anything new, would be done with "new system", of course. (Once we "switch"). If/when any old, existing bundle in Orbit needed to be *changed* then it would have to be "re-done" in new system.
This approach of compositing "old" (CVS/PDE Build output) and "new" (Git/EBR output) and to only allow new content to use Git/EBR seems very good to me.
(In reply to Mickael Istria from comment #86) > This approach of compositing "old" (CVS/PDE Build output) and "new" (Git/EBR > output) and to only allow new content to use Git/EBR seems very good to me. +1
Moving target, since we didn't make any widespread transition for SR1, though, appreciate those of you who have tried the new method. Let's start a list of "things to do" ... 1) Is there agreement that "orbit" should not be part of the bundle name? As one source of support for not having it that way, is this blog entry from an OSGi expert: http://blog.osgi.org/2015/08/release-6-of-osgi-compendium-osgi.html?showComment=1439468504763#c7602007936621922721 While he was specifically talking about maven artifacts in the blog, it seemed clear he thought "... the artifact id to be identical to the Bundle-SymbolicName of the jar". 2) Settle the issue of "where the source jar comes from". I feel strongly it needs to be in some sort of version controlled system at Eclipse.org. My concern is many years of LTS (even decades!). It would be good to be reassuring to anyone that we could reproduce the bundle, with some change in manifest, or signature, for example. I've heard it said "well, if it needs to be re-created, we do have a bundle we previously produced in our own Orbit p2 repositories. BUT, the problem with that, is that once signed (and conditioned by pack200) there is no "going back". Its not the signature that is the problem, but it is the pack200 conditioning. I've seen real life cases of this when we moved up our signing service from Java 6 to Java 8 last year. Many of the "old bundles" can not be re-conditioned, whereas if we used the original jar, we could. So, besides CVS :) where can we keep these version controlled original jars? Have a "binary repository" in Git? Say with each jar in its own directory, with a name of <symbolic-id>_<original-version>. I mention this "separate repo" since it seems that usually committers would only rarely have to work that that repository, so main-stream work, making branches, or what ever, would still be easy). 3) "We" need to implement conditioning and signing for the new build. (Last I looked, they were not signed.) 4) "We" do need some list of bundles and associated CQ's. While the Eclipse Foundation's goal is to automate some of that (instead of project's opening a piggy back CQ) that work isn't done yet, and even when done, they will need some way of "looking up" the CQ for a given version of a given bundle. (I've seen "ip log files" in some of the "source directories" I've looked at, but am not aware if that work is documented, or settled). 5) I believe we need to change our process of "committership" a little. Not so much "becoming a committer", but when to remove someone. Not sure we ever specified that, for our project, but seems this is a good time. Something like, even now, "anyone who has not committed in past year, get's automatically dropped, and must re-apply if they want to commit again". My main concern with current system, is we have a lot of bundles, with someone named as "contact", and some of those people are no longer interested in committing fixes in Orbit, perhaps not even Eclipse, and in some cases, they might not even be coding any longer! So, removing a committer listed as a "contact", would change the "contact" to "no one" (or, some similar wording). What do others think? 6) How does current build work? I'd like it to work so that "once a bundle is produced, is is no longer produced each build". (Or, at least have some strict limit ... if nothing else, perhaps "once released"). Or, perhaps better, "only if Git shows a change since last build tag"? 7 ... ? ) What else?
(In reply to David Williams from comment #88) > 1) Is there agreement that "orbit" should not be part of the bundle name? See here for a related discussion: https://dev.eclipse.org/mhonarc/lists/ebr-dev/msg00016.html I bought into the argument of easier identifying the source of a bundle (jar) when looking into the file system. I think it's helpful to easily identify bundles coming from Orbit. > While he was specifically talking about maven artifacts in the blog, it > seemed clear he thought "... the artifact id to be identical to the > Bundle-SymbolicName of the jar". I'm not sure that is related. BJ talks about aligning the Maven artifact if with the BSN. That's not an issue for Orbit recipes. We already have Maven artifactId == BSN == jar file name. > 2) Settle the issue of "where the source jar comes from". I feel strongly it > needs to be in some sort of version controlled system at Eclipse.org. My > concern is many years of LTS (even decades!). It would be good to be > reassuring to anyone that we could reproduce the bundle, with some change in > manifest, or signature, for example. Wayne and I talked about having a Maven repository at Eclipse.org as source of truth containing all the original, IP approved jars. > 3) "We" need to implement conditioning and signing for the new build. (Last > I looked, they were not signed.) added to the TODOs https://wiki.eclipse.org/Orbit/Transition_to_Git_Plan > 4) "We" do need some list of bundles and associated CQ's. While the Eclipse > Foundation's goal is to automate some of that (instead of project's opening > a piggy back CQ) that work isn't done yet, and even when done, they will > need some way of "looking up" the CQ for a given version of a given bundle. > (I've seen "ip log files" in some of the "source directories" I've looked > at, but am not aware if that work is documented, or settled). added to the TODOs https://wiki.eclipse.org/Orbit/Transition_to_Git_Plan > 5) I believe we need to change our process of "committership" a little. Not > so much "becoming a committer", but when to remove someone. Not sure we ever > specified that, for our project, but seems this is a good time. Something > like, even now, "anyone who has not committed in past year, get's > automatically dropped, and must re-apply if they want to commit again". My > main concern with current system, is we have a lot of bundles, with someone > named as "contact", and some of those people are no longer interested in > committing fixes in Orbit, perhaps not even Eclipse, and in some cases, they > might not even be coding any longer! So, removing a committer listed as a > "contact", would change the "contact" to "no one" (or, some similar > wording). What do others think? I agree that it might be about time to have a process for inactive committers. Maybe we should review the "contact" information thing. It's easy to get stale. Keep in mind that we now have Gerrit. Thus, in theory there is no need to grant committership first. People can submit reviews at any time. However, it might still be good to have someone as committer who feels ownership for (a selection of) recipes. > 6) How does current build work? I'd like it to work so that "once a bundle > is produced, is is no longer produced each build". (Or, at least have some > strict limit ... if nothing else, perhaps "once released"). Or, perhaps > better, "only if Git shows a change since last build tag"? It doesn't work that way. The Maven Tycho build does not work incrementally. What we could do (as a workaround) is to add a final mirroring step which will only mirror those bundles into the final target p2 repository which actually changed (using a comparator during mirroring). Added to the TODOs https://wiki.eclipse.org/Orbit/Transition_to_Git_Plan
(In reply to Gunnar Wagenknecht from comment #89) > (In reply to David Williams from comment #88) > > 1) Is there agreement that "orbit" should not be part of the bundle name? > > See here for a related discussion: > https://dev.eclipse.org/mhonarc/lists/ebr-dev/msg00016.html > > I bought into the argument of easier identifying the source of a bundle > (jar) when looking into the file system. I think it's helpful to easily > identify bundles coming from Orbit. > You mean for humans scanning a list, right? There's other ways for machines to do it. Even other ways for OSGi to do it, if anyone felt it was needed as a runtime constraint. But, I think it looks tacky .. and would require people to have to "change their code" for very little reason (when they use 'require bundle). Such as when moving from an "old" version to a "newer" version. I think that's almost a non-starter ... but, I'll ask around. > > > While he was specifically talking about maven artifacts in the blog, it > > seemed clear he thought "... the artifact id to be identical to the > > Bundle-SymbolicName of the jar". > > I'm not sure that is related. BJ talks about aligning the Maven artifact if > with the BSN. That's not an issue for Orbit recipes. We already have Maven > artifactId == BSN == jar file name. So, are you saying we change the bundle symbolic name, from what the third party called it? (In those cases where they already have an OSGi bundle). That doesn't seem right. > > > 2) Settle the issue of "where the source jar comes from". I feel strongly it > > needs to be in some sort of version controlled system at Eclipse.org. My > > concern is many years of LTS (even decades!). It would be good to be > > reassuring to anyone that we could reproduce the bundle, with some change in > > manifest, or signature, for example. > > Wayne and I talked about having a Maven repository at Eclipse.org as source > of truth containing all the original, IP approved jars. Doesn't this imply the jars have to be "mavenized" first, if not already? That seems like extra work. (And, I'm not saying it's not the least of all evils, just doesn't seem quite right, to make a mavenized version for long term storage, then an OSGI version (and, I'd assume, with BND, a different mavenized version too? The first mavenized version would still have to be "raw" (unsigned, unconditioned, so wouldn't be very useful, except for our Orbit build input). > > > 3) "We" need to implement conditioning and signing for the new build. (Last > > I looked, they were not signed.) > > added to the TODOs > https://wiki.eclipse.org/Orbit/Transition_to_Git_Plan > > > 4) "We" do need some list of bundles and associated CQ's. While the Eclipse > > Foundation's goal is to automate some of that (instead of project's opening > > a piggy back CQ) that work isn't done yet, and even when done, they will > > need some way of "looking up" the CQ for a given version of a given bundle. > > (I've seen "ip log files" in some of the "source directories" I've looked > > at, but am not aware if that work is documented, or settled). > > added to the TODOs > https://wiki.eclipse.org/Orbit/Transition_to_Git_Plan > > > > 5) I believe we need to change our process of "committership" a little. Not > > so much "becoming a committer", but when to remove someone. Not sure we ever > > specified that, for our project, but seems this is a good time. Something > > like, even now, "anyone who has not committed in past year, get's > > automatically dropped, and must re-apply if they want to commit again". My > > main concern with current system, is we have a lot of bundles, with someone > > named as "contact", and some of those people are no longer interested in > > committing fixes in Orbit, perhaps not even Eclipse, and in some cases, they > > might not even be coding any longer! So, removing a committer listed as a > > "contact", would change the "contact" to "no one" (or, some similar > > wording). What do others think? > > I agree that it might be about time to have a process for inactive > committers. Maybe we should review the "contact" information thing. It's > easy to get stale. Keep in mind that we now have Gerrit. Thus, in theory > there is no need to grant committership first. People can submit reviews at > any time. However, it might still be good to have someone as committer who > feels ownership for (a selection of) recipes. There's no need, as long as a committer is willing to review and commit it. And, given Orbit's history ... I know how that'd turn out. We must have someone assigned to be responsible ... and if the original person is no longer "around", willing or able, then I'd suggest no changes until someone else volunteers. > > > 6) How does current build work? I'd like it to work so that "once a bundle > > is produced, is is no longer produced each build". (Or, at least have some > > strict limit ... if nothing else, perhaps "once released"). Or, perhaps > > better, "only if Git shows a change since last build tag"? > > It doesn't work that way. The Maven Tycho build does not work incrementally. > What we could do (as a workaround) is to add a final mirroring step which > will only mirror those bundles into the final target p2 repository which > actually changed (using a comparator during mirroring). > > Added to the TODOs > https://wiki.eclipse.org/Orbit/Transition_to_Git_Plan Yes, comparator is a must. (and, even then, must ignore some things, like a difference in signatures, based on TSA timestamp.) You say "it doesn't work that way" ... what I am saying is "it should work that way". What's the inhibitor? Can't you change the pom modules, or feature, or something? Seems there's still plenty of room for discussion, and plenty of work still to do. (i.e. volunteers feel free ... this won't happen without help).
(In reply to David Williams from comment #90) > You mean for humans scanning a list, right? Yes. For humans debugging an issue ... and I actually think of the issues we have duplicate bundles in the release train now knowing if the come both from Orbit or just one or none... > [..] But, I think it looks tacky .. and would require > people to have to "change their code" for very little reason (when they use > 'require bundle). Such as when moving from an "old" version to a "newer" > version. If they change to a newer version they should be updating the lower version bounds. Manifests can be update in one go (in case they still use Require-Bundle instead of the recommended Import-Package ;). I can see the additional work for features, though. > So, are you saying we change the bundle symbolic name, from what the third > party called it? (In those cases where they already have an OSGi bundle). > That doesn't seem right. Yes. An we already do that in Orbit to align the BSN with the bundle naming conventions for Orbit. There are quite a few which use a BSN that isn't fully qualified and we changed it in Orbit to a fully qualified one. In fact, sometimes an Orbit bundle is modified from an original bundle to replace a piece of code that isn't IP clean. Having a different BSN is IMHO the better option here to differentiate the modified bundle from the "original" one. > Doesn't this imply the jars have to be "mavenized" first, if not already? No they don't. There is a simple upload capability in Maven to deploy any artifact (doesn't even have to be a jar, could be a zip, image, whatever) under any coordinates. The upload is a one line mvn command. It will create a pom as part of the upload but all it needs is the artifact and some coordinates. The coordinates don't even have to be official ones. We could use anything we want. > That seems like extra work. I think it will be less work than committing them into a repository and doing branching and so on. It also leaves room for automation. IPzilla could publish approved attachments automatically into such a repository. > There's no need, as long as a committer is willing to review and commit it. > And, given Orbit's history ... I know how that'd turn out. We must have > someone assigned to be responsible ... and if the original person is no > longer "around", willing or able, then I'd suggest no changes until someone > else volunteers. Agreed. > You say "it doesn't work that way" ... what I am saying is "it should work > that way". What's the inhibitor? Can't you change the pom modules, or > feature, or something? By "It" I was talking about Maven and Tycho. You go into a folder and say "mvn" and it builds all the modules. There are other ways, though. We can make the build process as fine grained as possible. It "just" needs a lot of work. For example, the Hudson job can analyze Git changes and only build the recipes which have been touched. But AFAIK we then need to generate an ad-hoc Tycho p2 repository build which would produce a repository only of this bundle. The easier approach (for now) is, just build it all and use the comparator to mirror only changes into the long-time storage area. > Seems there's still plenty of room for discussion, and plenty of work still > to do. (i.e. volunteers feel free ... this won't happen without help). +1
(In reply to Gunnar Wagenknecht from comment #91) > (In reply to David Williams from comment #90) > If they change to a newer version they should be updating the lower version > bounds. Manifests can be update in one go (in case they still use > Require-Bundle instead of the recommended Import-Package ;). I can see the > additional work for features, though. I don't think this statement is true, in practice or in theory. You should update the lower bound only if you used a new API from the new third party bundle. Otherwise it is more modular to "leave wide". In fact, in practice, many Eclipse projects (not to even mention adopters) intentionally want their code (or, at last some of their bundles) to work with the N, and N-1 and N-2 releases -- all unchanged. So we would make that impossible for "require bundle" users, if we changed the naming convention. Now that I think about it, I think we are stuck with it. And, don't see much value in adding "orbit" to the name ... certainly not enough to break all those that use "require bundle" and want to work on multiple releases (which might have different versions of the third party bundle). > No they don't. There is a simple upload capability in Maven to deploy any > artifact (doesn't even have to be a jar, could be a zip, image, whatever) > under any coordinates. The upload is a one line mvn command. It will create > a pom as part of the upload but all it needs is the artifact and some > coordinates. The coordinates don't even have to be official ones. We could > use anything we want. > > > That seems like extra work. > > I think it will be less work than committing them into a repository and > doing branching and so on. It also leaves room for automation. IPzilla could > publish approved attachments automatically into such a repository. That all sounds great! When can we say that's the plan? I assume the Eclipse Foundation is on the critical path for this one -- not us? That is, I assume the "build system" can be modified for that easily enough? (And, I'm assuming, not to just "get the latest", but to get by location and specific ?dated? version? Such as if something is needed "for maintenance" in a few years? Thanks for that education. > By "It" I was talking about Maven and Tycho. You go into a folder and say > "mvn" and it builds all the modules. There are other ways, though. We can > make the build process as fine grained as possible. It "just" needs a lot of > work. For example, the Hudson job can analyze Git changes and only build the > recipes which have been touched. But AFAIK we then need to generate an > ad-hoc Tycho p2 repository build which would produce a repository only of > this bundle. The easier approach (for now) is, just build it all and use the > comparator to mirror only changes into the long-time storage area. Yes, the "easier approach for now" is what we said about Orbit's original build, 10 years ago. :) I was thinking of something simpler such as to remove it from the "modules" list, and move it to the "pre-req" list, if there were other bundles that needed it -- to "resolve" -- since we don't really build the source, we don't need it for compiling. Er, I didn't miss anything, did I? ... you are not proposing to (re)compile from source, right? As nice as that would be ... it would be very hard to do in practice. Thanks again for all your work on this.
(In reply to David Williams from comment #92) > [...] In fact, in practice, > many Eclipse projects (not to even mention adopters) intentionally want > their code (or, at last some of their bundles) to work with the N, and N-1 > and N-2 releases -- all unchanged. This is getting religious, but if they want there code to be open ... why do they use require bundle in the first place and not import package? ;) > That all sounds great! When can we say that's the plan? I assume the Eclipse > Foundation is on the critical path for this one -- not us? That is, I assume > the "build system" can be modified for that easily enough? (And, I'm > assuming, not to just "get the latest", but to get by location and specific > ?dated? version? Such as if something is needed "for maintenance" in a few > years? Maven coordinates: groupId+artifactId+version ... so yes, versioned to the very specific point. > Yes, the "easier approach for now" is what we said about Orbit's original > build, 10 years ago. :) I was thinking of something simpler such as to > remove it from the "modules" list, and move it to the "pre-req" list, if > there were other bundles that needed it -- to "resolve" -- since we don't > really build the source, we don't need it for compiling. Er, I didn't miss > anything, did I? ... you are not proposing to (re)compile from source, > right? As nice as that would be ... it would be very hard to do in practice. I need to think more about this "pre-req" thing you are talking about. It might make things easier.... Yes, no re-compilation is happening. Class files from the jars are used as is. Only re-packaging might happen (eg., sub-set of a jar). In theory, one could include Java source code in a recipe which gets compiled into a class file, which could then replace a class file in the original jar. I've played with it. Works "by accident" (just create src/main/java). It's a convenient way of patching libraries to improve their support in an OSGi environment (eg., give them an activator that does some magic, help with ServiceLocator, etc.). However, I'm not sure if we should go down that path. :)
(In reply to David Williams from comment #88) > Moving target, since we didn't make any widespread transition for SR1, > though, appreciate those of you who have tried the new method. > > Let's start a list of "things to do" ... > > 1) Is there agreement that "orbit" should not be part of the bundle name? Yes, "orbit" must not be added to the bundle name. This might maybe seem like a little help for those who maintain/produce the Orbit bundles but for all consumers/users of those bundles it will be confusing and even misleading. Most of the consumers never heard of "orbit" and seeing this in a bundle name would be disturbing for them. They want to see 'org.junit' in their workspace and not something like 'org.junit.orbit' or 'org.junit-orbit'.
(In reply to Dani Megert from comment #94) > (In reply to David Williams from comment #88) > > Moving target, since we didn't make any widespread transition for SR1, > > though, appreciate those of you who have tried the new method. > > > > Let's start a list of "things to do" ... > > > > 1) Is there agreement that "orbit" should not be part of the bundle name? > > Yes, "orbit" must not be added to the bundle name. This might maybe seem > like a little help for those who maintain/produce the Orbit bundles but for > all consumers/users of those bundles it will be confusing and even > misleading. Most of the consumers never heard of "orbit" and seeing this in > a bundle name would be disturbing for them. They want to see 'org.junit' in > their workspace and not something like 'org.junit.orbit' or > 'org.junit-orbit'. +1 "orbit" should not be a part of bundle names
For the record. I don't see the big deal with adding orbit suffixes to the bundle names. Nobody should be using require-bundle on these things. Even if they do then they will want to make sure they are getting the 'orbit' one over some random one that has not been tested with the rest of the eclipse project consumers. I don't see why indicating to the developer where the bundle came from is confusing. At a minimum it will make it obvious that the bundle jar did not come directly from the open source project that developed the code.
IMO orbit should be a part of it since it clearly shows the lineage of the artifact. Projects may publish osgi bundles but if it has to go through orbit then odds are it is getting modified in some way outside of the control of that project. At that point it is no longer that projects artifact, it is someone else's...orbits. but I have no horse in this game so whatever you like
(In reply to Jesse McConnell from comment #97) > IMO orbit should be a part of it since it clearly shows the lineage of the > artifact. Projects may publish osgi bundles but if it has to go through > orbit then odds are it is getting modified in some way outside of the > control of that project. At that point it is no longer that projects > artifact, it is someone else's...orbits. > > but I have no horse in this game so whatever you like I agree with Jesse.
For what it is worth in https://github.com/jbosstools/jbosstools-locus (Similar to Orbit but for JBoss Tools where Orbit does not always make sense to use) we have this rule: "Do always use org.jboss.tools.locus.<libraryname> as bundleid The bundle id is set to Locus to avoid any potential conflicts with Orbit." Which funny enough we put in place since Orbit naming at least historically used https://wiki.eclipse.org/Bundle_Naming which meant we could not safely use the existing namespaces. So for me it would be more sane if orbit identified its custom bundles by having orbit in the name rather than reusing i.e. org.junit making it impossible for org.junit to provide their own since Orbit already taken that namespace.
I'm not sure where this discussion is going. I'm about to add freemarker and the latest Antlr using EBR (which is totally awesome, thanks Gunnar!). Do I add .orbit or not? Since org.antlr-runtime already exists in Orbit, I'm tempted to keep the same BSN. And then do that with freemarker. Yes, it would be confusing if they started making their own bundles upstream, but how likely is that? And are there ways to mitigate crossing streams?
(In reply to Doug Schaefer from comment #100) > I'm not sure where this discussion is going. ... Well, let me resolve that now. I think there is not consensus on making a change. Each "camp" has put forth good arguments, but since there is not consensus, as Project Lead, I'll decide the issue by stating the "status quo wins" (when there is not consensus for a change). So, we will not append "orbit" to the Bundle Symbolic Name. I hope to respond to some of the comments in more detail later, but for now just wanted to say anyone doing new work for Neon -- new system, or old -- should NOT append ".orbit" (and, if you already are, please remove it). I think better to have it decided -- and move on to accomplishing the main goal -- since nothing seems to be leading toward consensus. In some informal talk chat with Gunnar, he could understand the rationale for not breaking those who use "require bundle" and, as Doug mentioned, the "mixed case" is the worst! Plus, this issue of the name, is not really part of "moving to Git" which is our primary objective. What I could see, if others felt strongly about it, is to open an enhancement request, so that in Neon+2 we would plan to move to having a suffix (as well as having the pro and con discussion all over again). But I think Neon+2 (meaning we'd make the change relatively shortly after the end of Neon+1) would give time to a) give notice to the community of of a "breaking change" coming and b) would allow the community, in Neon, and Neon+1, to move to the "import packages" method of using Orbit bundles. I am not saying we will do all this ... just saying that would be a reasonable proposal, if anyone felt strongly about it. But, for Neon, and Neon+1, we'll use the naming convention that Orbit already has. Thanks,
(In reply to David Williams from comment #101) > But, for Neon, and Neon+1, we'll use the naming convention that Orbit > already has. +1 We'll decouple the discussion from the actual move to Git. I'll clean up the recipes in Git.
Mars SR2 is long gone. What's the plan for this bug?
Recipes are being used. We will not actively migrate old CVS bundles into Git. This should be done when a bundle is updated. There is still a lot work left to restore the same functionality that the scripts did based on CVS - mostly around generating nice HTML for the download area. Roland, do you think this can be closed?
Yes, I think so. CVS should now be read-only, and all new/updated bundles are being done on the orbit-recipes git repository. We have a build process in place using Hudson that builds+signs and promotes the build. We should go back through older Orbit bugs and ask if the ones requesting things be contributed under orbit-recipes can be closed. This, esepcially since we are releasing milestone repositories that can be consumed. I'm marking this as CLOSED (FIXED).
(In reply to Roland Grunberg from comment #105) > CVS should now be read-only Please communicate this directly to the webmaster.
(In reply to Wayne Beaton from comment #106) > (In reply to Roland Grunberg from comment #105) > > CVS should now be read-only > > Please communicate this directly to the webmaster. Sorry for the confusion, I meant that the step should have already been taken, although I was having a hard time finding the bug. It's Bug 499568.
