Bugzilla – Full Text Bug Listing
| Summary: | Please set up maven.eclipse.org | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Technology] Dash | Reporter: | Aaron Digulla <digulla> | ||||
| Component: | Maven | Assignee: | Project Dash Incoming bugs <dash-inbox> | ||||
| Status: | RESOLVED FIXED | QA Contact: | |||||
| Severity: | enhancement | ||||||
| Priority: | P3 | CC: | alex.blewitt, anders.g.hammar, beyhan.veliev, caniszczyk, d_a_carver, eric.gwin, glyn.normington, holger.staudacher, ian, joakim.erdfelt, lshanmug, magne.rasmussen, pascal, pwebster, remy.suen, robert.munteanu, sbouchet, steffen.pingel, t-oberlies, wayne.beaton | ||||
| Version: | unspecified | ||||||
| Target Milestone: | --- | ||||||
| Hardware: | All | ||||||
| OS: | All | ||||||
| Whiteboard: | |||||||
| Bug Depends on: | 336864 | ||||||
| Bug Blocks: | 361396, 364469 | ||||||
| Attachments: |
|
||||||
|
Description
Aaron Digulla
What are the requirements for this? How much processing power is required? How much RAM? How much disk space? I'm adding Bug 336864 as a blocker on this as disk space is at a premium right now and I figure that this is going to require a lot of it on an ongoing basis. We also need to think about backups. Can somebody also summarize the differences between Open source Nexus and Nexus Pro? Pascal should be able to tell if I'm wrong but: > How much processing power is required? Once per day, the server searches all folders for obsolete artifacts. Other than that, it's a web server that serves mostly static content plus a search engine for artifacts. I'm not sure how much CPU the search engine will need but searches are for group and artifact IDs (no full text search), so it's not heavy. > RAM? 64MB should be plenty. > Disk space About the same as a p2 repository. Maven needs about 1-2k of meta data per artifact and each bundle maps to one artifact. > Diff. between Nexus and Nexus Pro http://www.sonatype.com/nexus-professional-vs.-nexus-open-source.html The pro version allows to mass-upload artifacts and it works as a p2 repository. Hmmm... maybe we can save the hassel and replace the current p2 repository with Nexus and serve the same artifacts from one source? Pascal: Can Nexus do everything that a standard p2 repository server can do? As I see it, the main difference is that a p2 server has no web interface to upload artifacts. > Hmmm... maybe we can save the hassel and replace the current p2 repository with
> Nexus and serve the same artifacts from one source?
>
> Pascal: Can Nexus do everything that a standard p2 repository server can do? As
> I see it, the main difference is that a p2 server has no web interface to
> upload artifacts.
At this point, the p2 support in Nexus only consists in proxying p2 repos. At this point I don't think this is a necessary feature.
I have still requested a pro license to be emitted for the foundation and it will be sent to Wayne.
A license for nexus pro has been sent to the webmaster earlier today. > A license for nexus pro has been sent to the webmaster earlier today.
Thanks for the license. However, we have a policy to not run closed-source, commercial software at Eclipse.org so if we go this route, we'd go for an open-source solution.
I'm also concerned about the long-term disk space and bandwidth requirements of running such a service.
(In reply to comment #5) > > A license for nexus pro has been sent to the webmaster earlier today. > > Thanks for the license. However, we have a policy to not run closed-source, > commercial software at Eclipse.org so if we go this route, we'd go for an > open-source solution. Then install the open source version of Nexus. For the purposes of this bug, it should be enough. If we need something else, we can open a bug against Nexus asking to have the feature ported to the OSS version. (In reply to comment #6) > Then install the open source version of Nexus. For the purposes of this bug, it > should be enough. If we need something else, we can open a bug against Nexus > asking to have the feature ported to the OSS version. I know it all sounds very easy. Just install the software. Unfortunately, webmaster has to support this thing moving forward. While every little bit of stuff we add individually seems like "not that big a deal", the collection of all the little bits of stuff adds up. We are very keen to do this. We just have to make sure that it's done right. And unfortunately that takes a little time. I'm adding Bug 336864 as a dependency. This should clear up the disk space problem. (In reply to comment #7) > I know it all sounds very easy. Just install the software. Unfortunately, > webmaster has to support this thing moving forward. While every little bit of > stuff we add individually seems like "not that big a deal", the collection of > all the little bits of stuff adds up. Unfortunately, my crystal ball is broken :-) My point is this: So far, we haven't migrated even a single Eclipse project to support this great new feature. That means we don't know if we can reuse the existing p2 bundles (using links, for example, or a smart Maven+p2 unified server). How many changes need to be done in the build process? Can we customize PDE? Or do we need patches in the build scripts? Maybe PDE doesn't even work and we have to migrate all projects to Tycho. It's unclear whether it makes sense to offer every nightly, integration build or only the releases. Should old nightly/i-builds be kept or can we flush them as soon as the next one is available? Or maybe we *have* to flush them, because of how Maven snapshots work. So from my point of view, it looks a bit like this: I'd like to get some basic answers so we can move the bug forward. Disk space isn't an issue right now nor will it be for the next 6 or even 12 months. Bandwidth is also not an issue since we won't offer the whole of the Eclipse ecosystem from the beginning. But instead of helping to start the learning curve to get answers for those important questions, always only new concerns are brought up. Therefore, I opened this bug: Try whether our ideas work and how to make them work under the constraints of the real world. Instead of seeing hundreds of "I doubt", "Maybe ...", "I feel", "My concerns are" which you can't rely on. Maybe we'll have to tear down the server serving maven.eclipse.org in three months. But I'm pretty sure the URLs won't change, only the technology that serves it. Does that make sense? (In reply to comment #5) > > A license for nexus pro has been sent to the webmaster earlier today. > > Thanks for the license. However, we have a policy to not run closed-source, > commercial software at Eclipse.org so if we go this route, we'd go for an > open-source solution. Nexus is available as an open-source version which shouldn't affect what needs to be done here: http://nexus.sonatype.org/ > I'm also concerned about the long-term disk space and bandwidth requirements of > running such a service. Disk space is already taken up with maven repo-ettes being published in random locations. At least by having a central place, that can be quantified. In addition, this can start out small; we don't need an instance huge amount of space to get started. The current Maven repository publishers (EclipseLink, Jetty, ECF) would love to get on board. We may need to ramp up after a proof-of-concept stage, though; at that point, disk space requirements can be re-evaluated. As an example, my 'org/eclipse' cache on my local machine is 15Mb in size (Jetty, EclipseLink). The org.eclipse.ecf providers in my Eclipse install is less than 1Mb, but with JavaDocs maybe double that. We're talking tens of Megabytes rather than gigabytes of required space. Furthermore, if we have a separate group for the release candidates vs the releases, we can archive the release candidates on an approximate timetable. And the snapshot holder could be cleaned periodically as well, if that were enabled. As for the bandwidth requirements; once it's in a Maven form, then the Sonatype guys (who run the main Central maven repository) could likely pick up and feed it into Central. That would take care of the bandwith and mirroring requirements, not to mention getting it out to a lot more people. But release candidates all get cached on the user's local disk after first access, so you're not likely to see repeated requests for the same artifact from the same person. In the interest of being open and transparent, Wayne and I have asked Aaron, Alex and David if they were interested in leading this effort. The Foundation would provide a minimalist virtual server and give these new Maven admins the keys. Alex and Aaron are in the process of becoming committers on the Dash project. The vserver is already done, so things should be moving forward shortly. The maven.eclipse.org server has been set up, and the new Dash committers (Dave, Alex, and Aaron) all have received the credentials. Many thanks for stepping up and offering help. Moving to Dash. Just to track the progress: We have imported a couple of download artifacts from the eclipse.org download site into a testing repository at http://maven.eclipse.org/nexus/content/repositories/testing/ Please open new bugs if you find problems in there (Product: "Dash", Component: "Maven") (In reply to comment #13) > Just to track the progress: We have imported a couple of download artifacts > from the eclipse.org download site into a testing repository at > http://maven.eclipse.org/nexus/content/repositories/testing/ The testing repository works great! When will the productive repositories become available? We would need releases (and possibly milestones) of org.eclipse.osgi and org.eclipse.jdt.core. (In reply to comment #14) > The testing repository works great! When will the productive repositories > become available? We would need releases (and possibly milestones) of > org.eclipse.osgi and org.eclipse.jdt.core. Looking at the current progress: A couple of months? But that shouldn't stop you to use the testing repo as your main repo for now. Just put that into a parent POM which you use everywhere (or in settings.xml or use a local Nexus proxy), so you have a central place to update the URL. When we more everything to the right place, we won't tear down testing right away, so you have time to update your URL (and even if you don't: Your builds will still work using the artifacts in your local cache). What's more important: We need feedback about problems and issues with the test repo. We'd rather avoid to push it to production just to find that something is broken. So *please* use testing for real work and let us know what you find. (In reply to comment #15) > But that shouldn't stop you to use the testing repo as your main repo for now. We can't do releases with the testing repo because the POM of the released artifact will eventually point to a non-existing Maven repository and hence be unusable. And as long as we need a workaround for our release (currently we upload the Eclipse artifact to our own Maven repository) we will probably also use it during development. So, for our use case, we need a Maven repo with a longer lifetime. (In reply to comment #16) > (In reply to comment #15) > > But that shouldn't stop you to use the testing repo as your main repo for now. > > We can't do releases with the testing repo because the POM of the released > artifact will eventually point to a non-existing Maven repository and hence be > unusable. And as long as we need a workaround for our release (currently we > upload the Eclipse artifact to our own Maven repository) we will probably also > use it during development. > > So, for our use case, we need a Maven repo with a longer lifetime. I don't expect to have something ready for production release use, until after Indigo is out. We also need to come up with a strategy of how long snapshots in the testing repo will stay around, how often we schedule Nexus to clean up the repos, and reindex them as well. So for now, I say create your own longer term repo as you are doing now, and then later we can setup a hosted repo that points to your existing repository and let Nexus manage it after that. (In reply to comment #16) > > But that shouldn't stop you to use the testing repo as your main repo for now. > > We can't do releases with the testing repo because the POM of the released > artifact will eventually point to a non-existing Maven repository and hence be > unusable. Depending on your use case, you have several options to work around this: 1. Use a property for the URL. That way, developers can override it from the command line. 2. Create a proxy repo using Nexus or a similar Maven proxy server and use the proxy URL in your builds. When testing is moved to the final place, just update the config of Nexus. Forgot one option: You can use the tools at https://github.com/digulla/org.eclipse.dash.m4e.tools to create your own repo. Aaron, could you post instructions how to use the tools? (In reply to comment #20) > Aaron, could you post instructions how to use the tools? Please have a look at http://wiki.eclipse.org/Maven_Tools_4_Eclipse and let me know what I missed. Created attachment 202711 [details]
fix
Not sure if the change to Pom.groovy is the right fix but I had to make this change to make it run (some tests stil fail).
Thanks for the link! I had to make a few changes to get the tool running (see patch). Then I followed these steps: * Install decentxml 1.4-SNAPSHOT into local maven repository hg clone https://code.google.com/p/decentxml/ cd decentxml mvn install -Dmaven.test.skip=true * Build m4e: git clone git://git.eclipse.org/gitroot/dash/org.eclipse.dash.m4e.tools.git cd org.eclipse.dash.m4e.tools mvn clean install -Dmaven.test.failure.ignore=true * Download zip archive to downloads/ directory * Run m4e: ./run.sh convert org.eclipse.dash:dependency-management:3.7.0 This creates a Maven repository in tmp/m2repo. I published the repository to http://mylyn.org/maven/testing/. I had to tweak a bunch of the version constraints in the poms but overall this looks very promising. Thanks a lot for providing the tool, Aaron! (In reply to comment #17) > I don't expect to have something ready for production release use, until after > Indigo is out. We also need to come up with a strategy of how long snapshots > in the testing repo will stay around, how often we schedule Nexus to clean up > the repos, and reindex them as well. > > So for now, I say create your own longer term repo as you are doing now, and > then later we can setup a hosted repo that points to your existing repository > and let Nexus manage it after that. Any status on having a release repo? I'm getting bugs to migrate our existing hosted repo to the maven.eclipse.org site. I'm also interested in docs. We do not currently build using Maven, but are publishing a repo for those that do. I'd like to get a better understanding of how my processes should change to utilize the central site instead. I would also be interested in the planned process. Does it involve what has been done in the testing repository, i.e. conversion from p2? If yes, I have some serious concerns which I need to see resolved first (see bug 361515). I'd be grateful for some clarification of the scope of this bug so that I can set the correct dependencies of bug 364469. Firstly, does this bug include populating the Maven repo hosted at maven.eclipse.org with all the bundles in Orbit? Secondly, bug 364469 assumes that once this bug is fixed, maven.eclipse.org and the Maven repo it serves will be available indefinitely. Is this assumption valid? Thanks! Sorry for the long downtime. We just had a release and pressure is starting to drop (... is that why they call it a "release"?). That said, I think this bug here has been fixed: The server is installed and we have Nexus running on it. I've created a couple of issues to track the new tasks: Bug 367121 - [maven.eclipse.org] Configure the Nexus server to use SSL Bug 367126 - [maven.eclipse.org] Define process how to deploy official Eclipse releases Bug 367128 - [maven.eclipse.org] Upgrade Nexus to 1.9.2.3 Bug 367130 - [maven.eclipse.org] Evaluate the nexus-p2-repository-plugin Feel free to open new bugs if I missed anything. (In reply to comment #26) > I'd be grateful for some clarification of the scope of this bug so that I can > set the correct dependencies of bug 364469. > > Firstly, does this bug include populating the Maven repo hosted at > maven.eclipse.org with all the bundles in Orbit? Yes. Eventually, maven.eclipse.org should contain all bundles necessary to fulfill all dependencies. Where possible, official Maven Central dependencies will be offered as profile options to allow users replace Eclipse (= EPL clean) dependencies with full-featured dependencies (= *NOT* EPL clean). The reason for that is that consumers of the repos sometimes build non-Eclipse software (i.e. no p2, no OSGi and no Orbit) which has existing dependencies and Orbit sometimes conflicts with that. > Secondly, bug 364469 assumes that once this bug is fixed, maven.eclipse.org > and the Maven repo it serves will be available indefinitely. Is this > assumption valid? Yes. The repo is here to stay. We're not sure about the layout. (In reply to comment #22) > Created attachment 202711 [details] > fix I'll have a look at this in a couple of days. > Not sure if the change to Pom.groovy is the right fix but I had to make this > change to make it run (some tests stil fail). Did you get the tests to pass? If not, open a new bug: https://bugs.eclipse.org/bugs/enter_bug.cgi?product=Dash&component=Maven and post the errors. (In reply to comment #27) > (In reply to comment #26) > > I'd be grateful for some clarification of the scope of this bug so that I can > > set the correct dependencies of bug 364469. > > > > Firstly, does this bug include populating the Maven repo hosted at > > maven.eclipse.org with all the bundles in Orbit? > > Yes. Eventually, maven.eclipse.org should contain all bundles necessary to > fulfill all dependencies. Thanks for the clarification, but I'm still not completely clear. ;-) Does maven.eclipse.org now contain all the bundles in Orbit? (The word "eventually" above implies this is not yet the case.) If not, bug 364469 will need to (a) populate maven.eclipse.org with the current Orbit content, and (b) set up a (preferably automated) process to ensure that additions, deletions, and other changes to Orbit are mirrored in maven.eclipse.org. (In reply to comment #29) > (In reply to comment #27) > > Yes. Eventually, maven.eclipse.org should contain all bundles necessary to > > fulfill all dependencies. > > Thanks for the clarification, but I'm still not completely clear. ;-) > > Does maven.eclipse.org now contain all the bundles in Orbit? The scope seems to be to convert all artifacts from the p2 release train reposiories to Maven artifacts. I'm not quite convinced that this is a good idead - one reason for my concerns is bug 361515. (In reply to comment #30) > (In reply to comment #29) > > (In reply to comment #27) > > > Yes. Eventually, maven.eclipse.org should contain all bundles necessary to > > > fulfill all dependencies. > > > > Thanks for the clarification, but I'm still not completely clear. ;-) > > > > Does maven.eclipse.org now contain all the bundles in Orbit? > The scope seems to be to convert all artifacts from the p2 release train > reposiories to Maven artifacts. I'm not quite convinced that this is a good > idead - one reason for my concerns is bug 361515. Huh! So, far from containing Orbit bundles, maven.eclipse.org actually contains "mavenised" JARs converted from the corresponding Orbit bundles, which is precisely the opposite of what we need in bug 364469, e.g. to build OSGi projects such as Virgo by getting their bundle dependencies directly out of Orbit. Unless someone can definitely state that maven.eclipse.org will eventually contain all Orbit bundles with no modifications, then I'll need to sever the dependency of bug 364469 on this bug and look elsewhere. (In reply to comment #31) > Huh! So, far from containing Orbit bundles, maven.eclipse.org actually contains > "mavenised" JARs converted from the corresponding Orbit bundles, which is > precisely the opposite of what we need in bug 364469, e.g. to build OSGi > projects such as Virgo by getting their bundle dependencies directly out of > Orbit. Let's continue this discussion in bug 364469. The server is up and running. Everything else is a new bug. |