Bugzilla – Full Text Bug Listing
| Summary: | CBI Common Build infrastructure should work without external internet access | ||
|---|---|---|---|
| Product: | Community | Reporter: | Markus Keller <markus.kell.r> |
| Component: | Architecture Council | Assignee: | eclipse.org-architecture-council |
| Status: | CLOSED MOVED | QA Contact: | |
| Severity: | normal | ||
| Priority: | P3 | CC: | daniel_megert, david_williams, denis.roy, pascal |
| Version: | unspecified | ||
| Target Milestone: | --- | ||
| Hardware: | All | ||
| OS: | All | ||
| Whiteboard: | stalebug | ||
|
Description
Markus Keller
> CBI should either replace Maven with a dependable technology, Going down the path of switching technology will be a big cost on the community that is already spread too thin. On top of that which technology? Back to PDE Build? I don't think we have the resources to develop something, and the reality is that all modern build systems get dependencies from an external source (gradle, npm, etc.). If Maven was so unreliable, people would have discarded it years ago and it would have never become that popular. In situation like these where a technology works for the biggest development orgs in the world and not for you, it is not the technology that you should question, but your usage. In our case, I think the problem is routed in the fact that we are not using a caching proxy, and consequently when our connection to the world goes missing, the artifacts can't be retrieved. Repository managers like Nexus and Artifactory have been invented almost 10y ago to solve this problem (and also the problem of build speed by retrieving artifacts from the internet instead of locally) and I don't understand why we are not using those to cache the artifacts. On top of that we already have an instance of Nexus running at https://repo.eclipse.org/ So what is preventing us to use it? I'm not an networking expert, but already today the HIPP servers are going through a proxy, so if the proxy could re-route the maven central requests to a proxy of central on Nexus, then we could benefit from the caching performed by Nexus and not worry when things are going done. The other advantage of such a solution is that it would be transparent to the developers. I have to agree with Pascal's assessment here. At Eclipse we have many moving parts to make this all work. My experience is that proxy access through layers of java apps (as is the case on Hudson) is the root cause of much of our flakiness. Looks like improvements are on their way. See bug 492412 comment 18 ff. To report some success here, our platform build no longer accesses anything "outside" eclipse.org. At least, directly. (And, at least as well as I can tell -- there are thousands of lines of "fetching" and "downloading" I need to analyze better but various "greps" and "sortings" showed nothing but "eclipse.org" URLs). Besides using the eclipse.org mirror of maven central, I learned we were specifying the location of "tycho" with a "sonatype" URLs -- for the simple reason it had always been done that way. Turns out, that is only needed if using "snapshots" of Tycho, which we did do a lot of in the beginning but now do very rarely. We will still have to "go external" when we test a "staged" version of Tycho before it's released but, for now, no other reason to. > To report some success here, our platform build no longer accesses anything
> "outside" eclipse.org.
Just out of curiosity, have you seen any performance improvements by doing that?
For the small build of EGerrit, I have been able to notice some improvements (and increased stability). (In reply to Denis Roy from comment #6) > > To report some success here, our platform build no longer accesses anything > > "outside" eclipse.org. > > Just out of curiosity, have you seen any performance improvements by doing > that? Not noticeably, but, we don't measure it. And, in a 2-hour build, I suspect it would be hard to notice the improvement in any one thing. I would guess the "fetching" (even when external) might be at most 20% of our time, so even if that improved by 25% that would be still just be 5% of the overall time, or about 6 minutes if I've done my math right. To better measure our builds performance is a long-standing work item, but even if that's done, the way Tycho "fetches" things, is sort of "whenever it wants to" so not sure we could capture that unique time (without some sort of full profiling, which is not planned). This bug hasn't had any activity in quite some time. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. If you have further information on the current state of the bug, please add it. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. -- The automated Eclipse Genie. This bug hasn't had any activity in quite some time. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. If you have further information on the current state of the bug, please add it. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. -- The automated Eclipse Genie. This issue has been migrated to https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/248. |