Community
Participate
Working Groups
For users behind restrictive firewalls, the git:// protocol occasionally poses problems. Thus, better use git's (dumb) http:// protocol instead. According to Denis (see bug #421207#c3), this makes no difference performance-wise.
Adding some other component owners to CC as this would effect a number of repositories. I count 48 occurrences of <properties> <tycho.scmUrl>scm:git:git://git.eclipse.org/gitroot/platform/eclipse.platform.runtime.git</tycho.scmUrl> </properties> Assuming this "makes no difference to us" ... might want to implement the change by defining a variable in parent pom, so that as a project we just define this once, then in future would be easy to change to something else if we ever found that necessary. But, to be honest, not sure of Tycho/Maven rules of "combining variable properties" inside of properties. Such as not positive it'd be valid to write something like <tycho.scmUrl>scm:git:${scmURLProtocol}://git.eclipse.org/gitroot/platform/eclipse.platform.runtime.git</tycho.scmUrl> If not valid, guess we'd be stuck making the 48 changes.
I'm fine with the change in principle. Where would we want to define the property, wouldn't that have to be in each git root pom? Also, does the EGit source plugin import work with http (I assume so)? PW
(In reply to Paul Webster from comment #2) > Also, does the EGit source plugin import work with http (I assume so)? I assume you mean File > Import > Plug-ins and Fragments > Projects from repository? Then yes, it works with my local testing for bug #421207.
(In reply to Paul Webster from comment #2) > I'm fine with the change in principle. Me too. > Where would we want to define the > property, wouldn't that have to be in each git root pom? I'd expect the property to be in the eclipse-parent-pom, so that we can change it at one location in the future. Otherwise it doesn't make much sense.
(In reply to Dani Megert from comment #4) > (In reply to Paul Webster from comment #2) > > I'd expect the property to be in the eclipse-parent-pom, so that we can > change it at one location in the future. Otherwise it doesn't make much > sense. Yes, eclipse-platform-parent POM. And, while I know its not good to "over engineer" a solution for hypothetical cases, I'm wondering if it'd be worth to turn some of the other part of this URL into variables too (also defined in eclipse-platform-parent). So, in each repo, we'd end up with something like <tycho.scmUrl>scm:git:${scmURLProtocol}://${scmHOST}/${scmROOT}/platform/eclipse.platform.runtime.git</tycho.scmUrl> I'm thinking of use-cases such as "Long term support" where, I'm assuming, not all source will be necessarily be in the "standard location". Plus, of less interest to me, it would allow some companies/adopters to have their own "private fork" where the host name and "root" may not be exactly what what they are on ellipse.org. [And, yes, I realize in LTS and private forks, the committers could just change this for every repo ... but ... just seems to help make it easier and less error prone] I'm just thinking if we are changing one thing ... might make sense to change other candidates, so we would not have to change again, in future. This doesn't have to be perfectly coordinated, but variables must exist in parent pom before anyone else changes ... I I suggest we just change one or two at first, to make a test run before changing them all. So, le me know if I should add just the one, or all three to parent pom and I'll do that later today. I'd be nice to figure out how to fix bug 413056 using this same sort of mechanism, but not sure how to do that.
(In reply to David Williams from comment #5) > And, while I know its not good to "over engineer" a solution for > hypothetical cases, I'm wondering if it'd be worth to turn some of the other > part of this URL into variables too (also defined in > eclipse-platform-parent). So, in each repo, we'd end up with something like > > <tycho.scmUrl>scm:git:${scmURLProtocol}://${scmHOST}/${scmROOT}/platform/ > eclipse.platform.runtime.git</tycho.scmUrl> The only drawback I see, is that it becomes less predictable what the real value in a repo will be, since repos can either not use any of those variables or just some. How about adding a ${scmURLPrefix}? This would still allow to change any aspect of the URL globally while not introducing too many variants.
(In reply to Dani Megert from comment #6) > (In reply to David Williams from comment #5) > > And, while I know its not good to "over engineer" a solution for > > hypothetical cases, I'm wondering if it'd be worth to turn some of the other > > part of this URL into variables too (also defined in > > eclipse-platform-parent). So, in each repo, we'd end up with something like > > > > <tycho.scmUrl>scm:git:${scmURLProtocol}://${scmHOST}/${scmROOT}/platform/ > > eclipse.platform.runtime.git</tycho.scmUrl> > > The only drawback I see, is that it becomes less predictable what the real > value in a repo will be, since repos can either not use any of those > variables or just some. How about adding a ${scmURLPrefix}? This would still > allow to change any aspect of the URL globally while not introducing too > many variants. I did a local test build just to make sure to basic idea worked ... and at first did not seem to ... then realized that the reason was that if we do this enhancement, each and every bundle in each and every repo will need to be "touched" to "get" the new value in the current, distributed bundle. (And confirmed did work, then). But, thought I'd point that out and ask "is it worth it?". I mean, fine by me ... but seems like a lot of work, and I'm not sure how wide spread this issue is? Have those with restrictive firewalls contacted their IT departments to see about the restriction lifted?
(In reply to David Williams from comment #7) > > I did a local test build just to make sure to basic idea worked ... and at > first did not seem to ... then realized that the reason was that if we do > this enhancement, each and every bundle in each and every repo will need to > be "touched" to "get" the new value in the current, distributed bundle. (And > confirmed did work, then). I'm fine with making this the new default, and as bundles are changed in Luna they pick up the scm reference. The active bundles (that people are most likely to use this against) will pick up the new scm information, and that will reduce the intersection of git+"restrictive firewall" to possibly a few bundles. The people that then hit the remaining problem bundles are no worse off than they are now, and as plugins are modified the situation will improve. PW
I'm not saying it has to be perfectly coordinated ... but ... we need to end up consistent. By M4? M5? ... ? These will show up as "unanticipated comparator errors" until they are consistent ... thereby cluttering and diminishing the value of the "unanticipated comparator list".
(In reply to David Williams from comment #9) > I'm not saying it has to be perfectly coordinated ... but ... we need to end > up consistent. By M4? M5? ... ? These will show up as "unanticipated > comparator errors" until they are consistent ... thereby cluttering and > diminishing the value of the "unanticipated comparator list". Ah, right, that would probably be worth fixing. Did we not touch everything with a pom at the beginning of Luna, to move from a 4.3 parent to a 4.4 parent? PW
(In reply to Paul Webster from comment #10) > (In reply to David Williams from comment #9) > > ... Did we not touch everything > with a pom at the beginning of Luna, to move from a 4.3 parent to a 4.4 > parent? > I believe so ... and its those memories that give me pause. :) Again, I'm not saying we should NOT do it ... just that someone (e.g. PMC) should decide it its worth the team's effort. In addition to the usual trouble it takes, there would probably be a period of failed builds where a feature has not been updated yet, but its bundles have, with the end result of "feature can't find its bundles" sort of build failure. Again, I don't mind ... just want to be sure we go into this with eyes open and really understand the extent of the problem. Such as if all the users having trouble are from the same one company ... then maybe that one company could reconsider their IT policy (I know, that's usually like pulling teeth) but if its a quarter of users/contributors from a dozen companies, then sounds more important for us to make the effort. And I simply don't know how many people and companies it effects and hope someone does.
That we have to touch every bundle is a too high price for the team at the moment. Everyone is very busy and in the past 1.5 years we did not get a single complaint regarding this issue. So, I wouldn't say 'WONTFIX' but defer it at the moment. If someone disagrees, I can bring it up in our PMC call.
(In reply to Dani Megert from comment #12) > So, I wouldn't say 'WONTFIX' but defer it at the moment. If someone > disagrees, I can bring it up in our PMC call. I agree, it's a large change to consume right now. We could time it with the next release (Mwhatever) after Luna. Unless we find a maven solution, we'll have to touch all the poms to move their parents from 4.4.0-SNAPSHOT to 4.5.0-SNAPSHOT so that would be a good time to do it. PW
(In reply to David Williams from comment #9) > I'm not saying it has to be perfectly coordinated ... but ... we need to end > up consistent. By M4? M5? ... ? These will show up as "unanticipated > comparator errors" until they are consistent ... thereby cluttering and > diminishing the value of the "unanticipated comparator list". Sound to me as if the problem is with the validation rather than with mixed protocols used in source ref headers. Thus, can't we just "fix" the validator and migrate chicken-little to "http://"?
(In reply to Markus Kuppe from comment #14) > > Sound to me as if the problem is with the validation rather than with mixed > protocols used in source ref headers. Thus, can't we just "fix" the > validator and migrate chicken-little to "http://"? The validator is working correctly (bundles with the same qualifier have different content). Since bundle+version+qualifier == immutable in p2 it indicate a problem with the build process. And then our process requires us to touch the affected bundles (in this case, all of them :-) which is a pretty high cost. PW
(In reply to Dani Megert from comment #12) > Everyone is very busy and in the past 1.5 years we did not get a > single complaint regarding this issue. At EclipseCon Europe, this issue was brought up during a session on how to contribute to Eclipse. IMO it is important that contributing has as few hurdles as possible. (And no, big corporations who have these kinds of firewalls are not going to change, it's just not going to happen.) > So, I wouldn't say 'WONTFIX' but defer it at the moment. If someone > disagrees, I can bring it up in our PMC call. Defer would mean that users get this at the earliest in 1.5 years? So if the solution of defining this in the parent pom is not feasible for the short term, how about changing it in tycho.scmUrl directly for Luna? With some shell commands (sed, commit, push), this should not be too hard assuming one already has all the repositories.
Can't we just (smart)replace git:// with http:// or whatever in the front-end? This should be a lot less invasive. It could be done in GitScmUrlImportWizardPage in the end of the class somewhere [1]? [1] http://git.eclipse.org/c/egit/egit.git/tree/org.eclipse.egit.import/src/org/eclipse/egit/internal/importing/GitScmUrlImportWizardPage.java?id=146c2f221b628c6dc67a77c0f5964ab362482224
(In reply to Wim Jongman from comment #17) > Can't we just (smart)replace git:// with http:// or whatever in the > front-end? This should be a lot less invasive. It could be done in > GitScmUrlImportWizardPage in the end of the class somewhere [1]? > > [1] > http://git.eclipse.org/c/egit/egit.git/tree/org.eclipse.egit.import/src/org/ > eclipse/egit/internal/importing/GitScmUrlImportWizardPage. > java?id=146c2f221b628c6dc67a77c0f5964ab362482224 It would have to be restricted to URLs that start with "git://git.eclipse.org/". But I'm not sure if we want to add such a hack to EGit instead of just simply changing the URLs.
(In reply to Wim Jongman from comment #17) > Can't we just (smart)replace git:// with http:// or whatever in the > front-end? This should be a lot less invasive. It could be done in > GitScmUrlImportWizardPage in the end of the class somewhere [1]? > > > [1] > http://git.eclipse.org/c/egit/egit.git/tree/org.eclipse.egit.import/src/org/ > eclipse/egit/internal/importing/GitScmUrlImportWizardPage. > java?id=146c2f221b628c6dc67a77c0f5964ab362482224 I think this idea is worth pursuing. And not sure what you mean by "smart replace", but I could imagine a heuristic that if initial attempt failed, then another attempt would be made using 'http' scheme instead as fall-back. I mention this since even though 'git' and 'http' are both implemented and equivalent on "git.eclipse.org", I am not sure it is "the same" on all git repositories and the import wizard is meant to be general purpose. If repeated tries, failures, and re-tries is "too much" (e.g. for performance reasons) perhaps another option is to have a preference where users could specify something like "Use http instead of git scheme in source reference URLs". Ideally, those with the need would do the deed ... i.e. supply some high quality patches and unit tests. :) [And, please note, this is a comment from one who knows very little about the specifics ... just conceptually, I think the general idea of preferences or fall-back heuristic makes sense in this case, but pretty sure we can't make any blanket assumptions about how repositories are setup/configured.]
How about simply fixing the comparator? The scm URL will always be generated, so, never really interesting to report this as an unanticipated change. With this in place, it would be OK for me to go ahead and fix this bug here. Maybe those who are interested in seeing this bug here fixed can start to work an a fix for the comparator? ;-)
(In reply to David Williams from comment #19) > And not sure what you mean by "smart > replace", I was thinking more in the lines of a from/to mask somewhere in the gui. From url: {git://}{git}@{github.com}:{ECF/Chat.git} To Url : https://$2/$3 Another method is to just to show the url and let the user correct it manually maybe with some replace options. I think simplicity is the key if we want to make this happen quickly.
(In reply to Wim Jongman from comment #21) > I think simplicity is the key if > we want to make this happen quickly. IMO this is exactly the reason why the best fix is to update the pom. The whole point of source ref is to help unexperienced users with finding the code. It defeats the purpose if we ask the user to correct a url they don't know about in the first place.
Thanks(In reply to Dani Megert from comment #20) > How about simply fixing the comparator? The scm URL will always be > generated, so, never really interesting to report this as an unanticipated > change. With this in place, it would be OK for me to go ahead and fix this > bug here. +1 > Maybe those who are interested in seeing this bug here fixed can start to > work an a fix for the comparator? ;-) Do you have pointers? I do not really know what to search for or where to look.
(In reply to Markus Kuppe from comment #23) > > Maybe those who are interested in seeing this bug here fixed can start to > > work an a fix for the comparator? ;-) > > Do you have pointers? I do not really know what to search for or where to > look. David, can you jump in?
(In reply to Dani Megert from comment #24) > (In reply to Markus Kuppe from comment #23) > > > Maybe those who are interested in seeing this bug here fixed can start to > > > work an a fix for the comparator? ;-) > > > > Do you have pointers? I do not really know what to search for or where to > > look. > > David, can you jump in? Anyone who wants to understand the comparator could start with http://wiki.eclipse.org/Platform-releng/Platform_Build_Comparator_Logs (And for the comparator itself I think there's been some that have written blogs on it, in the past, or search bugzilla) ... BUT, the comparator is simply saying two bundles have exact same qualifier, but different bits ... there are a few cases that might be ok to "ignore" indefinitely, such as when signature has different timestamp, but normally any "comparator error" needs to be fixed at some point, such as by a milestone, or by a release ... So, not sure what's being said here. This does not seem like a case to ignore indefinitely and we would want all the bundles touched at some point ... say by M4 or M5. Is this a discussion about what point in time we need to be accurate? Or if it could be "incorrect indefinitely"? IMHO, it is not a good case to allow to be incorrect indefinitely. [FWIW, I think there is some "real" comparator code in Eclipse SDK, but believe Tycho made its own copy and uses that. Not sure. My "releng" code, is simply some parsing code that "categorizes" the warnings from the comparator in the log into buckets of "not too serious" to "more serious". That log parsing code is in http://git.eclipse.org/c/platform/eclipse.platform.releng.buildtools.git/ ]
(In reply to David Williams from comment #25) > So, not sure what's being said here. This does not seem like a case to > ignore indefinitely and we would want all the bundles touched at some point > ... say by M4 or M5. > > Is this a discussion about what point in time we need to be accurate? Or if > it could be "incorrect indefinitely"? IMHO, it is not a good case to allow > to be incorrect indefinitely. Yes, I think this is a case where we can just ignore some information, because it is generated. Like the signature.
(In reply to Dani Megert from comment #20) > How about simply fixing the comparator? The scm URL will always be > generated, so, never really interesting to report this as an unanticipated > change. With this in place, it would be OK for me to go ahead and fix this > bug here. Actually, this won't work, i.e. it would only help to hide the "noise" in the reports. Without touching a bundle that has no real change it will be considered unchanged. That's actually why we have the comparator logs: they tell us that we have to touch a certain bundle so that the changes are picked up and put into our repo.
To cross-reference, see bug 420696 for discussion of 'gitroot' going away ... someday.
(In reply to David Williams from comment #28) > To cross-reference, see bug 420696 for discussion of 'gitroot' going away > ... someday. I'm wondering if this bug should be moved to EGit? I'm thinking the best solution is one that gives user some choice to change the URL "on the fly". I know if someone imports a "team project set", a dialog pops up that gives user a chance to change the URL (the whole repository specification) that is "hard coded" in the team project set file to some other form that the user might prefer? I'm beginning to think that (or similar) is the only good solution to this issue, in general, especially for the long term.
(In reply to David Williams from comment #29) > (In reply to David Williams from comment #28) > > To cross-reference, see bug 420696 for discussion of 'gitroot' going away > > ... someday. > > I'm wondering if this bug should be moved to EGit? > > I'm thinking the best solution is one that gives user some choice to change > the URL "on the fly". I know if someone imports a "team project set", a > dialog pops up that gives user a chance to change the URL (the whole > repository specification) that is "hard coded" in the team project set file > to some other form that the user might prefer? > > I'm beginning to think that (or similar) is the only good solution to this > issue, in general, especially for the long term. You can create a new enhancement for EGit and link it to this one, but the URLs still need to be changed in the poms, see comment 22 for why.
We briefly discussed this on the Eclipse PMC today: https://wiki.eclipse.org/Eclipse/PMC We agree that due to the read-only nature of those references, the 'http:' protocol would make more sense than the 'git:' protocol. But we are concerned about the impact of changing this wholesale. Due to the requirement of never shipping different content with the same version, we would have to up-version all bundles when we change the Eclipse-SourceReferences header. And that would only catch the next version Eclipse Platform. But there's a couple other projects around which also use the Eclipse-SourceReferences header, and there's older versions of the Platform. Thus fixing this on the consumer side (egit team provider) seems like a better approach. There was concern about making the import too complex for consumers; perhaps there could be a simple fallback that tries git://hostname first but falls back to http://hostname in case the connect fails ? Another option would be a simple extension point that allows contributors of the Team framework to rewrite Eclipse-SourceReferences . Pass in the bundle ID and original SourceReference, return the rewritten SourceReference. Could be chained over multiple extensions. This idea is already tracked here: https://bugs.eclipse.org/bugs/show_bug.cgi?id=351817 (marked 'helpwanted'). The EPP packages could then provide a "rewrite" from http: to git: . The rewrite support would also potentially address one of the concerns that Jeff McAffer had when the SourceReferences idea originally came up -- namely vendor support of patched bundles from a different repository than the original: [[https://bugs.eclipse.org/bugs/show_bug.cgi?id=243582#c1 ]] This would also address rewriting old cvs: references to git: (or http:)
An interested contributor for my project ran into this problem: https://bugs.eclipse.org/bugs/show_bug.cgi?id=434443#c14 So I've finally filed the respective ER against EGit: https://bugs.eclipse.org/bugs/show_bug.cgi?id=450835 EGit team, if you could let us know what's the right place where to contribute this we can probably put together a contribution. Thanks! Martin
(In reply to Martin Oberhuber from comment #32) > EGit team, if you could let us know what's the right place where to > contribute this we can probably put together a contribution. I commented on the bug. Still -1 on doing a random workaround in EGit instead of changing the metadata. Switching the scheme just works because Eclipse happens to use the same paths for all schemes (may be different for non-Eclipse projects). Also, does connecting to git:// behind a proxy result in a timeout (don't know)? If yes, you just made their experience worse because the import will hang for a time before falling back to http://.
Source references have been borked with multiple changes.
