Community
Participate
Working Groups
Httpclient version 3 (org.apache.commons.httpclient) has been discontinued upstream. Eclipse Platform should move to hc-httpclient version 4 (org.apache.http.client). Thanks!
ECF already has bug 251740 (which could be a depends on for this bug, as having a httpcomponents-based provider for ECF is tantamount to having it for P2/this bug). Given ECF's provider architecture and the existing codebase currently used by p2 (for both jre-based provider and httpclient 3.1.0), creating an httpclient 4-based provider is not a large development task. It would not involve any ECF filetransfer API changes...or even additions...rather it would just be a new implementation of the existing filetransfer API. Further, my expectation is that much of the httpclient 3.1.0-based provider code could be reused (with probably simple/straightforward refactorings) for a httpclient 4.0-based provider. The 3.1 provider code is here: http://git.eclipse.org/c/ecf/org.eclipse.ecf.git/tree/providers/bundles/org.eclipse.ecf.provider.filetransfer.httpclient However, ECF currently has no resources to create/debug, test, and support such a provider. As the discussion on bug 251740 details, we would like to see a httpclient 4.1/httpcomponents provider...but unfortunately wanting it doesn't make resources available. If contributions can be made and/or other resources identified, then I will personally assist with the testing and support...to the degree that I can. Note my expectation is that any change in provider will result in a period of testing, deployment, some regression, and support...since each transport implementation seems to have their own distinct bugs and issues...and many of these issues will only surface after deployment (e.g. proxys). I'm adding Henrich Kraemer to this bug, as he did a large part of the httpclient 3.1 provider work...and may have some additional comments about an httpclient 4 provider. Henrich: amazing timing, eh :)?
Note that httpclient 4.1 now has native support for NTML.
Our product was intended to migrate httpclient 4.x, but it's deferred due to release plan. We already used P2 and ECF provider based on httpclient 4.x to download more than 1GB artifacts for Amazon S3, they worked well to download files from Internet with basic proxy support. Though I don't have chance to test it with NTLMv2 proxy, I still think it's very good to migrate ECF provider based on httpclient 4.x in M5. We can get more feedback from community users.
Created attachment 225378 [details] A patch for P2 to use 4.x provider
Created attachment 225379 [details] A patch enabling proper ecf repo in the parent pom.xml Those two patches were used to produce Eclipse builds for testing described in http://eclipseandlinux.blogspot.com/
Eclipse is currently not build with the CBI. Could you please create a patch for the Eclipse build. Thx.
(In reply to comment #3) > Our product was intended to migrate httpclient 4.x, but it's deferred due to > release plan. We already used P2 and ECF provider based on httpclient 4.x to > download more than 1GB artifacts for Amazon S3, they worked well to download > files from Internet with basic proxy support. Though I don't have chance to > test it with NTLMv2 proxy, I still think it's very good to migrate ECF > provider based on httpclient 4.x in M5. We can get more feedback from > community users. Thanks Meng for the testing and input. Good to hear. Will there be other opportunities for you/team you work with/test further once the new provider is distributed with p2/Eclipse? Thanks again.
Of course, we will do it. (In reply to comment #7) > > Will there be other opportunities for you/team you work with/test further > once the new provider is distributed with p2/Eclipse? Thanks again.
(In reply to comment #6) > Eclipse is currently not build with the CBI. Could you please create a patch > for the Eclipse build. Thx. I wish I could. I'm not very familiar with the Eclipse-build - but I thought it would be enough to change p2.map file to include proper ecf bundles. Unfortunately I could not found a stable ecf repo - just last successful. Scott, could you create a long-living version of a ecf build for the purpose of including it into Eclipse build?
(In reply to comment #9) > (In reply to comment #6) > > Eclipse is currently not build with the CBI. Could you please create a patch > > for the Eclipse build. Thx. > > I wish I could. I'm not very familiar with the Eclipse-build - but I thought > it would be enough to change p2.map file to include proper ecf bundles. > Unfortunately I could not found a stable ecf repo - just last successful. > > Scott, > could you create a long-living version of a ecf build for the purpose of > including it into Eclipse build? Hi Daniel. Yes...eventually we will produce a long-lived version of the httpclient4 repo for consumption into Eclipse build. However...at the moment we/ECF are currently working out (with Denis/EF) the security arrangements WRT our builder (which is located at https://build.ecf-project.org/jenkins/ ). Also...we need to continuously update things (e.g. in response to bug reports, testing results, etc), and then produce a new build, and so I suggest that once we get the security stuff worked out with Denis that we work with you, Pascal, and Ian to arrange for a location at download.eclipse.org to put the most recent build of the necessary httpclient4 repos (either in ECF's area at download.eclipse.org or at p2/Equinox's area there...is what we've done in the past). In the mean time (until we get the security stuff worked out) we will have to point you and others to the last successful artifacts at: https://build.ecf-project.org/jenkins/job/C-HEAD-filetransfer.httpclient4.feature/
Pascal, will CBI build of platform enable soon? I also see some bugs for merging CBI build code to master? If so, this work can start after CBI build is enabled. (In reply to comment #6) > Eclipse is currently not build with the CBI. Could you please create a patch > for the Eclipse build. Thx.
Scott, do we also need to consume the new version of the org.eclipse.ecf.filetransfer.feature feature?
(In reply to comment #12) > Scott, do we also need to consume the new version of the > org.eclipse.ecf.filetransfer.feature feature? I think there have been a couple of minor bug fixes since the Juno release...so it would probably be good to include the new version of the plugins in the o.e.e.filetransfer.feature...in other words 'yes'.
(In reply to comment #13) > (In reply to comment #12) > > Scott, do we also need to consume the new version of the > > org.eclipse.ecf.filetransfer.feature feature? > > I think there have been a couple of minor bug fixes since the Juno > release...so it would probably be good to include the new version of the > plugins in the o.e.e.filetransfer.feature...in other words 'yes'. if you can provide details which bundles need to be replaced and where to get them from, tycho could offer some help with testing: - we could include the httpclient v4 bundles in the development (-SNAPSHOT) version of the tycho distro e.g. starting with 0.18.0-SNAPSHOT from end of March on - many of our users use the development version of tycho so this would give the code some exposure out in the wild for several months until we release tycho 0.18.0
CBI has already test build. I'll apply those two patches to including ECF provider based on httpclient 4.x. Then we can use CBI build to test proxy support.
We need firstly update repository url of ECF in parent pom before updating p2's feature. David, could you please review the patch of parent pom.xml then apply it if you feel it's fine?
There is now a signed integration build of the ECF HttpClient4 provider and dependencies available here: http://download.eclipse.org/rt/ecf/int/site.p2 One question going forward: would you prefer that this URL be the same for each subsequent integration build (i.e. with new qualifiers?)...or should there be a separate subdirectory for each integration build (e.g. with date/qualifier in path?...e.g. http://download.eclipse.org/rt/ecf/int/v20130221-0726.jar. In this repository is the entire ECF SDK, which includes the Httpclient4 plugins, but also has the HttpClient 3.1 plugins and lots of other plugins for ECF that have nothing to do with file transfer. Below is a listing of the plugins specifically for the HttpClient4 provider (and dependencies). HttpClient4 ECF plugins The following are the ECF binary and source plugins that make up the ECF HttpClient4 provider. org.eclipse.ecf_3.2.0.v20130221-0726.jar org.eclipse.ecf.source_3.2.0.v20130221-0726.jar org.eclipse.ecf.ssl_1.1.0.v20130221-0726.jar org.eclipse.ecf.ssl.source_1.1.0.v20130221-0726.jar org.eclipse.ecf.identity_3.2.0.v20130221-0726.jar org.eclipse.ecf.identity.source_3.2.0.v20130221-0726.jar org.eclipse.ecf.filetransfer_5.0.0.v20130221-0726.jar org.eclipse.ecf.filetransfer.source_5.0.0.v20130221-0726.jar org.eclipse.ecf.provider.filetransfer_3.2.0.v20130221-0726.jar org.eclipse.ecf.provider.filetransfer.source_3.2.0.v20130221-0726.jar org.eclipse.ecf.provider.filetransfer.ssl_1.0.0.v20130221-0726.jar org.eclipse.ecf.provider.filetransfer.ssl.source_1.0.0.v20130221-0726.jar org.eclipse.ecf.provider.filetransfer.httpclient4_1.0.200.v20130221-0726.jar org.eclipse.ecf.provider.filetransfer.httpclient4.source_1.0.200.v20130221-0726.jar org.eclipse.ecf.provider.filetransfer.httpclient4.ssl_1.0.0.v20130221-0726.jar org.eclipse.ecf.provider.filetransfer.httpclient4.ssl.source_1.0.0.v20130221-0726.jar HttpClient 4 Orbit Deps The ECF provider has dependencies on the following Orbit plugins. These plugins (and versions) *are* in the ECF repository, but we do not include the source plugins, so if those are needed then they will have to be gotten from Orbit. org.apache.httpcomponents.httpcore_4.1.4.v201203221030.jar Deps: none org.apache.httpcomponents.httpclient_4.1.2.v201203221030.jar Deps: org.apache.commons.logging_1.1.1.v201101211721.jar org.apache.commons.codec_1.3.0.v201101211617.jar HttpClient 3.1 It's possible to have *both* the HttpClient4 and HttpClient3.1 provider present at the same time. So if that's desired (for fallback in case of regression) the following plugins make up the HttpClient3 provider (note that the other org.eclipse.ecf.* plugins above are also required) org.eclipse.ecf.provider.filetransfer.httpclient_4.0.300.v20130221-0726.jar org.eclipse.ecf.provider.filetransfer.httpclient.source_4.0.300.v20130221-0726.jar org.eclipse.ecf.provider.filetransfer.httpclient.ssl_1.0.0.v20130221-0726.jar org.eclipse.ecf.provider.filetransfer.httpclient.ssl.source_1.0.0.v20130221-0726.jar HttpClient 3.1 Orbit Deps For the HttpClient provider the following Orbit bundles are required: org.apache.commons.httpclient_3.1.0.v201012070820.jar Deps: org.apache.commons.logging_1.1.1.v201101211721.jar org.apache.commons.codec_1.3.0.v201101211617.jar
P2 picks up httpclient4 provider. http://git.eclipse.org/c/equinox/rt.equinox.p2.git/commit/?id=0fed49265ace8a6bbca9e33c3a9b4a1ddf145ebe
I downloaded 4.3.0 I-Build: I20130303-2000, that build contains httpcomponent 4 and ecf provider based on httpclient4. I successfully installed CDT from built-in repo.
That's great Meng, thanks for taking care of the integration. Do you know if all the debugging flags listed at http://wiki.eclipse.org/Equinox/p2/TransportDebugging are still applicable?
(In reply to comment #19) > I downloaded 4.3.0 I-Build: I20130303-2000, that build contains httpcomponent 4 > and ecf provider based on httpclient4. Does anyone happen to know why the I20130303-2000 is not part of the I-builds p2 repository [1]? [1] http://download.eclipse.org/eclipse/updates/4.3-I-builds
(In reply to comment #20) > That's great Meng, thanks for taking care of the integration. > > Do you know if all the debugging flags listed at > http://wiki.eclipse.org/Equinox/p2/TransportDebugging are still applicable? Here are the transport debug flags: http://hc.apache.org/httpcomponents-client-ga/logging.html
(In reply to comment #21) > (In reply to comment #19) > > I downloaded 4.3.0 I-Build: I20130303-2000, that build contains httpcomponent 4 > > and ecf provider based on httpclient4. > Does anyone happen to know why the I20130303-2000 is not part of the > I-builds p2 repository [1]? > > [1] http://download.eclipse.org/eclipse/updates/4.3-I-builds They just aren't be auto-copied yet ... was still sanity checking. Hope to get that started tonight.
(In reply to comment #22) > (In reply to comment #20) > > That's great Meng, thanks for taking care of the integration. > > > > Do you know if all the debugging flags listed at > > http://wiki.eclipse.org/Equinox/p2/TransportDebugging are still applicable? > Scott, I found the ecf properties have been named to "xxx.httpclient4.xxx". For example, org.eclipse.ecf.provider.filetransfer.httpclient.browse.connectTimeout => org.eclipse.ecf.provider.filetransfer.httpclient4.browse.connectTimeout. Should we still use the original property name for backward compatibility?
The "org.eclipse.equinox.p2.tests" bundle [1] still incorrectly refers to "org.eclipse.ecf.provider.filetransfer.httpclient" whereas it should require "org.eclipse.ecf.provider.filetransfer.httpclient4" now. (This is obviously breaking runtime tests) [1] http://git.eclipse.org/c/equinox/rt.equinox.p2.git/tree/bundles/org.eclipse.equinox.p2.tests/META-INF/MANIFEST.MF
(In reply to comment #25) > The "org.eclipse.equinox.p2.tests" bundle [1] still incorrectly refers to > "org.eclipse.ecf.provider.filetransfer.httpclient" whereas it should require > "org.eclipse.ecf.provider.filetransfer.httpclient4" now. > > (This is obviously breaking runtime tests) > > [1] > http://git.eclipse.org/c/equinox/rt.equinox.p2.git/tree/bundles/org.eclipse. > equinox.p2.tests/META-INF/MANIFEST.MF Thanks for finding it. That was fixed. http://git.eclipse.org/c/equinox/rt.equinox.p2.git/commit/?id=c2719b94e074e8d43cb95ae7d7d4bc1fb4b43b56
The ECF version consumed in M6 breaks access to p2 repositories requiring basic authentication (see bug 251740). IMHO, this is a blocker for shipping M6 with that ECF version.
Bug 402740 has been fixed and a new integration build of ECF has been created. ECF integration build with this fix, available here: http://download.eclipse.org/rt/ecf/int3/site.p2 The qualifier for all bundles (same set as described in comment 17 ) is now: v20130309-0714 The only bundle version number changed was for org.eclipse.ecf.provider.filetransfer.httpclient4 And it's version went from 1.0.200 to 1.0.300
(In reply to comment #28) > Bug 402740 has been fixed and a new integration build of ECF has been > created. > > ECF integration build with this fix, available here: > > http://download.eclipse.org/rt/ecf/int3/site.p2 > Hi Scott, Could you please reuse existing url[1] to publish new ECF build? If ECF always uses a new url for new build, we have to ask release team to update the url of ECF in parent pom[2]. [1] http://download.eclipse.org/rt/ecf/int2/site.p2/ [2] http://git.eclipse.org/c/platform/eclipse.platform.releng.aggregator.git/tree/eclipse-platform-parent/pom.xml#n28
I've put this new 'int3' build in our Platform build scheduled for Monday morning, 8 AM (0311-0800). I personally like having the new build at new URL ... so I'll say thanks for doing that ... since then the content of our builds are a little more predictable, easier for me to sanity check them, and we have to explicitly mark a change in our SCM to reflect a change has taken place. [And, I am aware, there's a long standing, sometimes heated debate on pros and cons of each approach, which I am not trying to rekindle here ... just recording my appreciation.]
(In reply to comment #30) > I've put this new 'int3' build in our Platform build scheduled for Monday > morning, 8 AM (0311-0800). > > I personally like having the new build at new URL ... so I'll say thanks for > doing that ... since then the content of our builds are a little more > predictable, easier for me to sanity check them, and we have to explicitly > mark a change in our SCM to reflect a change has taken place. [And, I am > aware, there's a long standing, sometimes heated debate on pros and cons of > each approach, which I am not trying to rekindle here ... just recording my > appreciation.] We can do it either way (one directory vs. separate dirs for each integration build). Please just let us know of consensus or decision...on this bug...and we'll do it. Just for calibration: ECF will probably do a new release build either later today Monday 3/11 or Tuesday 3/12.
(In reply to comment #31) > (In reply to comment #30) > > We can do it either way (one directory vs. separate dirs for each > integration build). Please just let us know of consensus or decision...on > this bug...and we'll do it. Just for calibration: ECF will probably do a > new release build either later today Monday 3/11 or Tuesday 3/12. Lets stick with separate directories. To be honest, I didn't realize until I'd just hit send that the "releng team" that Meng was talking about was _me_! :) I'm sure just trying to save me the effort ... but, Meng, I don't mind ... honest ... if I ever seem slow getting to something, just ping me or send a note to platform-releng list and if its one of the few times I can't get to it in the time you want, I'm sure someone else could. Team work. But thanks for trying to save the effort. And, thanks to you Scott for your flexibly and advance notice. We will be doing what we hope is final M6 builds on Wednesday ... our main, final M6 testing on Tuesday, with regressions, etc., fixed on Wednesday ... so a change now is definitely "last minute", so you might clarify what's changed, what impact would be (if we didn't pick it up), etc., what re-testing might need to be done, etc., just so its well documented for all (not that I doubt its important).
Hi David, (In reply to comment #32) <stuff deleted> > > this bug...and we'll do it. Just for calibration: ECF will probably do a > > new release build either later today Monday 3/11 or Tuesday 3/12. > > Lets stick with separate directories. To be honest, I didn't realize until > I'd just hit send that the "releng team" that Meng was talking about was > _me_! :) > I'm sure just trying to save me the effort ... but, Meng, I don't mind ... > honest ... if I ever seem slow getting to something, just ping me or send a > note to platform-releng list and if its one of the few times I can't get to > it in the time you want, I'm sure someone else could. Team work. But thanks > for trying to save the effort. > > And, thanks to you Scott for your flexibly and advance notice. We will be > doing what we hope is final M6 builds on Wednesday ... our main, final M6 > testing on Tuesday, with regressions, etc., fixed on Wednesday ... so a > change now is definitely "last minute", so you might clarify what's changed, > what impact would be (if we didn't pick it up), etc., what re-testing might > need to be done, etc., just so its well documented for all (not that I doubt > its important). It's now looking as though we will *not* be doing another build today (Monday) or Tuesday...essentially because there are no further additional ECF changes. Earlier today I had thought there would be, but there are not. Further, ECF is having it's 3.6.0 release today, and it will be based upon this same build. So...unless some new emergency pops up, we will not be contributing another build to M6. And until we hear otherwise, post M6, we will create new integration builds in new directories.
Thanks to Jan and Scott for fixing the problem, and thanks to David for integrating the fix. We have integrated the new p2 version into Tycho (bug 402100), and it looks as if the new httpclient is now working well.
(In reply to comment #24) > (In reply to comment #22) > > (In reply to comment #20) > > > That's great Meng, thanks for taking care of the integration. > > > > > > Do you know if all the debugging flags listed at > > > http://wiki.eclipse.org/Equinox/p2/TransportDebugging are still applicable? > > > Scott, > > I found the ecf properties have been named to "xxx.httpclient4.xxx". For > example, > > org.eclipse.ecf.provider.filetransfer.httpclient.browse.connectTimeout => > org.eclipse.ecf.provider.filetransfer.httpclient4.browse.connectTimeout. > > Should we still use the original property name for backward compatibility? What (if anything) was ever decided here? By chance, I ran across this wiki http://wiki.eclipse.org/Equinox/p2/TransportDebugging which would seem to need updating if there are new properties. As far as "should they or shouldn't they" while "we" don't use both, it sounds like someone could, so maybe you do want the httpclient4 version, and just update the docs appropriately?
FWIW, I've ran into at least one case where lack of http3 support will cause people's "Juno versions" to be "incompatible" with Kepler ... for packaging, if nothing else. See bug 405057.
