Community
Participate
Working Groups
Build Identifier: N20100727-2000 Currently in Eclipse IDE it's impossible to use other transport than ECF because API and implementation are both packed in one bundle (p2.repository) - so they could not be used separately. An abstraction layer should be defined to make possible providing different transport implementations. Reproducible: Always
Created attachment 175904 [details] Patch provides decoupling between repository transport API and its implementation API and implementation are decoupled via additional bundle - p2.repository.transport - thus p2.repository doesn't depend on ecf packages. Implementation is registered as OSgi service under IRepositoryTransport interface The patch provides default repository transport implementation which to be used only if no implementation is registered. If there is error while retrieving implementation service from OSGi service registry, an IllegalStateException is thrown. Patch provides a new test in p2.repository group which verifies that if there is no registered implementation, the default one will be used.
Created attachment 175905 [details] Provides connection between transport API and current ECF transport implementation
Isn't one of the major reasons for using ECF in the first place that it decouples the API from the actual implementation?
(In reply to comment #3) > Isn't one of the major reasons for using ECF in the first place that it > decouples the API from the actual implementation? Yes, ECF does decouple API (org.eclipse.ecf.filetransfer) from implementation (protocol-specific providers that implement this API). ECF was specifically introduced in order to support the p2 design requirement of allowing/supporting the usage of multiple/alternative transports...i.e. other than/in addition to http. ECF filetransfer's asynchronous/non-blocking nature also was an original p2 transport design requirement...in order to allow for multi-threaded retrieval and user-cancellation. Katya seems to be referring to the p2 repository download code (which uses the ECF filetransfer API) as both 'transport and implementation'. This is a higher level of notion of 'transport'...represented primarily by the IRepositoryTransport interface included in the patch...which essentially exposes the p2 Transport utility class as API. I'll just note that IRepositoryTransport abstraction presents a synchronous/stream-based download API, which doesn't have any notion of support for asynchronous/multithreaded download, nor protocol-level multiple transport handling...i.e. two of the original p2 requirements (as well as lacking other original p2 requirements like authentication, user cancellation, etc).
In fact IRepositoryTransport is not a brand new interface - it's just extracted from the original version of RepositoryTransport class. In p2 ECF has always been used via the methods of this interface - the only difference is the way of transport retrieving: static method RepositoryTrasport.getInstance() vs OSGi service lookup. I thought that defining an interface is a standard way to declare transport's features, but it would work just fine if I register the transport service under RepositoryTransport class. My point is that introducing the interface doesn't restrict existing functionality in any way. Do you have any considerations why it shouldn't be exposed? I understand that ECF is a level of abstraction but in the same time it introduces a lot of dependencies to other components (i.e. equinox.registry) which aren't really needed. I agree also that ECF provides a lot of useful features, but there are cases in which the provisioning mechanism could work without them. For example, user cancellation is necessary if there is a user who could trigger it, but could be missing if provisioning operation is executed in a "black box". Again, what I propose is a default functionality, a bare minimum, that allows you to use p2. It's just a default and should be discarded in favor of presented ECF for example. Refactoring existing source in that way allows us to use p2 not only in Eclipse IDE, but outside as well (for standalone usage) and helps keeping p2 bundles as less as possible. The refactoring is inspired by the idea of minimal p2 installer which is being discussed in the mailing lists.
One danger in exposing a very limited API and make the p2 codebase dependent on it is that it will eliminate any chance to improve the p2 functionality when moving forward. User cancellation, authentication, async downloads etc. are areas that can be greatly improved. It would be bad if such efforts were blocked. Why is there a desire to run p2 without ECF? After all, ECF is very small and it doesn't have any external dependencies.
The intention was to isolate the bare minimum and make all other existing functionality pluggable. If you need authentication, monitoring, etc you can add the corresponding bundles and you get that functionality immediately. Our efforts are in p2 modularization direction. I agree that the areas you both mention bring a lot of value and we also plan to benefit from them in our scenarios. But we believe that the right way ahead is to make them pluggable so every user could use that subset of p2 functionality which best fits to his requirements and nothing more. Currently it's not possible. ECF could be considered as small with regards to bundle size, but according to Eclipse dependency calculation it depends on 50 other bundles. On the other hand our modularized proof of concept contains 20 bundles and is able to install even Eclipse. Don't you think this justifies the abstraction we are talking about?
(In reply to comment #7) > ECF could be considered as small with regards to bundle size, but according to > Eclipse dependency calculation it depends on 50 other bundles. On the other > hand our modularized proof of concept contains 20 bundles and is able to > install even Eclipse. Don't you think this justifies the abstraction we are > talking about? Any abstraction that decreases the number of required bundles from 50 to 20 is certainly worth while. I'm very surprised to learn that ECF would require 50 bundles! I thought that by using the simplest configuration (the URL based file transfer), ECF would be more or less self contained. Scott, can you shed some light on this?
(In reply to comment #5) > The intention was to isolate the bare minimum and make all other existing > functionality pluggable. If you need authentication, monitoring, etc you can > add the corresponding bundles and you get that functionality immediately. Our > efforts are in p2 modularization direction. I agree that the areas you both > mention bring a lot of value and we also plan to benefit from them in our > scenarios. But we believe that the right way ahead is to make them pluggable so > every user could use that subset of p2 functionality which best fits to his > requirements and nothing more. Currently it's not possible. > > ECF could be considered as small with regards to bundle size, but according to > Eclipse dependency calculation it depends on 50 other bundles. This calculation is off. ECF's filetransfer dependencies are very small...and it's possible to make them even smaller through some approaches I will describe. Here are the basic dependencies: ECF Core Filetransfer org.eclipse.ecf o.e.e.identity o.e.e.filetransfer o.e.e.provider.filetransfer These 4 core bundles have the following dependencies (on Equinox/OSGI) org.eclipse.equinox.common org.eclipse.core.jobs org.eclipse.equinox.concurrent org.eclipse.osgi org.eclipse.osgi.services optional - org.eclipse.core.net optional - org.eclipse.equinox.registry Although I haven't checked recently, I'm pretty sure that all of these dependencies (except perhaps the extension registry...and perhaps org.eclipse.equinox.concurrent) are required by p2/Equinox. Core.net (proxy support) is optional. As indicated above, the extension registry is also optional. It's (unfortunately) not marked as optional yet, but that should be considered a bug that will be likely be addressed by bug 310114. With the above four bundles alone you can get all the primary features of the filetransfer for p2. These bundles have a total on-disk size of ~300k. If SSL support is desired, these fragments can be added (and in Eclipse distro, they are included): o.e.e.ssl o.e.e.provider.ssl These two are a total of 13k, and are only needed for https/ssl support. If the httpclient provider is used (which it is in Eclipse, to get greater reliability over the JRE-based provider...particularly in older VMs), these ECF bundles are also needed: o.e.e.provider.filetransfer.httpclient o.e.e.provider.filetransfer.httpclient.ssl 74k The httpclient provider depends upon the Apache httpclient codebase, which has the following extent of dependencies: org.apache.commons.httpclient (314k) org.apache.commons.logging (45k) org.apache.commons.codec (54k) As you can see, more than half of the code size of ECF in the Eclipse distribution is because of the httpclient provider. In terms of modularization, dependencies, and code size, I think this is doing pretty good. All the features required by p2 (auth, asynchronous, multi-transport, etc) are available in ~300k of code in 4 bundles. Yes, things can be improved...e.g. the require-bundle dependency on the extension registry can be made optional (the code already allows this)...and suggestions and contributions along these lines are most welcomed. But there are actually quite a few extant options for eliminating/reducing things down (e.g. proxy support, ssl support, httpclient provider) while still allowing for p2's full use of multi-threaded (asynchronous) transfer, authentication, cancellation, etc. >On the other > hand our modularized proof of concept contains 20 bundles and is able to > install even Eclipse. Don't you think this justifies the abstraction we are > talking about? Given the above-described dependency structure...along with the modularization and deployment options with ECF (i.e. proxy support, multiple providers, ssl support, etc), I'm not as sure. I admit that there is some utility in having the smallest possible functioning, reasonably reliable p2 client installation, but I suspect that removing all of ECF and introducing another transport abstraction may not be the most effective way to get to that goal. One point about a synchronous file transfer API (IRepositoryTransport): since p2's usage of filetransfer is internal (i.e. to the planner and the engine) the use of blocking/synchronous I/O (IRepositoryTransport) internally means that the cancellation provided by the IProgressMonitor parameter (e.g. in calls like IPlanner.getProvisioningPlan) *may* not be honored. That is, calls to getProvisioningPlan could block for a very long time (or stop indefinitely)...because of I/O...without the ability to cancel these operations as implied by the IProgressMonitor.
OK, so to summarize, the total overhead of ECF is 4-6 bundles and about 300k. That's more in line with what I though. Does that really justify maintaining yet another abstraction layer for transport?
Guys, thanks for the detailed explanation. I admit I have been mislead by Eclipse dependency calculation - my first impression was that things are a lot more complicated. I've played a bit with what you have proposed and it seems reasonable. I understand now what you mean by introducing unnecessary abstraction layer. But there is something interesting I have noticed - p2 APIs are URL based, so one may expect that Java URL handling would be the default implementation. It's standard and what's more important - it provides extensibility. That's what we tried to achieve with the new abstraction layer, but I agree that this is maybe not the best approach. I'll appreciate very much any hints how it could be done better. Maybe ECF is worth to be switched to Java URL handling and all its features to be plugged on top? It seems pretty easy to provide protocols support for example. When talking about size and number of bundles, ~300K in 4 bundles may seem negligible in Eclipse IDE context, but currently the "isolated" version of p2 is ~1MB in 10 bundles. In that case ECF would be 1/3 of all p2 code. Maybe explaining how we are going to use the isolated version could help understanding why we find that further "modularization" so important. Our use case is to use p2 in automatic procedures. So there won't be anyone to benefit from most of the features in ECF - no need of monitoring, asynchronous processing, authentication, etc. That's why we could do pretty well with default Java URL handling. I agree with you that another abstraction layer is not the right approach. But could we do something in that direction with the existing one? Thanks again, looking forward to your further comments.
(In reply to comment #11) > Guys, thanks for the detailed explanation. I admit I have been mislead by > Eclipse dependency calculation - my first impression was that things are a lot > more complicated. I've played a bit with what you have proposed and it seems > reasonable. > > I understand now what you mean by introducing unnecessary abstraction layer. > But there is something interesting I have noticed - p2 APIs are URL based, so > one may expect that Java URL handling would be the default implementation. It's > standard and what's more important - it provides extensibility. That's what we > tried to achieve with the new abstraction layer, but I agree that this is maybe > not the best approach. I'll appreciate very much any hints how it could be done > better. Maybe ECF is worth to be switched to Java URL handling and all its > features to be plugged on top? It seems pretty easy to provide protocols > support for example. The main reason that the Java URL handling is not used in ECF filetransfer API is that it is based upon blocking/stream-based I/O (i.e. InputStream/OutputStream). The ECF filetransfer API was designed expressly to be non-blocking/asynchronous based-so that clients (like p2) could get a) multithreaded downloads b) reliable cancellation c) use jobs (i.e. the equinox jobs API, which is heavily used in Eclipse) I'm probably not listing them exactly as expressed in the original p2 requirements (which were conveyed some time ago), but these are the gist of the requirements on ECF WRT blocking/asynchronous file transfer. > > When talking about size and number of bundles, ~300K in 4 bundles may seem > negligible in Eclipse IDE context, but currently the "isolated" version of p2 > is ~1MB in 10 bundles. In that case ECF would be 1/3 of all p2 code. > Maybe explaining how we are going to use the isolated version could help > understanding why we find that further "modularization" so important. Our use > case is to use p2 in automatic procedures. So there won't be anyone to benefit > from most of the features in ECF - no need of monitoring, asynchronous > processing, authentication, etc. That's why we could do pretty well with > default Java URL handling. I agree with you that another abstraction layer is > not the right approach. But could we do something in that direction with the > existing one? 123k of 300k is in the org.eclipse.ecf.provider.filetransfer bundle. This is the JRE-based implementation of the filetransfer API (as well as other code to support other providers...like the httpclient provider), and it *could* be replaced by something much smaller, if desired. As the primary author I can say that the reason it's the size it is is to support the requirements put on ECF filetransfer by p2 originally (e.g. support alternative transports, asynchronous/non-blocking/cancelable, etc). So if a new/smaller version was created/contributed, I believe it could be used. The p2 code that accesses the ECF filetransfer provider might have to be refactored slightly (since there is some code there that uses API in o.e.e.provider.filetransfer...i.e. to switch providers for NTLMv2 proxy handling). Further, we (i.e. ECF) would be willing to entertain breaking the ECF filetransfer API into multiple bundles. Some further shrinkage might be possible. But without some injection of resources and/or external contributions we cannot commit to doing such work currently. But frankly I suspect that there will be a point of diminishing returns here...as honestly 300k doesn't seem like a lot to me given the requirements originally placed on ECF filetransfer by p2 (e.g. multi-transport, non-blocking, supports proxies, etc., etc.). But all contributions/patches along these lines will be considered.
Thanks for the hint - maybe a smaller version of o.e.e.provider.filetransfer bundle would work for us. I'll get back to you when we have a proposal.
A few months back I have explored something similar to what Katya is doing. The problem I was trying to solve is that when I'm embedding p2 in Maven/Tycho or in Nexus, I want to have the "host" application transport layer to be used (for example because of the user-oriented p2 way of things does not quite fit). It is indeed something that IBM also wanted to see happen in order to embed p2 in their webserver. Now even though I understand about introducing an API for transport that could limit our use in the future, I think this will be a recurring question from users, and we may find a compromise by creating a provisional API.
Pascal, do you mean a solution that would enable p2 to run without OSGi? I think that would be a fairly large step given that we then must abandon the use of ds and such.
(In reply to comment #14) > A few months back I have explored something similar to what Katya is doing. The > problem I was trying to solve is that when I'm embedding p2 in Maven/Tycho or > in Nexus, I want to have the "host" application transport layer to be used (for > example because of the user-oriented p2 way of things does not quite fit). It > is indeed something that IBM also wanted to see happen in order to embed p2 in > their webserver. Given the pluggable transport for ECF file transfer, I don't really understand this. Anyone can create an implementation of the ECF filetransfer API using whatever transport they wish (proprietary or open). Given the asynchronous nature of the API (which was one of the requirements for p2...i.e. for cancellability, , authentication, proxy, etc), I would say it's not all that complicated (it uses a fairly common listener/callback/event structure for asynchronous event delivery), and there is now a fair amount of support code to aid in the creation of providers based upon other transports (http-based or not). If the original p2 requirements that led to the development of an asynchronous file transfer API are still present (i.e. need for cancellation, non-blocking I/O, multi-threaded transfers, etc...on clients *or* servers...and I would suggest that even in servers that non-blocking I/O...and other features of the ECF API...are useful for a server-side filetransfer API), and the existing API allows (and in fact supports) the total replacement of all the transport with whatever OSGi code is desired, then I don't really understand why it's necessary to introduce a new API at all. Rather, why not just create a provider...using whatever code desired...e.g. if code size/deployment is your concern, then create a new JRE-URLConnection-based implementation that leaves off some of the features of the existing one (and so has smaller code size). Or create one using your favorite database (as I understand from ECF mailing list that some already have done), or create one using some other file transfer protocol (e.g. bittorrent...which we already have, incidently...but are not using in p2/Eclipse), or create one using your own custom file transfer API that uses a proprietary transport. > > Now even though I understand about introducing an API for transport that could > limit our use in the future, I think this will be a recurring question from > users, and we may find a compromise by creating a provisional API. It seems to me it would make more sense to a) implement new providers underneath that API (as described above) to meet specific requirements (like smaller deployment size), or introduce proprietary/private transports, etc. b) identify any areas where further *API* flexibility or changes are desired...and change/add to the existing API as needed in future versions of the API. So far, I haven't heard anything on this bug that suggests a limitation in the API itself (represented by org.eclipse.ecf.filetransfer code)...rather just a request to have alternative implementations (e.g. for code size, etc). In other words, I don't think new API is needed...especially since a new blocking-I/O API would i) have to be maintained (and I'm not going to maintain it) ii) create confusion (what API do you use to replace the p2 transport?) iii) since it would presumably be blocking I/O-based...such an API would potentially break the contracts of the existing p2 API (e.g. cancellability) In any event...I and others went to a fair amount of trouble to create a clean separation between an asynchronous, transport-independent file-transfer API and implementations (based upon http, or whatever protocol). It seems like a waste to introduce yet another API, and end up likely re-doing many of the things that were already done (e.g. authentication, proxy, etc., etc., etc). So my vote on new filetransfer API is -1 (if that's not obvious).
(In reply to comment #16) Hi Scott, Hi folks, IMHO that would not be a matter of voting and, moreover, arguing what "small" is. Together we have to think over how to make the things modular, pluggable and fulfill our needs. If we compare the current situation with purchasing small city car (simple file download), finally we will get a truck (4 bundles). Yes it is true - the truck could provide us with lots of functionality - power, draw-bar, cargo, etc. ... but finally it does not fit the desired needs - it could not be parked in your garage(or city center) and moreover most of its functionality is not used. Moving down to the problem, we have great framework with lots of abstraction and utilities but is that all necessary for simple download from an URL? I strongly believe, that we should make framework modular and its features possible for add, i.e. if you need more than simple download, that functionality should be pluggable. If it is not available, for example than exception might be thrown. Looking quickly through four bundles I noticed what is elaborate for a simple downloading: 1. identity bundle has unused additions - GUID, StringUtils, LongID, URIID, etc. 2. o.e.ecf has large and complex layer of abstraction (Container, ContainerManager, ContainerListener) for much more complex use cases 3. o.e.e.filetransfer - uses a few classes from ecf (ECFException, Event) and identity (ID, BaseID, IIdentifiable). These classes pull lots of bundles and functionality (remember the city car->truck example) - lots of events (I guess, few use them) 4. o.e.e.provider.filetransfer - unnecessary abstraction - wrapped URL into IFileID in AbstractRetrieveFileTransfer.sendRetrieveRequest():881. Why IFileID (identity bundle) is needed than? A possible proposal for solution that quickly comes to my mind (I guess a better could be invented): 1. To remove dependencies of "o.e.ecf.filetransfer": o.e.ecf.filetransfer -> o.e.ecf and o.e.ecf.filetransfer -> o.e.e.identity 2. Hence, rework some of the "o.e.ecf.filetransfer"’s classes(exceptions not to implement ECFException, other - IAdaptable). Rework IFileId usage. I believe FileTransfer bundle’s interfaces should stay mostly the same as they are now. 3. As a result, anybody might be provider of a so called own file transfer "o.e.ecf.provider.filetransfer". If some of its service is not available, then UnsupportedException might be thrown. Finally, it is obvious how large and functional the things are now which is great, However, I hope we might modulate them. As a result, it will be possible to have the functionality that it is needed (for example, only 2 bundles - filetransfer + provider.filetransfer) and make all other pluggable and still available. Regards Rado
(In reply to comment #17) > (In reply to comment #16) > Hi Scott, Hi folks, > > IMHO that would not be a matter of voting and, moreover, arguing what "small" > is. Together we have to think over how to make the things modular, pluggable > and fulfill our needs. > > If we compare the current situation with purchasing small city car (simple file > download), finally we will get a truck (4 bundles). Yes it is true - the truck > could provide us with lots of functionality - power, draw-bar, cargo, etc. ... > but finally it does not fit the desired needs - So I would like to point out that the existing core functionality of the file transfer API...i.e. asynchronous/non-blocking download, along with transport independence, support for authentication, use of core proxy API, etc. were dictated *by the original p2 transport requirements*. They weren't imposed by me or anyone else. Yes, those requirements were based upon Eclipse's needs for p2/update manager, but I believe many of those core requirements (e.g. non-blocking I/O/reliable cancellation) are likely to exist...even in small environments...especially since the existing API is quite modular, and not at all 'large' by objective measures. >it could not be parked in your > garage(or city center) and moreover most of its functionality is not used. > Moving down to the problem, we have great framework with lots of abstraction > and utilities but is that all necessary for simple download from an URL? Like I said above, the framework/functionality present was added to meet the original p2 requirements...for reliability, cancellability, transport-independence, etc. As a useful analogy, OSGI BundleContext.installBundle can be used to dynamically install bundles in a framework (i.e. without p2 at all). But of course it's very limited relative to p2. Similarly, simple download from an URL can of course be implemented with URLConnection...but by doing so you *lose* the things that led to the original requirements on ECF filetransfer from p2: e.g. cancellability/asynchrony/non-blocking i/o, authentication, etc. If you want to do this with p2 for your product then I wouldn't try to stop you but I would question the wisdom of doing it as API...that has to be maintained...as well as explained when it's limitations are encountered by other clients. >I > strongly believe, that we should make framework modular and its features > possible for add, i.e. if you need more than simple download, that > functionality should be pluggable. If it is not available, for example than > exception might be thrown. > > Looking quickly through four bundles I noticed what is elaborate for a simple > downloading: > 1. identity bundle has unused additions - GUID, StringUtils, LongID, URIID, > etc. > 2. o.e.ecf has large and complex layer of abstraction (Container, > ContainerManager, ContainerListener) for much more complex use cases > 3. o.e.e.filetransfer > - uses a few classes from ecf (ECFException, Event) and identity (ID, BaseID, > IIdentifiable). These classes pull lots of bundles and functionality (remember > the city car->truck example) Referring to these classes as 'lots of bundles and functionality' is a mischaracterization. The other bundles referred to are minimal Equinox dependencies (common and jobs). The total size of the unused classes you refer to is less than 10k of code. Yes, there are some of the ECF core classes that are not used by P2 filetransfer, but again this is not at all a large codebase, and the abstraction does provide the transport independence required (e.g. the ID and subclasses). But even so, it is possible that some of the ECF core interfaces/classes that are unused in P2 could be refactored so that they could be used in separate bundles...but like I said the total amount of these unused (in p2) classes does not seem that large...and it seems to me like a waste of effort relative to looking to other areas for code size reductions. > - lots of events (I guess, few use them) > 4. o.e.e.provider.filetransfer > - unnecessary abstraction - wrapped URL into IFileID in > AbstractRetrieveFileTransfer.sendRetrieveRequest():881. > Why IFileID (identity bundle) is needed than? Same comment as above...yes these event classes are unused by p2, but they are used for other parts of ECF. Yes they could be refactored, but would the benefit be worth it? Any proposed refactorings for size improvements would be considered...as long as resources are identified to actually do them. > > A possible proposal for solution that quickly comes to my mind (I guess a > better could be invented): > 1. To remove dependencies of "o.e.ecf.filetransfer": o.e.ecf.filetransfer -> > o.e.ecf and o.e.ecf.filetransfer -> o.e.e.identity > 2. Hence, rework some of the "o.e.ecf.filetransfer"’s classes(exceptions not to > implement ECFException, other - IAdaptable). Rework IFileId usage. I believe > FileTransfer bundle’s interfaces should stay mostly the same as they are now. > 3. As a result, anybody might be provider of a so called own file transfer > "o.e.ecf.provider.filetransfer". If some of its service is not available, then > UnsupportedException might be thrown. This doesn't really work. The dependencies on o.e.ecf and o.e.e.identity and equinox common are integral. > > Finally, it is obvious how large and functional the things are now which is > great, However, I hope we might modulate them. As a result, it will be possible > to have the functionality that it is needed (for example, only 2 bundles - > filetransfer + provider.filetransfer) and make all other pluggable and still > available. Again, I'm open to bundle-level refactorings that reduce size while maintaining required functionality. But I can't commit to doing that work myself without some resources identified, and I don't happen to believe that the size reductions we are discussing are large enough to justify the amount of work involved (i.e. diminishing returns)...but if others disagree about the size of benefits relative to cost then they are welcome to do that work.
Guys, we have tried to implement what we've talk about above. Here's the summary: There are three different approaches: 1. Remove dependencies from "ecf.filetransfer" to "ecf" (actually used classes are only ECFException, Event ) and "ecf.identity" (only classes Namespace, ID) That approach works for p2.repository downloads but breaks compatibility of current "ecf.filetransfer" bundle 2. Extract "ecf.filetransfer.core" (from "ecf.filetransfer") that does not depend on ecf and "ecf.identity". That approach works for p2.repository downloads but breaks compatibility of current "ecf.filetransfer" bundle 3. Implement new "filetransfer" bundle api that has no dependencies. Moreover, p2.repo will use it and "ecf.provider.filetransfer" will implement it (forward to current implementation). In order to decouple p2 from unused functionality, a new lean "provider.filetransfer" might be implemented. Last time we talked, we agreed that the third option is the best of all - on one hand it would allows us to use p2 independently from ecf, on the other - we won't introduce unneeded level of abstraction. It seemed that no matter that the idea is clear, the implementation turns out to be clumsy. A lot of (dummy) interfaces should be wrapped in order to keep the current communication between ecf bundles. That wrapping makes the source unreadable and increases complexity a lot. (I can provide technical details here if you are interested) Taking into account the result, I don't think this is the right approach. Especially considering the fact that ECF is going to be used standalone(outside Eclipse) ever more and more and for other use cases besided p2. A great example for that is ECF considered as the best implementation for discovery mechanism in distributed OSGi services use case. As I understood, for that purpose ECF should be refactored not to use Eclipse extension points. I guess the direction is changing now and it's worth to discuss again the creation of provisional API and whatever other refactoring is needed? We can help for removing the extension points for example. It would be pretty easy because we can use the same approach as used in the proposal for touchpoints (https://bugs.eclipse.org/bugs/show_bug.cgi?id=325328) We can start with the existing use cases: - creating provisional API - removing extension points from ECF and discuss any oncoming requirements when they appear. What do you think about that?
(In reply to comment #19) > Guys, we have tried to implement what we've talk about above. Here's the > summary: > > There are three different approaches: > > 1. Remove dependencies from "ecf.filetransfer" to "ecf" (actually used classes > are only ECFException, Event ) and "ecf.identity" (only classes Namespace, ID) > That approach works for p2.repository downloads but breaks compatibility of > current "ecf.filetransfer" bundle > > 2. Extract "ecf.filetransfer.core" (from "ecf.filetransfer") that does not > depend on ecf and "ecf.identity". > That approach works for p2.repository downloads but breaks compatibility of > current "ecf.filetransfer" bundle > > 3. Implement new "filetransfer" bundle api that has no dependencies. Moreover, > p2.repo will use it and "ecf.provider.filetransfer" will implement it (forward > to current implementation). > In order to decouple p2 from unused functionality, a new lean > "provider.filetransfer" might be implemented. I don't really agree that these are your only options. Another option: Simply use (and improve and/or shrink) the JRE-based provider that's already in org.eclipse.ecf.provider.filetransfer. Another one (this may be what you are saying with option 3, but I'm not sure): Create your own JRE-based provider and remove/eliminate org.eclipse.ecf.provider.filetransfer. Because of a workaround for the NTLMv2 proxy issue this may require some changes to p2 (code written/worked on by Cloudsmith people I believe), but I expect that would be doable as well. <stuff deleted> > Taking into account the result, I don't think this is the right approach. > Especially considering the fact that ECF is going to be used standalone(outside > Eclipse) ever more and more and for other use cases besided p2. A great example > for that is ECF considered as the best implementation for discovery mechanism > in distributed OSGi services use case. As I understood, for that purpose ECF > should be refactored not to use Eclipse extension points. Although it's very true that other parts of ECF (e.g. discovery, remote services, etc) are useful for other distributed OSGi use cases, it seems beside the point to me...as these other parts of ECF are not *in* p2 at the moment. They are separate modules (bundles) that are only needed (and only need to be present) for those other use cases. And WRT the extension registry...many/most of ECF already has the extension registry as an optional dependency. For example, ECF core/identity can run without the extension registry...as can the discovery API, and other parts of ECF. The filetransfer provider currently does depend upon the extension registry, but this can/could possibly be relaxed to make it optional as well. > > I guess the direction is changing now and it's worth to discuss again the > creation of provisional API and whatever other refactoring is needed? We can > help for removing the extension points for example. It would be pretty easy > because we can use the same approach as used in the proposal for touchpoints > (https://bugs.eclipse.org/bugs/show_bug.cgi?id=325328) > > > We can start with the existing use cases: > - creating provisional API > - removing extension points from ECF > and discuss any oncoming requirements when they appear. > > What do you think about that? I would have no problem with an effort to make the dependency on the extension registry optional in the ECF filetransfer API (although I sort of question whether it's worth it, but that's an individual judgment call). I don't, however, have any personal resources to dedicate to such an effort, and I am the ECF committer who has been maintaining the filetransfer work for a while now. I'm happy to answer questions, but without some contributions I doubt that I'm going to have resources in this release cycle even for testing of the refactorings. Also...just as a caution...I don't intend to jeopardize the stability of ECF filetransfer for p2/Eclipse, so am hesitant to have large-scale refactorings undertaken without some direct involvement by people who are familiar with the existing code. If you wish to contact me directly, I welcome you to do so at slewis at composent.com.
Katya, I'm sorry but I don't follow where you are suggesting to add the new API. In the past we talked about introducing a thin transport layer (along the line of RepositoryTransport) to p2 that would decouple from ECF. Is this what you are talking about?
(In reply to comment #21) > Katya, I'm sorry but I don't follow where you are suggesting to add the new > API. In the past we talked about introducing a thin transport layer (along the > line of RepositoryTransport) to p2 that would decouple from ECF. Is this what > you are talking about? a clarification... The main requirement (from p2) that led to ECF creating a filetransfer API in the first place was asynchronous/non-blocking/cancellable filetransfer. There were other requirements as well (e.g. use of platform proxy API for proxy support, support for partial filetransfers, integration with jobs API, support for multiple protocols...e.g. http, file, torrent, foo, etc), but the primary one that implies an API separate from URL.getContent() was the asynchronous support. Now...given this, I don't think another transport layer for p2 is well-advised. Reason: If it's asynchronous/non-blocking, it will not be 'thin' in the sense being asked for...i.e. it will be another/new API (i.e. not using the JRE blocking I/O)...essentially duplicating ECF's IRetrieveFileTransferContainerAdapter and IBrowseFileTransferContainerAdapter. If it's synchronous/blocking-IO-based it will mean that p2's API will then be unable to meet it's cancellability contracts (as filetransfers can/do occur during planning and provisioning operations, I believe...and if those filetransfers can't be cancelled, then the planning can't be cancelled). Another Reason: In either case (synchronous or asynchronous) it will have to be supported and maintained as a separate API...I/ECF can't/won't support it...and I don't suspect the p2 team really wants an long stream of new bugs of the form 'p2 install can't be cancelled'. Also...we've already got a JRE/URLConnection-based implementation of the filetransfer API...which could be improved, made more simple/smaller, subclassed, or even just used as a guide to creating a new, much small/simpler JRE-based (or other lib-based) implementation. P2 can/does use this other provider with no changes...and it's usage reduces the overall size significantly, by eliminating the httpclient bundles and their requirements (apache httpclient 3.1.0). And finally, we/I am open to making doing things like adding optional dependencies (e.g. on Equinox registry) for the filetransfer API...it's just that I can't commit to doing it without community resources identified to do so.
Merry Xmas! I just released a fix for this. Recap of the changes: - *no* new API. Given that the audience for this is limited (Sonatype, IBM server team, SAP), I have decided to *not* create an official API and simply separate things up so someone could hook a new transport without ECF (yes it is not trivial). - new bundle org.eclipse.equinox.p2.transport.ecf - lookup of the Transport service through the agent. - updated products and features to include the new bundle Now, someone need to go through the exercise of verifying that a new transport can actually be used. Tags in CVS: - pre_Bug_321819 for the state before I changed anything - post_Bug_321819 for the state after I changed things.
