Community
Participate
Working Groups
The ECF file transfer API does not expose a method to start (and end) a partial file transfer...i.e. starting from a point in the file other than at the beginning, and/or ending the transfer other than at the end of the given file. To support the Equinox provisioning work, and the download manager, this functionality will be added, as described by notes here http://wiki.eclipse.org/Equinox_Summit_2007_Results#ECF.2FTransports.2FRepositories
Added following method to IRetrieveFileTransferContainerAdapter /** * Send request for transfer of a remote file to local file storage. This * method is used to initiate a file retrieve for a remoteFileID (first * parameter). File transfer events are asynchronously delivered a file * transfer listener (third parameter). The given remoteFileID and * transferListener must not be null. * * @param remoteFileID * reference to the remote target file (e.g. * http://www.eclipse.org/index.html) or a reference to a * resource that specifies the location of a target file. * Implementing providers will determine what protocol schemes * are supported (e.g. ftp, http, torrent, file, etc) and the * required format of the scheme-specific information. If a * protocol is specified that is not supported, or the * scheme-specific information is not well-formed, then an * IncomingFileTransferException will be thrown. Typically, * callers will create IFileID instances via calls such as: * * <pre> * IFileID remoteFileID = FileIDFactory.getDefault().createID( * ftc.getRetrieveNamespace(), "http://www.composent.com/index.html"); * </pre> * * Must not be <code>null</code>. * @param rangeSpecification a range specification for retrieving a portion of the given * remote file. If <code>null</code> the entire file will be retrieved (as per {@link #sendRetrieveRequest(IFileID, IFileTransferListener, Map)}. * If non-<code>null</code> the given file range will be used to retrieve the given file. For example, if the * rangeSpecification has a start value of 1 and end value of 3, and the total length of the file is * 5 bytes with content [a, b, c, d, e], a successful retrieve request would transfer bytes 'b', 'c', and 'd', but not 'a', and 'e'. * @param transferListener * a listener for file transfer events. Must not be null * @param options * a Map of options associated with sendRetrieveRequest. The * particular name/value pairs will be unique to the individual * providers. May be <code>null</code>. * @throws IncomingFileTransferException * if the provider is not connected or is not in the correct * state for initiating file transfer */ public void sendRetrieveRequest(IFileID remoteFileID, IFileRangeSpecification rangeSpecification, IFileTransferListener transferListener, Map options) throws IncomingFileTransferException; IFileRangeSpecification (new interface) looks like this: public interface IFileRangeSpecification { /** * Get the start position to start from. The position is in bytes, and byte 0 is the first byte * of the file, N-1 is the last position in the file, where N is the length of the file in bytes. * @return the position in the file (in bytes) to start from. If the returned start position is * less than 0, or equal to or greater than N, then it is an invalid range specification and * will result in a InvalidPartialFileTransferRequestException. */ public long getStartPosition(); /** * Get the end position of transfer range. The position is in bytes, and byte 0 is the first byte * of the file, N-1 is the last position in the file, where N is the length of the file in bytes. * @return the position in the file (in bytes) to indicate the end of range to retrieve. If equal to -1, * then this means that no end position is specified, and the download will continue to the end of file. If >= 0, * but less than the {@link #getStartPosition()} then this range specification is invalid. If greater than or * equal to N (where N is length of the file in bytes), then the remaining part of the given file will * be downloaded. */ public long getEndPosition(); } Any comments/criticisms welcome of this API for initiating partial file retrievals (i.e. retrieval of the part of a file defined by startRange.getStartPosition(), startRange.getEndPosition() welcome. One question coming out of this: for implementations that *do not* support retrieving file ranges what should this API do? Should it throw an IncomingFileTransferException, complete by respond immediately (via listener) with an InvalidFileRangeSpecificationException? Or should it retrieve the entire file without complaint? I'm currently leaning toward throwing an InvalidFileRangeSpecificationException immediately upon calling sendRetrieveRequest.
Added implementation of new API described in comment 1. Currently, the only provider that supports this new API is the org.eclipse.ecf.provider.filetransfer 'default' provider, based upon the JRE UrlConnection class. This is the provider that is currently being used for the provisioning work. For this provider, partial retrieve only works for http/https, and then only if the server is an http 1.1-compliant server. Other providers (or 1.0 servers) throw an exception on usage. We will soon persue adding implementations for other providers (e.g. httpclient and bittorrent), but would like to get some feedback on the API before doing other implementations. Also added/tested new test case class PartialFileRetrieveServiceTest in org.eclipse.ecf.tests.filetransfer plugin. This test case tests the basic API. Will be enhancing these tests as we go along. So please have a look at API and try out implementation/tests and provide feedback for API on this bug and/or file new bugs (if problems found through testing, etc).
(In reply to comment #2) > We will soon persue adding implementations for other providers (e.g. httpclient > and bittorrent), but would like to get some feedback on the API before doing > other implementations. I don't think that this API makes a lot of sense in the context of BitTorrent. Can the p2 team provide some insight if this API is needed in the BitTorrent world?
(In reply to comment #3) > (In reply to comment #2) > > We will soon persue adding implementations for other providers (e.g. httpclient > > and bittorrent), but would like to get some feedback on the API before doing > > other implementations. > > I don't think that this API makes a lot of sense in the context of BitTorrent. > Can the p2 team provide some insight if this API is needed in the BitTorrent > world? > Of course Tim, Pascal, Jeff, and others should comment further on this, but as I understand it the requirement for this API coming out of the Equinox summit was the desire to do partial downloads and have across-Eclipse session caching of those partial downloads. Obviously, this will only work for providers that can support it. If bittorrent can't support it then we'll obviously limit to providers that can.
(In reply to comment #4) > Of course Tim, Pascal, Jeff, and others should comment further on this, but as > I understand it the requirement for this API coming out of the Equinox summit > was the desire to do partial downloads and have across-Eclipse session caching > of those partial downloads. Obviously, this will only work for providers that > can support it. If bittorrent can't support it then we'll obviously limit to > providers that can. BitTorrent already supports pause/resume. When you start a file transfer pointing at the exact same target file / directory, it will automatically resume if applicable. At the moment, if you want to start from scratch, you have to delete all of the output files (the target file itself or the contents of the target directory).
What you describe makes perfect sense in the context of the download manager strategy I am looking at for p2. It's perfectly acceptable if only a subset of providers can support this functionality. In the case of BitTorrent, I'm not sure if this is possible -- but again, the algorithm that we were looking to use this was specifically for transfer types that would benefit by switching from one mirror to another. In the case of BitTorrent, the torrent algorithm itself is already going to optimize getting the best connection -- having the p2 download manager strategy attempt to further optimize would not make sense.
(In reply to comment #6) > In the case of BitTorrent, I'm not > sure if this is possible -- but again, the algorithm that we were looking to > use this was specifically for transfer types that would benefit by switching > from one mirror to another. In the case of BitTorrent, the torrent algorithm > itself is already going to optimize getting the best connection -- having the > p2 download manager strategy attempt to further optimize would not make sense. If you're just going to switch mirrors (trackers, in BT terms) and point it at the same file/folder, assuming the torrent is defining the same files and has the same hash contents (hash does not _have_ to be the same given some scenarios, but to keep a long story short...), it should automatically resume as is after performing a hash comparison.
This API (IFileTransferContainerAdapter.sendRetrieveRequest(IFileID remoteFileID, IFileRangeSpecification rangeSpecification, IFileTransferListener transferListener, Map options) is now working with both the UrlConnection-based provider (org.eclipse.ecf.provider.filetransfer) and with the Apache httpclient 3.0.1-based provider (org.eclipse.ecf.provider.filetransfer.httpclient). If the httpclient provider bundle is present, it will override the org.eclipse.ecf.provider.filetransfer implementation. Test code for the partial file download is here: http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.ecf/tests/org.eclipse.ecf.tests.filetransfer/src/org/eclipse/ecf/tests/filetransfer/PartialFileRetrieveServiceTest.java?root=Technology_Project&view=log Test code for the associated pause/resume (bug #204836) is here http://dev.eclipse.org/viewcvs/index.cgi/org.eclipse.ecf/tests/org.eclipse.ecf.tests.filetransfer/src/org/eclipse/ecf/tests/filetransfer/PauseResumeServiceTest.java?root=Technology_Project&view=log This API and impls will be in ECF 1.2, release 10/19/2007.
I'm resolving this bug. The described API and the two implementations will both be in ECF 1.2 release scheduled for 10/19/2007. Please reopen if necessary.
