Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 334698

Summary: Batch import and export
Product: [ECD] Orion Reporter: John Arthorne <john.arthorne>
Component: ClientAssignee: John Arthorne <john.arthorne>
Status: RESOLVED FIXED QA Contact:
Severity: enhancement    
Priority: P2 CC: mail
Version: 0.2   
Target Milestone: 0.2   
Hardware: PC   
OS: Windows 7   
Whiteboard:
Bug Depends on: 334824    
Bug Blocks: 334720    

Description John Arthorne CLA 2011-01-18 15:38:34 EST
We should support importing a large set of files into a project in one shot. For example uploading a zip file, or import from arbitrary URL.

We should also support exporting to zip. This would be some form of GET on a directory tree that returns a zip of the entire tree.
Comment 1 John Arthorne CLA 2011-01-18 15:40:47 EST
I have specified and implemented a server API for performing import and chunked file upload:

http://wiki.eclipse.org/Orion/Server_API/Transfer_API
Comment 2 John Arthorne CLA 2011-01-18 15:42:00 EST
What's still missing:

 - Upload client implementation

 - Export server API and implementation
Comment 3 Sundar CLA 2011-02-01 18:23:04 EST
Thought about the import, export concept and i would like to add the following after reading the Transfer API specification (http://wiki.eclipse.org/Orion/Server_API/Transfer_API).

As far as i understood, API does not cover the error conditions like the following yet,
- Error response if there is a server failure. So that client can be notified. Will it be HTTP 500?
For e.g. Lets say user uploaded a password protected file and server could not unzip it. What is the error message provided as a response?
- What if the file/folder already exists on the server side. Will the user be warned before overwriting?


--------------------------------------------------------------

The proposal i have regarding export API is as below:
- User should be able to download the whole folder and its contents in a zip ile.
- User should be able to download individual files separately. Individual files need not be exported as zip files.
- A separate header for differentiation which type of file user wants to download.


I am very much interested in working on this offline and submit the code changes. So if code changes are welcome please let me know, i will start work on this immediately. Any pointers to go forward with the implementation is highly appreciated.
Comment 4 John Arthorne CLA 2011-02-02 15:58:43 EST
(In reply to comment #3)
> As far as i understood, API does not cover the error conditions like the
> following yet,
> - Error response if there is a server failure. So that client can be notified.
> Will it be HTTP 500?

That depends on the failure ;) All HTTP error codes are possible. Generally we wouldn't specify a 500 response in particular, since that is for unexpected server failures. However we can/should certainly specify any 4xx response codes that are applicable here. The transfer API was written hastily last week and could use some clean up like that.

> For e.g. Lets say user uploaded a password protected file and server could not
> unzip it. What is the error message provided as a response?

Hmm, if we fail to read the zip perhaps 415 Unsupported Media Type is appropriate.

> - What if the file/folder already exists on the server side. Will the user be
> warned before overwriting?

By default, no. However we should support a "no-overwrite" option that would fail in this case, like we do for our other creation methods:

http://wiki.eclipse.org/Orion/Server_API/File_API#Notes_on_POST_method
> 
> --------------------------------------------------------------
> 
> The proposal i have regarding export API is as below:
> - User should be able to download the whole folder and its contents in a zip
> ile.
> - User should be able to download individual files separately. Individual files
> need not be exported as zip files.
> - A separate header for differentiation which type of file user wants to
> download.

I don't quite follow the reason for this. The "raw" file can be obtained via the Orion file API. The reason we have something like this for import is to support resumable or chunked upload of large files which is atomic (either the entire upload completes, or the file on the server is left untouched). For download none of this seems applicable. The file API could support range headers to support downloading in chunks if that's what you are looking for.

> I am very much interested in working on this offline and submit the code
> changes. So if code changes are welcome please let me know, i will start work
> on this immediately. Any pointers to go forward with the implementation is
> highly appreciated.

Contributions to both specification and implementation are most welcome! To get started see:

http://wiki.eclipse.org/Orion/Getting_the_source

The implementation is in package org.eclipse.orion.internal.server.servlets.xfer.

Current unit tests can be found in org.eclipse.orion.server.tests.servlets.xfer

You can ping on IRC #eclipse-orion if you need any help getting going (or on this bug).
Comment 5 John Arthorne CLA 2011-02-03 14:13:28 EST
Marking this fixed, since we had basic import/export working in M5. I have opened bug 336274 for the issue of handling overwrite. Sundar, feel free to open other bugs for suggested changes here.