Bugzilla – Full Text Bug Listing
| Summary: | [server] provide simple metadata storage | ||
|---|---|---|---|
| Product: | [ECD] Orion | Reporter: | Anthony Hunter <ahunter.eclipse> |
| Component: | Server | Assignee: | Anthony Hunter <ahunter.eclipse> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | enhancement | ||
| Priority: | P3 | CC: | john.arthorne, ken_walker, malgorzata.tomczyk, simon_kaegi |
| Version: | 4.0 | ||
| Target Milestone: | 4.0 M2 | ||
| Hardware: | PC | ||
| OS: | Linux | ||
| Whiteboard: | |||
| Bug Depends on: | 413655, 414592, 414596, 415000, 415505, 415700, 415796, 415985, 415989, 416089, 416661, 416952, 416956, 418230 | ||
| Bug Blocks: | |||
|
Description
Anthony Hunter
I tried to roll up what we have talked about so far into http://git.eclipse.org/c/orion/org.eclipse.orion.design.git/plain/mock-ups/Projects/Bug412995.html . Please let me know if there are any comments. Design Highlights: Each user has their own folder with a user.json and a list of folders, each of which is a workspace. Workspaces and projects are organized in folders in a hierarchy under the user they belong to. A workspace is a folder containing workspace.json. A project is a folder containing project.json. I tried to roll up what we have talked about so far into http://git.eclipse.org/c/orion/org.eclipse.orion.design.git/plain/mock-ups/Projects/Bug412995.html . Please let me know if there are any comments. Design Highlights: Each user has their own folder with a user.json and a list of folders, each of which is a workspace. Workspaces and projects are organized in folders in a hierarchy under the user they belong to. A workspace is a folder containing workspace.json. A project is a folder containing project.json. Second draft of the design is at http://git.eclipse.org/c/orion/org.eclipse.orion.design.git/plain/mock-ups/Projects/Bug412995.html?id=1b30056e1ad4cc642ef99a4c2b0524a0791a620a This is a summary of the conversation I have with Anthony offline: The GET on REST API /workspace/workspaceId/project/projectId is currently not used by the client. The only place we are using this location is in a param in Rename and Delete commands. We could easily replace it with /file/... location and the rid of this API at all. (In reply to comment #4) > This is a summary of the conversation I have with Anthony offline: > The GET on REST API /workspace/workspaceId/project/projectId is currently > not used by the client. The only place we are using this location is in a > param in Rename and Delete commands. We could easily replace it with > /file/... location and the rid of this API at all. Agreed, I spoke with Simon and he recommends we get rid of the workspace API. All the time there is a terminology problem to distinguish between "top level folder" and "project". We agreed that not every top level folder is a project, but only the one that has project.json in it. Is there a possibility that the new API solves this overlap and and uses "Project" for the actual project and something else (like "Top Folder") for the top level folder? (In reply to comment #6) > All the time there is a terminology problem to distinguish between "top > level folder" and "project". We agreed that not every top level folder is a > project, but only the one that has project.json in it. Is there a > possibility that the new API solves this overlap and and uses "Project" for > the actual project and something else (like "Top Folder") for the top level > folder? The top level for a user is discovered by calling a GET /workspace It returns a list of workspaces and Orion selects the first one as the top level. When you create a new user it creates a workspace with the name "Orion Content" You then effectively create folders/projects under the "Orion Content" folder, so the workspace is the top level folder. (In reply to comment #5) > Agreed, I spoke with Simon and he recommends we get rid of the workspace API. This work is Bug 415700 [Projects] reimplement Workspace API from server-side to client-side A third draft of the design is at: http://git.eclipse.org/c/orion/org.eclipse.orion.design.git/plain/mock-ups/Projects/Bug412995.html?id=1e988c779a6650fbb638ce551c76d415be48d343 In terms of the metadata, the work is complete and we have a migration. I have updated the Next Steps in the design. I want to reiterate that we want to complete the final two design goals: A workspace and project folder tree is a sharable resource (commit into git, zip and copy). A workspace and project tree can be created and updated client-side. The work for this is described in Bug 415700 . I supposed we should confirm we are all on the same page here. (In reply to Anthony Hunter from comment #9) > A third draft of the design is at: > http://git.eclipse.org/c/orion/org.eclipse.orion.design.git/plain/mock-ups/ > Projects/Bug412995.html?id=1e988c779a6650fbb638ce551c76d415be48d343 > > In terms of the metadata, the work is complete and we have a migration. I > have updated the Next Steps in the design. > > I want to reiterate that we want to complete the final two design goals: > A workspace and project folder tree is a sharable resource (commit into git, > zip and copy). > A workspace and project tree can be created and updated client-side. > > The work for this is described in Bug 415700 . I supposed we should confirm > we are all on the same page here. I forgot two additional comments: We wanted to to allow for the saving of preferences against the workspace or project, currently all preferences are saved server side with the user. Once the preferences are available in the workspace.json or project.json, they can be shared. We have a lot of server side code that depends on IWebResourceDecorator, not exactly sure how to handle this client side yet (which makes it a good time to ask if we are working towards the same design). (In reply to Anthony Hunter from comment #9) > The work for this is described in Bug 415700 . I supposed we should confirm > we are all on the same page here. Anthony, Simon and I got together today and talked about this. Here is my summary of what we talked about: - From the client's perspective, the server will expose a simple tree of resources (files/folders) for each user. Most of the functionality currently provided by the "workspace" service can evolve to be covered by the file service. I.e., to create a "project" you don't need to use a separate service, it is just creating a folder under that user. Most of the Orion client code will move to using the file service for everything. The possible exception is the special support we have on the server for "linked projects", and "SFTP-backed projects". Since these are specific to the Java Orion server maybe it is ok to keep these in a separate service and it is not a capability that other Orion servers (e.g., node.js) need to implement - All user settings, and metadata needed by client side tools, will be reified as simple json files in the user's file tree (project.json, perhaps also workspace.json). The server will not know anything about this metadata, and will provide no API or capability for manipulating it. This enables the client side metadata to be shared, and lowers the burden on server implementations (since this metadata is just another file for the server to store). - There will still need to be some very minimal server-side metadata that is not exposed to clients. This is what the IMetaStore was intended to capture. Examples of information in this store: server file system paths, authorization data, metadata used by server side tools (e.g., server-based deployment tooling). We will store this metadata in the same file system location on the server to make cleanup and management easier, but will not return these files to the client. The idea is that they contain nothing of interest to either user or client plugins, and in some cases there is a security problem with exposing them. We seemed to be on a path to converge server metadata and client metadata into one concept, but I strongly thing we will actually need both server-based and client-based metadata, at least in the Java server implementation. We should be using client-based metadata everywhere we possibly can, but there are some limited cases where server metadata is needed. I have delivered http://git.eclipse.org/c/orion/org.eclipse.orion.server.git/commit/?id=fa13eaf991b98bd5db38c09f975f553e7006e5ea The goal of this commit was to expose the project.json to the client when the Workspace API is used to create a project. So now when you create a top level folder on the client you get the project.json. Gorza, you can experiment with this by changing the setting in your orion.conf (web-ide.conf) to orion.core.metastore=simple This is preliminary, since this project.json includes the "server metadata", specifically the project's content location that should not be exposed to the client. It is at least start to begin sharing the work. (In reply to Anthony Hunter from comment #12) > Gorza, you can experiment with this by changing the setting in your > orion.conf (web-ide.conf) to I'll try it out tomorrow. (In reply to Anthony Hunter from comment #12) > This is preliminary, since this project.json includes the "server metadata", > specifically the project's content location that should not be exposed to > the client. It is at least start to begin sharing the work. I think we have a difference here. If your project.json contains "server metadata" and lives in the main folder then this file will be shared in by Git, for instance. When I was handling the project.json I made sure that stuff stored in it can be reused in other workspaces, the same way .project can be reused in desktop Eclipse. (In reply to Malgorzata Janczarska from comment #14) > (In reply to Anthony Hunter from comment #12) > > This is preliminary, since this project.json includes the "server metadata", > > specifically the project's content location that should not be exposed to > > the client. It is at least start to begin sharing the work. > > I think we have a difference here. If your project.json contains "server > metadata" and lives in the main folder then this file will be shared in by > Git, for instance. When I was handling the project.json I made sure that > stuff stored in it can be reused in other workspaces, the same way .project > can be reused in desktop Eclipse. You are correct and as I said we should to not expose the "server metadata" to the client. I key here is to work on our Orion 4.0 story. I expect we are going to stick with using the server workspace API to create workspaces and projects/folders, so now we need to finalize how this will work. OK, I did some more testing and I am going to have to rethink the layout, since we cannot git clone into a folder that already contains a project.json. So automatically adding a project.json on a top level folder creation server side is not going to work. Fourth draft of the design is at: http://git.eclipse.org/c/orion/org.eclipse.orion.design.git/plain/mock-ups/Projects/Bug412995.html?id=fa5c21a4bb1e451b7bc10baf0590b5d289498dda This is what I plan to deliver and migrate to for 4.0. When I was originally planning this out, I had imagined a design where the client and server would share the same meta data and there was one project.json. This is not going to work, so we are going to have to have a server project.json and a client project.json. The server project metadata is maintained in the same folder as the workspace.json. See the above design page for the details. (In reply to Anthony Hunter from comment #12) > Gorza, you can experiment with this by changing the setting in your > orion.conf (web-ide.conf) to > > orion.core.metastore=simple I tried this today and looks like you are setting wrong ContentLocation in [ProjectName].json, I get 404 for every project created in the workspace. When I looked inside the user name in ContentLocation was missing, it was "file:/C:/some/path/foo/OrionContent/SomeProject" and should be "file:/C:/some/path/foo/gosia/OrionContent/SomeProject" (In reply to Malgorzata Janczarska from comment #18) > (In reply to Anthony Hunter from comment #12) > > Gorza, you can experiment with this by changing the setting in your > > orion.conf (web-ide.conf) to > > > > orion.core.metastore=simple > > I tried this today and looks like you are setting wrong ContentLocation in > [ProjectName].json, I get 404 for every project created in the workspace. > When I looked inside the user name in ContentLocation was missing, it was > "file:/C:/some/path/foo/OrionContent/SomeProject" and should be > "file:/C:/some/path/foo/gosia/OrionContent/SomeProject" Sorry, you are hitting work in progress caused by Bug 416653 For now you need to set both: orion.core.metastore=simple orion.file.layout=userTree The design http://git.eclipse.org/c/orion/org.eclipse.orion.design.git/plain/mock-ups/Projects/Bug412995.html?id=3c06733153e2c6a726e8c094e119cb9363c0b0d8 Lists the final storage layout. |