Bugzilla – Full Text Bug Listing
| Summary: | [server][progress service] support querying for running progress tasks | ||
|---|---|---|---|
| Product: | [ECD] Orion | Reporter: | Susan McCourt <susan> |
| Component: | Server | Assignee: | Malgorzata Janczarska <malgorzata.tomczyk> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | enhancement | ||
| Priority: | P3 | CC: | billhigg, john.arthorne, malgorzata.tomczyk, mamacdon, simon_kaegi, Szymon.Brandys, tomasz.zarna |
| Version: | 0.2 | ||
| Target Milestone: | 0.4 M2 | ||
| Hardware: | PC | ||
| OS: | Windows 7 | ||
| Whiteboard: | |||
| Bug Depends on: | 364830 | ||
| Bug Blocks: | 344012, 344055 | ||
|
Description
Susan McCourt
iirc this is something we decided to consider for 0.4, isn't? Let me know if there is anything I can help you with. I can even take the bug over, if you don't think you'll have enough time in 0.4. John, are you planning to work on this bug? I would like to take it over alog with Bug 344012 and Bug 344047, are you OK with it? OK, never mind. I talked to Szymon and it occurred that I misunderstood something and I'm not in a plan for it. Sorry for confusion. Those are my thoughts about progress service. This is how it works now: a client site requests for creation of a task, in return it gets the task id. It uses task id to periodically ask for the task status and updates the progress on the UI, when task is finished the result of status request returns the task result and client stops asking for the task status. Client may also use the task result to perform some action: for instance display the git log. Current problems with this approach * High traffic: every site that started a task generates a request every 2 seconds * Delay: when task finishes we have to wait up to 2 seconds for the status, there are some tasks that will finish after less than 1 second * Client site can’t run more than 1 task at a time – although this could be handled by server, client could do more requests and ask for 2 or more tasks * Page reload makes the task untraceable: the running task id is kept in the runtime of current site, after page reload user has absolutely now way to check the task status, the action was supposed to be performed after the task is finished is also lost * Security: if a different user gets the task id somehow he can access to the task result that may contain sensitive data. There is no access control on tasks. Changes I think could be made: * As John said there is a need for some kind of tasks grouping. Client would ask for the group identifier and would get the status of all tasks in this group in one request. The group identifier shouldn’t be kept in runtime, because of the reload page problem. The natural identifier seems to be a user name, but I think it has some disadvantages: for some reasons there may be many sessions open for the same user account at a time, even by different users. The example is our anonymous user. Many users may also share one user account for different reason, for instance admin account. I think that safest way would be to group tasks by session. If we get the session id directly from the request that it solves also a security issue. * There are some actions like git clone that should trigger more than one action. For instance git clone: this should refresh git repositories, but also when cloning is finish it should refresh your workspace, because you have a new project in there now. Git merge should refresh the git log for remote branch, git log view for local, small logs on git status. To address this apart from an action that is normally registered we could add something like an action_id with a resource_id maybe. Navigator could register for file_update for his current resource and repositories view could register for git_repositories_update with the workspace id. * We could change getting the status from polling to long-polling. The client would send the request for status to server. Server doesn’t reply immediately, but keeps the request “hanging” until something changes. If the request gets timed-out client sends another request that replaces the old request on the server. This way can send requests less frequent but get updates faster. * Normally users open many Orion windows at a time (I have usually between 5 and 10 opened). If every window in the same session was requesting for the current status this highly increases traffic. We could think of shearing the information between browser windows, for instance by localStorage. So that only one window would be responsible for making the requests and if this window is closed another window takes over. Gosia already started working on the UI part of the progress stuff. I talked to John and he is fine with Gosia working on the server side too. Thanks for the excellent detailed summary of the current state and its pros/cons Gosia. Some general comments: - We should give some thought to what the client UI progress display will look like before designing in too much detail the implementation of the client/server communication. Do we want every page in Orion showing everything that is happening in the background, or is it enough to have some kind of minimal "Busy" message in the header, with a link to another page that shows detailed progress information. Imagine for example a world where there are a lot more tools plugged into Orion, some of them running for quite long periods. I can picture having a Hudson-like dashboard page showing everything that is running for the case where the user really wants to know what is going on (eg, https://hudson.eclipse.org/). - I definitely agree we could do some kind of caching, once we decide the level of detail to give on each page. For example a progress message could be stored in local storage along with a timestamp. Each page could use the progress message from storage if the timestamp is recent enough. Probably better to worry about the caching a bit later though, once we see how it is working. - I don't like the idea of using session id for grouping tasks. Session can be a very volatile thing - maybe there are multiple back-end servers and each request can go to a different one. If the session is somehow lost I can lose ability to keep track of long running work that I started in previous session. I also often use different browsers at once.. that might be a rare case today but others might do it. I prefer the idea of user id for grouping tasks... show me all the long running tasks associated with me, regardless of where/when they started. I think in most cases anonymous users won't be allowed to run long-running tasks, or maybe we fall back to use session if user id is not available. (In reply to comment #6) > - We should give some thought to what the client UI progress display will look > like before designing in too much detail the implementation of the client/server > communication. Do we want every page in Orion showing everything that is > happening in the background, or is it enough to have some kind of minimal "Busy" > message in the header, with a link to another page that shows detailed progress > information. I attached some propositions to Bug 344012. > - Probably better to worry about the caching a bit later though, once we see how it is working. Probably the last thing to do on this bug. (In reply to comment #6) > - I don't like the idea of using session id for grouping tasks. Session can be a > very volatile thing - [...] > or maybe we fall back to use session if user id is not available. The good thing with using a userId as a grouping for tasks is that user may start a task, then log out, go home and the next day check how did it finish. I pushed the first change: tasks are now grouped by user id. This is now transparent for everyone using API, but I want to force everyone using tasks passing the user name (or session id) from now on. MQ pointed out one important thing: we may have more than one server providing log running tasks. Until now this wasn't a problem because we asked for each task individually. We asked the URL provided at the task startup. Now we would like to have one grouping URL for task (at least from one server). We also would like to request information from this URL even if we didn't start any tasks yet (for instance to check if our tasks scheduled in the previous session finished). The obvious way to handle it is to provide an extension along with any other plugin (for instance file plugin) that would provide the URL to tasks list. We could handle multiple plugins like this and report all tasks on one control. The second thing is that we don't have to require to add this tasks plugin. Task could also provide this URL on scheduling. Than if we don't have this URL yet on the list, we could add it and track tasks from in from now on. I'm changing this bug for a master bug for server-site progress service bugs. I will be opening smaller bugs depending on this bug for each step I'm implementing. I wrote a blog entry a while back that talks about a RESTful protocol for running a task that could be some combination of: - Asynchronous, long-running - Composite http://billhiggins.us/blog/2011/04/27/resty-long-ops/ (In reply to comment #12) > I wrote a blog entry a while back that talks about a RESTful protocol for > running a task that could be some combination of: > > - Asynchronous, long-running > - Composite > > http://billhiggins.us/blog/2011/04/27/resty-long-ops/ Thanks Bill! described protocol seems to be similar to what we are using now in Orion. Did you manage to address in jazz losing track to the task when page scheduling it was closed/reloaded? (In reply to comment #13) > Thanks Bill! > described protocol seems to be similar to what we are using now in Orion. > Did you manage to address in jazz losing track to the task when page scheduling > it was closed/reloaded? Yes the client/server interaction for progress was deliberately modeled on the Jazz approach. From what I have seen the jazz web UI doesn't handle progress reporting across page transitions at all (although I believe it does prompt when you try to leave a page that is "busy"). Malgorzata/John: Here's what's in Jazz Web UIs: 1. Simple notion of "unsaved change" - this is what triggers the "Are you sure you want to leave this page?" message. 2. Simple progress message that can be tied to an XHR callback (i.e. when it returns, the progress message disappears). 3. The async/composite task framework in Lifecycle Project Administration as documented in my "REST-y long-ops" blog entry. I think 3. is the most interesting one because it allows for a persistent job-tracking resource that theoretically could survive a page transition though I never thought to implement this because the Jazz web UIs rarely use real page transitions... just Ajax-y pseudo-transitions. Here is how I would implement progress monitoring that could survive page transitions: 1. Use the REST-y long-op protocol described above. 2. When you POST the job and get back the 202 and job URL, store the job URL somewhere in the browser (cookie, HTML5 storage, etc.); the presence of a job URL in client storage means "I think there is an incomplete job that I should be tracking" 3. When a page loads, look to see if any progress URLs exist in client-side storage; if there is one, run a GET on the job URL and see if it's still in progress (it may have finished in the case where you are returning to the UI after a break or something). 4. If the job is no longer in progress, clean up the client-side state. If the job is still in progress, show the job-tracking UI. Everything we need for now seems to be currently available, this is new API: GET /task - get the list od tasks options: RunningOnly=true - return only tasks that are running Longpolling=true - start getting tasks though long-polling, returns a list of tasks and a LongpollingId needed in next request Longpolling=true&LongpollingId=[ID] - continue long-polling LocalTimeStamp=[dateAsLong] - tasks modified since given timestamp results=true - list contains task results (results=false list contains results only for tasks that failed) GET /task/[taskId] - get the task status DELETE /task - remove completed tasks DELETE /task/taskId - remove task (only for completed tasks) PUT /task/taskId, put data: Cancel=true - cancel task (only for tasks that support canceling) more API changes will be made on new bugs |