Community
Participate
Working Groups
The dashboard needs a new architecture similar to that described by bug 129861. The dashboard should consists of the following components: 1. An information database. A sql database containing the extracted and summarized data for each of the projects. This database will probably contain a number of tables. This data is long-lived, probably forever. 2. Log files for the extraction routines. These _text_ files will contain complete logs of how each bit of data in the database was extracted. The logs will be maintained for a month and then will be recycled. 3. A REST web api for extracting data from the database and log files from the archive. The returned database information will be CSV files; the returned log files will be text files. 4. A (single) nightly processing routine. This routine will run a number of information extraction modules and store the results in the database. 5. Information extraction modules. There are standard modules (scan a mailing list, scan a newsgroup, look at Bugzilla, etc.) and there can be project custom modules. The project custom modules will be stored in project’s own directory and will be described in the project-info.xml file. These can be used to scan non-standard, but still public, repositories of information. (This is a future feature; not version 1). 5. Various web pages that display dashboard information. These pages will use the REST api to extract the data from the database. Thus these pages are easily testable on local machines while the data resides on the central server.
Even though there is the possibility that this new architecture might not be finalized on your side I have a couple of questions about your views on the new architecture as some of the things in the backend will apparently change. 1. You mention that the DB will probably contain multiple tables. Are these tables the ones that hold the computed statistics information? If so what is the deciding factor on multiple tables vs one table like it currently is? 2. Am I understanding correctly that the log files will no longer live as rows in the DB but will be kept in some filesystem? Will there be REST APIs to access these too or will they just be available for download as is now the case?
Re: comment #1 1a. Yes. 1b. One table versus multiple tables: I don't really know the answer. With one table it is more difficult to add a new statistic type because it requires modifying the schema. But it is easier to query one table. So perhaps one table is the way to go. In the end, it shouldn't matter if we abstract out the idea that we can query and update just a few of the statistics. 2a. I envision a set of text files named "log01.txt" and "log02.txt" and up to "log31.txt". The logs are written to the logDAYOFMONTH.txt file and thus automatically rotate. So, no, not in the database because we don't need to store them forever. 2b. Yes, a REST API. Of course the return value for http://www.eclipse.org/projects/dashboard/dashboard-log.php?day=03 will just be the text file iteself, so the return value from the API is quite simple.
Does having log files named logDAYOFMONTH.txt mean that there won't be per project log files and that they will all be concentrated in one single file? Or will they be something like: birt-log01.txt, gef-log28.txt, etc?
project-logNN.txt
For this bug I see two parts: 1) Internal code changes: new way of storing log files, one piece of code to call all computation scripts. and: 2) Change the dashboard so it uses REST APIs for the clients. I would like to split this into two separate bugs making #2 above higher priority than #1. The reason being that the changes to #2 will be reflected sooner on the client side and when changes are to #1 the visible changes to an external viewer should be minimal. Is this a possibility?
Re: comment #5 - If you think that's best, go ahead.
Two new bugs created: Change Dashboard to use REST APIs https://bugs.eclipse.org/bugs/show_bug.cgi?id=130623 Change Dashboard internals https://bugs.eclipse.org/bugs/show_bug.cgi?id=130624
Created attachment 36839 [details] General patch to solve bugs 130623, 130624 and this one.
Attached patch should fix issues.
I'm confused by this patch. Here are some questions: dashboard/dashboard_csv_download.php - doesn't seem to return any page at all much less a CSV file. dashboard/dashboard_detail.php - includes a /projects/common/csv2dashboardinfo.class.php file that the patch does not include common/dashboard.class.php - includes a /projects/common/file-utils.php that the patch does not include ...and then I stopped looking
Created attachment 37934 [details] Complete patch for bugs 130623, 130624, 109221 and this one.
Latest patch should be complete and include various fixes for other bugs.
Created attachment 38040 [details] New version of the files against the revised CVS HEAD
Created attachment 38293 [details] Patch with more debug info This new patch adds debug information to many of the components of the dashboard.
debug patch applied
The dashboard is completely blank.
Created attachment 38934 [details] DB Alteration routine.
Changing status to fixed to trigger patching.
Created attachment 39131 [details] Newest dashboard patch including fixed for other bugs (#135290, #125395, #134273 and #113716) Marking as reso/fixed to trigger patching. This patch has some code that related to bugs #135290, #125395, #134273 and #113716.
Patch applied.
Dashboard CSV files are empty and log files are empty thus I cannot tell if it works.
Created attachment 41935 [details] Patch with latest items and documentation This patch contains fixes for bugs: bug 129862 bug 106559 bug 134275
Marked reso/fixed to trigger review
Reopening so that I can re-close this bug.
The current dashboard is being decommissioned and thus this bug is no longer relevant.
Closed.
