Bugzilla – Full Text Bug Listing
| Summary: | expose download information gathered from connector discover | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | z_Archived | Reporter: | Robert Munteanu <robert.munteanu> | ||||||
| Component: | Mylyn | Assignee: | Steffen Pingel <steffen.pingel> | ||||||
| Status: | RESOLVED FIXED | QA Contact: | |||||||
| Severity: | enhancement | ||||||||
| Priority: | P3 | CC: | steffen.pingel | ||||||
| Version: | unspecified | Keywords: | helpwanted | ||||||
| Target Milestone: | --- | ||||||||
| Hardware: | All | ||||||||
| OS: | All | ||||||||
| Whiteboard: | |||||||||
| Attachments: |
|
||||||||
|
Description
Robert Munteanu
We'll have to figure out where to track these. I'll probably set something up on mylyn.eclipse.org for now as it's tricky to evaluate the passed parameters on download.eclipse.org. Download information is now collected in the Apache logs of mylyn.eclipse.org. We would need some sort of the script to create reports. Should be fairly straight forward to record the hits in a database and to generate some graphs. Can you attach a sample log file with a couple of entries? Time permitting I'd be willling to take a shot at a log parser. pre.. # grep stats /var/log/apache2/access_log w.x.y.z - - [11/Apr/2011:11:22:44 -0400] "HEAD /stats/mylyn/discovery/com.foglyn?id=com.foglyn&discovery=3.5.0&product=com.adobe.flexbuilder.standalone.product&os=win32&arch=x86&ws=win32&nl=en_US HTTP/1.1" 404 - "-" "Jakarta Commons-HttpClient/3.1" w.x.y.z - - [12/Apr/2011:10:07:45 -0400] "HEAD /stats/mylyn/discovery/org.eclipse.mylyn.trac?id=org.eclipse.mylyn.trac_feature&discovery=3.5.0&product=org.eclipse.epp.package.jee.product&buildId=M20100909-0800&os=win32&arch=x86&ws=win32&nl=de_DE HTTP/1.0" 404 - "-" "Jakarta Commons-HttpClient/3.1" w.x.y.z - - [13/Apr/2011:07:33:58 -0400] "HEAD /stats/mylyn/discovery/com.itsolut.mantis?id=com.itsolut.mantis_feature&discovery=3.5.0&product=com.springsource.sts.ide&buildId=2.6.0.201103161000-RELEASE&os=win32&arch=x86&ws=win32&nl=it_IT HTTP/1.1" 404 - "-" "Jakarta Commons-HttpClient/3.1" w.x.y.z - - [13/Apr/2011:10:19:48 -0400] "HEAD /stats/mylyn/discovery/com.itsolut.mantis?id=com.itsolut.mantis_feature&discovery=3.5.0&product=org.eclipse.epp.package.cpp.product&buildId=M20110210-1200&os=win32&arch=x86&ws=win32&nl=ja_JP HTTP/1.1" 404 - "-" "Jakarta Commons-HttpClient/3.1" OK, thanks. What is the preferred implementation language for the parser script? We have perl and python available on the server and could probably make php available as well. Your choice :). Perl/Python should be familiar enough. Is the database backend MySQL? Yes, MySQL. OK. two more items to clear up and I should be set to go: * which pieces of information will be stored from the request log? * should I take into account the same file being parsed twice e.g. by taking into account the last recorded timestamp? (In reply to comment #9) > OK. two more items to clear up and I should be set to go: > > * which pieces of information will be stored from the request log? All parameters of the request. It would be nice if we could generate some statistics to show the operation system and language distribution etc. > * should I take into account the same file being parsed twice e.g. by taking > into account the last recorded timestamp? That probably makes sense. The alternative would be to not parse the log but to record requests directly in a database. (In reply to comment #10) > (In reply to comment #9) > > OK. two more items to clear up and I should be set to go: > > > > * which pieces of information will be stored from the request log? > > All parameters of the request. It would be nice if we could generate some > statistics to show the operation system and language distribution etc. OK, so all parameters from e.g. @com.foglyn?id=com.foglyn&discovery=3.5.0&product=com.adobe.flexbuilder.standalone.product&os=win32&arch=x86&ws=win32&nl=en_US@ will be stored . Anything else - IP address, user-agent? To make sure what I don't double parse I would need to store the date of the request anyway. Additionally, in the request above, what is the difference between the the @com.foglyn@ in the request path and the one assigned to the id parameter? No need to store the IP address or user agent. We can freely choose the request path. Requests are mapped to different locations to make it potentially easier to use tools such as webalizer or awstats which we could also consider. Created attachment 193628 [details] First parser implementation First attempt at parsing the data using Python. This works just fine with the provided sample data. A few notes: * the parser extracts all the fields, and skips previously recorded entries by timestamp; * the columns are all nullable, as I am not sure which fields will always be set. In the provided sample file there is an entry with no build id; * the script can be invoked using 'python src/DiscoveryParser.py' and it will run with the sample data and default MySQL connection credentials, but it is trivial to customize it and/or import it into another python file * the code is also in a git repository at https://github.com/rombert/discovery-log-parser . I have not added any license headers but I am happy to license it under the EPL. Do I only need to add the standard EPL header to the .py file? As for reading the data and presenting it in a web page, how should I proceed? Steffen, did you have the chance to try the script? Sorry for not getting back earlier Robert. We are in the midst of wrapping up the Indigo release. From taking a quick glance the script looks good to me. Is the idea to run it periodically (e.g. using cron)?
I got the following errors when running the script:
EE
======================================================================
ERROR: test_parseLines (__main__.LogParserTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test/test_parse.py", line 7, in test_parseLines
r.readFile('test/data/log.txt')
File "/home/spingel/rombert-discovery-log-parser-a929a96/src/DiscoveryParser.py", line 16, in readFile
self.parseLine(line)
File "/home/spingel/rombert-discovery-log-parser-a929a96/src/DiscoveryParser.py", line 29, in parseLine
request_datetime = datetime.strptime(ts[:-6], "%d/%b/%Y:%H:%M:%S")
AttributeError: type object 'datetime.datetime' has no attribute 'strptime'
======================================================================
ERROR: test_skip_already_parsed (__main__.LogParserTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "test/test_parse.py", line 24, in test_skip_already_parsed
r.readFile('test/data/log.txt')
File "/home/spingel/rombert-discovery-log-parser-a929a96/src/DiscoveryParser.py", line 16, in readFile
self.parseLine(line)
File "/home/spingel/rombert-discovery-log-parser-a929a96/src/DiscoveryParser.py", line 29, in parseLine
request_datetime = datetime.strptime(ts[:-6], "%d/%b/%Y:%H:%M:%S")
AttributeError: type object 'datetime.datetime' has no attribute 'strptime'
We only have Python 2.4.2 available on the server. Maybe that's the problem?
For the front-end a python or perl script that aggregates data from the database with some simple visualization, e.g. though http://code.google.com/apis/chart/ would probably work?
(In reply to comment #15) > Sorry for not getting back earlier Robert. We are in the midst of wrapping up > the Indigo release. From taking a quick glance the script looks good to me. Is > the idea to run it periodically (e.g. using cron)? > That's fine, I'm just eager to see the data :-) . Yes, the script is designed to be run from cron, dumping only the new data in the database. > AttributeError: type object 'datetime.datetime' has no attribute 'strptime' > > We only have Python 2.4.2 available on the server. Maybe that's the problem? Yes, it most likely is. That method is available from Python 2.5 . I'll submit an update which _should_ work on Python 2.4 shortly. I only have Python 2.7 to test for now. > > For the front-end a python or perl script that aggregates data from the database > with some simple visualization, e.g. though http://code.google.com/apis/chart/ > would probably work? Yes, it would. Do you prefer a script which generates on demand , i.e. accesibly from a browser , or a cron script which pre-generates the data? Created attachment 196278 [details]
Second version, compatible with Python 2.4
Steffen, it would be nice if you found the time to re-test the script. I would then start working on the visualisation. Thanks for remiding me. The second patch must have escaped the radar. I'll try to get around that next week after the 3.6.2 release. Robert, I'm very sorry for dropping the ball on this. We have meanwhile lost the mylyn.eclipse.org server and hence the logs. I have setup something on mylyn.org where we can now see installs on a per connector/version basis: http://stats.mylyn.org/awstats.pl . It looks like Mantis is the second most popular open source connector :). I hope that offsets the lack of responsiveness on this bug a little bit. That's great Steffen, thanks! |