Community
Participate
Working Groups
API tooling is planning to augment its toolset to produce reports of what API and *non*-API is referenced. This analysis could be used to obtain a better understanding of what parts of your API and non-API is used by others. This is different from current discouraged access reporting, as the report is oriented from the perspective of the producer rather than the consumer (currently, if you use something illegally, you are told - but the producer has no idea). We should report non-API access as well as illegal API use (illegal extension, etc). Typically, these reports would be run as part of a releng build - but could also be integrated into the IDE's search facilities.
Created attachment 120973 [details] patch Mike and I had discussed the need for some way to store/collate a potentially large set of references as analysis is being performed. This patch provides one API that could be used.
Created attachment 121711 [details] work in progress This patch builds on some of the ideas from the previous patch. This patch contains a search menu item in the PDE Tools popup menu, available for API enabled projects. The results of the search currently are just dumped out to the console. Added code in this patch: -ApiSearchEngine -IApiSearchRequestor (+ default impl) -IApiSearchReporter (+ default impl) -progress reporting -search action for API projects Moving forward, I think we need a more abstract way to represent a search result other than a raw listing of references, provide a good UI for the IDE world, and create an ant task to run the search engine in (for a start)
Created attachment 122157 [details] more work This patch builds further on the previous one, adding: 1. the API use ant task, which runs the search engine on an SDK location 2. updated internals for searching, allowing exclude lists to be used 3. a console progress monitor, which reports progress to the console work still to be done: - an XML search reporter is needed to turn the mass data into something useful - an XSLT is required to make the data useful to the end user - better performance Some interesting findings so far - preliminary testing showed it took ~13MB of memory to store the API use graph for the entire SDK (3.4.1) and was computed in 39 minutes - caching extracted references did not work at all, since the miss rate is very high and persisting / restoring references causes another whole resolve cycle.
Example build file for the ant task: <?xml version="1.0" encoding="UTF-8"?> <project name="api_use_reporting" default="run" basedir="."> <property name="baseline" value="/eclipse/eclipse_3.4.1" /> <property name="report_location" value="/eclipse/tests_api/reports/use" /> <property name="exclude_list_location" value="/eclipse/tests_api/exclude_list_external.txt" /> <target name="run"> <apitooling.apiuse profile="${baseline}" report="${report_location}" excludelist="${exclude_list_location}" considerapi="true" considerinternal="false" debug="true" /> </target> </project>
Created attachment 122415 [details] update Continued effort, includes an xml reporter and some performance improvements
Created attachment 122461 [details] update further update: 1. better memory management writing XML output 2. increased performance by only scanning dependent projects that have access to packages from the component context 3. better collated data results
Created attachment 122758 [details] update updated patch against HEAD with: 1. better memory usage 2. improved search scope handling 3. report conversion ant task
Created attachment 122821 [details] updated patch apparently the last patch was missing some, perhaps something went wrong creating it...
Created attachment 122979 [details] updated patch Patch with updates to the report conversion task to support user defined XSLTs
Created attachment 122981 [details] example .xsl file This is an example *.xsl file for transforming the outputted references to html.
Created attachment 123120 [details] example report Example report for the usage graph of org.eclipse.debug.core
Created attachment 123280 [details] updated patch This patch updates the search engine with far better performance (incremental reporting) and updated use and report conversion tasks.
Created attachment 124167 [details] updated patch This update provides the complete use and report conversion tasks
Example build file to run the tasks <?xml version="1.0" encoding="UTF-8"?> <project name="api_use_reporting" default="run" basedir="."> <property name="baseline" value="/eclipse/eclipse.tar.gz" /> <property name="report_location" value="/eclipse/apiuse-both/XML" /> <property name="html_report_location" value="/eclipse/apiuse-both/HTML" /> <property name="exclude_list_location" value="/eclipse/exclude_list_external.txt" /> <property name="xslt_file" value="/home/mrennie/Desktop/references.xsl"/> <target name="run"> <apitooling.apiuse profile="${baseline}" report="${report_location}" excludelist="${exclude_list_location}" considerinternal="true" considerapi="true" debug="true" /> <apitooling.apiuse_reportconversion xsltfile="${xslt_file}" htmlfiles="${html_report_location}" xmlfiles="${report_location}" debug="true" /> </target> </project>
Created attachment 124299 [details] updated patch updated patch with: 1. documentation for the ant tasks 2. small fix for windows paths 3. provide the default XSLT file in the api-tooling JAR file work continues on the DB reporting tasks.....
Created attachment 124302 [details] small update builds on the previous patch by updating the requestor reference kinds that will be ignored
Committed the latest round of changes to HEAD which include: 1. a new vector of meta-data: Internal-Permissible is no used to better separate Internal (illegal) references from Internal (allowed) references 2. two DB ant tasks: apiusedb (performs a search and reports to the specified DB) and apiusedb_reportconversion (which takes the output from the apiuse task and dumps it into a DB from XML). 3. robust debug reporting of progress for searching / reporting 4. complete documentation for all the new tasks
example build file for the DB use task: <project name="db_api_use_reporting" default="run" basedir="."> <property name="baseline" value="/eclipse/eclipse.tar.gz"/> <property name="exclude_list_location" value="/eclipse/exclude_list.txt"/> <property name="db_domain" value="localhost"/> <property name="db_name" value="mydatabase"/> <property name="db_port" value="3306"/> <property name="db_driver" value="mysql"/> <property name="db_username" value="username"/> <property name="db_password" value="password"/> <target name="run"> <apitooling.apiusedb dbdomainname="${db_domain}" dbname="${db_name}" dbport="${db_port}" dbdrivername="${db_driver}" username="${db_username}" password="${db_password}" baseline="${baseline}" excludelist="${exclude_list_location}" considerinternal="true" considerapi="true" debug="true"/> </target> </project>
example build file for the DB report conversion task: <project name="api_use_db_report_conversion" default="run" basedir="."> <property name="report_location" value="/eclipse/apiuse-both/XML" /> <property name="db_domain" value="localhost"/> <property name="db_name" value="mydatabase"/> <property name="db_port" value="3306"/> <property name="db_driver" value="mysql"/> <property name="db_username" value="username"/> <property name="db_password" value="password"/> <target name="run"> <apitooling.apiusedb_reportconversion dbdomainname="${db_domain}" dbname="${db_name}" dbport="${db_port}" dbdrivername="${db_driver}" username="${db_username}" password="${db_password}" xmlfiles="${report_location}" debug="true"/> </target> </project>
to use the DB tasks, there are two small requirements: 1. you have to have a DB to connect to 2. you have to have the driver for said DB on your Ant classpath
moving forward there are some items still to be done: 1. the DB reporter does not actually dump data into a database yet, as we are not sure what to the formation of the tables, etc will be: the reporter is complete, it connect to the specified database and sends queries, the only missing piece is that the queries are empty. 2. as it is now you can point the task to an Eclipse install and it gives you back the reference graph for the SDK; I could envision that a user might want to say "take this group of bundles and give me the reference graph compared to this other group of bundles". For example taking the WTP bundles and extracting their usage of the SDK. 3. it would be very cool if you could do similar graph analysis in the IDE with some kind of API view or something.
committed last set of updates. 1. The usage tasks can be used to scan one baseline against another (i.e. bundle group A vs. group B) 2. The reports have been jazzed up 3. the explicit DB reporting has been dropped in favour of a pluggable API use task. This way consumers of the task can provide whatever kind of reporter they want: if it writes to a DB or files or whatever. marking as fixed. please verify Olivier
+1
