Bugzilla – Full Text Bug Listing
| Summary: | avoid 'bundles' directory in GET HTTP style map files | ||
|---|---|---|---|
| Product: | [Tools] Orbit | Reporter: | David Williams <david_williams> |
| Component: | releng | Assignee: | David Williams <david_williams> |
| Status: | RESOLVED FIXED | QA Contact: | Project Inbox <orbit.releng-inbox> |
| Severity: | normal | ||
| Priority: | P3 | CC: | aniefer, dj.houghton, gunnar |
| Version: | unspecified | ||
| Target Milestone: | Indigo M6 | ||
| Hardware: | PC | ||
| OS: | Windows 7 | ||
| Whiteboard: | |||
x When using unpack=true on the GET map file entry, PDE/Build will unzip the archive according to its extension. If it ends in ".jar" then we will create a directory (named by remove the .jar extension) and then extract the jar there. In this case we expect no top level directory entry in the archive.
There is a "dest" attribute, and it does get passed directly to the ant get task. It should be an absolute path, so generally something like ${buildDirectory}/plugins/bundleName_version
(In reply to comment #2) > ... > > There is a "dest" attribute, and it does get passed directly to the ant get > task. It should be an absolute path, so generally something like > ${buildDirectory}/plugins/bundleName_version Thanks Andrew, it sounds like then there might be a way to make a compatible version ... and still do away with the *.zip extensions? So, to be explicit where we now have an GET entry that ends like .../I20110117143708/bundles/org.apache.ant_1.8.2.v20110112.zip,unpack=true we should be able to have one as follows, and all still work as expected? .../I20110117143708/repository/plugins/org.apache.ant_1.8.2.v20110112.jar,unpack=true,dest=${buildDirectory}/plugins/org.apache.ant_1.8.2.v20110112 Cool. We'd have to find a new way to compute if the bundle should be unpacked or not, and not rely on *.zip extension, ... but that info is all right there, in several places (the p2 repo, and the feature.xml file). Sounds like we could even start to test this new format very soon (temporarily using the redundant zip files to compute unpack or not). Thanks again, (In reply to comment #3) Not quite. If you get rid of the zip extensions and have normal jars, then you should be able to just go from > .../I20110117143708/bundles/org.apache.ant_1.8.2.v20110112.zip,unpack=true to > .../I20110117143708/repository/plugins/org.apache.ant_1.8.2.v20110112.jar,unpack=true The change of ".zip" to ".jar" will change how pde/build unpacks the archive. You don't need the "dest" attribute unless you wanted to keep .zip and remove the top level directory entry inside the archive. In this case, you would do something like .../plugins/org.apache.ant_1.8.2.v20110112.zip,unpack=true, dest=${buildDirectory}/plugins/org.apache.ant_1.8.2.v20110112/ant.zip Here the ant.zip would get extracted into the org.apache.ant_1.8.2.v20110112 directory. (In reply to comment #4) > (In reply to comment #3) > Not quite. > Glad I asked ... sounds even easier then ... we would be getting rid of the zip extension and top level directory inside the jar/archive ... just an ordinary jar file as any other repository jar file ... so, only "todo" is fix up the map generator to better know how to compute "unpack=true". (That is, compute it without the zip file extension, since it'd no longer exist or be created during the build.). So ... guess I should not mark http get formatted map file as "deprecated"? Or should we still ... just so people know they can/should migrate? Or are there cases where GET/HTTP is better/preferred/or only one that works? (In reply to comment #5) > So ... guess I should not mark http get formatted map file as "deprecated"? Or > should we still ... just so people know they can/should migrate? Or are there > cases where GET/HTTP is better/preferred/or only one that works? I would suggest that the p2 repo should be the favoured format. The only case I can think of where HTTP-get might work better is for bundles containing nested jars that need to be compiled against and the bundle is still meant to be installed as a jar. (The p2 repo2runnable which is used to transform the repo only unpacks the bundles that are marked to be unpacked when installed). I've made this change for current builds in 'committers' area .. none promoted to 'downloads' yet. If anyone has the ability to test from 'committers' area, that'd be great. I'd done a small test from 'webtools' and seemed to work as expected. Previously, we used a heuristic based on 'jar' and 'zip' extensions, to know to mark map entry line with with "unpack=true". While the information is there in other forms (e.g. the feature.xml, and the repository metadata) I took the easy way out and simply "hard coded" the list of bundles to mark with a new file in maps directory, called "bundlesToUnzip.properties. I'll open a feature enhancement to improvement this in the future, by reading/parsing the faeture.xml or repository metadata but wanted to be sure the basics got done for M6. There's currently only 10 entries in this file. It may be rare enough, it's not worth automating. to cross reference, bug 338363 is the enhancement request to compute the list from feature.xml instead of hand editing the file. For the record, I've confirmed that creating download sites without the /bundles directory saves about 200M which seems about right. The download site is now 400M instead of 600M ... 30% savings. There is, still, a little (intended) redundancy in that we have the normal p2 repository at /repository and we also zip up that directory and offer as a downloadable archive file. That file is about 180M ... so times 2 roughly adds up to 400M. |
Currently the "get http" style map files refer to the bundles in the 'bundles' directory. To avoid redundancy, we'd like to stop producing that 'bundles' directory and use instead only the 'repository' directory, or 'repository/plugins' to be exact. This style of map file has "unpack=true" added to the end of the map entry, if the file is to be unzipped in an install instead of used as a jar. Plus, the zip files themselves in the 'bundles' directory have a "top level" folder in them (consisting of bundle name_and_version) to consumers can easily install into their installation location without they themselves worry about the directory name. While looking to implement this, I noticed a few issue that have to be solved: 1) currently the code that produces these map files takes advantage of the zip extension to know if to add the "unpack=true" attribute to the line of the map entry. That info will now have to come from else where ... either read/parse the feature.xml ... or better read/parse the content.jar/xml file. 2) PDE build itself _might_ have to be improved to handle this case. The bundles pointed to in 'repository/plugins' will all be jars, and even though it will unzip the entries, based on on the unpack=true attribute, it presumably will unzip these on top of each other, directly into what ever <eclipseLocation> it is using. 2a) one solution might be to also specify a dest=<bundlename_version> attribute, if PDE build will transpartently pass this on to ant, then it'd unzip in the right place. 2b) Perhaps PDE build can (or already does?) know to read the "shape" attribute in an OSGi bundle ... and then we'd simply have to make an effort so that we started using the shape attribute for those bundles that are intended to be unpacked (and not use only the feature.xml file attribute). Or ... we might avoid all these problems if we simply stop providing the get-http style of the map file. As far as I know, the p2 style of map file can be used easily. It would take a change in consumer build scripts, though, since to use the p2 style, the build has to specify the 'transformedRepoLocation' property, to use for a working directory to initially store the downloaded (and possibly unpacked) bundle. transformedRepoLocation=${buildDirectory}/transformedRepo But that's a pretty easy migration to make ... if that really works for everyone. I propose we encourage people to switch/change to use the p2 style of map file, and see if there are any problems, or cases where people can not switch. The old get-http style will be marked as deprecated on download page, and a pointer to this bug provided.