Bugzilla – Full Text Bug Listing
| Summary: | Add update site performance enhancement utility | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Eclipse Project] Platform | Reporter: | Jeff McAffer <jeffmcaffer> | ||||||||||||||||
| Component: | Update (deprecated - use Eclipse>Equinox>p2) | Assignee: | Platform-Update-Inbox <platform-update-inbox> | ||||||||||||||||
| Status: | RESOLVED FIXED | QA Contact: | |||||||||||||||||
| Severity: | enhancement | ||||||||||||||||||
| Priority: | P2 | CC: | alex.blewitt, aniefer, chris, david_williams, ed.burnette, francois, john.arthorne, kim.moir, Michael.Valenta, pascal, pombredanne | ||||||||||||||||
| Version: | 3.2 | ||||||||||||||||||
| Target Milestone: | 3.2 RC2 | ||||||||||||||||||
| Hardware: | PC | ||||||||||||||||||
| OS: | Windows XP | ||||||||||||||||||
| Whiteboard: | |||||||||||||||||||
| Attachments: |
|
||||||||||||||||||
|
Description
Jeff McAffer
Andrew has been doing some investigation on Pack200 and initial impressions are that this has huge potential. Notice the net transfer savings on the full eclipse SDK is 38%!! And if you remove the source and doc plugins that ratio climbs to 60%. So we are proposing that pack200 be used for Callisto. Here is some detail - Callisto update feature.xmls remain unchanged - The site.xml be augmented with an attribute that says "it may have some pack200 content" - the Callisto update site contains both standard and packed jars for plugins (we could do features too but the main win is on code jars) - Update is changed to detect that the site it is using may have packed content so when looking for foo_1.0.0.jar it would first look for foo_1.0.0.pack.gz (the standard suffix for packed things) - If the packed version is found, it is downloaded and unpacked. Unpacking returns the JAR to its original shape - If the packed vesion is NOT found then we download the normal JAR - Either way, processing continues as normal. Notes - Pack200 is an implementatoin of JSR200 and is a standard part of Java 5. - currently this approach would only work if the client is running 1.5. That is one reason to continue having both forms on the update site - we can look at implementing the unpack part of the JSR but that will likely be time consuming and not happen for 3.2 (we have not looked at the JSR yet to see how complicated it is) - it appears that the unpack.exe runs without the JRE. I suspect it is not possible/legal to ship just the unpacker but it would be worth looking into. - This approach would add about 50% to the size of the Callisto update site as a whole but depending on how many people are using Java 5, can significantly reduce download times. From Andrew's note: pack200 is a 1.5 feature, but the unpack200.exe can work standalone. We decide if we can run unpack in update as follows: 1) use reflection to see if we can load the classes 2) if we can't, check a system property "java.util.jar.Pack200.Unpacker" for the location of unpack200.exe 3) Try to exec unpack200 either in the location from 2, or from the search path Someone running in a 1.4 vm can take advantage of the unpacking by getting an unpack200.exe and putting it on their search path or setting the system property. If we fail to load the 1.5 classes and fail to exec the unpack200, then we can't unpack and we just download the full jar from the update site. From dejan: I am a bit concerned about the 1) through 3) because we cannot control what people have on their workstations when they want to use Update Manager. This is similar to javaw.exe.manifest file that you need to drop in your JDK in order to make SWT use XP skins. People need to be told that unpack200.exe will cut their download time in half PRIOR to trying to download Callisto, and to be given a chance to install it first (it would be great if we offer to install it right there and allow them to proceed to installing Callisto without leaving Update Manager). Andrew answers: It seems that unpack has the same memory problems as pack (not cleaning up between jars) and we will need to exec the executable instead of using reflection and the 1.5 API. This means that your point is doubly important because now we will depend entirely on people either having unpack200.exe on their search path or specified in a property. Jeff's musings: Some thoughts - somehow this is being seen as complicated. it is really very simple - unpack200.exe can (and typically will) be found in JAVA_HOME/jre/bin. If the person is using 1.5, that is where it will be. If they are not, we can do some deluxe searching around etc but that is not the mainline case. That is, normal people will not have to mess with paths or properties or... The statement is easy, "if you run 1.5 to do the update operations, it is faster." - if unpack is not available, we download full JARs. - if the site you are conecting too does not have packed content, download the full JARs. - if unpack is available and the site claims to have some packed content, try for the .pack.gz file first. If that fails, go for full JARs - Callisto will have pack.gz files for most/all so we will never/seldom double trip - we cannot install unpack prior to downloading as we cannot ship unpack by itselff (if we could we would just put it in update.core) and we do not ship JREs. I have renamed the defect to include the work on update site digests. Digests and pack200 support are complementary in that digests improve search and browsing of the site on the client, while pack200 reduces the payload size and thus makes the actual download phase faster. I will soon post a link to a small document describing the proposal for both. We would like to bundle them together into one utility that update site administrators can run after they make changes to the site. The utility would generate digests, pack the jars and update site.xml accordingly. Just as a quick note on pack200 on MacOS The exe is found at /System/Library/Frameworks/JavaVM.framework/Versions/1.5/Commands/pack200 And will be ther only on MacOSx 10.4 and up. Thanks, Philippe. Mac OS X 10.4 is our supported version on Mac so it is ok. I have put together a small proposal and published it on the Update home page in 'working' directory: http://dev.eclipse.org/viewcvs/index.cgi/%7Echeckout%7E/platform-update-home/doc/working/site-performance-enhancement.html Comments are welcome. Re comment #7; although Mac OS X 10.4 is required for Java 1.5, the default install for end users is still Java 1.4. You have to download a separate Java 1.5 updater, so you can't assume that all Mac OS X 10.4 users will have Java 1.5 (though if they do, it will be in the path that Phillipe mentioned). This sounds like a very cool mechanism for downloading data. As a matter of interest, does anyone know why Pack200 is a separate executable? It would seem that a Java implementation would be possible, and even could be used as a classloader directly since it's just a binary (class-aware) packing mechanism. After all, GZip decompression is implemented in Java ... Re: document; Jeff's comment #1 suggests that the standard suffix is .pack.gz, but the working document says that "The utility will use digest and pack200 attributes as input ... plug-in fragment or archive (say, com.example.xyz.jar) and generate a packed version (com.example.xyz.jar.gz)." Should this last one be com.example.xyz.pack.gz? Also, "At this point, payload size seizes to be trivial" should be "ceases to be trivial". Re: comment #10 (to answer my own question) http://java.sun.com/j2se/1.5.0/docs/api/java/util/jar/Pack200.html So, yes, it's possible :-) I guess the unpacker allows it to run on older VMs too, since the implementation is in a java.* package and thus can't be added directly to the classpath. Another possibility is a clean-room re-implementation of the unpack200 algorithm, which could then be bundled with Eclipse (and/or Harmony). Some Comments about doc in comment #9: - the generated packed version of com.example.xyz.jar would be com.example.xyz.jar.pack.gz - for the site utility, would we want to only pack those jars that show up in the digest, or would it be enough to pack all the jars on the site. I can imagine 2 site.xml's that share directories. -The location of the unpack200 executable can be specified to update using a system property "org.eclipse.update.jarprocessor.pack200". The value can be one of: "@jre" - find unpack200 in java.home/bin "@path" - find unpack200 on the search path "@none" - don't use unpack200, just download the .jar "path/to/dir" - path to the directory containing the unpack200 executable If the property is not set, then we look for unpack200 first in java.home/bin, then on the system path, otherwise we don't use unpack. Alex we shortly investigated the implementation of unpack, however given the complexity and the time we had, we decided not to. I guess that when you say: "Another possibility is a clean-room re-implementation of the unpack200 algorithm, which could then be bundled with Eclipse (and/or Harmony)." you are proposing to author this don't you? ;-) (In reply to comment #11) > Re: document; Jeff's comment #1 suggests that the standard suffix is .pack.gz I put the doc together quickly, and it shows :-). The extension should be whatever is appropriate or pack200 (it appears that .pack.gz is the right suffix). > Also, "At this point, payload size seizes to be trivial" should be "ceases to > be trivial". Ditto :-). Should be 'ceases'. Re comment #14, yes, that was my intention. I probably need to touch base with the Harmony guys first regarding it (I'd like to ensure that it's compliant with their license too). I also tried to find the spec on the 'net, and although I found the link from the JSR homepage, I didn't find the actual document from the Sun download site. The other approach would be to reverse engineer the pack mechanism and the produce a spec for someone else to implement; but if there's a Sun spec that I can use for this (and it's within the IP remit of Harmony) then that would be a more efficient use of time. Unfortunately, I couldn't find the spec of how the pack algorithm works, so I've not got very far in that process. (Anyway, it's late and I have to get up for work in the morning ...) I have updated the proposal to fix the pack extension, the typo and to add the details described in comment 13. Here's some comments/questions on the proposal.
1. Yea!, improved update performance is hard to say no to! :)
2. I am a little confused about what's input and what's output.
A. you mention new site attribute digest=<url>
this is apparently meant as a directive to the digest creation utility, and as an informative attribute to the update manager as well? Is there a need to have those so strongly coupled?
What if the site.xml file has digest=<url> but it is really not there? The update manager reverts to normal update manager behavior? (recommended :) or does it give user an error that there is error at the update site?
B. Why, exactly, are the the digest locale's in the site.xml file? (I know you say its to avoid multiple trips to server, looking for non-existent files, but .... what exactly do they do?). It almost sounds like you could have an English-only version of the site.sml, but multiple languages of the digest. So, the end user would see, in English, some choices to install another language?
Shouldn't they be in the digest.xml file, if needed at all? Are these needed by digest utility? Or update manager? Or both? Something about it just strikes me as very un-web-like. More practically, kind of inflexible. But, again, I do not really know what they are, so maybe missing whole point.
Worst, you say "he value of this attribute is generated by the utility based on all the different digest locales." ... so, its output of utility, input to UM?
But, this modifies the site.xml file? That's not good! :) Especially if this is the only known case of doing that. (The reason I say its not good, is that the site.xml file is an authored document, likely under version control, etc., so its not a good design to modify it).
Other comments (not related to my input/output confusion :)
You say "Digest will have a processing instruction listing the digest version to allow future enhancements while retaining backward compatibility.". I do not see why this has to be a processing instruction. It should be, IMHO, an attribute of what ever the root tag in the digest is. (But, I can not tell you happy I am you are thinking of "version" even for the first version, most people forget! and then end up having to assume "abscense of version means version 1" which is always dicey).
There is one sentence in the proposal I didn't like: "It is the responsibility of the utility caller to ensure that it is run regularly in order to keep the generated content in sync with the source. Failure to do so will result in stale content and browsing or installation errors"
Stale content maybe, but installation errors are not acceptable. We have too many of those as it is :) If something could cause an installation failure, please consisder using some "fast fail" mechanism ... perhaps some CRC property in the digest that says "all is in synch" and if that doesn't match, then don't do anything. (And it would still be "the utility caller's" responsiblilty to be sure that CRC was up to date :) But could also, eventually, be done "on the fly" by a smart server that detected changes had been made.
Ready for an enchancement request? :) You say java.home/bin and search path will be queried for unpack200. What about all the JREe's/JDK's in workbench, as "installed JRE's". I know I try to avoid defining java.home, and adding things to my search path when ever possible. I realize this wouldn't work for RCP's ... but, would be nice for the other 95% of us (java developers :) not to have to change our system settings for this.
My last big uncertainty ... you dont' say so, but but I somehow get the feeling there's an assumption in here that all the content is on the same server. Is that true? I'd hate to see that codified in this utility, but ... maybe I'm mis-reading.
Thanks, hope these comments help.
Sorry, Did I set this to assigned? Didn't mean to. (I am _not_ volunteering :) And, I do not see an option to set back to 'new'. Is there a need to > have those so strongly coupled? Not really, but it helps because it makes things simpler. > What if the site.xml file has digest=<url> but it is really not there? The update manager reverts to normal behaviour. It almost sounds like you could have an > English-only version of the site.sml, but multiple languages of the digest. The digest is a mirror of what is available on the site. Therefore, there cannot be more locales in digests than there is in features via feature.properties. We are not changing the design here, just ensuring that we get the right translated strings based on the desired locale (assuming it existed at the time of digest creation). > all the different digest locales." ... so, its output of utility, input to UM? > But, this modifies the site.xml file? That's not good! :) Especially if this is > the only known case of doing that. (The reason I say its not good, is that the > site.xml file is an authored document, likely under version control, etc., so > its not a good design to modify it). This is a tough nut. The actual list of locales is a direct product of the digest creation process by parsing all the features jars, we can locate all the available feature.properties files and compose this list. If we don't put it here, we will need to create another file near digests (something like 'locales.properties') that contains this list. UM will load this file first, find out which local is available. Alternatively, we can have a digest list (digests.properties) that simply lists all the generated digests. The good thing of this alternative is that it does not modify site.xml. The bad thing is that it adds another level of indirection (opening of a digest index file to locate the digest to download). see why this has to be a processing instruction. It should be, IMHO, an > attribute of what ever the root tag in the digest is. Yes, it can be a root attribute too - anything that allows us to evolve the digest without breaking backward compatibility. > Stale content maybe, but installation errors are not acceptable. Digests contain all the information needed to perform pre-install validation without downloading full features. If the digest is stale, this validation may not be correct. When you proceed with the installation, you may find that some newly added dependencies are not met. I don't see why this is a problem. Whenever you are creating derived information, the derived information becomes invalid when the original changes. 'Smart server' or anything that detects changes and reruns the utility to update packs and digests is needed. > Ready for an enchancement request? :) You say java.home/bin and search path > will be queried for unpack200. I will let Andrew answer this one because I just copied his comment :-). > My last big uncertainty ... you dont' say so, but but I somehow get the feeling > there's an assumption in here that all the content is on the same server. Is > that true? The digest is linked to the site.xml. Unless others think otherwise, I expect that digests and packed archives sit next to the source files i.e. site.xml nd original archives. I don't expect a digest that contains features sitting on another site. Branko has additional proposal for 'friendly sites' that link a site to other sites that contain additional content but those sites should have their own digests and packed files. Created attachment 37769 [details]
First patch
Attached is the first version of a patch to update.core providing the pack/unpack functionality.
There are 2 parts to it. First is the org.eclipse.update.jarprocessor & internal packages which provide pack/unpack/sign functionality and which can be run independently via org.eclipse.update.jarprocessor.Main. This functionality would be used for the update site performance utility, and by eclipse.org to sign jars. Recursing into nested jars is currently turned off because of issues getting the signed jar to verify after packing/unpacking.
Second part is the change to update to support downloading packed jars from the update site.
Created attachment 37820 [details]
update - command line options
Updated patch. Clear up and simplify the command line options. New pack-readme.html, and a jarprocessor.jardesc file for exporting a standalone jar.
Created attachment 38035 [details]
Handle nested jars
This version now correctly handles nested jars.
Any feedback will be appreciated, especially on the changes to FeaturePackagedContentProvider.
+1 for inclusion in 3.2. Hi Andrew, I looked at the patch more closely yesterday evening and there seem to be several problems: 1. It adds new public methods to the class SiteModel which should be pushed down to ExtendedSite in order not to change SiteModel since it is public API 2. Adds new static protected method getLock to FeatureContentProvider which also should be changed since this class is also part of public API 3. org.eclipse.update.jarprocessor.internal package should change to org.eclipse.update.internal.jarprocessor to confirm to package naming convention in update 4. I am not 100% sure but org.eclipse.update.jarprocessor should be internal package since I think we are not allowed to add new APIs now I think these changes will be relatively simple and otherwise it seems to me that functionally it is working. We should make every effort to refrain from adding APIs at this point unless it is critically required by the new function. Agreed. I don't think API is needed here because it is typically run as a standalone application that has its own main class, or from within update core. I.e., I don't think there are any external clients who need to call it programatically. Created attachment 38421 [details] Updated patch Here is an update patch with the changes from Branko (comment #25) 1) public methods moved from SiteModel to ExtendedSite. 2) locks moved from FeatureContentProvider to internal LockManager 3) org.eclipse.update.internal.jarprocessor is now the name of the package. This now has the contents of both of the old org.eclipse.update.jarprocessor and org.eclipse.update.jarprocessor.internal packages. I also updated some copyrights the latest patch is out. good work. Note: we have been running this same jarprocessor utility as part of the build process all of this week, so it has had some measure of testing in a production environment. So far, so good. Re Comment #26 and Comment #27 : Not making Api at that stage is fine, but I think having it available also as an ANT task would make a lot of sense. It could be valuable for other projects, and make building smoother. I took build I20060413-0010 and using this jarprocessor I packed all the jars and created an update site. I then downloaded just the platform runtime binary and connected to my update site and found that update does not check for the pack.gz files because the siteModel field is null. We are setting this field by getting the site off the feature in FeaturePackagedContentProvider.setFeature, however the site on the feature is null, so we need a better way to get the site model so we can check if it supports pack200. After a workaround to not perform this check, I was able to successfully download the pack.gz versions of the jars, unpack and then install them. One thing I noticed is that there is a pause between each download while the jar is unpacked, we should have some feedback in the progress monitor so the user can see that we are still doing something. The unpack step need to be accounted for and given a SubProgressMonitor so that it can move the monitor and also show the right message. Just out of curiosity, why wasn't this NULL problem cought earlier? How did you test the patch above i.e. if it worked in your tests, what is different now? Created attachment 38861 [details]
siteOptimizer Application
I did the testing on that part of the code early on. Most of my testing was around the actual jarProcessor. I can only guess that I must have changed the check without retesting it, likely in the refactoring.
The attached path creates a siteOptimizer application. It will delegate off to the internal.jarprocessor.Main when a -jarProcessor argument is specified. A standalone jarprocessor.jar can still be exported if desired. There is a placeholder method runDigestBuilder for when a -digestBuilder argument is specified.
the patch also does the following
1. Change the FeaturePackagedContentProvider to get the site in its constructor
2. Change the progress monitor subtask to reflect the unpacking, currently there is no way to judge the progress of the unpack, so I did not use a SubProgressMonitor.
3. use JarFile(input, false) to avoid verifying the jar after signing.
Created attachment 39181 [details]
Site Optimizer + verbose
This patch adds a -verbose option for the jar processor to the above patch.
Andrew, I just submitted the previous patch ,can you please create patch against new HEAD. Plus to create digest using Andrew's app arguments are: -digestBuilder -digestOutputDir="D:\eclipse\digest" -siteXML="D:\eclipse\sitecopy\site.xml" of course change output directory and site.xml locations to correspond to your local settings. Created attachment 39196 [details]
Verbose
As requested, here is the verbose patch resynched against head.
in the head. thanks andrew. Thanks Branko. Regarding the digest builder, I was suprised that so much of the functionality was in the SiteOptimizerApplication itself. If we want to provide other entry points (UI Action, Ant Task) it would be usefull to have this refactored into its own class/package. yes, the digest function needs to be available as an ant task, Eclipse application , action , ... Having it seperated out would be good. let's close this bug. I think we finished the work in RC2, so i am marking it as rc2. Incidentally, although this has been marked as closed, I'm still working on the pack200 decompressor for harmony. Once that's done, it should be possible to redistribute it and to perform unpacking even if there isn't a 1.5 VM installed, though given that 1.5 may be the standard by the next Eclipse release it may be moot at this point ... Alex, that is great. Please do let us know when you have something. (In reply to comment #43) > Alex, that is great. Please do let us know when you have something. I unpacked my first pack200 file the other day :-) http://alblue.blogspot.com/2006/10/java-pack200-coming-along-nicely.html Mind you, it was only an empty interface, so there's a bit of a way to go along yet. And at an average of 2 bytes per day, I'm not exactly Bletchley Park material at the moment. Still, I'll keep this bug updated with my progress. Hopefully it will be before the 3.3 milestones end... For those that want to track the status of my pack200 implementation; you can add this feed to your reader: http://beta.blogger.com/feeds/6705935/posts/full/-/pack200 or browse via: http://alblue.blogspot.com/search/label/pack200 Note that any of my posts regarding non-Eclipse stuff aren't synced via planeteclipse, so there won't be any of those entries posted via PE. Anyway, I've now got to a stage where I can reconstitute a minimal Jar from a pack200 file. There's still a long way to go, but I think I'm on the final lap ... As and when I've got something that can be used by others, I'll file a separate bug with the code for inclusion in Eclipse. Hopefully I can get it finished in time for Eclipse 3.3. I won't bother updating this bug with any more status reports. |