Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 222751

Summary: Ganymede M5 contains obsolete files and invalid *.pack.gz files
Product: Community Reporter: Martin Oberhuber <mober.at+eclipse>
Component: Cross-ProjectAssignee: Bjorn Freeman-Benson <bjorn.freeman-benson>
Status: RESOLVED FIXED QA Contact:
Severity: major    
Priority: P2 CC: aniefer, bjorn.freeman-benson, denis.roy
Version: unspecified   
Target Milestone: ---   
Hardware: PC   
OS: Windows XP   
URL: http://download.eclipse.org/releases/ganymede
Whiteboard:
Attachments:
Description Flags
List of obsolete files on Ganymede M5 none

Description Martin Oberhuber CLA 2008-03-14 10:22:47 EDT
Created attachment 92571 [details]
List of obsolete files on Ganymede M5

The current Ganymede M5 site on http://download.eclipse.org/releases/ganymede
has following three small problems which unnecesarily degrade performance:
1.) several obsolete files not referenced in site.xml
2.) several invalid plugins/*.jar.pack.gz files which have length 0 bytes.
3.) Unnecessary features/*.jar.pack.gz files which cannot be consumed by
    update manager as per bug 222667

These three problems do not make the site unusable, but they do add unnecessary burden on the servers, since mirrors get loaded with obsolete files, and clients cannot consume invalid pack.gz files leading to duplicate requests.

I would recommend following steps:

1.) Purge obsolete files as per attached list. Attachment lists those files 
    which currently reside in releases/ganymede/features|plugins but are not
    referenced in site.xml so they are obsolete.

2.) Find out why there are pack.gz files with 0 bytes size. It seems as if
    exactly those files have 0-length where the contributing projects perform
    signing and packing themselves, so the Eclipse.inf file specifies that 
    jars have been processed already. It looks as if JarProcessor does not
    process these files, resulting in invalid 0-length files.
    Either the invalid 0-length *.jar.pack.gz files should be purged from the
    site, or (better yet) JarProcessor should be re-run to create valid 
    *.pack.gz files.

3.) rm -f features/*.jar.pack.gz
Comment 1 Martin Oberhuber CLA 2008-03-14 13:16:33 EDT
FWIW, I ran the siteOptimizer on dsdp.eclipse.org to generate the bad pack.gz
files, and uploaded the result to
  http://download.eclipse.org/dsdp/tm/div/ganymede_m5_fixpack.zip

The archive is 43MB, and can be extracted onto releases/ganymede, it will
overwrite only the 0-byte-length .pack.gz files as well as fix the invalid
digest (see bug 221993).

So this would be a quickfix for now, and I verified on Windows that the
resulting ganymede can be installed completely.

If anybody wants to have a complete archived ganymede site for local installs,
it's now available on 
  http://download.eclipse.org/dsdp/tm/div/ganymede_m5.zip
Size is 298MB -- 12MB larger than the archive I had created on 25-Feb, because
this time I chose not to pack any jar files that were not packed on
releases/ganymede, just to be safe. This means that ECF, Mylyn, STP as wellas
most of EMFT are not packed in this version.

More observations, compared to the 25-Feb version:
* Buckminster core version bump from 1.0.0.r08951 to 1.0.0.r08952
* Buckminster cvs version bump from 1.0.0.r08949 to 1.0.0.r08953
* net.sourceforge.lpg.lpgjavaruntime.source was Jarred, now extracted
Comment 2 Bjorn Freeman-Benson CLA 2008-03-14 16:07:30 EDT
(1) and (3) - I deleted the whole staging directory and started over.
(2) is due to bug 187396
Comment 3 Martin Oberhuber CLA 2008-03-14 16:13:55 EDT
(In reply to comment #2)
> (2) is due to bug 187396

I'm confused... I'm running the same command as in bug 187396 comment 18 and it works for me on Ganymede, since I'm using Eclipse-3.4M5 as the builder. I thought that the Ganymatic also uses a recent Eclipse as the builder? Bug 187396 seems to be relevant for Europa only.
Comment 4 Bjorn Freeman-Benson CLA 2008-03-14 18:49:05 EDT
(In reply to comment #3)
I only update the Ganymatic (and its underlying Eclipse) when necessary: I prefer to keep a stable build system and not run on the bleeding edge.
However, at your urging, I have updated the Eclipse that I use to run the siteOptimizer to 3.4M5.
When it runs, I get errors like this:

Running on /home/data/users/bfreeman/downloads/releases/ganymede/staging/features/org.eclipse.rse.ftp.source_3.0.0.v20080212-79378_kE77a9C9E535C.jar
Running Pack on /home/data/users/bfreeman/downloads/releases/ganymede/staging/features/org.eclipse.rse.ftp.source_3.0.0.v20080212-79378_kE77a9C9E535C.jar
STDERR: Exception in thread "main" java.lang.SecurityException: SHA1 digest error for META-INF/eclipse.inf
STDERR:         at sun.security.util.ManifestEntryVerifier.verify(ManifestEntryVerifier.java:253)
STDERR:         at java.util.jar.JarVerifier.processEntry(JarVerifier.java:225)
STDERR:         at java.util.jar.JarVerifier.update(JarVerifier.java:212)
STDERR:         at java.util.jar.JarVerifier$VerifierStream.read(JarVerifier.java:438)
STDERR:         at java.io.InputStream.read(InputStream.java:112)
STDERR:         at com.sun.java.util.jar.pack.Package$File.readFrom(Package.java:789)
STDERR:         at com.sun.java.util.jar.pack.PackerImpl$DoPack.readFile(PackerImpl.java:529)
STDERR:         at com.sun.java.util.jar.pack.PackerImpl$DoPack.run(PackerImpl.java:490)
STDERR:         at com.sun.java.util.jar.pack.PackerImpl.pack(PackerImpl.java:91)
STDERR:         at com.sun.java.util.jar.pack.Driver.main(Driver.java:279)
Error: 1 was returned from command: /opt/public/ganymede/apps/ibm-java2-ppc-50/jre/bin/pack200 -E4 /home/data/httpd/download.eclipse.org/releases/ganymede/staging/features/org.eclipse.rse.ftp.source_3.0.0.v20080212-79378_kE77a9C9E535C.jar.pack.gz /home/data/httpd/download.eclipse.org/releases/ganymede/staging/features/temp.org.eclipse.rse.ftp.source_3.0.0.v20080212-79378_kE77a9C9E5

Additionally (or perhaps it is the same issue), I get zero length files, e.g., 
-rw-rw-r-- 1 bfreeman callistoadmin       0 2008-03-14 18:46 plugins/org.eclipse.gmf.runtime.gef.ui_1.1.0.v20080114-1111.jar.pack.gz

The command line is:
/shared/ganymede/apps/ibm-java2-ppc-50/bin/java -jar /home/data/users/bfreeman/ganymede/eclipse-34M5/plugins/org.eclipse.equinox.launcher_1.0.100.v20080205.jar -application org.eclipse.update.core.siteOptimizer -jarProcessor -pack -verbose -outputDir /home/data/users/bfreeman/downloads/releases/ganymede/staging/ /home/data/users/bfreeman/downloads/releases/ganymede/staging/
Comment 5 Martin Oberhuber CLA 2008-03-14 19:03:15 EDT
Hm.. it works ok for me on x86 / Sun Jvm. Are you running the latest IBM JVM? See bug 220943. Does the error also occur when running the "pack200" command verbatim as specified on the commandline?
Comment 6 Martin Oberhuber CLA 2008-03-14 19:04:46 EDT
If you want to try on an x86 vserver yourself, my latest mirroring / packing / digesting script is in http://download.eclipse.org/dsdp/tm/div/mirrorGanymede.tar
Comment 7 Bjorn Freeman-Benson CLA 2008-03-15 14:07:17 EDT
(In reply to comment #5)
> Are you running the latest IBM JVM?

Yes.
See the complete log of a run: http://build.eclipse.org/ganymede/temp.log

Given that this "sort of works" now (latest JVM, latest Eclipse 3.4) (modulo the fact that the IBM PPC JVM doesn't seem to work with pack200) and the fact that I'm not planning to pack200 every staging build (the pack200 step takes over an hour and thus each Ganymede build would take over an hour longer if we did that), I'm closing this bug.