Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 309624

Summary: [publisher] PDE Updatesite Build all generates wrong categories
Product: [Eclipse Project] Equinox Reporter: ekkehard gentz <ekke>
Component: p2Assignee: Ian Bull <irbull>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: P3 CC: aniefer, bugs.eclipse.org, johan.wannheden, john.arthorne, joostvanpinxten, nbaccount, pascal
Version: 3.6   
Target Milestone: 3.6 M7   
Hardware: All   
OS: All   
Whiteboard:
Attachments:
Description Flags
zip of updatesite
none
screenshot: the empty update site project - only site.xml is there
none
screenshot: site project after sync all features and build all
none
screenshot: all exported into filesystem in new directory
none
screenshot: added site into fresh SDK M6 installation
none
screenshot: install new software - wrong categories
none
screenshot: without P2 metadata categories are ok
none
Proposed patch
none
Patch v2 none

Description ekkehard gentz CLA 2010-04-19 02:48:12 EDT
Created attachment 165234 [details]
zip of updatesite

as discussed at
http://ekkescorner.wordpress.com/2010/04/18/who-eats-the-categories-from-update-sites/#comments
I tried to make it reproducable that an update site has wrong category informations.

so I created a new updatesite with 3 categories containing some features.
generated as usual. setup fresh Helios SDK M6.
tested the site: all Categories are there.

had this in the past, too: sometimes the categories work, sometimes they are totally lost, sometimes wrong.

now I found out when this happens. it happens after generating again.

here's what I've done:

1: re-arranged the site.xml, so I only have one Category with the Features I had before in 3 Categories.
2: deleted all from Updatesite project without the site.xml
3: hit "sync" all features, hit "build all"
4. exported the site project into filesystem
5. deleted all from my FTP site, uploaded new site in 2 versions:
    a) as generated with p2 metadata (http://ekkes-updates.de/redview-sandbox-category)
    b) same but removed conten.jar and artifacts.jar (http://ekkes-updates.de/redview-sandbox-category-no-p2)
6. unzipped SDK M6 with fresh workspace
7. added the URL of updatesite
8. install new software from this site 
--> now the 3 categories from first generation are displayed together with the new features looking at timestamp
9. edited installed updatesite to the url of the site without p2 metadata
--> now the correct site was there: one Category with all features
(tried it also using a local update site instead my uploaded)

screenshots and a zip of the site are attached
Comment 1 ekkehard gentz CLA 2010-04-19 02:49:43 EDT
Created attachment 165235 [details]
screenshot: the empty update site project - only site.xml is there
Comment 2 ekkehard gentz CLA 2010-04-19 02:50:25 EDT
Created attachment 165236 [details]
screenshot: site project after sync all features and build all
Comment 3 ekkehard gentz CLA 2010-04-19 02:51:14 EDT
Created attachment 165237 [details]
screenshot: all exported into filesystem in new directory
Comment 4 ekkehard gentz CLA 2010-04-19 02:51:52 EDT
Created attachment 165238 [details]
screenshot: added site into fresh SDK M6 installation
Comment 5 ekkehard gentz CLA 2010-04-19 02:52:33 EDT
Created attachment 165239 [details]
screenshot: install new software - wrong categories
Comment 6 ekkehard gentz CLA 2010-04-19 02:53:14 EDT
Created attachment 165240 [details]
screenshot: without P2 metadata categories are ok
Comment 7 Pascal Rapicault CLA 2010-04-19 08:11:35 EDT
Moving to Ian since he usually takes care of category generations.
Comment 8 Andrew Niefer CLA 2010-04-20 17:33:53 EDT
Problem seems to be UpdateSite#load, it keeps a cache of the sites based on location and makes no attempt to check if the cache is up to date.

In this case, when generating categories a second time after changing the site.xml, the cache just returns the original site 

The cached site object does have a checksum on it that we could conceivably check against the checksum of the file we are supposed to load, but I'm not sure that recalculating a checksum is any cheaper than just always reloading the file.

The checksums are used by the UpdateSite repositories that are used for backwards compatibility with site.xml repositories.
Comment 9 Ian Bull CLA 2010-04-20 17:57:49 EDT
(In reply to comment #8)
> Problem seems to be UpdateSite#load, it keeps a cache of the sites based on
> location and makes no attempt to check if the cache is up to date.
> 
Good find Andrew, Seems like a dup of bug 244060
Comment 10 ekkehard gentz CLA 2010-04-21 00:46:18 EDT
(In reply to comment #8)
> Problem seems to be UpdateSite#load, it keeps a cache of the sites based on
> location and makes no attempt to check if the cache is up to date.
> 
> In this case, when generating categories a second time after changing the
> site.xml, the cache just returns the original site 
> 

yes - exactly what happened

great - that you found the reason

ekke
Comment 11 Andrew Niefer CLA 2010-04-22 14:33:58 EDT
Created attachment 165823 [details]
Proposed patch

This patch doesn't cache file: URIs, and also changes the cache to use SoftReferences.  It doesn't fully address bug 244060, but it does fix building categories from the UI.
Comment 12 Andrew Niefer CLA 2010-04-22 14:37:50 EDT
Created attachment 165825 [details]
Patch v2

Patch v2, remove the SoftReference from the cache if it has been cleared.
Comment 13 Andrew Niefer CLA 2010-04-22 16:42:55 EDT
I have released this patch.  Update site building is working for me now.
Comment 14 ekkehard gentz CLA 2010-04-22 17:52:06 EDT
(In reply to comment #13)
> I have released this patch.  Update site building is working for me now.

great news :)

so this should work with the next N-Build or I-Build ?

ekke
Comment 15 Andrew Niefer CLA 2010-04-23 10:24:29 EDT
The change is tagged v20100422, should be in the M7 warmup I20100423-0906 whenever it finishes
Comment 16 ekkehard gentz CLA 2010-04-23 10:33:57 EDT
(In reply to comment #15)
> The change is tagged v20100422, should be in the M7 warmup I20100423-0906
> whenever it finishes

great !
I also have to wait for this I-Build, because of another bug where I got a stack overflow while installing software using I20100422 ;-)
Comment 17 ekkehard gentz CLA 2010-04-24 09:35:25 EDT
(In reply to comment #15)
> The change is tagged v20100422, should be in the M7 warmup I20100423-0906
> whenever it finishes

just tested:
OSX10.6.3, cocoa64
Helios SDKM6 + I-Build 20100423-0906

did some test with an update site: sync, build-all, exported to filesystem
tested from other installation - also SDKM6 + I-Build 20100423-0906
the Categories were wrong:
only 2 of 3 displayed and not all features below the categories

generated again
tested again in a fresh installation
this time only 1 Category and not all features of this category visible

ekke
Comment 18 Ian Bull CLA 2010-04-24 13:27:54 EDT
(In reply to comment #17)
I released a few changes yesterday for category publishing, but they were released around 23:59 EST and were not part of the I20100423-0906.

I'm just posting this here so we can rule that out as a possible cause.

The fact that you are not getting the correct categories with a fresh install (and thus a restart) is a little worrisome.  In the past the problem was due to cached update sites, but if this is happening the first time you build and sync then there might be a regression here.

Can you post your site.xml file (or is it the same one that's in attachment 1 [details])?
Comment 19 ekkehard gentz CLA 2010-04-24 14:37:49 EDT
(In reply to comment #18)
> (In reply to comment #17)
> I released a few changes yesterday for category publishing, but they were
> released around 23:59 EST and were not part of the I20100423-0906.
> 
> I'm just posting this here so we can rule that out as a possible cause.
> 
> The fact that you are not getting the correct categories with a fresh install
> (and thus a restart) is a little worrisome.  In the past the problem was due to
> cached update sites, but if this is happening the first time you build and sync
> then there might be a regression here.
> 
> Can you post your site.xml file (or is it the same one that's in attachment 1 [details])?

Ian,

I did the tests in fresh installations, but the generation always from the same,
so it could be that there were generations before.

think I wait for the next build (N-build or I-build ?) to test again
and I also can do a test in a fresh install

ekke
Comment 20 Joost van Pinxten CLA 2013-10-25 05:31:51 EDT
Has this really been resolved? It seems to me that building update sites with Kepler (4.3.0) is still affected with this bug... If this has been resolved, it's probably a regression now? 

That is, categories are hidden, unless the P2 meta-information is deleted manually.
Comment 21 Joost van Pinxten CLA 2013-10-25 06:41:50 EDT
(In reply to Joost van Pinxten from comment #20)
> Has this really been resolved? It seems to me that building update sites
> with Kepler (4.3.0) is still affected with this bug... If this has been
> resolved, it's probably a regression now? 
> 
> That is, categories are hidden, unless the P2 meta-information is deleted
> manually.

To be more specific, this occurs when you are creating an empty project through the New-Wizard ("Feature Project" and "Update Site Project").

Additionaly: this problem does not seem to occur when using the Exporting wizard (as "Deployable feature") with a (wizard-)configured category.
Comment 22 Max Hacking CLA 2015-09-02 13:56:13 EDT
Maybe this is a regression but this bug still seems to be alive and well in the Mars Release (4.5.0 20150621-1200).