Bugzilla – Full Text Bug Listing
| Summary: | [generator] [publisher] Meta data generator cannot handle existing meta data | ||
|---|---|---|---|
| Product: | [Eclipse Project] Equinox | Reporter: | Stephan Herrmann <stephan.herrmann> |
| Component: | p2 | Assignee: | Ian Bull <irbull> |
| Status: | RESOLVED WONTFIX | QA Contact: | |
| Severity: | normal | ||
| Priority: | P3 | CC: | dj.houghton, jeffmcaffer, pascal |
| Version: | 3.4.1 | ||
| Target Milestone: | 3.5 | ||
| Hardware: | Other | ||
| OS: | Linux | ||
| Whiteboard: | |||
|
Description
Stephan Herrmann
Perhaps the bug you are referring to is bug 224480. If you have steps to reproduce the problem you are seeing, that would be good. Thanks. (In reply to comment #1) > Perhaps the bug you are referring to is bug 224480. What I remembered was actually a mixture of bug 224480 and bug 220494 ;-) > If you have steps to reproduce the problem you are seeing, that would be good. I uploaded a tar to this location: http://www.objectteams.org/distrib/updates_bug251888.tgz (~40MB). This file contains three features in four versions each. content.xml and artifacts.xml stem from a previous run with only three versions. The latest version exists in features/, plugins/ and has been added to site.xml. I'm using a shell script that basically feeds default values into this call: ${ECLIPSE}/eclipse -application ${APP} \ -source ${LOCATION} \ -metadataRepository file:${LOCATION} -metadataRepositoryName "${NAME} Updates" \ -artifactRepository file:${LOCATION} -artifactRepositoryName "${NAME} Artifacts" \ ${MOREARGS} Depending on the value of ${MOREARGS} different things happen: 1.) MOREARGS=-site site.xml -append Effect: no artifacts deleted but resulting *.xml don't work: After uploading everything, new feature is not displayed in the p2 UI. Although it happened to me many times that features were not displayed (after "refresh"), this part of the effect is not strictly reproducable. 2.) on result of 1.) repeat generator now with MOREARGS=-site site.xml Effect: plugins/ is empty now restore files to original, then 3.) MOREARGS=-site site.xml Effect: all but the newest features/plugins are removed. restore files to original, then remove content.xml and artifacts.xml, then 4.) MOREARGS=-site site.xml Effect: success. While writing this I get the impression from items 2.) and 3.) that all artifacts mentioned in an existing yet unused artifacts.xml are actually removed. While this suggests a viable workaround it still looks like a bad bug, doesn't it? Is -append an option many people use? I repeatedly had the _impression_ that it does not work reliably. After several more attempts I just succeeded to reproduce the broken xmls using the scenario that I would actually favor (if it worked): I have a directory with the newest jars in features/ and plugins/ plus the artifacts/content.xml from previous builds and generate using "-site site.xml -append". I'm uploading the resulting broken xml to http://www.objectteams.org/distrib/xml_bug251888.tgz Interestingly the non-showing-features-in-the-UI effect is kind-of sticky: after restoring correct xml on the server, one client machine still didn't show the newest feature, while on a different machine 2 or 3 refreshes eventually revealed the missing feature... Only restarting Eclipse and once more refreshing "Available Software" finally gave me back what I need. While I'm at it: the "Available Software" page at this point started showing duplicate features etc, next refresh feature again missing, next refresh: list is OK, I click "Install", the first wizard page after that shows that it will update zero features with a total size of 9,910 KB, but then it actually worked. Just one of those normal workflows with p2. In a fresh download of 3.4.1, I get the following when attempting (2) from above. MalformedURLException: no protocol: site.xml Change the -site site.xml to -site file:C:/dev/test/otdt-updates-1.2.2/site.xml Run again, get the following: java.lang.IllegalArgumentException: Not appending to artifact repository (file:C:/Dev/test/otdt-updates-1.2.2) will destroy input files. This is as expected according to the fix in bug 220494. Can you confirm you have 3.4.1? Particularly org.eclipse.equinox.p2.metadata.generator 1.0.1.R34x_v20090819 (In reply to comment #3) > MalformedURLException: no protocol: site.xml > > Change the -site site.xml to -site file:C:/dev/test/otdt-updates-1.2.2/site.xml Sorry, I overlooked that my 'smart' script is doing the substitution behind the scenes, so yes the argument is actually: -site file:/tmp/bug251888/otdt-updates-1.2.2/site.xml > Run again, get the following: > > java.lang.IllegalArgumentException: Not appending to artifact repository > (file:C:/Dev/test/otdt-updates-1.2.2) will destroy input files. much to my surprise today I see an error dialog pointing me to a log file containing exactly this error message. This point goes to you :) Looking into eclipse/configuration I find old log files (from the very day when I logged comment 2) containing this error message, too. However, I really can't remember having seen any error dialog. Either, I had been staring at the screen too long, or the log message was written without a notice, OR, I was working remotely and simply didn't see the dialog because I was using xvfb. In short: I think it is wrong if the metadata generator reports messages with not trace on stdout/stderr (while success messages ARE printed to stdout). Isn't it intended to be a headless tool? > This is as expected according to the fix in bug 220494. Hm, I don't see any similarity in the input: I'm using neither -inplace nor -publishArtifacts. Additionally, I don't see any mistake in my usage of the metadata generator, what's the illegal argument? How to avoid this error? > Can you confirm you have 3.4.1? Particularly > org.eclipse.equinox.p2.metadata.generator 1.0.1.R34x_v20090819 I do have it (as witnessed by the error message I now see :-/ ). Perhaps some of my experiments while searching for steps on how to reproduce accidentally used 3.4.0. I don't have traces of what exactly I used :( My apologies. Summing up: Item 1) produces unusable metadata (just reproduced a minute ago) Items 2) and 3) throw an error and to nothing Vaporization indeed never happened in today's experiments. I'm changing the title to better match the remaining issues. Has anybody ever succeeded in producing valid metadata using "-append" and "-site"? I haven't. If someone can help me to get this working on our build machine I could agree on closing this issue. Hm? Ian, could you please verify that this scenario is working with the publisher and also help Stephan setting this up. Thx. Using 3.5M7 all scenarios in this bug work OK. Thanks. I'm afraid I spoke too soon.
I was mostly successful with the _metadata generator_,
until I found that categorization doesn't work with
-append (only the first version of each feature
appears in the category defined in site.xml).
After learning that the metadata generator is dead,
I'm trying to migrate to the publisher, but I'm still missing
some bits:
- does the publisher support cumulative mode like -append?
Also, when using a category.xml (created using the PDE/UI -
is the format of category.xml documented somewhere?),
the CategoryPublisher only includes the newest version of
each feature in the given category.
In my category definition I tried both 1.3.0.qualifier notation
and 0.0.0. Still only one version of each feature categorized.
Maybe I'm barking up some wrong tree, so what would be the
recommended workflow for adding a new version of a set of
features to a repository including categorization?
I tried:
1. FeaturesAndBundlesPublisher to create {content,artifacts}.xml
2. CategoryPublisher to enhance content.xml
but neither seems to work incrementally.
I'm trying to work with
- metadata from previous builds
- features/ and plugins/ from current build
in order to generate merged metadata. Is that supported somehow,
or _must_ I suck the entire update site in order to generate
meta data?
(In reply to comment #8) > - does the publisher support cumulative mode like -append? yes, you should be able to use -append in most publisher applications. Does this appear broken? > > Also, when using a category.xml (created using the PDE/UI - > is the format of category.xml documented somewhere?), The format is not API, and will likely change next year. > the CategoryPublisher only includes the newest version of > each feature in the given category. > In my category definition I tried both 1.3.0.qualifier notation > and 0.0.0. Still only one version of each feature categorized. I will look at this there could be a bug here. (I have opened bug 277340 to track). > > Maybe I'm barking up some wrong tree, so what would be the > recommended workflow for adding a new version of a set of > features to a repository including categorization? Ideally you would take your existing repository and run the CategoryPublisher with the category file. This should add (append) the categories to your repository. Another option would be to generate an update site, and use the UpdateSite publisher (http://wiki.eclipse.org/Equinox/p2/Publisher#UpdateSite_Publisher_Application) and create a full p2 repo. If the only bug here is the multiple versions, then I would like to close this bug and we can focus on bug 277340). (In reply to comment #9) > (In reply to comment #8) > > - does the publisher support cumulative mode like -append? > yes, you should be able to use -append in most publisher applications. Does > this appear broken? Looking at http://wiki.eclipse.org/Equinox/p2/Publisher I found no indication that, e.g., the FeaturesAndBudlesPublisher would support -append. A quick experiment seems to prove you right. While I'm at it, on the same wiki page, I believe this line -categoryDefinition /home/irbull/workspaces/p2/mail/category.xml is missing a "file:" prefix, no? And, what's the meaning of -categoryQualifier? > I will look at this there could be a bug here. (I have opened bug 277340 to > track). thanks. > Ideally you would take your existing repository and run the CategoryPublisher > with the category file. This should add (append) the categories to your > repository. But we basically have two orthogonal append operations: - append new feature versions - append category information Aren't both operations needed for each new version? Can you advise to append to a repository that already contains category information, or must two variants of the repository be maintained: 1: pure feature/bundle information 2: with categories > If the only bug here is the multiple versions, then I would like to close this > bug and we can focus on bug 277340). I see your point, but I'd prefer to wait for the fix for bug 277340 and then come back to see whether alternating runs of FeaturesAndBudlesPublisher with -append and CategoryPublisher produce the desired result. (In reply to comment #10) > Looking at http://wiki.eclipse.org/Equinox/p2/Publisher I found no indication > that, e.g., the FeaturesAndBudlesPublisher would support -append. > A quick experiment seems to prove you right. It's a wiki, you can always update it ;-). > While I'm at it, on the same wiki page, I believe this line > -categoryDefinition /home/irbull/workspaces/p2/mail/category.xml > is missing a "file:" prefix, no? > And, what's the meaning of -categoryQualifier? Yes it is, thanks. -categoryQualifier provides a way to give unique qualifier to category ids. For a variety of reasons, it is important that categories are unique. We also use version timestamps, so you really shouldn't need to specify this from the command line. > > Ideally you would take your existing repository and run the CategoryPublisher > > with the category file. This should add (append) the categories to your > > repository. > > But we basically have two orthogonal append operations: > - append new feature versions > - append category information > Aren't both operations needed for each new version? Can you advise to > append to a repository that already contains category information, > or must two variants of the repository be maintained: > 1: pure feature/bundle information > 2: with categories > Just so I understand, if you have category Foo with feature Bar (v1.0) in a repo. If you publish feature Bar (v2.0) you want that to appear in the same category? The entire story around categories is not solved yet (and won't be for 3.5). However, you might be able to use composite repositories to solve your problem. Create: 1. repo with pure features / bundles 2. repo with categories 3. composite repo that points to both of these. In theory, if you connect to your composite repo, you will see categorized content. If you update, you can create new category repositories. While we can append to repositories, we cannot append to categories (add a new feature to an existing category) since categories are IUs and IUs are immutable. You could write a custom publisher that removes old versions of the categories and publishes new, updated ones. Ian, thanks for your comments. > Just so I understand, if you have category Foo with feature Bar (v1.0) in a > repo. If you publish feature Bar (v2.0) you want that to appear in the same > category? yes. > The entire story around categories is not solved yet [...] > [...] > While we can append to repositories, we cannot append to categories (add a new > feature to an existing category) since categories are IUs and IUs are > immutable. All right, that's where the limits to -append are. From this I conclude that I'd best apply some simple staging: I'll maintain the history of versions in a category-free repository, and before publishing a new version, I'll make a copy of that repository, add categories and upload the result to the server. Until the category story is finalized this is a fair workaround. (In reply to comment #12) > All right, that's where the limits to -append are. > From this I conclude that I'd best apply some simple staging: > I'll maintain the history of versions in a category-free repository, > and before publishing a new version, I'll make a copy of that > repository, add categories and upload the result to the server. > > Until the category story is finalized this is a fair workaround. > Category free repositories also have the advantage that you can create different "category repositories" for the same data. Different consumers may have a different view of the world. It is this requirement that makes category publishing more difficult (and more flexible) than the simple story we had in update manager. Thanks for your patience on this. |