Community
Participate
Working Groups
The category.xml replacement of site.xml's site/@associateSitesURL (Bug 346926), site/repository-reference/@location should be supported by the eclipse-repository packaging. Hence this request for an enhancement (as suggested in Bug 452053).
If some committer can give me some pointers here, I might be willing to give this bug a try, as I'd really like this feature in Tycho soon.
Is this supported by p2 itself? Ie is this property kept in metadata when you export a category.xml as p2 repo from Eclipse? Why can't you use associateSitesURL ?
(In reply to Mickael Istria from comment #2) > Is this supported by p2 itself? Ie is this property kept in metadata when > you export a category.xml as p2 repo from Eclipse? > Why can't you use associateSitesURL ? As far as I can see, associateSitesURL is no longer supported by category.xml; it only works with old-style site.xml update sites. And no, just setting the property and searching the artifact.xml/contents.xml generated by Tycho for the URI didn't turn up a match (with Tycho 0.22).
Ok. It seems like this is available on p2 side: cf bug 436318. Tycho 0.22 includes a good version of p2 to support this feature (bug 444112 ). Looking at the PublisherServiceImpl (that directly takes the file and passes this to p2 ), it seems to me that Tycho 0.22 should already support that out-of-the-box...
(In reply to Mickael Istria from comment #4) > Ok. > It seems like this is available on p2 side: cf bug 436318. Tycho 0.22 > includes a good version of p2 to support this feature (bug 444112 ). Looking > at the PublisherServiceImpl (that directly takes the file and passes this to > p2 ), it seems to me that Tycho 0.22 should already support that > out-of-the-box... I am afraid I cannot get this to work: I used the PDE UI to add a site/repository-reference/@location to my repositories/stable/category.xml After the build, I grepped for my test URL: $ grep -r 'example\.org' repositories/stable/target repositories/stable/target/category.xml: <repository-reference location="http://www.example.org/" enabled="true" /> Moreover, neither repositories/stable/target/repository/artifacts.jar nor repositories/stable/target/repository/content.jar contain the test URL.
(In reply to Andreas Sewe from comment #5) > I am afraid I cannot get this to work: this may be a similar issue as bug 341744 comment 32
(In reply to comment #4) > It seems like this is available on p2 side: cf bug 436318. Tycho 0.22 includes a > good version of p2 to support this feature (bug 444112 ). Looking at the > PublisherServiceImpl (that directly takes the file and passes this to p2 ), it > seems to me that Tycho 0.22 should already support that out-of-the-box... The reason why this doesn't work is that PublishingRepository.getMetadataRepository() returns a ModuleMetadataRepository which only stores installable units and nothing else. So p2 probably calls addReferences on that repository, but that call is ignored. This is intentional: the ModuleMetadataRepository contains the build results of a module which can be used by other modules, and it doesn't make sense to copy "repository references" defined in a different module (or even in a remote p2 repository) to an eclipse-repository module. So really the problem is that the category.xml publisher result currently is copied at all. The category.xml publisher could directly write into the p2 repository at target/repository. (This is what Jan tried out bug 341744.) This would make the references show up in the build result. The side-effect of changing the publisher output repository is that these results can no longer be referenced by other modules (which can only read the p2metadata/p2artifacts.xml written by the ModuleMetadata/ArtifactRepository). But this is no problem because it anyway doesn't make sense (and was technically almost impossible) to reference category IUs in other modules.
Any ETA for this?
Just to place the information where it's the most relevant, a possible workaround is to use org.jboss.tools.tycho-plugins:repository-utils:generate-repository-facade mojo. https://github.com/mickaelistria/aCute/commit/2053ab221f5eaf1dfd3c28122691b99e4cc4dd03
New Gerrit change created: https://git.eclipse.org/r/96098
What is exactly expected in the result ? I create a unit test and all the parsing methods. But i see a snapshoot and resolution in the LocalMetadataRepository (publishRepositoryReferences call by initialize while closing tag reading). Does we must resolve it like this king of class of p2 repo. I don't know yet the internal p2 process sorry. And if yes, when we do it? just after reading it, just like LocalMetadataRepository ? I push my WIP, you can have a look
after investing a little, i am a little disappointed, Why tycho used his own MetadataIO instead of the MetadataRepositoryIO directly from p2 ? Because if we use it directly, we benefit of all their codes and must implements the repo but it's already done with the LocalMetadataRepository. no ? did I miss something ?
(In reply to guillaume dufour from comment #12) > after investing a little, i am a little disappointed, > > Why tycho used his own MetadataIO instead of the MetadataRepositoryIO > directly from p2 ? > Because if we use it directly, we benefit of all their codes and must > implements the repo but it's already done with the LocalMetadataRepository. > no ? > did I miss something ? I can't tell why and from where exactly it was copied but e.g. the name "Parser35M7" https://github.com/eclipse/tycho/blob/2694138a1611b81eaf4cc63a5f90dc3e86c091ed/tycho-bundles/org.eclipse.tycho.p2.maven.repository/src/main/java/org/eclipse/tycho/p2/maven/repository/xmlio/ArtifactsIO.java#L49 seems to suggest this was copied as a snapshot from p2 (and possibly adapted/patched for Tycho needs) the code has been in Tycho from day 1, so this one goes to Igor I guess
@guillaume if you are interested in software archeology, [1] is the commit where I introduced MetadataIO in its current shape (more or less, it used to share even less core with p2 back then). The reason I had to write Tycho-specific implementation was because MetadataRepositoryIO could not be used outside of running OSGi platform. You'd need to refactor MetadataRepositoryIO to allow standalone use if you want to replace MetadataIO with it. [1] https://github.com/sonatype/sonatype-tycho/commit/23d1f55398353ee803be14912c7db2dc19f8f025#diff-a9608a3f73f40b964c46b9f42691b6d2
Thanks a lot for your reply. I finish this without touching p2. But if I have enough time, i will do a PR to p2 to benefits of their code without re-implementing their. Is that a good idea, are you ok ?
I see the jenkins build fail but no error in the log and all test pass. Any idea?
(In reply to guillaume dufour from comment #16) > I see the jenkins build fail but no error in the log and all test pass. Any > idea? looks like there was a problem deleting the target/ folde. I retriggered the voter job.
(In reply to guillaume dufour from comment #15) > Thanks a lot for your reply. I finish this without touching p2. But if I > have enough time, i will do a PR to p2 to benefits of their code without > re-implementing their. Is that a good idea, are you ok ? if you can refactor p2 so that it can be reused without having to copy and adapt code for Tycho, yes that's a good idea. Tycho committer Tobias Oberlies is also a p2 committer so I hope he would be available to review your patch.
@Jan : to reply to the comment/review on patch 10 with the incompatibilty in MetadataIO. I see it from MetadataRepositoryIO. But I don't know if this class is used for all parsing/writing from p2. Is there any reference doc for this? Because it seems to always be this format since 10 years...https://github.com/eclipse/rt.equinox.p2/blob/master/bundles/org.eclipse.equinox.p2.metadata.repository/src/org/eclipse/equinox/internal/p2/metadata/repository/MetadataRepositoryIO.java So I think, maybe, i am on the wrong way... I can do a compatible parser to all formats (with units root and repository root), but is it enought ? Is that what we want... What i understand is, - category defined repo ref - we must get it and add it to repo - we must save them in the content.xml - we must use them for resolution ? The 3 first points are closed to be treated. The 3rd introduce a breaking change on the way tycho dump content.xml in p2 repo. The last one not yet treated, because I don't know if i must do this.
(In reply to guillaume dufour from comment #19) > @Jan : to reply to the comment/review on patch 10 with the incompatibilty in > MetadataIO. > > I see it from MetadataRepositoryIO. But I don't know if this class is used > for all parsing/writing from p2. Is there any reference doc for this? > Because it seems to always be this format since 10 > years...https://github.com/eclipse/rt.equinox.p2/blob/master/bundles/org. > eclipse.equinox.p2.metadata.repository/src/org/eclipse/equinox/internal/p2/ > metadata/repository/MetadataRepositoryIO.java I am not aware of official docs but de facto the content.xml format defined by p2 is API in the sense that different p2 clients in different eclipse versions rely on it to be parseable. > So I think, maybe, i am on the wrong way... > I can do a compatible parser to all formats (with units root and repository > root), but is it enought ? Is that what we want... > What i understand is, > - category defined repo ref > - we must get it and add it to repo > - we must save them in the content.xml > > The 3 first points are closed to be treated. The 3rd introduce a breaking > change on the way tycho dump content.xml in p2 repo. I think the problem is actually that Tycho initially decided to use a different format _internally_ for content.xml in target/p2content.xml and local maven repo. Note that this format is only used for intra-reactor and reactor-> local maven repo dependencies. This was a choice that is probably hard to change now. The way I see it there are 3 options: 1. revert back to the format given by p2 pros: we would get all the parser patches/improvements from upstream cons: incompatible change i.e. you can't exchange Tycho artifacts using the local maven repo if Tycho versions are different, you need to silently ignore the older/newer format,... 2. find a way to add the repo references in the given Tycho content.xml format pros: compatible change i.e. transparent to users cons: have to double-maintain content.xml parser, further divert from p2 content.xml format 3. Try to avoid the whole content.xml parser problem by publishing the category.xml directly into target/repository, similar as per bug 341744 comment 32 . Although this didn't seem to work for p2.statsURI I think it could work for repo references Looking at what we are trying to achieve here vs. what we are potentially breaking I tend to prefer option 3 (or 2, in that order). > - we must use them for resolution ? > > The last one not yet treated, because I don't know if i must do this. Not sure if I got you right, but here is my take: Dependency resolution of a Tycho project is a different story (governed by its target platform, see [1]). repo references in category.xml are only there for dependency resolution *at install time*, i.e. when someone is using the eclipse-repository build result to install what has been built by Tycho into an eclipse installation. I think the main usecase here is you provide a plugin that depends on another third-party plugin (not published by eclipse) and you can't/don't want (e.g. for legal reasons) to include the third-party plugin(s) in your own p2 repo but rather just reference their public p2 repo for dependency resolution at install time. Also, repo references in category.xml are only used and IMHO only make sense in an eclipse-repository module as a module-local property. I don't see a use-case for an upstream module having a dependency on the eclipse-repository module and referring to its repo references. [1] https://wiki.eclipse.org/Tycho/Target_Platform
So I create a new gerrit review without all my crap and i just push the category.xml to the target/repo with the publisher.
(In reply to guillaume dufour from comment #21) > So I create a new gerrit review without all my crap and i just push the > category.xml to the target/repo with the publisher. I hope this will work. Sorry for the detour but that's how it goes sometimes
after some days, I get back to this dev and to be sure what to implement. here is my dev: - keep my impact on AbstractSiteDependenciesAction & ProductDependenciesAction to read repo ref - provide them by repo properties (P2GeneratorImpl) - read this repo ref while executing the tycho-p2-director:materialize-products mojo (in DirectorMojo#getBuildOutputRepository) , no ? Is that it in your head ?
(In reply to guillaume dufour from comment #23) > after some days, I get back to this dev and to be sure what to implement. > here is my dev: > - keep my impact on AbstractSiteDependenciesAction & > ProductDependenciesAction to read repo ref > - provide them by repo properties (P2GeneratorImpl) > - read this repo ref while executing the > tycho-p2-director:materialize-products mojo (in > DirectorMojo#getBuildOutputRepository) , no ? > > Is that it in your head ? I'm not into the details right now but the repo references in category.xml should not affect the build itself, i.e. the repo refs are not added the build's target platform and not read anywhere during build except for being published along with category.xml. I think we should just make sure that the repo refs are published and end up in the target/repository build result (as opposed to the tycho-internal target/{p2content|p2artifacts}.xml). I'd start by changing PublishCategoriesMojo to publish the category.xml directly into target/repository (similar to bug 341744 comment 32 ) and see if that helps.
I stop working on this ticket. I think i don't know enought p2 and tycho to treat. I will treat other ticket and if no one have treat it, maybe in some month/year, I will do it. sorry
Is there any progress on this?
New Gerrit change created: https://git.eclipse.org/r/139698
Hello, i am back on this jira, before doing it, can some one provide a test project. Because I create one, but i am not sure it's a good one. Can someone check if it's ok or correct/provide me a good one: https://git.eclipse.org/r/#/c/139698/
Gerrit change https://git.eclipse.org/r/139698 was merged to [master]. Commit: http://git.eclipse.org/c/tycho/org.eclipse.tycho.git/commit/?id=9d52c62abba51475f1aa426db757c4b6f7037b08
New Gerrit change created: https://git.eclipse.org/r/141135
Gerrit change https://git.eclipse.org/r/141135 was merged to [master]. Commit: http://git.eclipse.org/c/tycho/org.eclipse.tycho.git/commit/?id=2b6a22a9f399d5f5236c1af4dfda977d9452dea2
New Gerrit change created: https://git.eclipse.org/r/141872
Gerrit change https://git.eclipse.org/r/141872 was merged to [master]. Commit: http://git.eclipse.org/c/tycho/org.eclipse.tycho.extras.git/commit/?id=5ac3d2d32ec7049f646c50c834c2d511259462d8
First of all thanks for taking care about this issue, I appreciate that a lot. Unfortunately the solution doesn't work out in the end for me with an Eclipse 2019-03 IDE into which I tried to install some content from a self-built update site defining referenced repositories. They are originally defined in my category.xml, like <repository-reference location="http://download.eclipse.org/tools/orbit/downloads/drops/R20180606145124/repository" enabled="true" /> <repository-reference location="http://download.eclipse.org/modeling/mdt/uml2/updates/5.4/" enabled="true" /> After confirming everything in the "New Software Installer", I get exceptions like !ENTRY org.eclipse.equinox.p2.engine 4 4 2020-01-10 00:07:37.428 !MESSAGE An error occurred while collecting items to be installed !SUBENTRY 1 org.eclipse.equinox.p2.engine 4 0 2020-01-10 00:07:37.428 !MESSAGE session context was:(profile=epp.package.dsl, phase=org.eclipse.equinox.internal.p2.engine.phases.Collect, operand=, action=). !SUBENTRY 1 org.eclipse.equinox.p2.artifact.repository 4 0 2020-01-10 00:07:37.428 !MESSAGE No repository found containing: osgi.bundle,com.google.inject,3.0.0.v201605172100 !SUBENTRY 1 org.eclipse.equinox.p2.artifact.repository 4 0 2020-01-10 00:07:37.428 !MESSAGE No repository found containing: osgi.bundle,org.eclipse.uml2.uml,5.4.0.v20180903-1400 !SUBENTRY 1 org.eclipse.equinox.p2.artifact.repository 4 0 2020-01-10 00:07:37.428 !MESSAGE No repository found containing: org.eclipse.update.feature,org.eclipse.uml2.uml,5.4.0.v20180903-1400 As far as I could drill down the misbehaviour, the problem seems to be rather simple: While the jboss mojo ("org.jboss.tools.tycho-plugins:repository-utils:generate-repository-facade") generates type="0" AND type="1" repository declaration tags into content.xml, tycho's eclipse-repository packager (in v1.5.0) generates type="0" repository declarations only. After I added the missing type="1" repository definitions manually in the content.xml (in the jar & xz archives) and removing the previous "Available Software Sites" entries in the IDE's preference page the installation works like a charm. It would be awesome if you guys could add that fix, as I'm not familiar with the Tycho code base at all.
https://github.com/eclipse/tycho/blob/master/tycho-bundles/org.eclipse.tycho.p2.tools.impl/src/main/java/org/eclipse/tycho/p2/tools/mirroring/MirrorApplication.java#L74 This seems to be the causing line. Only METADATA repo reference is added, but the ARTIFACTS is also/instead needed. I don't know if adding both types/changing type to ARTIFACTS is a correct fix, but it sure would be a quick one. I'd appreciate if someone with more knowledge on p2 would fix this.
I also get the errors described by Christian in Comment #34. And what's even worse: If the user now goes to Preferences->Install New Software, it really seems like the referenced site is fully added. I tried to "reload" that site from the preferences, which doesn't work. The only way to fix errors is to remove the site, and add it again. Maybe even a restart required?
In case it helps anyone else, as a workaround until this Tycho issue is fixed, in the Epsilon project [1] we've added a Groovy program [2] to our POM [3] that clones the <repository> elements in the Tycho-generated content.xml and sets the type of the cloned repository elements to 1 (i.e. adds artefact repositories with the same URLs). Following this change, installation works as expected both from P2 Director and from the Eclipse UI. [1] https://eclipse.org/epsilon [2] https://git.eclipse.org/c/epsilon/org.eclipse.epsilon.git/tree/releng/org.eclipse.epsilon.updatesite/patch-content-xml-and-zip-site.groovy [3] https://git.eclipse.org/c/epsilon/org.eclipse.epsilon.git/tree/releng/org.eclipse.epsilon.updatesite/pom.xml
Eclipse Tycho is moving away from this bugs.eclipse.org issue tracker to https://github.com/eclipse/tycho/issues/ instead. If this issue is relevant to you, your action is required. 0. Verify this issue is still happening with latest Tycho 2.4.0-SNAPSHOT if issue has disappeared, please change status of this issue to "CLOSED WORKFORME" with some details about your testing environment and how you did verify the issue; and you're done if issue is still present when latest release: * Create a new issue at https://github.com/eclipse/tycho/issues/ ** Use as title in GitHub the title of this Bugzilla ticket (may include the bug number or not, at your own convenience) ** In the GitHub description, start with a link to this bugzilla ticket ** Optionally add new content to the description if it can helps towards resolution ** Submit GitHub issue * Update bugzilla ticket ** Add to "See also" property (up right column) the link to the newly created GitHub issue ** Add a comment "Migrated to <link-to-newly-created-GitHub-issue>" ** Set status as CLOSED MOVED ** Submit All issues that remain open will be automatically closed next week or so. Then the Bugzilla component for Tycho will be archived and made read-only.
Migrated to https://github.com/eclipse/tycho/issues/141
