Bugzilla – Full Text Bug Listing
| Summary: | Support embedding repository information in released bundles | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | [Eclipse Project] PDE | Reporter: | Martin Oberhuber <mober.at+eclipse> | ||||||||||||||||
| Component: | Build | Assignee: | pde-build-inbox <pde-build-inbox> | ||||||||||||||||
| Status: | RESOLVED FIXED | QA Contact: | |||||||||||||||||
| Severity: | enhancement | ||||||||||||||||||
| Priority: | P3 | CC: | alex.blewitt, aniefer, bokowski, caniszczyk, contact, curtis.windatt.public, daniel_megert, darin.eclipse, david_williams, digulla, don, gunnar, jeffmcaffer, john.arthorne, kim.moir, mariot.chauvin, markus.kell.r, mn, oisin.hurley, pascal, pwebster, stephan.herrmann, tjwatson | ||||||||||||||||
| Version: | 3.5 | ||||||||||||||||||
| Target Milestone: | 3.6 M6 | ||||||||||||||||||
| Hardware: | All | ||||||||||||||||||
| OS: | All | ||||||||||||||||||
| Whiteboard: | |||||||||||||||||||
| Bug Depends on: | |||||||||||||||||||
| Bug Blocks: | 195729, 306637 | ||||||||||||||||||
| Attachments: |
|
||||||||||||||||||
+1 for the idea/direction. 0/-1 for putting the info in the binary bundles. We need the info to be available but putting it in the binary means that it is not generic. For example, say WindRiver is shipping an Eclipse-based product and that they offer their customers support. When a problem happens, the customers should see WindRiver information about how to report bugs and where to get source (it may have been patched, changed, rebuilt, ...). I can see several options for where the information could go. Most of these involve putting it in p2 metadata and then having a bit of p2 infrastruture that "install" the projects in the workspace. There is likely some Buckminster code that would do part of this. One of the issues that comes up for me is whether this information goes in individual new elements (e.g., IUs), is some how in an IU fragment, or is in "lumps" more or less equivalent to the map files. Anyway, mostly implementation details. I like the idea. The main issue I see here is how to get the latest code (e.g. HEAD for CVS) into Eclipse. For CVS, it is Team>Replace with Tag or Branch (or switch) but how would a non-CVS user know this? What is it for other repository providers (e.g. I don't know how this would be done for Subversion)? Also, what happens if the required repository provider isn't installed? Thanks for the comments! (In reply to comment #1) Having the info provided by the final distributor does in fact make some sense, and the idea of embedding bugzilla component information (or a bugzilla "new bug" URL) into the proposed IU comes to mind for those bundles directly obtained from Eclipse.org. In the concrete case, however, my idea was just to make a clear connection between the binary bundle and its provenience i.e. where it came from. Even if Wind River is the final distributor of some Eclipse based technology, and chose to patch some bundle, it would be their obligation to also patch the provenience / repository information (I'd actually think that the EPL even requires them to patch the provenience info in that case, as per http://www.eclipse.org/legal/eplfaq.php#SOURCEWOBJ ). One other option might be that, if a bundle has been patched by a distributor, the distributor keeps the original repository info (which points into the Eclipse.org repository) intact but also ships a source bundle. When invoking the "Import from Repository" wizard, the wizard would create a project with CVS Team Sharing enabled for the Eclipse repository, but the patched source files would show up as "Modified". So, I personally don't see a connection between who provides support for some technology, and if or how to bundle the binaries together with source provenience information. Bundling provenience information together with the binaries seems in strict obedience with the EPL for me. Note that provenience info could also be fuzzy just saying "source for this bundle is available from foo.bar". This would not be as helpful for users, but would be the correct solution in cases where patched EPL code must be made available, but the actual Repository is not public. (In reply to comment #2) The idea is that users wouldn't need to know how to operate the team provider to see the version which they are about to import. The tooling would care for importing exactly the tag that was used to build the binaries. For an N-build made out of CVS, embedding a branchtag (HEAD) together with a timestamp would seem appropriate. If the person invoking the "Import from Repository" function does not have a CVS team provider installed, the import wizard could simply issue an error in that case ("Source is in CVS, but CVS Team provider is not installed"). Perhaps the Team API would need to be extended in order to support programmatically switching to some branchtag w/ some timestamp independent of the repository provider? Feels like it is getting complicated. In an effort to keep it simple I was trying to a) keep the concerns separate (producer/consumer) and b) avoid shipping the dev structure. By putting it in the bundles you need to know the information at build time. Bundles are often built by people who do not ship stuff (ie., they do not control the outside view of things) so they would not have this information or it changes after the fact. I would stay away from the "provenience" discussion. That's dangerously close to lawyer talk. We simply want to enable people to get code in their workspace that they can work with. That location may be different for different teams in a company, different geographies, different customers, ... Hm, these are good points. In fact I was a bit scared of the provenience discussion myself (does a vendor who patches an Eclipse bundle also have to patch the "Bundle-Vendor" attribute in the Manifest?) - Also, in the case of a Platform Source Build, the Repository info is available when the source build ZIP is created but not when the binaries are actually built.
So it looks like some artifact needs to be produced by the build to hold the Repository information. This artifact could be the Mapfile (aka directory.txt). Some missing steps seem to be
(1) Define markup in the Mapfile to hold info for N-builds (branch and
timestamp, instead of tag)
(2) Create a builder that takes a Mapfile and packs it into a P2 artifact
(e.g. an IU like "Source Info" similar to "Source Features")
(3) Create a consumer (PDE wizard) that takes this artifact and is able to
create projects with proper Team Sharing from it, and/or adds the Team
Sharing information to source projects when they are created with the
existing "Import Plug-ins and Features" wizard.
Would the PDE team make a comment at this point?
It looks like the Knopflerfish OSGi implementation has something done along the lines of this idea: The meta-inf of their bundles seems to include a slot named "bundle-subversionurl" I didn't check that in any more detail but thought it were worth noticing. http://www.knopflerfish.org/images/repo_desktop_host.gif In general, I tend to agree with Jeff that this information should better not live in the bundle itself but in the (p2) meta-info. We've seen more often than not that whole projects move or change their CM system... we don't want to change already released bundles if that happens, so the meta info is the right place for this. I'm actually wondering where this enhancement request should really live. Should it go into Equinox in order to get the foundation for this done, or does p2 already provide everything needed once we've got bundle meta-info support end to end? Or should we leave this one open on PDE, and clone a new bug for Equinox? We want to make progress on this in the 3.6 time frame. Does this bug also include the UI enhancements for this? We discussed this at the PDE call and the p2 call last week. A bundle's metadata is immutable: the metadata for a bundle should not be different based on which source repo it was built from (if the metadata is different, the bundle may arguably be different). Thus, for the purpose of providing different source repository information for the same bundle, metadata does not seem like the right answer. As well, if one is building a bundle, it seems like they should have the source repository information at build time - the source had to come from somewhere. Having the information in the binary bundle feels most flexible, as one can use the information without requiring metadata. I'm assuming the MANIFEST.MF is fine?
Note, Maven embeds this into their POM...
http://maven.apache.org/pom.html#SCM
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">
...
<scm>
<connection>scm:svn:http://127.0.0.1/svn/my-project</connection>
<developerConnection>scm:svn:https://127.0.0.1/svn/my-project</developerConnection>
<tag>HEAD</tag>
<url>http://127.0.0.1/websvn/my-project</url>
</scm>
...
</project>
Bug 195729 tried to solve the same problem 2 years ago. Sweet ... big +1 for a feature like this. I also think that putting the information into the MANIFEST.MF is fine. It could be used by PDE out of the box. Dealing with the provisioning process could be delegated to SCM implementors. They know best how to fetch source from XYZ into a project ABC. (In reply to comment #4) > By putting it in the bundles you need to know the information at build time. True. But if you're building and you don't konw where the source came from, you should not include the information. > Bundles are often built by people who do not ship stuff (ie., they do not > control the outside view of things) so they would not have this information or > it changes after the fact. After discussing with build/team, I think a reasonable approach would be to put the information in the manifest of the binary bundle. Using a .psf format appears to be the most flexible - to support different repositories. A new Eclipse header in the manifest could identify the repository (team provider) and memento required to retrieve source. Currently, PDE build uses map files to fetch source. It passes map file entries to "fetch factories". A fetch factory is repository specific, and generates an Ant script to fetch source. It could also be the job of the fetch factory to generate source headers. Source header information would be accumulated in a separate file that would then be fed to PDE build to use for creating source headers. This provides some flexibility, since others could simply provide a "source" file, rather than having to use fetch factories (if needed). Since metadata for a bundle should be the same no matter what the bundle was built for, it seems more appropriate to have this information in the binary bundle manifest than metadata. If a product were shipping a branched version of the source from a different repository, they could decide whether or not to have PDE build generate source repo information. From a tooling point of view, it would be simpler to have the information with the bundles, rather than in a separate index (i.e. yet another configuration file to ship). I have to say I like the simplicity of storing this information directly in the binary bundle. Keeping the information elsewhere just seems to invite problems with that information getting out of sync, and problems with ensuring the correct source information is paired up with the correct binary. As Darin mentioned, the p2 metadata for a given bundle is considered immutable for a given bundle bsn and version, so the metadata is no easier to alter than a file inside the bundle itself. If someone redistributes a binary bundle without modifying it, then they haven't changed where the source of that binary code came from so I'm not seeing the producer/consumer dichotomy here. If someone redistributes a bundle with modification, then they can alter the source header to match the new location (or delete the header entirely if they don't want to provide it). This also dovetails nicely with signing. Since the binary bundle is signed by the producer, I can be sure that if the signature is intact then the source information will be correct. Created attachment 158774 [details]
patch
Example patch that adds project reference generation to the CVS fetch factory.
(In reply to comment #14) > I have to say I like the simplicity of storing this information directly +1 for putting repository information into the MANIFEST.MF. I filed this request for dealing with "public open source" kind of software, such that contributors could contribute more easily. The original needs are satisfied with this simple solution. Independent of the question where to put the information, I'm a little concerned about how to deal with "tainted" bundles. Consider the case where an Eclipse adopter patches a single source file to fix a particular issue. 99% of the bundle's sources would still be available from the original repository, but not the one patched file (the EPL requires the adopter to "make the source available" somewhere instead). If the original repository information remains in the MANIFEST, there is some risk that a person looks at incorrect source for the binary, which is IMO worse than looking at no source at all. I think there should be a policy or recommended best practice how an adopter should decorate or modify the repository information to avoid this scenario. I'm assuming that the adopter does want to play by the rules. They could remove the repository information altogether, but then the feature is gone for 99% of the code where it would match. There could be an exception list; any other ideas? +1 for putting repository information into the binary bundle's MANIFEST.MF. (In reply to comment #16) > Independent of the question where to put the information, I'm a little > concerned about how to deal with "tainted" bundles. Consider the case where an > Eclipse adopter patches a single source file to fix a particular issue. 99% of > the bundle's sources would still be available from the original repository, but > not the one patched file (the EPL requires the adopter to "make the source > available" somewhere instead). If the original repository information remains > in the MANIFEST, there is some risk that a person looks at incorrect source for > the binary, which is IMO worse than looking at no source at all. > I think there should be a policy or recommended best practice how an adopter > should decorate or modify the repository information to avoid this scenario. > I'm assuming that the adopter does want to play by the rules. > They could remove the repository information altogether, but then the feature > is gone for 99% of the code where it would match. There could be an exception > list; any other ideas? When a bundle is "tainted", the shipped source bundles will become invalid - just the same as the project set information. When this situation happens, do you re-build source bundles, or do you ship the original (incorrect) source bundles? If you are re-building source bundles, but not making it available publicly, then I would say the project set information should not be shipped/generated in the manifest. If the source is available publicly, the project set information should be changed to reflect the origin of the source. Over time, some of the "tainted" bundles/fixes will become part of the platform. However, if you always ship with some set of extra fixes, then you are in a perpetual state of being unable to include project set information with the bundles. In the 3.6 time frame, I don't think we'll have time to solve this problem. Looking to the future, we might be able to come up with some story where patch files are shipped with the platform to indicate the extra set of fixes that have already been applied/included. Putting it in the binary is obviously the most convenient. The OSGi standard even has a header related to this. (are you going to use that header?) The producer/consumer dichotomy is the standard one that drives us to not put the word "Eclipse" in various places. In this context it is likely somewhat less but "where to get the source" feels like a close friend of "I need support for this, who do I talk to" and so should be set in the context of the product, not the components. Having said that, it really depends on who we think is consuming this info. If it is developers using the individual components then sure, in the bundle is fine. If it is end-users of some product the, not so much. Some things to keep in mind: - the build system may have a different path to getting source than is publicly available - The build may require authentication/login info that should not be published - What is the format of the header? I think the standard OSGi header is pretty free form) - What SCM systems will be supported? CVS, SVN, GIT, ... and is there a way to normalize the info needed by these (In reply to comment #18) > When a bundle is "tainted", the shipped source bundles will become invalid - > just the same as the project set information. When this situation happens, do > you re-build source bundles, or do you ship the original (incorrect) source > bundles? Well, here is what we (Wind River) are doing: - We try hard feeding patches back into Eclipse, but still (due to different release cycles) with every release of our product there's 3 or 4 extra patches that we ship which are not (yet) in Open Source. - We generate correct source bundles for every modification, since this seems to be the most convenient way of complying with the EPL requirement of "making source avaialable". Most end users do not install any source bundles, though (not even the original Eclipse ones). Since the "source pointer" is going to be shipped with every binary bundle, I think that the "tainted source" problem will be a real one. Again, I think it's most important to make consumers aware that the source they are going to look at may not match their binaries. I agree that an automated way of getting correct source is likely out of scope for 3.6. But we should think about some format (probably just a free-form comment) that allows product builders annotate the source pointer. For instance just a comment like "Source for the following files has been modified by Wind River and is available via the Wind River support portal: Foo.java, Bar.java, yoyo.properties " If you are modifying the source, you should also be modifying the manifest to at least increment the version number. Then at the same time, the simplest action would be to just remove the source header entirely. If you are shipping source bundles then the user can still get source from there as they do today. Yes, we do modify the Manifest for stuff we're patching today (and we do increment the qualifier). So technically, it would be easy to remove the repository pointer. I could live with this for the sake of consistency, but IMO it would be a pity losing this feature for all of the bundle just because 1 or 2 files are modified. I'd prefer having an option to annotate the repository pointer, such that my clients could still import source through the project set, compare open source changes, create patches... All I want is that they KNOW that some sources have been changed locally (note that as soon as they build from the imported sources, they'll have the unpatched version of binaries so they are fine to that end). And no, Wind River is not interested in hosting a clone of the CVS repository on our public servers, such that we could put a different project set info into the Manifest. We'll just want to let clients know where they can obtain the patches. (In reply to comment #19) > Having said that, it really depends on who we think is > consuming this info. If it is developers using the individual components then > sure, in the bundle is fine. If it is end-users of some product the, not so > much. This feature is really intended for the developers - to easily get projects into a workspace such that patches can be made. > Some things to keep in mind: > - the build system may have a different path to getting source than is publicly > available Yes, PDE build will have a file that drives the content of the headers. The file can be generated by a task in the build, or simply provided. > - The build may require authentication/login info that should not be published We're planning to use the team "project set format" which supports a team provider specific format. For CVS, login/password information is not included (however, I don't think the specification dictates this). In general it would be poor for a team provider to include this (so I don't believe this is a problem). > - What is the format of the header? I think the standard OSGi header is pretty > free form) The header will identify the Eclipse team provider (basically the extension ID associated with the team support) and project reference(s) using the "project set capability" (a team's project set capability can generate a reference/memento for a project such that the project can be re-created in a workspace). The header name itself needs to be agreed upon. This is Eclipse specific support, since we are basing it on Eclipse team provider extensions and the Eclipse .psf format. I would suggest "Eclipse-ProjectSet:". What is the standard OSGi header you (Jeff) mentioned? > - What SCM systems will be supported? CVS, SVN, GIT, ... and is there a way to > normalize the info needed by these We will provide support for CVS out of the box. It is up to each team provider to implement the necessary support in their fetch factory. (In reply to comment #23) > The header name itself needs to be agreed upon. This is Eclipse specific > support, since we are basing it on Eclipse team provider extensions and the > Eclipse .psf format. I would suggest "Eclipse-ProjectSet:". I like the idea of using an Eclipse specific header for the "automatic" (project set reference) feature. With respect to the "tainted source" problem, this would allow contributors to still use the standard OSGi header for freeform comments. An import wizard could then leverage the Eclipse automatic functionality, but still display the OSGi comment. With respect to the actual name, this is no project set but a single reference, so I'd propose any of Eclipse-ProjectReference Eclipse-TeamProjectReference Eclipse-CMReference Eclipse-SourceReference Eclipse-SourceRef ... > What is the standard OSGi header you (Jeff) mentioned? In Apache Commons Net, which uses the Apache "Bnd" utility to generate Manifest headers, I have seen Bundle-DocURL: http://commons.apache.org/net/ being used for a somewhat similar use, since the given URL does not reference the actual documentation but the Library's homepage on the Internet. > this is no project set but a single reference
True. I wondered if there's any reason to allow for a bundle to reference more than one project. I don't have a real use case for this, but could imagine a single bundle provided to pull in many projects... however, I suppose that's the same as simply providing a .psf file.
+1 for 'Eclipse-ProjectReference'
(In reply to comment #25) > +1 for 'Eclipse-ProjectReference' I actually feel like 'Eclipse-SourceReference' may be more accurate, since this is going to ship with the binary bundle and reference where the source can be found. > We're planning to use the team "project set format" which supports a team > provider specific format. For CVS, login/password information is not Is there a particular reason why you are not using the PDE/Build mapfile format with associated fetch factories? Somehow this functionality feels closer to what we want to accomplish (provision a specific version/tag of source out of a repository) as well as where to accomplish it (during the build). I recently noticed that fetch factories support provisioning out of a p2 repository. This might open up a chance for no longer shipping source bundles, but shipping a reference to provision a source bundle on the fly out of p2 when needed. With its mirrors, p2 provisioning would always be faster than CVS. > I wondered if there's any reason to allow for a bundle to reference more > than one project. Well, perhaps not one project, but multiple source locations might make sense. This could also address the "tainted source" issue: - primary source location with "CVS" mapentry - secondary one with "P2" to get source bundle - third one with "COPY" mapentry to provision tainted sources directly shipped in the binary bundle - fourth one pointing to external Jar sources needed ... (In reply to comment #26) > Is there a particular reason why you are not using the PDE/Build mapfile format > with associated fetch factories? Somehow this functionality feels closer to > what we want to accomplish (provision a specific version/tag of source out of a > repository) as well as where to accomplish it (during the build). Fetch factories generate an ant script to retrieve source for the purpose of compilation - however, they don't do eveything required to create a project in Eclipse with team support. For example, the CVS fetch factory does not generate CVS directories/metatada required for synchronizing projects, nor does it set up the connection between the project and the repository. This is what the "project set capability" does, and that's why I prefer the term "Project Reference". Ok, I see your point. The main goal of this work is setting up a project with team support, so that's what we should mainly be after now.
On the other hand, fetch factories may be interesting in the future, as an alternative to "import project with source from source bundles". Perhaps the Manifest header could be in a format that would allow supporting fetch factories/mapfile entries / other means of acquiring source in the future? For instance:
Eclipse-SourceReferences=type:"spec",type:"spec",type:"spec"
allowing multiple source references of different type. A first implementation could only allow:
Eclipse-SourceReferences=PSF:"projectSetMemento"
but in the future, there could be:
Eclipse-SourceReferences=PSF:"projectSetMemento",MAP:"fetchMemento",
COPY:"src/path/to/tainted/source",TEXT:"refToPlaintextInfo"
The particular user story I have in mind is the "tainted source" scenario, where the PSF reference could provision the version-controlled project, but then a COPY rule could copy the "tainted" source files out of the binary bundle into the project such that they look like locally modified CVS files and source exactly matches the binary.
Again, the main goal should be the one outlined in comment 0 for now, but I think it would be wise adding a "Type" to the manifest header such that the header can potentially be extended in the future.
(In reply to comment #28) > Again, the main goal should be the one outlined in comment 0 for now, but I > think it would be wise adding a "Type" to the manifest header such that the > header can potentially be extended in the future. I agree with this. Based on the osgi spec for common header syntax, the manifest entry format will be:
Eclipse-SourceReferences: clause (',' clause )*
clause ::= reference (';' directive )*
reference ::= '"' ref-string '"'
directive ::= token ':=' argument
argument := token | quoted-string
token ::= (alphanum | '_' | '-' )*
We will need a "type" directive, additional directives may be required depending on the type. In particular, the "psf" type (which is the only one we are doing right now) needs a "provider-ref" directive.
example:
Eclipse-SourceReferences: "1.0,:pserver:dev.eclipse.org:/cvsroot/rt,org.eclipse.equinox/p2/bundles/org.eclipse.equinox.p2.publisher,org.eclipse.equinox.p2.publisher";type:=psf;provider-ref:="org.eclipse.team.cvs.core.cvsnature"
We are using directives instead of attributes (token '=' argument) because directives generally have implied semantics and attributes are for matching and comparison purposes.
The proposed syntax looks great to me! Why do you propose "provider-ref" rather than just "provider" ? Created attachment 160008 [details]
updated patch
Updated Patch. I have no particular reasons for "provider-ref", this patch uses "provider". Darin, could you take a look at the changes to Constants and IFetchFactory.
Still need testing and properties to turn this on and off.
One outstanding question is what to do if the manifest already contained the Eclipse-SourceReferences header. We can keep the original, append, or replace, this patch currently replaces.
(In reply to comment #32) > One outstanding question is what to do if the manifest already contained the > Eclipse-SourceReferences header. We can keep the original, append, or replace, > this patch currently replaces. I would vote for "append". If the header already exists during a build, then I'd assume somebody has manually generated the header in order to convey some information in there, so that should not be overwritten. Another option might be using a placeholder (a bit similar to the ".qualifier"). For instance: Eclipse-SourceReferences=%pdeGenerateRef,"MyManualRef";type:=manual The "%pdeGenerateRef" placeholder would be replaced by PDE Build with the actually generated sourceReference. One advantage of using the placeholder is that the order of entries can be specified by the user. Another advantage is that it also solves the turn off / turn on issue, since the source-reference-generator would only run when the Eclipse-SourceReferences header already exists in the manifest and the %generateRef placeholder is there. For simplicity, my vote is to replace. If the "%pdeGenerateRef" placeholder has to be present, it means all developers have to add this header/placeholder to their manifest to get the feature to work? I'd rather see this controled by the build process so developers don't have to manage the tag. It's not clear what append means (at least to me). Since PDE build will allow the headers to be generated by any means (fetch factories, or simply provided in a file), it seems that the ability for a particular build process to control/specify the headers already exists. (In reply to comment #34) Good point, it makes no sense to require all developers update their MANIFEST.MF in order to benefit from the feature. On the other hand, though, *if* an Eclipse-SourceRef header has been added manually, it's not polite to trample on it, so what about this: * Global Property governs whether source info is generated or not * If generation is turned on, and a MANIFEST.MF has no existing Eclipse-SourceRef header, then generate it * If an existing Eclipse-SourceRef header is in the MANIFEST.MF, then replace the placeholder only but leave unchanged if no placeholder is there. This would allow global generation without touching any Manifest, but still give full flexibility for turning off the feature on an individual bundle, or merge generated information with manually written one just as needed. (In reply to comment #32) > Darin, could you take a look at the changes to Constants and > IFetchFactory. The API contants and IFetchFactory specification updates look good to me. (In reply to comment #35) > (In reply to comment #34) > Good point, it makes no sense to require all developers update their > MANIFEST.MF in order to benefit from the feature. On the other hand, though, > *if* an Eclipse-SourceRef header has been added manually, it's not polite to > trample on it, so what about this: > * Global Property governs whether source info is generated or not > * If generation is turned on, and a MANIFEST.MF has no existing > Eclipse-SourceRef header, then generate it > * If an existing Eclipse-SourceRef header is in the MANIFEST.MF, then > replace the placeholder only but leave unchanged if no placeholder is there. > This would allow global generation without touching any Manifest, but still > give full flexibility for turning off the feature on an individual bundle, or > merge generated information with manually written one just as needed. This make sense to me. Andrew? As I noted on my recent blog post on the subject, it's a concern that we're not using a standard way of referring to version control systems, but rather, laying on eclipse-specific semantics and knowledge of the PSF format and Eclipse IDE natures. As well as being highly specific to Eclipse, it's also pretty much a guarantee that it's not going to be adopted by non-Eclipse tools that might consume bundles and want to know their source (e.g. NetBeans OSGi support or IntelliJ's Osmorc plugin). http://alblue.blogspot.com/2010/02/link-to-source-repository-in-bundles.html Furthermore, this does somewhat fly in the face of having a perfectly well defined mechanism to refer to SCMs by URL that Maven has popularised. http://maven.apache.org/scm/scm-url-format.html This has a per-provider set of components; in other words, what applies to CVS might not necessarily need to be used for SVN, or vice-versa. And unlike the highly specific nature that only PDE knows about (and hey, good luck if you want to refactor that ever), it's possible to define a mapping from the known SCM implementors to a given Nature/FetchFactory. One could even publish the FetchFactory service with a service property, such that given an SCMURL, you could get anything that implements the FetchFactory, rather than ensuring an only-one approach hardcoded/distributed across thousands of bundles. There's standard support for both CVS and SVN (which would be the two initially supported by Eclipse), but if a registry/service approach was taken, would permit others to contribute implementation(s) of service providers as well. http://maven.apache.org/scm/cvs.html http://maven.apache.org/scm/subversion.html It would also give the choice of implementation to defer to either Subversive or Subclipse, depending on what's installed, rather than a hard-coded specific implementation. Or would you encode 'org.tigris.subclipse' in a header, and then force people who have already installed Subversive to install Subclipse in order to get at the source? Here is the proposed example: Eclipse-SourceReferences: "1.0,:pserver:dev.eclipse.org:/cvsroot/rt,org.eclipse.equinox/p2/bundles/org.eclipse.equinox.p2.publisher,org.eclipse.equinox.p2.publisher";type:=psf;provider-ref:="org.eclipse.team.cvs.core.cvsnature" Here is the equivalent in a Maven SCM type format: Eclipse-SourceReferences: scm:cvs:pserver:dev.eclipse.org:/cvsroot/rt,org.eclipse.equinox/p2/bundles/org.eclipse.equinox.p2.publisher Note the lack of any Eclipse-specific information needed in order to define, or access, the URL. On the assumption that some registry/service published with a property that has 'scm=cvs', one can pass the URL on and be done with it. Furthermore, this is something that can (relatively trivially) be added to the maven-bundle-plugin - one, which I can happily provide a patch for and/or implementation of this bug should we come to an agreement on a common format, which will facilitate getting Apache sources out of an Apache-provided bundle, and potentially therefore also other IDE support. Arguably, one of the weaknesses of the CVS SCM is that it doesn't support a branch (or tag) directly; we could fix that in an Eclipse specific way by adding the additional OSGi/Eclipse fields: Eclipse-SourceReferences: scm:cvs:pserver:dev.eclipse.org:/cvsroot/rt,org.eclipse.equinox/p2/bundles/org.eclipse.equinox.p2.publisher;branch:=1.0;projects:=org.eclipse.equinox.p2.publisher I'd also like to observe that this approach means that teams no longer need to provide a PSF file, which is not necessarily the case across all projects. Instead, they can just publish a single SCMURL on their website, and a File -> Import Projects from SCMURL can be kicked off to bring things down. It has been commented elsewhere that it may be necessary to have multiple source locations per bundle. Whilst it's true that some projects (like SWT) have multiple projects and/or multiple source folders, these are all stored in the same repository, and normally, under some common root that can be checked out. So rather than assuming that the SCMURL has a one-to-one relationship with a project, I'd suggest treating the URL as either a project, or a folder that contains nested projects elsewhere. In the Maven case, it's very common that the SCMURL points to a parent project which then has a number of modules underneath. In addition, DVCS like Git result in repositories which can't have a partial check-out (it's all or nothing) and it's uncommon for there to be a one-project-per-repository case (though clearly, one project is a degenerate case of multiple projects). So, in terms of realising a single Bundle-SCMURL, it may be necessary to associate that with a checkout which includes multiple Eclipse projects. In the default case, it would be beneficial to perform the full checkout and then execute the 'import existing projects from workspace' approach, which scans folders for .project files and then offers the opportunity to bring those in. As a default case, that would then permit most Maven SCMURLs to be used directly in this format (bringing multiple projects in where necessary) as well as perhaps adding a directive to restrict the set of projects to just one, where one is known at a subpath as above. Finally, I'd just like to observe that being able to understand the Maven SCMURL format, even if the work is passed off to the Eclipse handler implementations, has advantages outside of the OSGi/PDE world. For example, one might associate the 'scm' prefix with a real windows handler, which invokes Eclipse and then checks the project(s) out on your behalf. So the approach could work just as well for non-OSGi Java projects, as well as CDT, PyDev ... (In reply to comment #38) > As I noted on my recent blog post on the subject, it's a concern that > we're not using a standard way of referring to version control systems Your concern is well understood, and that's why the format of the "Eclipse-SourceRef" manifest entry is flexible enough to support other referral methods than PSF in the future. For now, this work item is to solve a very concrete request to support Eclipse developers. Let's take one step at a time and solve the scenario for Eclipse (CVS) developers end-to-end first, before we extend the feature for other repository access providers than PSF in the future. > pretty much a guarantee that it's not going to be adopted by non-Eclipse tools > that might consume bundles and want to know their source (e.g. NetBeans OSGi > support or IntelliJ's Osmorc plugin). It's not our job at Eclipse to define standards to be used everywhere at OSGi -- I'd say this is the job of the OSGi alliance. People can use Maven or Buckminster today for solving their workspace provisioning needs -- or a future iteration of this feature, but for now we're only after the user story in comment 0. > Furthermore, this does somewhat fly in the face of having a perfectly well > defined mechanism to refer to SCMs by URL that Maven has popularised. > > http://maven.apache.org/scm/scm-url-format.html Thanks, that's an interesting link, and it will be easy to use e.g. Eclipse-SourceRef="scm:mavenprov,mavenspec";type:=mavenurl in the future, i.e. simply using a different type:= than PSF. > It would also give the choice of implementation to defer to either Subversive > or Subclipse, depending on what's installed, rather than a hard-coded Now that's a very good point, and I agree it's a weekness of the PSF format that only one provider is supported. I agree that a mapping from "provider-specific-format" to "generic-format" and back would be highly desirable. But I guess that the code for this mapping would need to reside in the provider. Or, a new extension point e.g. repositoryReferenceMappers is introduced. Great idea for a new enhancement request to file, but unfortunately out of reach for 3.6 with the API freeze just 2 weeks away. > Arguably, one of the weaknesses of the CVS SCM is that it doesn't support a > branch (or tag) directly; we could fix that in an Eclipse specific way by Oops, am I missing something, or is this re-introducing something non-generic, non-standardized? > It has been commented elsewhere that it may be necessary to have multiple > source locations per bundle. Whilst it's true that some projects (like SWT) > have multiple projects and/or multiple source folders, these are all stored in > the same repository Not in the "tainted source" scenario, where e.g. Wind River applies a patch to SWT and there is no repository for the patched sources. See some comments futher up on that. > Finally, I'd just like to observe that being able to understand the Maven > SCMURL format, even if the work is passed off to the Eclipse handler > implementations, has advantages outside of the OSGi/PDE world. Definitely yes! I'm all in favor of standardizing repository references, especially in the light of the problem that PSF's are provider dependent. Would you mind filing a new bug requesting such standardized references? Let's keep this bug focused on ideas and concerns in support of comment 0, that are realistic to address end-to-end in Eclipse 3.6. If somebody has an idea addressing the subclipse-versus-subversive issue in this time frame, I'm all for it! PDE/Build's API treats the specific contents of this header as opaque. It is up to specific repository providers to decide on the whatever specifics make sense for them. As a client of its own API, build will provide a CVS implementation for this. As Martin says we need something concrete so we use the psf format because it can be immediately leveraged for 3.6 with very little effort. (In reply to comment #39) > (In reply to comment #38) > > As I noted on my recent blog post on the subject, it's a concern that > > we're not using a standard way of referring to version control systems > > Your concern is well understood, and that's why the format of the > "Eclipse-SourceRef" manifest entry is flexible enough to support other referral > methods than PSF in the future. > > For now, this work item is to solve a very concrete request to support Eclipse > developers. Let's take one step at a time and solve the scenario for Eclipse > (CVS) developers end-to-end first, before we extend the feature for other > repository access providers than PSF in the future. The Eclipse CVS map is trivially translatable to a Maven SCM format, though. Indeed, it also solves the other problem regarding the providers; specifically, by prepending with scm:cvs: then you identify the provider to use for the remainder of the URL. > It's not our job at Eclipse to define standards to be used everywhere at OSGi That's true; there's https://www.osgi.org/bugzilla/show_bug.cgi?id=48 for that. However, it would be prudent to base a design on something that is already widespread in the Java universe. > ... but for now we're only after the user story in comment 0. There's nothing incompatible with what I said in comment 0, nor bug 195729 which was filed almost three years ago (and a year before this bug was opened). > > Furthermore, this does somewhat fly in the face of having a perfectly well > > defined mechanism to refer to SCMs by URL that Maven has popularised. > > > > http://maven.apache.org/scm/scm-url-format.html > > Thanks, that's an interesting link, and it will be easy to use e.g. > Eclipse-SourceRef="scm:mavenprov,mavenspec";type:=mavenurl > in the future, i.e. simply using a different type:= than PSF. Note that the scm: identifies the maven url; the characters after that is the subprotocol which determines whether it's CVS, SVN or some other more esoteric version control system. Again, this is possible for use with this without having to have a ;type qualifier. > > It would also give the choice of implementation to defer to either Subversive > > or Subclipse, depending on what's installed, rather than a hard-coded > > Now that's a very good point, and I agree it's a weekness of the PSF format > that only one provider is supported. I agree that a mapping from > "provider-specific-format" to "generic-format" and back would be highly > desirable. But I guess that the code for this mapping would need to reside in > the provider. Or, a new extension point e.g. repositoryReferenceMappers is > introduced. Great idea for a new enhancement request to file, but unfortunately > out of reach for 3.6 with the API freeze just 2 weeks away. The format is still something that could be applied now, even if the implementation is internal, right? That's much better than having to revisit this down the line to provide a different implementation. > > Arguably, one of the weaknesses of the CVS SCM is that it doesn't support a > > branch (or tag) directly; we could fix that in an Eclipse specific way by > > Oops, am I missing something, or is this re-introducing something non-generic, > non-standardized? The Maven SCM CVS provider doesn't list a way of having a specific branch/tag. Normally, this is because the Maven pom is checked in as is without updating the branches, so they don't have a way of representing which-of-a-set to checkout. Other providers, like the Maven SCM SVN provider, implicitly use the branch as part of the URL. If we need to encode a specific branch/tag in the manifest. In the previous example: Eclipse-SourceReferences: scm:cvs:pserver:dev.eclipse.org:/cvsroot/rt,org.eclipse.equinox/p2/bundles/org.eclipse.equinox.p2.publisher would assume a 'head' of the project(s). There's no scope in the format for a branch/tag name. So, we'd have to add something like: Eclipse-SourceReferences: scm:cvs:pserver:dev.eclipse.org:/cvsroot/rt,org.eclipse.equinox/p2/bundles/org.eclipse.equinox.p2.publisher;tag:=ECLIPSE_35 The point would be to make the easy case easy (by just listing the SCMURL) and to make additional cases (like checking out a PSF) possible. For example, one could use: Eclipse-SourceReferences: scm:cvs:pserver:dev.eclipse.org:/cvsroot/rt,org.eclipse.equinox/p2/bundles/org.eclipse.equinox.p2.publisher to check out everything under the 'org.eclipse.equinox/p2/bundles/org.eclipse.equinox.p2.publisher' folder from CVS, and prompt to import the files, whilst: Eclipse-SourceReferences: scm:cvs:pserver:dev.eclipse.org:/cvsroot/rt,org.eclipse.equinox/p2/bundles/org.eclipse.equinox.p2.publisher;project:=org.eclipse.equinox.p2.publisher might just check out one project underneath that URL. > > It has been commented elsewhere that it may be necessary to have multiple > > source locations per bundle. Whilst it's true that some projects (like SWT) > > have multiple projects and/or multiple source folders, these are all stored in > > the same repository > > Not in the "tainted source" scenario, where e.g. Wind River applies a patch to > SWT and there is no repository for the patched sources. See some comments > futher up on that. Arguably this kind of problem is solved with DVCSs, but I don't think having this URL will help or hinder that case. If we need to do this, one could imagine a comma-separated list of repositories, rather than just one: Eclipse-SourceReferences: scm:cvs:pserver:dev.eclipse.org:/cvsroot/rt,org.eclipse.equinox/p2/bundles/org.eclipse.equinox.p2.publisher,scm:cvs:pserver:internal.windriver.com:/root/swt,org.eclipse.swt/super/sercret/repo/org.eclipse.swt I don't see any reason with allowing this to be a one-or-more repositories. One could even use that approach to support a specific subset of projects from a common root (i.e. where there's too many under a single root to want to check them all out): Eclipse-SourceReferences: scm:cvs:pserver:dev.eclipse.org:/cvsroot/rt,org.eclipse.equinox/p2/bundles;project:=org.eclipse.equinox.p2.publisher, scm:cvs:pserver:dev.eclipse.org:/cvsroot/rt,org.eclipse.equinox/p2/bundles;project:=org.eclipse.equinox.p2.consumer,... > Definitely yes! I'm all in favor of standardizing repository references, > especially in the light of the problem that PSF's are provider dependent. Would > you mind filing a new bug requesting such standardized references? Already have. See e.g. https://www.osgi.org/bugzilla/show_bug.cgi?id=48 and bug 195729 > Let's keep this bug focused on ideas and concerns in support of comment 0, that > are realistic to address end-to-end in Eclipse 3.6. If somebody has an idea > addressing the subclipse-versus-subversive issue in this time frame, I'm all > for it! The concern regarding API is valid, as far as the release train is concerned. However, that shouldn't prevent standardising on the URL. We could even support the mixed-case where a PSF is needed, by putting the type as a parameter: Eclipse-SourceReference: scm:cvs:pserver:dev.eclipse.org:/cvsroot/rt,org.eclipse.equinox/p2/bundles/org.eclipse.equinox.p2.publisher;type:=psf;psf:=p2.psf However, it would be great to support an out-of-the-box case where a PSF is not necessary. For example, if I wanted to check out all of P2, I would use: Eclipse-SourceReference: scm:cvs:pserver:dev.eclipse.org:/cvsroot/rt,org.eclipse.equinox/p2/bundles Even if this gets translated to a PSF to implement (like the solution provided in bug 195729 did), having the format use a widely adopted standard across the Java space rather than a roll-our-own approach will facilitate the adoption/use by other communities. Following a PSF one will only be of use to Eclipse projects, and as noted, not all of those have a PSF. I'd also like to caution against depending on a URL being able to do partial checkouts. Git and Hg don't support this. In fact, most Maven SCM URLs don't have any per-bundle/project breakdown; they point to the root of the project. Then, one checks out the entire root, and each module is its own subfolder (usually one level down, but in practice can be multiple). (In reply to comment #40) > PDE/Build's API treats the specific contents of this header as opaque. It is > up to specific repository providers to decide on the whatever specifics make > sense for them. > > As a client of its own API, build will provide a CVS implementation for this. > As Martin says we need something concrete so we use the psf format because it > can be immediately leveraged for 3.6 with very little effort. Format != implementation. I used the PSF format to execute the import for a representation which was different in bug 195729. Getting the format right and parseable and allowing different implementations (e.g. translating into PSF internally to hand off to the providers) is not only possible, it's already been done. Comment from Jason van Zyl on my blog post, which may be of interest here: "Tycho actually recently added support for POM-first bundles. It's not complete but I think that both approaches are going to be used for a while given the prevalence of both the maven-bundle-plugin and PDE. Tycho will interoperate between the two as we are doing with the JGit/EGit project. There will be a massive rift created by tools that do not support both modes gracefully and allow the two worlds to interoperate." Hm... thanks for the bug references, but I'm still not quite sure what you expect us to do right now, targeting Eclipse 3.6 and an API freeze in 2 weeks. In my opinion, it is OK to have a source format with "type:=PSF" today but be flexible to use a more standardized URI format in the future, when there is API to register handlers appropriately. Could you summarize in just 5 lines what you think we need to do now? (In reply to comment #44) > Hm... thanks for the bug references, but I'm still not quite sure what you > expect us to do right now, targeting Eclipse 3.6 and an API freeze in 2 weeks. > > In my opinion, it is OK to have a source format with "type:=PSF" today but be > flexible to use a more standardized URI format in the future, when there is API > to register handlers appropriately. > > Could you summarize in just 5 lines what you think we need to do now? * Use the Maven SCM format for the URLs, i.e. scm:cvs:pserver: instead of just :pserver: * Use a mapping (can be internal API or hard-coded for the time being, which maps cvs-> org.eclipse.team.cvs.core.cvsnature, or whatever the name of the FetchFactory is) * Allow a default SCM with no other arguments to mean 'check out this directory and prompt to import all project(s) found' * Allow many repositories to be listed to cater for comment 39 None of these seem difficult to do; the first is just a bit of string manipulation (i.e. drop the prefix), and you can use the existing IFetchProvider extension point that CVS provides: http://help.eclipse.org/galileo/topic/org.eclipse.pde.doc.user/reference/extension-points/org_eclipse_pde_build_fetchFactories.html This defines <factory class="org.eclipse.pde.internal.build.fetch.CVSFetchTaskFactory" id="CVS"/> So just take what's between scm: and :, and look up an existing IFetchFactory extension point. If you can't find it, capitalise it, and repeat. That will permit the small-cased variants of 'cvs' to be used in the URL whilst using the same provider as currently exists in Eclipse. It will also permit later uses of others to be registered against fetch providers in a way that uses the existing API. Then, generate an equivalent PSF in-memory and hand that off to the fetch factory for initialisation. Created attachment 160150 [details] Provides import-from-Maven-SCM-URL functionality This is a proof-of-concept code (i.e., the error handling, UI polish and background threading are non existent) of a file importer that uses a Maven SCM URL. This is a bundle which contains both the classes and the source (and yes, I'm well aware of the irony...). If you import this into your workspace (either dropping in to the dropins/ folder, or by importing as source) then you'll get a File -> Import -> Team -> Project from Maven SCM URL. The URLs can be as follows: scm:cvs:pserver:dev.eclipse.org:/cvsroot/eclipse:org.eclipse.ant.optional.junit scm:cvs:pserver:anonymous@dev.eclipse.org:/cvsroot/eclipse:org.eclipse.ant.optional.junit scm:cvs:pserver:anonymous:@dev.eclipse.org:/cvsroot/eclipse:org.eclipse.ant.optional.junit scm:cvs:pserver:user:password@dev.eclipse.org:/cvsroot/eclipse:org.eclipse.ant.optional.junit It may well support other access (e.g. ssh) but I've not tried that. If you need credentials, and you pass in the top one, you'll get asked as per normal for the auth code. Note that the anonymous@dev isn't really a valid Maven URL; you'll see anonymous:@ instead, but I have to mangle this explicit case because Eclipse CVS doesn't support that. If you have Subclipse installed, then you can check out: scm:svn:http://svn.apache.org/repos/asf/felix/trunk/io scm:svn:http://svn.apache.org/repos/asf/felix/trunk/http.jetty ... The purpose of this proof of concept is to show that it's possible to generate a PSF file on the fly from a Maven SCM URL and use that to import a project. This code only does one at a time, but the goal would be for the Manifest header to support a comma-separated list of repos (addressing earlier concerns) and for you to specify a subset of projects from a large repo (rather than defaulting to everything). Lastly, some mangling needs to be done to parse out a tag from a CVS other than assuming HEAD for Eclipse's use cases (and others); I envisage that being done by an OSGi header on top of the URL, like: Eclipse-SCMURL: scm:cvs:pserver:dev.eclipse.org:/cvsroot/eclipse:org.eclipse.ant.optional.junit;tag:=ECLIPSE35 The project name defaults to the last segment in the module, but one could imagine overriding that with: Eclipse-SCMURL: scm:cvs:pserver:dev.eclipse.org:/cvsroot/eclipse:org.eclipse.ant.optional.junit;tag:=ECLIPSE35;project:=JUnit since the 'project' is specific to Eclipse, and the 'tag' isn't supported by the Maven SCM URL format directly. An alternative would be to use URL parameters for the values, e.g.: Eclipse-SCMURL: scm:cvs:pserver:dev.eclipse.org:/cvsroot/eclipse:org.eclipse.ant.optional.junit?tag=ECLIPSE35&project=JUnit but I think for 'tag', it's really a property of the URL, whilst the project is a property of the Eclipse-SCMURL manifest entry. +1 for use of SCMURL format Here are my observations/opinions: (1) PDE is not in the business of understanding the source/project reference URLs/tags formats and creating equivalent project set references from them. That is up to the fectch factories and team providers - project set references were designed to have a provider specific format. It would be a bad design to have PDE responsible for generating them. We need a design where PDE can take a bundle header and pass it off to a team provider/extension to do the right thing. PDE is just the middle man (person). (2) I like the fact that SCMURLs already have a well specified format. It would be nice to leverage this. However, it's unfortunate that they don't provide tag/version infomration. Referring to HEAD has limited use when it comes to the various development scenarios, branches, manintenance streams of Eclipse. We *must* have the ability to specify tags in the header. (3) If we need to add extra information to the SCMURL (tags, project names), then how useful will these headers really be to other tools? Will other tools be able to use them if we're not following the public spec? The proposal to add tag/project attributes to the SCMURL polutes/modifies the SCMURL format - and provides what is alread present in the project set format. If SCMURLs are enhanced to provide tag information, they could be added as another "type", or as a completely new header (or they could be used as it to get stuff from HEAD). To implement the SCMURL properly, we'd need a new extension point (in team), that provides parsers for the differnt SCMURL's, to allow PDE to delegate to the the appropriate handler (i.e. parse the URL into an equivalent .psf). Becauase the SCMURL format does not contain all the required information, and because we don't have time to add new API/extensions in team in the 3.6 timeframe, and because we can solve a real problem for Eclipse developers, I propose to proceed with the implementation described in comment#30. (In reply to comment #48) Thanks Darin - I basically agree though I'd like to add some observations: (1) To me, the PSF-versus-URI discussion is mostly about syntactic sugar. Would it make people happy if we'd encode the PSF String + provider / project bits as an URI? That might actually be fairly easy, using the URI "query" attributes: scm:providerid:(URI-encoded-psf-string)?tag=v12&project=ab this would leverage the common URI format for parsing while still being able to convey the information we need. (2) The main problem with the PSF format today is that it's provider specific. That's a real problem but should be considered indpendent of what we are trying to accomplish here. > To implement the SCMURL properly, we'd need a new extension point (3) Actually I think that an extension of the existing API would be better, i.e. have every repository provider implement a #fromURI(URI scmuri) method. As an intermediate hack, I'm wondering whether the old, deprecated team.core.projectSets extension point could help being the middle man: - Register a projectSets extension for id = "svn" which knows both the subversive and subclipse psf formats, performs the String conversion and delegates to the proper provider Finally, I still do not understand why people are so excited about the format we choose for the "Eclipse-SourceReferences" bundle entry. What is the problem with choosing some eclipse-internal format because it's easiest for us to process automatically today... when we make the door wide open for adding any other formats in the future? In other words: What, specifically, are we breaking today that we can't fix later, if we proceed with the approach from comment 30? (In reply to comment #49) > In other words: What, specifically, are we breaking today that we can't fix > later, if we proceed with the approach from comment 30? What is the point of creating something new when projects like Maven have something that already works? To solve the argument "there is no version information": The version information is a must-attribute of the pom.xml in which you also find the SCM URL, so Maven has no need to duplicate this. On top of that, CVS and SVN projects have no standard how to tag a version (CVS can't cope with some characters in a tag, SVN has no tags at all but relies on an arbitrary sub path for tagging). So I'm not sure whether it's really smart to put the version information into the SCM URL when a user can easily select the desired version from the Team Share dialog. (In reply to comment #50) > So I'm not sure whether it's really smart to put the version information into > the SCM URL when a user can easily select the desired version from the Team > Share dialog. It's important because it identifies the version of the source associated with the bundle. This is not something the user should have to choose - it's known by build and is important in terms identifying the correct source. (In reply to comment #49) > (In reply to comment #48) > (1) To me, the PSF-versus-URI discussion is mostly about syntactic sugar. > Would it make people happy if we'd encode the PSF String + provider / > project bits as an URI? That might actually be fairly easy, using the > URI "query" attributes: > scm:providerid:(URI-encoded-psf-string)?tag=v12&project=ab > this would leverage the common URI format for parsing while still > being able to convey the information we need. Or could we just add "psf" as a new scm-provider? scm:psf:eclipse-provider-id:reference-string (In reply to comment #48) > Here are my observations/opinions: > > (1) PDE is not in the business of understanding the source/project reference > URLs/tags formats and creating equivalent project set references from them. Where the code resides should not be a reason to not do it in a forward-thinking way. I wouldn't think it would be part of PDE's job anyway - it should really live in Team Core (that way, people can invoke it from a headless console). Note that PDE would have to have knowledge of Team's internals to execute the code anyway, so it's not like this would be adding a different dependency; and it could/should be an internal class at this stage. > We need a design where PDE can take a bundle header and pass it off to a team > provider/extension to do the right thing. PDE is just the middle man (person). Agreed, but we know that that isn't going to happen in the timeframe for 3.6. However, that doesn't prevent us from coming up with an embedded solution that works for a known format now, and then introduce a 'proper' way of doing it in the future with the same format. Bundles are long-lasting. Once we start going down a PSF specific format, they're going to be embedded for ever. Look at org.eclipse.osgi.util_3.2.100 - that's pretty much the same as it was in Eclipse 3.3. The information in the header is also going to be long-lasting as well; so if we get it right now, then newer versions of Eclipse can understand more providers but still allow to parse older ones without needing to use a different URL format. What if ECF Identity never changes from hereon? Would we want to put out a new bugfix/micro version just to change the SCM URL down the line? > (2) I like the fact that SCMURLs already have a well specified format. It would > be nice to leverage this. However, it's unfortunate that they don't provide > tag/version information. My proposal was not to add tag info to the SCM URL, but rather as a property of the Eclipse-SourceReference manifest. For example: Eclipse-SourceReference: scm:cvs:....;tag:=V201002251234 That way, the meaning of the URL is unchanged, but the ;tag is part of the Manifest entry. In the situation where the tag was missing, for the cvs provider, it would default to HEAD, though as you point out, it makes sense for this to be filled with the CVS tag used on binary deliveries. Does that solve the problem you were concerned about? Just to make it clear, the Eclipse-SourceReference: is a Manifest header, which contains a SCMURL: Eclipse-SourceReference: SCMURL(;tag:=...)(;project:=)... [, SCMURL(;tag:=...)?(;project:=...)?]* It's not the SCMURL's parsers job to understand the manifest entry; we split it up in Eclipse (to permit (a) tags, (b) project names if they are different from folder, and (c) multiple repos in the 'tainted source' example) > (3) If we need to add extra information to the SCMURL (tags, project names), > then how useful will these headers really be to other tools? Will other tools > be able to use them if we're not following the public spec? The SCMURL is a part of, not the entirety, of the Eclipse-SourceReference. This has two important benefits: * The URL is the same for everyone, once it's extracted from the manifest, and * The same URL format can be used to *generate* it, particularly for those (open-source) projects who default to HEAD. Like UTF-8, it's backward compatible with a plain SCMURL on its own, but it supports additional constructs as well. > Becauase the SCMURL format does not contain all the required information, and > because we don't have time to add new API/extensions in team in the 3.6 > timeframe, and because we can solve a real problem for Eclipse developers, I > propose to proceed with the implementation described in comment#30. 1) You don't need to pollute the SCMURL format. I hope we've cleared that up. 2) You don't need to add new API/extensions. The code I attached in the bug shows you that it's possible now to import from a SCMURL without needing extensions. In any case, should we go down the PSF route now and the Maven one later, we still need to revisit the hard-coded-solution in favour of a more open/extension based mechanism anyway; so whether we're hardcoding CVS or not in either solution doesn't matter. Lastly, the SCMURL bundle I attached has the ability to import both CVS and SVN folders, which is going to cover the majority of Eclipse (and other Apache-type) projects. Further, it will be simple to get a field generated automatically from the Apache bundle projects which we'll then retroactively be able to take advantage of in Eclipse after its release. (In reply to comment #50) > (In reply to comment #49) > > > In other words: What, specifically, are we breaking today that we can't fix > > later, if we proceed with the approach from comment 30? > > What is the point of creating something new when projects like Maven have > something that already works? And not only that, attached source code and an executable plugin which can be downloaded into your Eclipse workspace *right now* that works? osgi> install https://bugs.eclipse.org/bugs/attachment.cgi?id=160150 You now have functionality to add in a project from a Maven SCMURL as per the File->Import dialog. Obviously from here to importing via the PDE extension point is needed, as is parsing out e.g. the tag information as noted above, but that's trivial - and if needed, I'm happy to give that a go. > To solve the argument "there is no version information": The version > information is a must-attribute of the pom.xml in which you also find the SCM > URL, so Maven has no need to duplicate this. Right, we don't need it in the Maven URL but we do need it in the EclipseSource header. > So I'm not sure whether it's really smart to put the version information into > the SCM URL when a user can easily select the desired version from the Team > Share dialog. We want to be able to bind a binary bundle to a specific version; but this is easily doable without changing the URL semantics. (In reply to comment #49) > (In reply to comment #48) > Thanks Darin - I basically agree though I'd like to add some observations: > > (1) To me, the PSF-versus-URI discussion is mostly about syntactic sugar. > Would it make people happy if we'd encode the PSF String + provider / > project bits as an URI? Maven's SCM format is specifically set up to allow this, so yes, having scm:psf:.... would follow the same form, and at some point later, permit people to move between other scm: providers. If we absolutely positively have to do this then using this approach is probably the right way to go. However, I'd argue that it's not; for the majority of projects, you'd be able to encode the information just as easily in a standard maven scm: url. > (2) The main problem with the PSF format today is that it's provider specific. > That's a real problem but should be considered indpendent of what we are > trying to accomplish here. Agreed. This is just as much about avoiding encoding Eclipse class names into each and every source bundle reference (i.e. using 'cvs' instead of 'org.eclipse.team.cvs.cvsnature' (or equivalent)). Using 'cvs' allows us to map this to another implementation/provider at a future stage. Using the class name does not. Furthermore, using 'cvs' then lends it self well for Eclipse 3.7 or E4 to use a newer function-point or service-based query to lookup a provider based on CVS, which is then responsible for parsing it in an independent way later. Ugliness is all but guaranteed in the 3.6 timeframe; we know we can't do it properly. Any solution is going to be hardcoded in some way. For example, the proposed implementation of the PSF-specific one sticks '1.0' on the front, which then fails if you check out with subversion. > > To implement the SCMURL properly, we'd need a new extension point To implement *Anything* properly, we'd need a new extension point. However, don't confuse a choice between a hacky implementation that uses a proprietary format versus a hacky implementation that uses an open and pre-existing format. > (3) Actually I think that an extension of the existing API would be better, > i.e. have every repository provider implement a #fromURI(URI scmuri) > method. As an intermediate hack, I'm wondering whether the old, > deprecated team.core.projectSets extension point could help being the > middle man: > - Register a projectSets extension for id = "svn" which knows both the > subversive and subclipse psf formats, performs the String conversion > and delegates to the proper provider That would also work. You will note that I use the core.team.repository extension point in my implementation rather than the projectSets format though. > Finally, I still do not understand why people are so excited about the format > we choose for the "Eclipse-SourceReferences" bundle entry. What is the problem > with choosing some eclipse-internal format because it's easiest for us to > process automatically today... when we make the door wide open for adding any > other formats in the future? Because bundles have a habit of lasting a long time, especially if they're bug free. Any approach taken in the 0.5 version of this implementation is still going to be around for a year or more to come. Yet we already have a de-facto standard that is not only usable, has been demonstrated to be implemented. Secondly, if Eclipse supports this format, then it will be easier to convince the maven-bundle-plugin folks to add the SCMURL format - without any hacking into a PSF-like-format - straight into their generated bundles. That's going to make it easier for any Eclipse hacker. Debugging into Ant, or JUnit, or ASM, or ... - it will have the potential to have this in if it's an open format rather than a closed one. > In other words: What, specifically, are we breaking today that we can't fix > later, if we proceed with the approach from comment 30? It's not breaking anything, it's guaranteeing that we're going to get non-standard data in bundles for at least the next year or two. Turning the question around, what specifically are we breaking today that we can't fix later, if we use the Maven SCM URL? It'd been demonstrably implemented - you've checked out the implementation, right? The implementation might be less generic now (assuming the psf API extension doesn't pan out) but the implementation can be improved in the future. However, the format will last a long time to come. Alex (In reply to comment #55) > To implement *Anything* properly, we'd need a new extension point. However, > don't confuse a choice between a hacky implementation that uses a proprietary > format versus a hacky implementation that uses an open and pre-existing format. Not true. With the original PSF format proposal, all required extension points exist: the fetch factory geneates the memento at build, and the team provider extension creates a project from the memento in the IDE. No other extension points are required - it's a completely extensible end-to-end solution. > Turning the question around, what specifically are we breaking today that we > can't fix later, if we use the Maven SCM URL? It'd been demonstrably > implemented - you've checked out the implementation, right? Using the SCM format the 3.6 implementation means it is not extensible - it is limited to PDE implementation of CVS and SVN. That means if anyone else (fetch factory/team provider pair) wants to make this work for their repository, they can't. This could be alleviated by having PDE provide an extension point to contribute some sort of Eclipse-SourceReference handler. A handler would return a PSF memento for a given header. I'd like to ask everyone here to take a small step back. As a user of Eclipse, this discussion has lost me. It's over my head. I can't follow what you're talking about and why this all has to be so damn complicated. To illustrate my point: Maven will create a source bundle when told so (it's just four lines in the pom.xml). The source bundle is installed along with the rest in the Maven repository. Example: http://repo1.maven.org/maven2/commons-lang/commons-lang/2.5/ As you can see, there is also javadocs and checksums. So when I need the source for something built with Maven, I get it out of the repository. Pro: It works. Since the source is packaged at the same time as the code, the source matches the code 100% all of the time. To break this is hard. With version tags, all it takes is a mistake in the Team menu, one careless check in. CVS servers can be moved, hard disks can die, Armageddon can get in the way. p2 has been released three years ago and it's still not on par with Maven. Maven is stable and reliable while p2 is ... well ... not. Do don't do as the p2 guys do and don't over-engineer this, okay? As a user, I'd prefer something that is simple and works over something that uses EBNF. Con: A Maven source bundle is just an archive of the sources, not the whole project. So it's nice as reference material but not so nice if you need to build the project. Maven has an assembly plugin which can archive the whole project but it's uncommon to upload that to the repository. Also, setup of the assembly plugin is a bit harder (you need 6-12 lines of code because you have to configure it). I mention that to be fair; I don't think that it's too hard to add all the necessary project files to the source bundle. But again: If SCM URLs turn out to be too hard and would only work for Eclipse projects anyway because everyone else uses a different SCM repository layout/tagging scheme/whatever, why not simply teach PDE to create "source project bundles" which contain everything we need to compile and supply them next to the compiled plugin. I'm sure the p2 guys will iron out the problems in their dependency resolver within the next 2-3 years and in the mean time, we can just download the source projects just like we download plugins (Google, and then install them from the local filesystem). (In reply to comment #57) PDE already does source bundles, this bug is not about that. The Eclipse Platform releng process also produces archived projects that can be built. This particular bug is to have PDE/Build insert a header into the manifest of a binary bundle to enable a simple checkout from repository for the user to start making changes for a patch. This bug was not even about the format of that header, bug 195729 is more properly the place for that discussion, though it is too late to move there now. I have released my changes but won't mark this fixed because of the discussion regarding the format. To summarize the changes: 1) New manifest header "Eclipse-SourceReferences". Format is opaque, which the exception that is should follow the osgi common header syntax, we are parsing it using org.eclipse.osgi.util.ManifestElement to figure out how many entries there are in the header. 2) new Constants#KEY_SOURCE_REFERENCES which is a key an IFetchFactory can place in its entry info map indicating a reference to be placed in the manifest. 3) set "generateSourceReferences" in the build configuration to turn this on. - Header will be inserted into manifests that don't already contain it - if the manifest contains the header, an entry of "${PDE_SOURCE_REF}" will be replaced, otherwise the header is left untouched. All that remains for build is our side of bug 195729, having our CVSFetchFactory implementation generate the header based on the map file. (In reply to comment #56) > Not true. With the original PSF format proposal, all required extension points > exist: the fetch factory geneates the memento at build, and the team provider > extension creates a project from the memento in the IDE. No other extension > points are required - it's a completely extensible end-to-end solution. But it also hard-codes the name of an Eclipse class, which makes future refactoring much more difficult than it needs to be. Also, you miss out that the code posted was just (a) a demonstration, and (b) was in fact extensible to allow other types as well; the code looked for an implementation of the Team repository that was *.(provider)nature, so it found *.cvsnature and *.svnnature automatically. A provider that registered a *.foonature as a team repository would be picked up without changes. Note that the auto-appending of "1.0," onto the front of a Subclipse-generated PSF-type format actually breaks the import. > Using the SCM format the 3.6 implementation means it is not extensible - it is > limited to PDE implementation of CVS and SVN. That means if anyone else (fetch > factory/team provider pair) wants to make this work for their repository, they > can't. Why would you think that? I'm suggesting changing: Eclipse-SourceReference: 1.0,:pserver:...;type:=psf into Eclipse-SourceReference: scm:psf:1.0,:pserver:... which also permits Eclipse-SourceReference: scm:cvs:pserver:... Eclipse-SourceReference: scm:svn:http://... Eclipse-SourceReference: scm:foo:... > This could be alleviated by having PDE provide an extension point to contribute > some sort of Eclipse-SourceReference handler. A handler would return a PSF > memento for a given header. That was the point of my 'proper API' comment above. However, as I've shown in my demo code, it can be done both easily and also in a way that permits external extensions. For the record, here's the source that identifies which provider to use from the content after the scm: --- 8< --- repo = "scm:cvs:pserver:..." if (!repo.startsWith("scm:")) return true; int colon = repo.indexOf(':', 4); String provider = repo.substring(4, colon); IFetchFactory fetchFactory = getFetchFactory(provider); private IFetchFactory getFetchFactory(String provider) { IFetchFactory factory = null; IExtensionRegistry reg = Platform.getExtensionRegistry(); IExtensionPoint point = reg .getExtensionPoint("org.eclipse.pde.build.fetchFactories"); for (int times = 0; times <= 1 && factory == null; times++) for (IExtension x : point.getExtensions()) { IConfigurationElement[] elements = x.getConfigurationElements(); for (int i = 0; i < elements.length && factory == null; i++) { IConfigurationElement element = elements[i]; String id = element.getAttribute("id"); if (times == 0 && provider.equals(id) || times == 1 && provider.equalsIgnoreCase(id)) try { factory = (IFetchFactory) element .createExecutableExtension("class"); } catch (CoreException e) { } } } return factory; } --- 8< --- Note that this is looking for the fetch factory's ID (first in the same case, then in the upper case, to match currently existing providers) and does not require any additional API hooks to give you the factory, from which, we can use the same approach as Andrew in order to parse the remainder of the URL. Lastly, note that the version parameter can be added as an OSGi attribute/directive, outside of the SCMURL itself: Eclipse-SourceReference: scm:cvs:pserver:...;tag:=v20100226 This does not break the SCMURL in any way (it's an attribute of the Eclipse-SourceReference, not part of the SCMURL itself) but there's a key difference with using a type:= or nature:= driven approach; any Maven SCMURL is automatically a valid Eclipse-SoruceReference. I still think inventing an scm:psf: is unnecessary, since all the information can be captured in an scm:cvs: or scm:svn: URL, along with supporting OSGi attributes/directives on the header. I checked out the PDE build code; I couldn't see some of Andrew's patch above. I'm not sure if Andrew released a partial version of this. (However, there was a typo in the JavaDoc of Eclipse-SourceRefernce, in case it does get pushed.) FYI, see bug 195729 where I've included a PDE core/UI patch for extensible importing of repository projects based on bundle manifests in the file system. The PDE core/ui patch uses "scm:cvs" style URLs with optional "tag" and "project" attributes. Others could provide extensions to handle other repositories. I will provide a patch for PDE build to create scm:cvs: style URLs. Created attachment 160528 [details]
patch
Modifies the CVS fetch factory to create SCMURLs (needs review & testing).
(In reply to comment #61) > Created an attachment (id=160528) [details] > patch > > Modifies the CVS fetch factory to create SCMURLs (needs review & testing). I've eyeballed it briefly and it looks good (though not tested myself). I note that the SCMURL can contain a userid (and optional password), for example: scm:cvs:pserver:dev.eclipse.org:/cvsroot/eclipse:org.eclipse.ant.optional.junit scm:cvs:pserver:alex@dev.eclipse.org:/cvsroot/eclipse:org.eclipse.ant.optional.junit scm:cvs:pserver:user:pass@dev.eclipse.org:/cvsroot/eclipse:org.eclipse.ant.optional.junit If you want to explicitly encode 'empty password' rather than 'prompt for password', it is left as a blank: scm:cvs:pserver:anonymous:@dev.eclipse.org:/cvsroot/eclipse:org.eclipse.ant.optional.junit I note that the generation of the SCM URL in attachment 160528 [details] explicitly strips out the username rather than putting that in the SCM URL itself. Is there a need to do that? Secondly, this always tags ;project= onto the end. We can probably infer a sensible default project name from the last segment of the modulename. So, for the PDE build, instead of: scm:cvs:pserver:anonymous:@dev.eclipse.org:/cvsroot/eclipse:pde/build/org.eclipse.pde.build;project=org.eclipse.pde.build we could have: scm:cvs:pserver:anonymous:@dev.eclipse.org:/cvsroot/eclipse:pde/build/org.eclipse.pde.build In this case, instead of tacking ;project=org.eclipse.pde.build on the end, we could do that if it were different from the last segment, e.g.: https://bugs.eclipse.org/bugs/attachment.cgi?id=160528&action=diff#src/org/eclipse/pde/internal/build/fetch/CVSFetchTaskFactory.java_sec4 250 buffer.append(module); 251 + String[] paths = module.split("/") //$NON-NLS-1$ + String defaultProjectName = paths[paths.length-1]; + if (!defaultProjectName.equals(projectName)) { 252 buffer.append(";project="); //$NON-NLS-1$ 253 buffer.append(projectName); + } We ideally need to do the reverse translation from the checkout (not all Maven SCM URLs will have a ;project= on them, so assuming it exists rather than inferring the default as above is necessary on the UI side as well) From this code: https://bugs.eclipse.org/bugs/attachment.cgi?id=160528&action=diff#src/org/eclipse/pde/build/internal/tests/FetchTests.java_sec3 90 assertEquals(sourceRefs.get("org.eclipse.cvs,0.0.0"), "scm:cvs:pserver:dev.eclipse.org:/cvsroot/eclipse:org.eclipse.sdk-feature/plugins/org.eclipse.cvs;project=org.eclipse.cvs,tag=v20090520"); That should be ";tag=", rather than ",tag=", shouldn't it? Also, as above, we should probably have test cases for the absence of a project/tag pair e.g.: assertEquals(sourceRefs.get("org.eclipse.cvs,0.0.0"), "scm:cvs:pserver:dev.eclipse.org:/cvsroot/eclipse:org.eclipse.sdk-feature/plugins/org.eclipse.cvs"); Finally, does it make sense to passthrough the userid and/or for PDE build put in an 'anonymous:@' to allow for checkouts without needing userid/password combinations? Otherwise we should get prompted each time for a userid/password combination. (In reply to comment #62) > Secondly, this always tags ;project= onto the end. We can probably infer a > sensible default project name from the last segment of the modulename. The project name may be important in order to avoid breaking up inter-project references. I find it a little dangerous to just "guess" the project name. Is there any technical reason why you dislike the ";project=" ? If the only reason is that it's not pretty, I'd rather keep it in there. > Not all Maven SCM URLs will have a ;project= on them Ok, this is a reason :) So the point is, we could always export the ;project=, but we still need to guess a suitable project name if it's not there. Taking the folder name is one option, trying to download a .project file from the repository and look in there would be another option. The CVS "Import As" wizard can do this today. > Also, as above, we should probably have test cases for the absence of a > project/tag pair e.g.: Makes sense in the light of above. > Finally, does it make sense to passthrough the userid and/or for PDE build put > in an 'anonymous:@' to allow for checkouts without needing userid/password > combinations? Otherwise we should get prompted each time for a userid/password > combination. It could be an option on export (properties file?) whether the userid/password should be exported or not, or whether it's mapped to a different it/password. This would be in line with previous comments that the login which builds things may be different than the login that it's built for (producer/consumer). (In reply to comment #63) > (In reply to comment #62) > > Secondly, this always tags ;project= onto the end. We can probably infer a > > sensible default project name from the last segment of the modulename. > > The project name may be important in order to avoid breaking up inter-project > references. I find it a little dangerous to just "guess" the project name. Is > there any technical reason why you dislike the ";project=" ? If the only reason > is that it's not pretty, I'd rather keep it in there. I have no objection to putting it there if it's required/needed. I don't think it's necessary if there's a known default. > > Not all Maven SCM URLs will have a ;project= on them > > Ok, this is a reason :) So the point is, we could always export the ;project=, > but we still need to guess a suitable project name if it's not there. Taking > the folder name is one option, trying to download a .project file from the > repository and look in there would be another option. The CVS "Import As" > wizard can do this today. The knowledge of the URL is all you have without credentials and can be parsed trivially. Furthermore, the standard layout follows this in most (all?) Eclipse projects. What I'd suggest is to make the export/import equal and opposite functionality. At the moment, if missing, the import uses the manifest bundle name (which is overly specific to OSGi). Yet at both we have the ability to use the last segment of the URL. We don't export the tag= when not needed; we should do the same for the project as well. Of course, when they're different, we must set the project. Finally, most OSGi values have defaults and the practice is not to state them if they are the default (e.g. ;bundle-version:=0.0.0). Why should we do the same with the project? > > > Finally, does it make sense to passthrough the userid and/or for PDE build put > > in an 'anonymous:@' to allow for checkouts without needing userid/password > > combinations? Otherwise we should get prompted each time for a userid/password > > combination. > > It could be an option on export (properties file?) whether the userid/password > should be exported or not, or whether it's mapped to a different it/password. > This would be in line with previous comments that the login which builds things > may be different than the login that it's built for (producer/consumer). (In reply to comment #64) > Finally, most OSGi values have defaults and the practice is not to state them > if they are the default (e.g. ;bundle-version:=0.0.0). Why should we do the > same with the project? I'm not against doing this, but the prerequisite is that there is a clear specification for what the default should be. I agree that "last segment of URL" is a simple and useful default, so I'm ok with specifying it as such. (In reply to comment #63) > > Finally, does it make sense to passthrough the userid and/or for PDE build put > > in an 'anonymous:@' to allow for checkouts without needing userid/password > > combinations? Otherwise we should get prompted each time for a userid/password > > combination. > It could be an option on export (properties file?) whether the userid/password > should be exported or not, or whether it's mapped to a different it/password. > This would be in line with previous comments that the login which builds things > may be different than the login that it's built for (producer/consumer). In general, bundles are built for the masses, so embedding user/password does not seem that useful (unless using 'anonymous'). Question is, do we need more API to specify whether to include this? Created attachment 160673 [details]
Use the default projectName as the last path segment in the URL
This calculates the default projectName as the last segment in the URL (which is a rule that can be applied to other providers as well, in the fullness of time; and matches what would happen in a dumb checkout too). I have attached a similar patch for the import functionality.
Created attachment 160676 [details]
patch
Removes CVS directives from SCMURLs which are no longer needed. The project name is only added if different from the last segment of the module name.
(In reply to comment #66) > In general, bundles are built for the masses, so embedding user/password does > not seem that useful (unless using 'anonymous'). Question is, do we need more > API to specify whether to include this? I was more wondering whether it would make sense to include a generic 'anonymous:@dev.eclipse.org' as part of the generation, to permit easier checkouts. This would only work for :pserver: type checkouts though. I'm OK with not doing this, but wanted to bring it up because (a) it might be warranted in some users, and (b) there's another colon in there (between the user and the password) which might throw the checking/parsing routines that are just strsplitting on : and looking for exact positioning. At the very least, it should be covered in a test case. (In reply to comment #68) > Created an attachment (id=160676) [details] > patch > > Removes CVS directives from SCMURLs which are no longer needed. The project > name is only added if different from the last segment of the module name. Yeah, I was thinking that ;-) (In reply to comment #68) > Created an attachment (id=160676) [details] > patch > > Removes CVS directives from SCMURLs which are no longer needed. The project > name is only added if different from the last segment of the module name. I released this patch. Darin is there anything else, or should I mark this fixed? Created attachment 160857 [details]
patch
Here's my final patch :-) It adds support for using the | separator vs : separator character based on what is present in the map file. As well, it adds quotes around the project name attribute in case of special characters/spaces.
I think we can call this fixed now. To keep things simple, let's not add the anonymous user for now. We could adjust that in the future or track it in a separate bug, if needed.
I'll keep the PDE UI bug open for 3.6 (M7), as we finish polishing the UI/import feature around this support.
Awsome. Thanks Darin! I can't wait trying out the feature. Is there a chance for Kim to run the M6 builds with this feature such that we'd get repository-annotated bundles for EclipseCon? Perhaps, if she can put it into the basebuilder, even inform cross-project such that many others can annotate their EclipseCon bundle builds too? (In reply to comment #72) > Is there a chance > for Kim to run the M6 builds with this feature such that we'd get > repository-annotated bundles for EclipseCon? That's up to Kim... but in general the builder is not updated until after the milestone to ensure stability during milestone week. I'd hate to break the build :-) last patch is released Kim, could you do a test build with this? Just to know whether there's any issue with any of the Platform+Equinox bundles... I'd love to use this for building by Helios+1 project, but a bit of testing would be good. (In reply to comment #71) > Here's my final patch :-) It adds support for using the | separator vs : > separator character based on what is present in the map file. As well, it adds > quotes around the project name attribute in case of special characters/spaces. Looks good to me! > I think we can call this fixed now. To keep things simple, let's not add the > anonymous user for now. We could adjust that in the future or track it in a > separate bug, if needed. Agreed, KISS is the way forward. > I'll keep the PDE UI bug open for 3.6 (M7), as we finish polishing the > UI/import feature around this support. Good stuff. (In reply to comment #75) > Kim, could you do a test build with this? Just to know whether there's any > issue with any of the Platform+Equinox bundles... I'd love to use this for > building by Helios+1 project, but a bit of testing would be good. Agreed, it would be great if we could do a test build with this enabled. I'm curious if there's any issues with the non-CVS checkouts (for those that use an SVN repository, for example). But it would be great if the bundles produced for the EclipseCon build had their source reference information in! (In reply to comment #76) > (In reply to comment #75) > > Kim, could you do a test build with this? Just to know whether there's any > > issue with any of the Platform+Equinox bundles... I'd love to use this for > > building by Helios+1 project, but a bit of testing would be good. > > Agreed, it would be great if we could do a test build with this enabled. I'm > curious if there's any issues with the non-CVS checkouts (for those that use an > SVN repository, for example). But it would be great if the bundles produced for > the EclipseCon build had their source reference information in! People using SVN repos in releng builds will first need for the SVN implementation of the IFetchFactory to start generating values for the source references. And I guess they also need to implement IBundleImporterDelegate for the UI extension point We should raise a bug on the svn guys with some guidance, I haven't followed the UI side close enough to do this myself. Perhaps a wiki would be useful. Please open a bug with platform releng for this request and I'll see what I can do. I have a lot of other bugs that I have to fix for M6 so I'm not promising anything. As Darin mentions, moving to a new builder does have risks. I usually test the test candidate milestone bundles on the Tuesday of milestone week to ensure that we can build from the milestone bundles. I had to try out new bundles for another bug. Here is test build running with I20100305-1011 bundles. https://build.eclipse.org/hudson/view/Eclipse%20and%20Equinox/job/eclipse-equinox-test/131/ The releng bug for testing this is bug 305157. *** Bug 195720 has been marked as a duplicate of this bug. *** Is there any downside to "everyone" using this? generateSourceReferences=true Any risk? Performance implications? Untested technology? Implications for adopting products? Mostly curious if WTP or Orbit should use? It's just adds another header in the manifest. That's all. I've had very positive feedback from people who have found it useful to load the source without digging around in map files etc. If you want your users to be able to find the source more easily that I would try it :-) (In reply to comment #82) > Implications for adopting products? I think it makes sense for the open source parts of a product to include the source headers. Obviously, the parts of a product that are proprietary will not provide source. It makes it simpler for developers and potential contributors to get the code. > Mostly curious if WTP or Orbit should use? I think it makes sense for WTP to use this, but not for Orbit - most bundles in Orbit (as far as I know), are just binary libaries that originate from some other source (apache, objectweb, etc). Orbit does not build/compile these bundles from source or intend developers to modify their source (Orbit just re-packages the binaries as OSGi bundles). However, an Orbit library could provide its own source header (SCMURL) that pointed back to the place that a developer would go to get/modify/patch source. Orbit would simply re-package that information - but that would be up to the project that produces the code. Thanks Kim, Darin. Is there a minimum "base builder" I'd need to use to have PDE Build generated and included them? I'm currently on v20100416. As for Orbit-like bundles .... that reminds me ... I assume PDE build will just ignore "binary" bundles automatically. No chance to break signed jars? Or will I have to manually provide and exclude list? :( Thanks again, David, the current version of builder I'm using is v20100427. The source reference bundles are in an earlier version, but I'm not sure of the exact tag. No, it won't change the binary bundles that you're consuming from Orbit. No exclude lists needed. (In reply to comment #77) > We should raise a bug on the svn guys with some guidance, I haven't followed > the UI side close enough to do this myself. Perhaps a wiki would be useful. I just filed bug 315011. Do you have any guidance for the SVN guys beyond the references I added to bug 315011? I'm afraid the comment in IBundleImporterDelegate might not encourage other SCM providers to start implementing at this stage? >I'm afraid the comment in IBundleImporterDelegate might not encourage
>other SCM providers to start implementing at this stage?
Darin, is this still experimental?
(In reply to comment #88) > >I'm afraid the comment in IBundleImporterDelegate might not encourage > >other SCM providers to start implementing at this stage? > Darin, is this still experimental? For 3.6, we did not publish a final API for the importing side of things. There is official API for generating the headers in PDE build. The feature was implemented late and we were not convinced that the API was in a good and final state. The feature is here to stay, the implementation may change slightly as we move to official API in a later release. However, we'll always try our best to maintain compatibility and assist those that need help migrating. FYI, here's a link to the relevant WIKI page: http://wiki.eclipse.org/PDE/UI/SourceReferences Hey, what if anything was the final conclusion on the shape of source references for SVN? I've been searching around and have yet to see any examples of this header pointing to an SVN repo. (In reply to comment #91) > [...] I've been searching around and have yet to see any > examples of this header pointing to an SVN repo. I supposed you haven't seen such an example, because - afaik - no svn plugin has implemented any support to actually use that header. I'd love to be corrected. Given that it was based around being compatible with Maven's SCM format, the SVN one is documented here: http://maven.apache.org/scm/subversion.html Specifically, here are the examples listed on that page: scm:svn:file:///svn/root/module scm:svn:file://localhost/path_to_repository scm:svn:file://my_server/path_to_repository scm:svn:http://svn.apache.org/svn/root/module scm:svn:https://username@svn.apache.org/svn/root/module scm:svn:https://username:password@svn.apache.org/svn/root/module However, I don't think either of the SVN providers do this yet. And, in case anyone asks, the same is true of Git as well http://maven.apache.org/scm/git.html scm:git:git://github.com/path_to_repository scm:git:http://github.com/path_to_repository scm:git:https://github.com/path_to_repository scm:git:ssh://github.com/path_to_repository scm:git:file://localhost/path_to_repository In the case of the SVN, we might want to consider appending ;repository=122345 and for Git appending ;id=a1b2c3d4 to uniquely identify the corresponding version as opposed to a more general one, much like we have ;tag= for CVS. |
This request is somehow cross-component and not PDE only, so I'm adding few people on CC for initial discussion. When getting a bug report that should be simple to fix, I'm often trying to find a way how the reporter could try fix the issue himself and come up with a patch. The two biggest obstacles explaining what he needs to do, are (1) How to obtain the correct sources, and (2) How to create a patch (preferably against HEAD). Given that the reporter apparently has some version of Eclipse up and running, he can easily import bundles from his running Eclipse into a workspace. If he has SDK installed, he can even import with source. While having the correct source for the actual runtime is great to debug, such imported bundles often don't build properly; plus, it's hard to find the correct bundle in the CVS or SVN Repository that matches the imported source. I propose that (1) Released binary bundles get Repository information added to their Manifest: Repository type, root, module and maptag -- basically exactly the information that currently is in the Mapfile. (2) Importing bundles into the Workspace gets an additional 4th Option, "Import with Source from Repository" which uses the known Repository type, root, module and maptag in order to import from that Repository. Now in most cases, it should be possible to build the bundle and much easier to create a patch. Plus, after patching against the released Maptag first, the submitter can even merge the patch onto HEAD himself just by doing Team > Switch to another branch or version. I'd suspect that this enhancement makes it easier to get community fixes for simple things, and take some unnecessary workload off committers. I'm actually assigning this request PDE-build first, assuming that once the Repository information is in the bundles, the 2nd step of leveraging it for the import could be a separate bug. If there is some OSGi / Equinox work needed to specify official markup for the Repository info in the Manifest, that might also be tracked with a separate depends-on bug.