Bugzilla – Full Text Bug Listing
| Summary: | Version/VersionRange classes need to handle new format | ||
|---|---|---|---|
| Product: | [Eclipse Project] Equinox | Reporter: | DJ Houghton <dj.houghton> |
| Component: | p2 | Assignee: | P2 Inbox <equinox.p2-inbox> |
| Status: | RESOLVED INVALID | QA Contact: | |
| Severity: | normal | ||
| Priority: | P3 | CC: | aniefer, david_williams, henrik.lindberg, irbull, thomas, tjwatson |
| Version: | 3.8.0 Juno | ||
| Target Milestone: | Juno M5 | ||
| Hardware: | PC | ||
| OS: | Mac OS X - Carbon (unsup.) | ||
| Whiteboard: | |||
|
Description
DJ Houghton
For details on the new version format for pre-release versions see RFC 175 section of the draft RFCs published by OSGi in september: http://www.osgi.org/download/osgi-early-draft-2011-09.pdf This document has some other rather stale RFCs but RFC 175 is pretty close to the final version as of today. Thomas/Henrik: We plan to discuss this on today's Equinox call if you would like to join in on the discussion. Sorry, I missed DJ's message (it was posted 8:18 pm in my timezone). Will there be any followup discussions? (In reply to comment #3) > Sorry, I missed DJ's message (it was posted 8:18 pm in my timezone). Will there > be any followup discussions? We figured as much. Sorry about that. We discussed this at the Equinox call today. DJ was going to do some initial investigation. I think it would be good to setup another call to discuss his findings soon, or we can discuss at the Equinox call next week. We are concerned that there is a lot of details we need to iron out to get pre-release OSGi versions to work in the omni version support in p2. At a minimum it seems the 4th segment of the OSGIVersion would need to be a comparable that understood if the qualifier is a normal release version or a pre-release version. If I understood it correctly (scanning the RFC) the change seems to that if separator between 3d and fourth segment is a '-' it is a prerelease such that:
a.b.c- is the smallest possible version (a prerelease and no qualifier)
a.b.c. is the smallest possible released version (unsure if '.' should/must be present)
a.b.c-[ANY] < abc.[ANY]
OmniVersion has the rule:
"If segments are of different type the rule MaxInteger > Integer > Comparable[] > MaxString > String > AbsoluteMin is used - the comparison is done and the version with the greater segment type is reported as greater."
Thus, if prerelease versions use a Comparable[] for the third segment, they would always be smaller than a release that uses a single integer in the third segment. e.g:
1.2.3.qualifier > 1.2.[3].qualifier
Question is if the omni format can express such a mapping ("is array if ends with -").
(In reply to comment #5) > Thus, if prerelease versions use a Comparable[] for the third segment, they > would always be smaller than a release that uses a single integer in the third > segment. e.g: > > 1.2.3.qualifier > 1.2.[3].qualifier > That was a really stupid suggestion since it would make 1.2.2 > 1.2.[3], which is wrong. Sorry However, since OSGi versions are stored as such (not in OmniVersion format IIRC), it is simply a matter of mapping to either a 5 segment version, or a 4 segment Omni Version where the last segment is an array. Another interesting aspect is what happens when a new OSGi version is handled by an old implementation. To make it work, the prerelease versions should be published in OmniVersion format (to make comparisons work by an older implementation). This will be difficult to get correct. Have been thinking about this for a bit, and here is an idea. We have one clear difficulty in the OmniVersion regarding parsing of the new pre-release format as the delimiters does not contribute any semantics in the result - they are simply delimiters. However, there is a bigger issue in that the new version format can not simply be stored as verbatim OSGi either as all old implementations of p2 reading newer repository content simply can not parse the pre-release versions! To solve this the repository format needs to be bumped with lots of consequences. If we instead solve this issue so that p2 publishes pre-release version strings as OmniVersions (instead of verbatim OSGi) the problem is now how to express a pre-release so it always compares smaller than any regular OSGi version. Well, that is not difficult: Regular OSGi N.N.N.S PreRelease OSGi N.N.N.-M.S But there is an edge case that may screw this up; I don't know how OSGi n.n.n is expressed, if it is N.N.N.-M, or N.N.N.'',-M. If it is the former there is a problem as that makes OSGi n.n.n < n.n.n-x and a small puppy will die. So, if that did not work, there no other option than adding the capabilities and backporting. At least if the requirement is to be 100% correct. One could also argue that old clients are not equipped to deal with pre-releases and they should never be exposed to such. As no expressed ranges were constructed with pre-releases in mind (see last section), they are likely to be bitten anyway! Worst case, if exposed, it is possible (by a user to modify their ranges using n.n.n to n.n.n.- to force a string compare. This may be a good enough solution in practice. If the puppy did not die (or the compromise is ok, and there are not other corner cases), there are still issues with the solution as the encoding can not correctly describe the bidirectional transformation using the old OmniVersion format. It will need to be just raw. For older version this is perhaps not a big deal (they can't handle the artifacts anyway), but it means that new clients are also presented with this more brutal form of version. This may be overcome (in a new implementation) by interpreting the raw pattern and constructing the format. (A bit magic though, but doable). I.e. when parsing raw N.N.N.-M.S the format is added, and when writing it out, the format is stripped. Irrespective of this, there are many other subtle problems. People that have specified ranges may now have them specified as including pre-releases. This seems to be by-design (I did not see any mention that it was possible to specify a pre-release free range) e.g. if range is 1.0.0. to 2.0.0 exclusive, and there are pre-releases of 1.9.9 but no release of 1.9.9 you may get the pre-release bits. But if this is by design we are fine (user would need to have an upper range of 1.9.9.ZZZZZZZZZZ... - ouch). If the intent is that a range must express if pre-releases are included in the range (at all), then we have other issues to deal with as well as versions then must be two dimensional. There are also a few other corner cases, such as how version ranges are specified. For example [L,R) actually gets parsed as [L-,R-) (see the attached doc for a more detailed explanation of why). Now, maybe we can assume that p2 (the publisher) always writes the fully qualified versions and the rest of p2 doesn't need to worry abou this, but I don't know if this is a valid assumption to make. There is also the problem of 'encoding' OSGi versions using some 'tricks' (like creating a 5 part version number), since OSGi and p2 serializations will now be different. That is, given a version V, what we write out and what OSGi writes won't be the same character for character representation. I don't know if this is a real problem (i.e. requirement), but I wanted to raise it. (In reply to comment #9) > For example [L,R) actually gets parsed as [L-,R-) (see the attached doc for a > more detailed explanation of why). > Yeah, this is a real bummer and AFAICT, it makes the new format impossible to handle with the current code base. It does not allow for interpreting versions differently when they occur in a range. We will need: Regular version parsing: L = L. Range parsing L = L- (In reply to comment #10) > (In reply to comment #9) > > For example [L,R) actually gets parsed as [L-,R-) (see the attached doc for a > > more detailed explanation of why). > > > Yeah, this is a real bummer and AFAICT, it makes the new format impossible to > handle with the current code base. It does not allow for interpreting versions > differently when they occur in a range. We will need: > > Regular version parsing: > > L = L. > > Range parsing > > L = L- To be more specific it all depends on the context: Regular stand-alone version used as a capability attribute: V => V. Versions in the context of Ranges: [L => [L- // All L versions are included (L => (L. // L versions with non-empty release qualifiers are included. R] => R.] // R versions with pre-release qualifiers are included. R) => R-) // No R versions are included The "So in summary" section of RFC 175 of the pdf http://www.osgi.org/download/osgi-early-draft-2011-09.pdf has some typos in the conversion. Here is the correct text from the RFC: [L,R) => [L-,R-) – All L versions are included. No R versions are included. This is the most common use of version ranges. [L,R] => [L-,R.] – All L versions are included. No R versions with non-empty release qualifiers are included. This is generally only used when L=R for exact versions. So, for the exact version of X, [X,X] => [X-,X.] includes all pre- release X qualifiers and the release qualifier of the empty string. (L,R) => (L.,R-) – L versions with non-empty release qualifiers are included. No R versions are included. (L,R] => (L.,R.] – L versions with non-empty release qualifiers are included. No R versions with non-empty release qualifiers are included. (In reply to comment #8) > Have been thinking about this for a bit, and here is an idea. Thanks Henrik! > If we instead solve this issue so that p2 publishes pre-release version strings > as OmniVersions (instead of verbatim OSGi) the problem is now how to express a > pre-release so it always compares smaller than any regular OSGi version. Well, > that is not difficult: > > Regular OSGi N.N.N.S > PreRelease OSGi N.N.N.-M.S > > But there is an edge case that may screw this up; I don't know how OSGi n.n.n > is expressed, if it is N.N.N.-M, or N.N.N.'',-M. If it is the former there is a > problem as that makes OSGi n.n.n < n.n.n-x and a small puppy will die. Poor puppy! I think I understand how the first suggestion (N.N.N.-M) would be a problem, but I am unsure how the second (N.N.N.'',-M) helps. The Minimum version is 0.0.0- which has an empty pre-release qualifier. And this must hold true: 0.0.0. > 0.0.0-X for all values of X. 0.0.0. has an empty release qualifier and a release qualifier must sort higher than all pre-release qualifiers. > > So, if that did not work, there no other option than adding the capabilities > and backporting. At least if the requirement is to be 100% correct. One could > also argue that old clients are not equipped to deal with pre-releases and they > should never be exposed to such. As no expressed ranges were constructed with > pre-releases in mind (see last section), they are likely to be bitten anyway! > Worst case, if exposed, it is possible (by a user to modify their ranges using > n.n.n to n.n.n.- to force a string compare. This may be a good enough solution > in practice. > > If the puppy did not die (or the compromise is ok, and there are not other > corner cases), there are still issues with the solution as the encoding can not > correctly describe the bidirectional transformation using the old OmniVersion > format. It will need to be just raw. For older version this is perhaps not a > big deal (they can't handle the artifacts anyway), but it means that new > clients are also presented with this more brutal form of version. This may be > overcome (in a new implementation) by interpreting the raw pattern and > constructing the format. (A bit magic though, but doable). I.e. when parsing > raw N.N.N.-M.S the format is added, and when writing it out, the format is > stripped. > > Irrespective of this, there are many other subtle problems. People that have > specified ranges may now have them specified as including pre-releases. This > seems to be by-design (I did not see any mention that it was possible to > specify a pre-release free range) e.g. if range is 1.0.0. to 2.0.0 exclusive, > and there are pre-releases of 1.9.9 but no release of 1.9.9 you may get the > pre-release bits. But if this is by design we are fine (user would need to have > an upper range of 1.9.9.ZZZZZZZZZZ... - ouch). This is by design, [1.0.0., 2.0.0) => [1.0.0., 2.0.0-) which means include all released versions of 1.0.0 and no versions of 2.0.0, but any version in between is included, including pre-release versions (e.g 1.0.1-x, 1.99.0-x etc). > > If the intent is that a range must express if pre-releases are included in the > range (at all), then we have other issues to deal with as well as versions then > must be two dimensional. Luckily this is not the case. (In reply to comment #12) > (In reply to comment #8) > > If the intent is that a range must express if pre-releases are included in the > > range (at all), then we have other issues to deal with as well as versions then > > must be two dimensional. > > Luckily this is not the case. Actually, that is really horrible. It means that it is impossible to specify a range where you are guaranteed to never include pre-releases. If a repository with pre-releases happens to be in sight, those will be included. The only way around it is to specify fully qualified lower/upper bounds where those point to known versions. THAT REALLY SUCKS, it is beyond useless! Thomas and I had a talk about this today, but have no suggestion to post yet. So, we talked about this again. We see no way to make an old version correctly compare prerelease versions. There will be cases where it gets it wrong. OTOH, it will never be able to do anything with pre-releases as the OSGi pre-release version format can not be parsed by the installed version of OSGi / PDE, etc. So, even if we managed to get the comparison correct, there is no way to guard against pre-releases. And it will blow up anyway if an attempt is made to install / use a pre-release. Therefore, we think that a solution that at least allows p2 to read the repository is good enough. People will have to take care not to expose old versions to pre-releases. We propose that: 1. pre-releases are published using raw N.N.N.-M.S 2. it may be possible to specify the format for this raw version in a backwards compatible way, if so the format will be included, otherwise it is just raw. 3. OmniVersion (new) will in addition parse release OSGi versions so that there is always a 4th S segment, and that it is set to empty string '' if missing. A new version will then always get the ordering correct, since the 4 segment is always compared against either a String (release), or -M (prerelease). All strings (including empty) is later than -M. N.N.N.-M.'' < N.N.N.-M.S < N.N.N.'' < N.N.N.S Which is exactly what is wanted. If for 2. above it is not possible to include the format (not backwards compatible), there will need to be special recognition in the new implementation to avoid passing a 5 segment raw to conversion to OSGi (which would not work). There would also need to be changes in the way how versions expressed in OSGi ranges is mapped to an OmniVersionRange depending on if it is constructed from two separate versions, or parsed from a string. Semantics needs to be specified (should range be adjusted if constructed out of two separate versions or not depending on include/exclude - etc). The "right way" is probable the way that causes the least changes in calling code :) (In reply to comment #14) > So, we talked about this again. > Thanks Henrik! I am still not liking the complexity of all this. Keep in mind I do not like the introduction of pre-release qualifiers in the OSGi version for the OSGi R5 specification either. I find all of this will only confuse existing users of OSGi and will provide very little value add. Therefore, I have opened an issue against the OSGi specification to get pre-release versions removed from the next version of the specification. I hope to know by next week if my efforts succeed. (In reply to comment #13) > > Actually, that is really horrible. It means that it is impossible to specify a > range where you are guaranteed to never include pre-releases. If a repository > with pre-releases happens to be in sight, those will be included. The only way > around it is to specify fully qualified lower/upper bounds where those point to > known versions. THAT REALLY SUCKS, it is beyond useless! "THAT REALLY SUCKS, it is beyond useless!" I agree! ;-) (In reply to comment #14) > So, we talked about this again. > > We see no way to make an old version correctly compare prerelease versions. > There will be cases where it gets it wrong. OTOH, it will never be able to do > anything with pre-releases as the OSGi pre-release version format can not be > parsed by the installed version of OSGi / PDE, etc. So, even if we managed to > get the comparison correct, there is no way to guard against pre-releases. And > it will blow up anyway if an attempt is made to install / use a pre-release. Except in the case where you are provisioning a new version of Eclipse / Equinox that supports pre-releases and a bundle that uses them. This would be the 'year-to-year' update scenario. > > Therefore, we think that a solution that at least allows p2 to read the > repository is good enough. People will have to take care not to expose old > versions to pre-releases. Although if p2 can't actually handle them, I think we will need to increment the metadata version and fail fast. Is there value in reading (parsing) the versions, but not processing them correctly? > > We propose that: > 1. pre-releases are published using raw N.N.N.-M.S > 2. it may be possible to specify the format for this raw version in a backwards > compatible way, if so the format will be included, otherwise it is just raw. > 3. OmniVersion (new) will in addition parse release OSGi versions so that there > is always a 4th S segment, and that it is set to empty string '' if missing. Can you give a concrete example of what this will look like? Is 3.8.1-SNAPSHOT.20120505-1234 a valid version number? or will the two qualifiers be mutually exclusive? (or does it not really matter in practice). Also, for #2 I'm not sure what you mean. I assume if we create a new version format (i.e. change the code and break forwards compatibility), anything is possible. That is, we could change the OSGi version class to handle both the new and old formats. Is that true, or is it even more complicated than that. (In reply to comment #16) > (In reply to comment #14) > > So, we talked about this again. > > > > We see no way to make an old version correctly compare prerelease versions. > > There will be cases where it gets it wrong. OTOH, it will never be able to do > > anything with pre-releases as the OSGi pre-release version format can not be > > parsed by the installed version of OSGi / PDE, etc. So, even if we managed to > > get the comparison correct, there is no way to guard against pre-releases. And > > it will blow up anyway if an attempt is made to install / use a pre-release. > > Except in the case where you are provisioning a new version of Eclipse / > Equinox that supports pre-releases and a bundle that uses them. This would be > the 'year-to-year' update scenario. > Not sure I understand. The problem as I see it is that even if p2 manages to do a reasonable job the installed OSGi version parser will not be able to deal with pre-releases in the installed meta data unless OSGi itself is also updated. > > > > Therefore, we think that a solution that at least allows p2 to read the > > repository is good enough. People will have to take care not to expose old > > versions to pre-releases. > Although if p2 can't actually handle them, I think we will need to increment > the metadata version and fail fast. Is there value in reading (parsing) the > versions, but not processing them correctly? > I can image there being value in being able to parse. If people are aware when publishing pre-releases that they are not handled 100% correct, they can take measures when making them available, such as requiring a newer OSGi version (I am even thinking that this is even enforced). Then, in practice, even if pre-releases could appear to be in range, they will not be selected. > > > > We propose that: > > 1. pre-releases are published using raw N.N.N.-M.S > > 2. it may be possible to specify the format for this raw version in a backwards > > compatible way, if so the format will be included, otherwise it is just raw. > > 3. OmniVersion (new) will in addition parse release OSGi versions so that there > > is always a 4th S segment, and that it is set to empty string '' if missing. > > Can you give a concrete example of what this will look like? Is > 3.8.1-SNAPSHOT.20120505-1234 a valid version number? or will the two > qualifiers be mutually exclusive? (or does it not really matter in practice). > 3.8.1-SNAPSHOT.blah.blah is a valid (new) OSGi pre-release, yes. Everything following '3.8.1' is the pre-release qualifier string, so would be encoded as 3.8.1.-M.'SNAPSHOT.blah.blah' (3 integers, minimum, string). Not sure what you meant by "mutually exclusive" here. The open issue is weather it is possible to include the format string (the omni version pattern that specifies the format and makes bi-directional transformation omni<->source possible). > Also, for #2 I'm not sure what you mean. I assume if we create a new version > format (i.e. change the code and break forwards compatibility), anything is > possible. That is, we could change the OSGi version class to handle both the > new and old formats. Is that true, or is it even more complicated than that. #2 is about the omni-version format string that is embedded in an omni version in string form. If the solution to the problem is to ignore backwards compatibility, then this is not an issue; old versions simply would not ever see the new OSGi versions types, and if some new format character/operator is required to handle the new OSGi version it is safe to introduce it. Hope that explains it. OTOH, bumping the repository version is perhaps not as bad as it sounds. Not sure what happens in practice - if such newer repositories are simply skipped if they are part of an composite update site, it may work just fine. I can imagine that publishing could bump the repository version if the new version format was used (and use the current format otherwise). (In reply to comment #15) > (In reply to comment #14) > > So, we talked about this again. > > > > Thanks Henrik! I am still not liking the complexity of all this. Keep in mind > I do not like the introduction of pre-release qualifiers in the OSGi version > for the OSGi R5 specification either. I find all of this will only confuse > existing users of OSGi and will provide very little value add. > > Therefore, I have opened an issue against the OSGi specification to get > pre-release versions removed from the next version of the specification. I > hope to know by next week if my efforts succeed. > +infinity for that If they still want pre-release support, one alternative is to allow an additional first character in the qualifier that is < any currently accepted character - e.g. '!'. If the spec then says, "Any qualifier starting with '!' is a prerelease" then almost the same is achieved as with the '-' separator. There are still backward compatibility issues as ranges written using old semantics would always include pre-releases. (Old OSGi version parsing would naturally choke on the '!' so pre-releases must always be published with a requirement on newer OSGi). There is still work to be done in p2 and in publishing. An even better solution is perhaps to use an available character. They could say that "A qualifier starting with '-' is a prerelease. In this case, the change is only in the specification! In any case, the proposal for pre-releases needs to go back to the drawing board due to the range issues. (In reply to comment #18) > (In reply to comment #15) > > (In reply to comment #14) > > > So, we talked about this again. > > > > > > > Thanks Henrik! I am still not liking the complexity of all this. Keep in mind > > I do not like the introduction of pre-release qualifiers in the OSGi version > > for the OSGi R5 specification either. I find all of this will only confuse > > existing users of OSGi and will provide very little value add. > > > > Therefore, I have opened an issue against the OSGi specification to get > > pre-release versions removed from the next version of the specification. I > > hope to know by next week if my efforts succeed. > > > +infinity for that > > If they still want pre-release support, one alternative is to allow an > additional first character in the qualifier that is < any currently accepted > character - e.g. '!'. If the spec then says, "Any qualifier starting with '!' > is a prerelease" then almost the same is achieved as with the '-' separator. > > There are still backward compatibility issues as ranges written using old > semantics would always include pre-releases. (Old OSGi version parsing would > naturally choke on the '!' so pre-releases must always be published with a > requirement on newer OSGi). > > There is still work to be done in p2 and in publishing. > > An even better solution is perhaps to use an available character. They could > say that "A qualifier starting with '-' is a prerelease. In this case, the > change is only in the specification! I don't think this works because 1.0.0."" < 1.0.0."-" which is not what the maven community wants. > > In any case, the proposal for pre-releases needs to go back to the drawing > board due to the range issues. I'm not sure there really is a good solution at this point in the specification's lifecycle. The notion of pre-release versions is adding a lot of complexity for little gain. We could handle it in p2, but it would be a lot of work to get it correct and I fear that other communities (products, implementations etc.) will also have a very hard time with this type of change. (In reply to comment #19) > > An even better solution is perhaps to use an available character. They could > > say that "A qualifier starting with '-' is a prerelease. In this case, the > > change is only in the specification! > > I don't think this works because 1.0.0."" < 1.0.0."-" which is not what the > maven community wants. > It would have to be expressed as (1.0.0.- ... to skip pre-releases. Irrespective how it is done, everyone must be aware of what will happen when old logic is exposed to pre-releases, and how pre-releases are included in ranges. > > > > In any case, the proposal for pre-releases needs to go back to the drawing > > board due to the range issues. > > I'm not sure there really is a good solution at this point in the > specification's lifecycle. The notion of pre-release versions is adding a lot > of complexity for little gain. We could handle it in p2, but it would be a lot > of work to get it correct and I fear that other communities (products, > implementations etc.) will also have a very hard time with this type of change. Agree. See bug369137. I am going to remove pre-release support from Equinox. This bug is no longer valid. |