Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 312976

Summary:	Aggregator should have COPY_BOTH option to copy artifacts, and their metadata
Product:	[Technology] CBI	Reporter:	David Williams <david_williams>
Component:	CBI p2 Repository Aggregator	Assignee:	CBI Inbox <cbi-inbox>
Status:	CLOSED WONTFIX	QA Contact:	David Williams <david_williams>
Severity:	enhancement
Priority:	P3	CC:	frederic.gurr, thomas
Version:	unspecified
Target Milestone:	---
Hardware:	PC
OS:	Windows 7
Whiteboard:

Description David Williams

2010-05-14 15:44:22 EDT

This is a continuation of the discussion in bug 312802.

In that bug, it was discovered some CRCs are wrong on /releases/helios.

The reason appears to be that some projects contribute unsigned jars, in pack200 form, that have not been conditioned. While there are several "project level" problems with this, it is also wrong for the aggregator/builder to use the wrong CRC.

I'm not sure if the bug is both buckyBuilder, and b3.aggregator, or only buckyBuidler (which is what produced current /releases/helios) ... but IMHO, it it should be fixed in both.

Also, this happens because we specify the unpack-as-sibling option, so the pack.gz file copied, then unpacked, but the crc metadata copied from original artifacts files, not recomputed. (Normally they match, but might not if a jar is not conditioned first ... not sure if there are other sources, e.g. if different VMs used?)

So, one fix is to re-compute the crc, maybe issue a warning if they do not match.

But, I think a better solution is to copy the original jar, and then also copy its meta data as we do now. Checking "integrity" could be a whole separate step, not done by the aggregator, per se.

The reason I think this is better is it makes less assumptions about what people should be doing, and does the copy/mirros/aggregation, etc.

While it is a good policy that jars and pack.gz files are binary equivalent, it is none the less a policy, and the aggregator should not build in these assumptions, if there's no reason to.

But, in either case, the CRC problem needs to be fixed.

Comment 1 David Williams

2010-05-17 10:25:43 EDT

Since bug 312802 was moved to b3.aggregator, this one is not required to track that specific problem with incorrect checksums.  

Instead, I'll transform it into a request to have COPY_BOTH options, instead of "UNPACK_SIBLING_AS_JAR". 

I think the copy-both is more conceptually correct and could avoid complications when the contributed pack.gz and jar are not perfectly equivalent.

Comment 2 Thomas Hallgren

2010-05-17 12:08:34 EDT

Not sure I agree with this. If we copy the packed version, and if we compute the checksum while we unpack it (bug 312976), then everything is OK. So copying both no longer bring any advantages once that bug has been fixed, instead it means that we don't "trust" the packed artifact. If we don't trust it, then we should avoid using it at all. Keep in mind that if we successfully copy and unpack the packed jar, then the packed jar is what p2 will use. Always.

We could of course copy both and then unpack too so that we can compare the unpacked versions and trap inconsistencies that are caused by unconditioned jars. But for unsigned and unconditioned jars, those inconsistencies are to be exptected and also OK.

Add that for a signed bundle the the aggregator already detects the inconsistency anyway. So singed + not conditioned will never be allowed.

Comment 3 David Williams

2010-05-17 12:44:04 EDT

(In reply to comment #2)
> Not sure I agree with this. If we copy the packed version, and if we compute
> the checksum while we unpack it (bug 312976), then everything is OK. So copying
> both no longer bring any advantages once that bug has been fixed, instead it
> means that we don't "trust" the packed artifact.

You mean bug 312802 I think. And, not sure I do trust them, completely, as bug 312802 reveals. They are not always perfectly equivalent ... and while we say they _should_ be, the reality is there is not guarantee all contributions will be ... and I just think the aggregator should limit itself on what it assumes, expects, and enforces. Let it aggregate. Let other things enforce. In this case, even once checksum is correct, the jar in helios central repo will be subtly different than the jar on the original site. 

Plus, p2 will use jar, not pack200 file, if someone happens to duplicate whole repo and uses locally.

Comment 4 David Williams

2016-09-16 15:42:34 EDT

[Bookkeeping change only. Moving bugs to the new "home" of aggregator, CBI.]

Comment 5 Frederic Gurr

2023-09-01 10:06:14 EDT

If this issue is still relevant, please move it to https://github.com/eclipse-cbi/p2repo-aggregator/issues.