Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.
Bug 316401 - aggregator fails silently, in some cases
Summary: aggregator fails silently, in some cases
Status: RESOLVED WORKSFORME
Alias: None
Product: CBI
Classification: Technology
Component: CBI p2 Repository Aggregator (show other bugs)
Version: unspecified   Edit
Hardware: PC Windows 7
: P3 normal (vote)
Target Milestone: ---   Edit
Assignee: CBI Inbox CLA
QA Contact: David Williams CLA
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-06-09 20:24 EDT by David Williams CLA
Modified: 2022-06-25 11:22 EDT (History)
3 users (show)

See Also:


Attachments
make error messages visible when using console logging (683 bytes, patch)
2013-11-07 13:35 EST, Stephan Herrmann CLA
no flags Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description David Williams CLA 2010-06-09 20:24:04 EDT
This is a continuation, of sorts, of bug 316357. 

There the ptp contribution was not being mirrored, for some reason, but there was nothing in the log. The content metadata was apparently captured, and reused, but not the artifacts. 

One "special" thing about this contribution is that it specifies version="" in the .build file, instead of a specific version. (That is, "get latest"). 

I'm currently thinking it was failing under the covers somewhere, because of a missing prereq from CDT (see bug 316343). I'm not sure what else changed, but once that missing pre-req was added, it started mirroring the artifacts as usual. 

Problem is, there was nothing in logs to indicate nothing was mirrored, initially, and I'm wondering if that's because it specifies "get latest" ... perhaps a different code path than if version = "xxx" had been specified?
Comment 1 Filip Hrbek CLA 2010-06-10 04:17:10 EDT
I doubt that the version="" pattern caused any problems. This one is translated as a version range of "0.0.0" which means "any version" and the planner chooses the latest version that is compatible with all other requirements in the provisioning plan.

We discovered a different problem in the aggregator. If p2 metadata describes an artifact that is finally missing in the artifact repository, the artifact is silently skipped (unless you require maven result - in that case maven metadata generator complains about that missing artifact). This may happen if
a) the metadata and its respective artifact repository are inconsistent
b) the artifact repository changes after the metadata repository is fetched (it takes some time until aggregator gets to actual mirroring)

This problem has been discussed here:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=312656#c21

We have already fixed this in case that the artifacts are mirrored. It would be a good idea to check artifact and metadata repository consistency even for non-mirrored repositories (so called "trusted contributions") but it has not been implemented yet.

The recent fixes have not been published though. There are more minor changes than this and we don't want to break things incidentally at this late stage of Helios release. The very latest aggregator (containing all the fixes) is available here:

https://build.eclipse.org/hudson/view/Modeling/job/emft-b3-build/lastSuccessfulBuild/artifact/site/headless
Comment 2 Stephan Herrmann CLA 2013-11-07 08:54:16 EST
I have an aggregation involving kepler artefacts which works fine with the 4.2 version of b3 and silently fails with the 4.3 version, giving the infamous:
  "Not all artifacts could be mirrored, see log for details"
with nothing in the logs.

While I was hacking on b3 anyway, I gave it a try to put more into the logs right when MirrorGenerator.run() detects errors.size() > 0. Result: The list 'errors' contains a few empty strings, nothing more.

This seems to indicate that Builder.getExceptionMessages(e) can produce an empty string, for which I could not see any simple explanation.

I'll put in a bit more logging to see what really happens.


Is it only me having trouble to use the 4.3 aggregator?
Comment 3 Stephan Herrmann CLA 2013-11-07 09:03:38 EST
(In reply to Stephan Herrmann from comment #2)
> Is it only me having trouble to use the 4.3 aggregator?

xref: bug 419402
Comment 4 David Williams CLA 2013-11-07 11:19:52 EST
(In reply to Stephan Herrmann from comment #2)
> I have an aggregation involving kepler artefacts which works fine with the
> 4.2 version of b3 and silently fails with the 4.3 version, giving the
> infamous:
>   "Not all artifacts could be mirrored, see log for details"
> with nothing in the logs.
> 
> While I was hacking on b3 anyway, I gave it a try to put more into the logs
> right when MirrorGenerator.run() detects errors.size() > 0. Result: The list
> 'errors' contains a few empty strings, nothing more.
> 
> This seems to indicate that Builder.getExceptionMessages(e) can produce an
> empty string, for which I could not see any simple explanation.
> 
> I'll put in a bit more logging to see what really happens.
> 
> 
> Is it only me having trouble to use the 4.3 aggregator?

I've put some comments in bug 419402 comment 2 that might help? 

Is this for you making a contribution to Sim. Release ... or, using for something else? I ask since if "for something else", may have to point to specific SR1 repo ... sadly, sometimes our "SR1" and "SR0" are not always "compatible".
Comment 5 Stephan Herrmann CLA 2013-11-07 13:35:09 EST
Created attachment 237284 [details]
make error messages visible when using console logging

(In reply to Stephan Herrmann from comment #2)
> While I was hacking on b3 anyway, I gave it a try to put more into the logs
> [...]

Here's a trivial patch to ensure that errors are logged at the end of mirroring. This is relevant when running b3 on the command line which uses console logging. When the final error is displayed the root causes have typically long scrolled out of the console's buffer. With this patch they're simply repeated to make sure they're visible.

(My ponderings in comment 2 about empty error strings were bogus :-/ )
Comment 6 Stephan Herrmann CLA 2013-11-07 13:50:24 EST
(In reply to David Williams from comment #4)
> (In reply to Stephan Herrmann from comment #2)
> > Is it only me having trouble to use the 4.3 aggregator?
> 
> I've put some comments in bug 419402 comment 2 that might help? 

Yes it did, thanks.
 
> Is this for you making a contribution to Sim. Release ... or, using for
> something else? I ask since if "for something else", may have to point to
> specific SR1 repo ... sadly, sometimes our "SR1" and "SR0" are not always
> "compatible".

It's "for something else" (my day job). In fact I'm already using (a mirror of) SR1. So I can't tell if that was relevant or not :)
Comment 7 David Williams CLA 2016-09-16 15:43:35 EDT
[Bookkeeping change only. Moving bugs to the new "home" of aggregator, CBI.]
Comment 8 Ed Merks CLA 2022-06-25 11:22:48 EDT
This seems stale.  I'd need to be able to reproduce it...