Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 348341

Summary:	rewrite http:// to file:// to improve aggregation build performance on build machine
Product:	Community	Reporter:	David Williams <david_williams>
Component:	Cross-Project	Assignee:	David Williams <david_williams>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	P3	CC:	irbull, mknauer, sbouchet
Version:	unspecified
Target Milestone:	---
Hardware:	PC
OS:	Linux
Whiteboard:

Description David Williams

2011-06-05 15:08:17 EDT

I've already done this work, and some test builds, so I'm just documenting here for awareness, and discussion, in case there are unknown implications. 

Our b3aggrcon file all used "http://download.eclipse.org/..." as location of the project repositories. And while I'd think there'd be some slick ways that could automatically be made "local" under the covers, that seems not the case. But, since our repository locations are uniform, we can do the transformation during a build, after checking out the b3aggrcon files from cvs; changing all occurrences of 

http://download.eclipse.org/ 

to 

file:///home/data/httpd/download.eclipse.org/

The savings in time, from a simple test run, appears substantial, approximately 30%, so seems worth it. Run number 511 took 2 hours 48 minutes, while run number 512 (otherwise the same, except for the new rewrite task) took 1 hour 50 minutes. 

One implication is now the logs will say things like 

Loading repository file:///home/data/httpd/download.eclipse.org/... 

instead of 

Loading repository http://download.eclipse.org/...

... might be a little confusing to some, but easy to get used to do the "reverse" mapping, when reading. 

Another implication, if, for some reason, the "symbolic" path to downloads ever changes, then the build script needs to be updated. As far as I know, it has been 

/home/data/httpd/download.eclipse.org/ 

since forever and unlikely to change. As a far as I know.

Comment 1 David Williams

2011-06-05 15:12:07 EDT

I should say explicitly, this shouldn't effect anyone running b3 aggregator locally, since the build scripts only do the conversion if the 'rewriteRepositoryURLValue' property has been defined. 

And, it goes without saying, contributors must still use http://download.eclispe.org in their contributions (and their own repositories URLs) since in general, this needs to "work" as is, at locations other than build.eclipse.org.

Comment 2 David Williams

2011-06-05 15:15:55 EDT

Oh, one cautionary note, the script currently does a blind substitution of all occurrences of "http://download.eclipse.org". As far as I can tell, that is only used in repository location attributes ... but, if that changes in future and is used somewhere else for some other purpose, then the "replace" task may have to be made more sophisticated.

Comment 3 Markus Knauer

2011-06-07 11:12:57 EDT

I need to reopen this one, at least I think this change causes some problems in the compositeArtifacts.jar that can be found in /releases/staging at the moment.

2011-06-07 10:39 compositeArtifacts.jar:

<repository name='Indigo artifacts' type='org.eclipse.equinox.internal.p2.artifact.repository.CompositeArtifactRepository' version='1'>
...
  <children size='2'>
    <child location='file:///home/data/httpd/download.eclipse.org/eclipse/updates/3.7milestones/S-3.7RC4-201106030909'/>
    <child location='aggregate'/>
  </children>
</repository>

The first child location is just a local one and should be a http://... location.

Comment 4 Markus Knauer

2011-06-07 11:19:18 EDT

I fixed this manually on the file system... this is a one-time fix only and will be broken with the next update of /releases/staging

Old file as a backup:
2011-06-07 10:39 compositeArtifacts.jar.backup

New (manually fixed) file:
2011-06-07 11:16 compositeArtifacts.jar

Comment 5 David Williams

2011-06-07 12:16:46 EDT

Thanks, Markus. I knew it was too easy :) 

I've fixed this by specifically excluding ep.b3aggrcon and equinox.b3aggrcon files from being rewritten. So, their "trusted repository" URLs should stay as provided when written in compositeArtifacts. 

Kind of hard coded ... but pretty sure those will be our only trusted repos, for a while. Probably won't even lose that much in performance gains, since we don't mirror those anyway. 

A new "staging" won't be ready for several hours ... I'll be sure to confirm before promoting. 

Let me know if you see any other issues. 

To cross-reference, see also bug 347956 for some possibly improved future support, directly in b3 aggregator. 

Thanks again,