Community
Participate
Working Groups
In talking over Eclipse TPTP's build process with some Intel build engineers, I again was faced with the enormity of the quantity of data we download and then upload that is really not absolutely necessary. I'd like to propose that perhaps our "2-stage" build become a "3-stage" one, whereby when we finish with our Intel architecture native builds, we upload our specific native packages, but all the resultant "all_platforms" packaging then takes place upstream. Right now I think we're doing an additional 170Meg for tptpdc, 130 for agntctrl, with lesser amounts (that still add up) for cbe and JVMTI all_platforms, not to mention the all_platforms.sdk packages as well.
Randy, can you explain what "all the resultant all_platforms packaging then takes place upstream" mean? Do you mean updating the all-platform zip files on the eclipse download server?
Right, what I'm saying is that all the {tptpdc|agntctrl|cbe|probekit|javaprofiler}.all_platform[.sdk] files are just rolled up versions of bigger trees containing exact replicas of what is in the smaller trees containing the individual packages. Do we really need to download that 150+MB of data and subsequently upload 350-400MB of data when that half-gig+ of data is redundant? Subsequent to my builds, an "all_platforms packing effort" could build those all_platforms files.
Randy, Why do you call the "all_platforms packing effort" the 3rd step? Are you just changing the way the all platform zip files are packaged (in the 2nd step)? I suppose the build is complete only after the all-platform zip files are all updated.
I call it the third step just as a proposal; that's the nature of the enhancement request. We have two steps now... one up at IBM Toronto (TPTP runtime build and some native), and one over here at Intel Oregon (remaining Intel architecture native builds)... I propose to add a third step back there whose sole purpose is to zip up those all_platforms files... but zip them up back at IBM Toronto to avoid submitting the same bits multiple times. I'd remove the creation and uploading of all_platforms zip files from *my* current build process, and you'd finish it there instead.
Randy, now I understand your proposal. (I didn't understand the 3rd step was to be done by me at the beginning.) Adding an extra step and hand shake between us may complicate the build, and make it more difficult to complete one build. I have a counter proposal. If the bottleneck is in transferring the big all-platform files, can you do the updating of the all-platform zip files on the eclipse download site? That is, you only download what's required to do your builds (the nativecode.zip) and create the AC and CBE packages for platforms that you are responsible for. Upload the individual platform zip files to eclipse, and you update the all-platforms zip files directly on eclispe download server. You can keep the script in your home directory. The script should be fairly straightforward. Do you like this approach?
Wow... I hadn't thought of that... I guess I just saw that box as a repository... I presume it's not unreasonable to do unzip/zips there script driven. Sounds like a good counter-proposal... so I guess just assign it to me then. Thanks for the suggestion!
Kiryl, another low priority, though even nicer to have than the last. Note it's an "enhancement request". Comment #5 below is most apropos... the best solution, probably, keeps the 2-stage approach. In this case, you wouldn't download BIG packages, overlay with our stuff, and upload. Instead, you'd build and upload the minimal necessary, and then run scripts at the upload site to repackage there. This will cut the bandwidth for the builds substantially!
Some work has been done in the context of bug #178141. Downstream build creates six agntctrl.* packages (for each supported platform), upload them to dev.eclipse.org. And then do remotely the following things: 1) updates them with some plug-ins taken from TPTP zip files 2) assemble tptpdc.all_platforms and agntctrl.all_platforms packages It save us from downloading big TPTP zip files to downstream build host and upload big *.all_platforms files.
Some more optimizations may be done. Mainly to prepare remained "all" packages remotely. To be reviewed on completion of bug 182085.
As of TPTP 4.6.0, TPTP is in maintenance mode and focusing on improving quality by resolving relevant defects and increasing test coverage through test creation, automation, Build Verification Tests (BVTs), and expanded run-time execution. As such, TPTP is not delivering enhancements. As part of the TPTP Bugzilla housecleaning process (see http://wiki.eclipse.org/Bugzilla_Housecleaning_Processes), this enhancement is resolved as WONTFIX. For this enhancement to be considered, please re-open with an attached patch including the Description Document (see http://www.eclipse.org/tptp/home/documents/process/development/description_documents.html), code (see http://www.eclipse.org/tptp/home/documents/resources/TPTPDevGuide.htm), and test cases (see http://www.eclipse.org/tptp/home/documents/process/TPTP_Testing_Strategy.html).
As of TPTP 4.6.0, TPTP is in maintenance mode and focusing on improving quality by resolving relevant enhancements/defects and increasing test coverage through test creation, automation, Build Verification Tests (BVTs), and expanded run-time execution. As part of the TPTP Bugzilla housecleaning process (see http://wiki.eclipse.org/Bugzilla_Housecleaning_Processes), this enhancement/defect is verified/closed by the Project Lead since this enhancement/defect has been resolved and unverified for more than 1 year and considered to be fixed. If this enhancement/defect is still unresolved and reproducible in the latest TPTP release (http://www.eclipse.org/tptp/home/downloads/), please re-open.
