Community
Participate
Working Groups
After all the parallel build goodness provided by galileo, assembly generation for various configs still happens serially. I investigated this a but and this seems to be because of the "main" target in generated files "package.org.eclipse.pde.build.container.feature.all.xml" and "assemble.org.eclipse.pde.build.container.feature.all.xml". I suppose these targets use the same file locations so that will need some parametrization as well.
Created attachment 141442 [details] Patch for the feature :) Hopefully this one works fine. I've created the patch against v20090527-1800 (eclipse 3.5)
Created attachment 141445 [details] updated patch A whole bunch of unit tests are failing because of this change. Could someone verify if I'm going in the right direction ? I also found that *some* tests are failing because the generated artifacts are generated in a tmp.ANY.ANY.ANY folder instead of a tmp folder. Do you think ANY.ANY.ANY is something that should be handled separately as empty. Any possibilities of some configuration like linux.gtk.* or linux.*.x86 ?
Step back, revert all code and the tests fail. What's happening did we just push out a build with tests broken ? Now I've no clue if the change I'm making is breaking any existing functionality or not, any answers ?
Andrew, can you look at this ;)?
Created attachment 141452 [details] Failing tests -- import in the junit view
My bad, did not notice that all the tests that were failing were failing at: File delta = Utils.findDeltaPack(); assertNotNull(delta); Installing the delta pack seems to resolve the issue so please ignore my earlier comments. I'll also file a patch for a more meaningful error message :)
Created attachment 141459 [details] updated patch updated patch with fixes for some of the failing tests. I still have about 18 errors and 3 failures (some because of missing Repo2RunnableTask) others caused because of folder renames for handling parallel tasks. Please have a look to see if I'm even going in the right direction in the first place.
Ping...
Created attachment 142938 [details] updated patch A change like this deserves a property to turn it on & off. We should also re-use the "parallelThreadCount" and "parallelThreadsPerProcessor" properties here. Ant has a race condition in the <mkdir> task, there is also a race in File#mkdirs in some vms. See bug 265654. Test failures caused by this look like: "Directory C:\Dev\Platform\Workspaces\junit-workspace\pde.build\284721\I.TestBuild creation was not successful for an unknown reason." The particular <mkdir>'s that we will need to do something about are printed by AssembleConfigScriptGenerator.generateInitializationSteps. The archive parent folder is likely to be common across configs and is the one I saw fail. We can create some of the common folders first in the assemble all script. The p2 publisher support has a "p2" configuration step which needs to finish before the rest of the assemble tasks can run. I moved the parallel to be after the p2 task. This was the cause of a lot of the failing PublishingTests The update patch removes the whitespace changes, moves the parallel task after the p2 config, and updates the PublishingTests to account for path changes. Right now I am seeing 1-2 failing tests that are caused by the ant mkdir race condition.
Also, before releasing, I would like to convince myself that 3.4 style p2 metadata generation is ok in parallel (the P2Tests pass which is at least a good sign). This is because of the static GeneratorResult used by IncrementalGenerator. Consider using a temp folder of "tmp/win32.win32.x86/" instead of "tmp.win32.win32.x86" so that everything is nicely contained under "tmp". We should also look at bug 143243 here. We will want to test the SDK releng build using this. Also perhaps e4.
What's your timeline? M1 or M2?
(In reply to comment #9) > A change like this deserves a property to turn it on & off. We should also > re-use the "parallelThreadCount" and "parallelThreadsPerProcessor" properties > here. This was more of a spike to see if things go the right way, and if you see that changes are going in the right direction. I'll ensure parameters are err... parametrized. For now I'd prefer to keep things simple. I'm on CVS, without commit access and would prefer to not keep track of multiple large patches, smaller the patch better for me. (Hint: GIT :P) > Right now I am seeing 1-2 failing tests that are caused by the ant mkdir race > condition. Which of these tests are you referring to, I have quite a few failures so I'm unable to pinpoint things here.
With this latest patch which I updated for head, for me the test suite is all green except for these <mkdir> failures. I just ran and got 4 failures: ScriptGenerationTests.testbug284721 ProductTests.test237747 PublishingTests.testbug268498 AssembleTests.testbug179612_default The first one fails regularly (except when run by itself). If you have compile problems due to the ant tasks, import org.eclipse.equinox.p2.repository.tools as a binary project into your workspace. Also, you need either the recent IBuild I20090729-1307 or the latest org.eclipse.equinox.p2.publisher from cvs.
Removing M2 milestone
Not going to make 3.6
Currently we are not actively enhancing PDE build anymore. Therefore, I close this bug as WONTFIX. Please reopen, if you plan to provide a fix.
