Community
Participate
Working Groups
This is a bug to track the work related to produce a Dali 3.1 build. This will be necessary to differentiate Dali 3.1 from Head, which will eventually be Dali 3.2 (Juno).
Transferred to Releng for better visibility.
It the status meeting, you said you were working on a "stand alone build", but, this is a request to have (also) a "public build", right? And our normal pages? (Both make sense, I'm just double checking). One issue, which may or may not matter to you, is that I'm not sure where a software repo would go. We can't put it in "indigo" (since, you said, current Indigo users should not get it automatically .... they'd get new function, and would be unable to udpate to SR2 ... and while this release will eventually be part of Juno, doesn't seem good at all to put it in there with a bunch of other Juno stuff. (Your users would have to known to get nothing else, except your stuff) ... one option is just to get it directly from your build page .... or, have some little "custom" spot, just for Dali.
(In reply to comment #2) > It the status meeting, you said you were working on a "stand alone build", but, > this is a request to have (also) a "public build", right? And our normal pages? > (Both make sense, I'm just double checking). Yes, a public build. Is there any other kind in open source? :) > > One issue, which may or may not matter to you, is that I'm not sure where a > software repo would go. We can't put it in "indigo" (since, you said, current > Indigo users should not get it automatically .... they'd get new function, and > would be unable to udpate to SR2 ... and while this release will eventually be > part of Juno, doesn't seem good at all to put it in there with a bunch of other > Juno stuff. (Your users would have to known to get nothing else, except your > stuff) ... one option is just to get it directly from your build page .... or, > have some little "custom" spot, just for Dali. A good question. I think having the download available directly from the build page is probably sufficient for this release. There are too many version headaches associated with these mid-cycle releases, and it is probably best to just avoid them altogether with a separate download that can be accessed by end users and adopters that want this particular update.
Created attachment 203946 [details] Proposed patch for releng.control
Created attachment 203947 [details] Proposed patch for releng.wtpbuilder
Created attachment 203948 [details] Proposed patch for releng R3_3_maintenance
Hi Carl and David, I am about to commit the patches to CVS and verify that I haven’t broken the main build. As for the new Dali build, I believe that you need to make some change to the cruise control machine for it to be operational. One other question, I guess that changes for the releng R3_3_maintenance branch does not need to be applied to HEAD since it will not be used. Thanks.
One small concern - you are calling it dali-3.1-I ... all of our other builds are 3 digit - can we make it dali-3.1.0-I ?
(In reply to comment #8) > One small concern - you are calling it dali-3.1-I ... all of our other builds > are 3 digit - can we make it dali-3.1.0-I ? Yes it is not a problem.
Another update that needs to be done by hand, but should be reflected in the CVS version - there are two buildbranches.php files that will need to be updated before the dali-3.1.0-I will be displayed- one for the committers page, the other for the official download site. See webtools.releng/downloadsites/buildServer-shared-webtools/committers/buildbranches.php webtools.releng/downloadsites/downloadServer-webtools/downloads/buildbranches.php (Note that updates to these CVS files are not automatically propogated to the actual servers- those files should be updated by hand: /shared/webtools/committers/buildbranches.php ~/downloads/webtools/downloads/buildbranches.php ) For historical purposes, can you attach patches to the two CVS files here before committing those changes?
Created attachment 204228 [details] Proposed patch for releng.control
Created attachment 204229 [details] Proposed patch for releng.wtpbuilder
I restarted CruiseControl, and the dali-R3.1.0-I build is showing.
(In reply to comment #13) > I restarted CruiseControl, and the dali-R3.1.0-I build is showing. Thanks Carl, I haven’t heard back from Denis yet, but I guess you made the changes for both buildbranches.php files and updated the build server.
Hi David and Carl, The Dali project have now 3 builds: 3.0 maintenance, 3.2 on Head, and 3.1 on Head for now but it will have its own branch sometime in November. Now for Releng I am not sure how we can handle this case since Releng is branched according to the WTP release schedule, and I don’t remember how David you managed it previously. It seems to me that using Releng R3_3_maintenance for the Dali 3.1 release it not a good idea.
Created attachment 204332 [details] Proposed patch for the download page scripts
Created attachment 204556 [details] Proposed patch for releng R3_3_maintenance addendum
Created attachment 204557 [details] Proposed patch for Dali packaging
Created attachment 204615 [details] Proposed patch for buildbranches.php files I also updated the files on the build server.
(In reply to comment #15) > Hi David and Carl, > The Dali project have now 3 builds: 3.0 maintenance, 3.2 on Head, and 3.1 on > Head for now but it will have its own branch sometime in November. > Now for Releng I am not sure how we can handle this case since Releng is > branched according to the WTP release schedule, and I don’t remember how David > you managed it previously. > It seems to me that using Releng R3_3_maintenance for the Dali 3.1 release it > not a good idea. Not sure what we did before (I don't see anything obvious in cvs history) but I'd just use something like R_3_3_1_maintenance_dali to be clear it's branched from SR1 and is just for dali. (And, probably best to load up what ever was released in webtools.maps/releng for SR1 ... did we tag it for SR1 ... to make sure it is a clean split. HTH
I tagged all of the map files with R3_3_1 for SR1 (which, in retrospect, was probably wrong for Dali).
Carl, Unless you want to take care of it, I am thinking to branch releng R3_3_maintenance as David suggested. After that I am not sure how it will translate in cc_config to load it up from that branch. Can you or David point me to an example. Thanks! name: R3_3_1_maintenance_dali root: Root_R3_3_1_maintenance_dali PS. I have tagged Dali's maps and code R3_0_1
Created attachment 204699 [details] cc_config.xml - Tentative to load R3_3_1_maintenance_dali branch I created the R3_3_1_maintenance_dali branch, and made a tentative to load it in cc_config.xml. Let me know if it makes sense.
(In reply to comment #23) > Created attachment 204699 [details] > cc_config.xml - Tentative to load R3_3_1_maintenance_dali branch > > I created the R3_3_1_maintenance_dali branch, and made a tentative to load it > in cc_config.xml. Let me know if it makes sense. Three comments: 1. I'm not sure why you added <cvs tag="R3_3_maintenance" module="${env.RELENGCOMMON}" reallyquiet="${env.CVS_REALLY_QUIET}"/> The "modificationset" elements controls what CC should "watch" to see if there are changes that should trigger a build. Seems RELENGDALI was already there, and that's all you'd need? Doubt you want to trigger a build if someone makes a change to "common" ... or, are you doing more changes than I'm aware of? 2. I see you changed dependencyFileLocation to point to a special location outside the "releng" folder, namely value="../releng.dali/maps/dependencies.properties"/> This might technically work, not sure how ant handles ".." in the middle of things, but you can see the build code is written to assume dependencies are under "releng" ... <property name="dependency.properties" value="${buildDirectory}/maps/${env.RELENG}/${dependencyFileLocation}"/> I'd keep it there, under "releng" to be consistent. If you have your own branch, as you do, you don't even need a new location ... I'd stick with "indigo" location since that's closes to what you depend on (even though conceptually it is an early "juno"). But if you really wanted your own, new file for some reason, it should just go in releng, say a sibling of 'indigo' and 'juno', say 'indigoDali' so your value would be value="indigoDali/dependencies.properties"/> But seriously ... you do NOT NEED a special directory. We have one for indigo and one for juno simply because we are building both from the same branch. 3. The mapVersionTag looks correct :) value="R3_3_1_maintenance_dali" />
(In reply to comment #24) Thank you David. 1. I misunderstood the "modificationset" 2. Yes now that we have our own branch, it makes things much easier. We will stick with the indigo location.
Created attachment 204760 [details] Redo the 3.3.2 buildBranches.php update Tran, in the update to buildServer-shared-webtools/committers/buildbranches.php , you undid the update of the 3.3.x builds from 3.3.1 to 3.3.2. (Both in CVS and on the build machine.) This hid our current 3.3.2 builds from the build.eclipse.org/webtools/committers page. 3.3.1 is done, we need to show 3.3.2. I am attaching this patch and updating both CVS and the build machine.
Hi Carl, Sorry for the confusion. I modified cc_config.xml in CVS, and after half hour it is not propagated to the build server. How can I force an update of it? Thanks.
(In reply to comment #27) > Hi Carl, > Sorry for the confusion. > I modified cc_config.xml in CVS, and after half hour it is not propagated to > the build server. How can I force an update of it? Thanks. Tran, cc_config.xml is the configuration file for Cruise Control. In order for Cruise Control to be updated, it needs to be stopped, the new files need to be fetched from CVS, and then it needs to be restarted. Right now, only David and I have access to the wtpBuild account to do so, so you will need to contact one of us. I will restart Cruise Control right now to get the updates.
(In reply to comment #28) Thanks Carl for restarting the server. Now I don't understand the last Dali build failure: The following error occurred while executing this line: /shared/webtools/projectBuilders/dali-R3.1.0-I/webtools.releng/releng.wtpbuilder/build.xml:118: build.distribution must be specified The build.distribution is specified in cc_config, so I am not sure what else am I missing?
I verified the build server and it contains the updated version of cc_config. So is it possible that something is cached? I also see: Errors/Warnings: (4) projectname: dali-R3.1.0-I checkout.builder.clean: true Version tag for webtools.releng/releng.wtpbuilder is: v201110042323 Version tag for webtools.maps/releng: HEAD Where "webtools.maps/releng" should be "R3_3_1_maintenance_dali" instead of HEAD?
(In reply to comment #30) > I verified the build server and it contains the updated version of cc_config. There is a syntax error in cc_config ... I have no idea why the XML/Ant Editor/Validator or Cruisecontrol itself doesn't flag an error ... but the following element isn't right, and causes whole job to be misinterpreted. <ant> antscript="${env.RELENG_CONTROL}/ant_med_priority.sh"> After fixing that, (changing '<ant>' to '<ant') you will also need to update the eclipse-builder prereq to "3.7.1" It will not longer find eclipsebuilder.id=3.7RC3 eclipsebuilder.dir=S-3.7RC3-201105261708 remember ... bug 360129? I'm guessing you copied from "maintenance" branch, which has not been fixed yet. Compare with HEAD and use that eclipsebuilder.id After that ... I'm sure there will be more issues :) I say that to set expectations ... in the past, when I've started new branches or build types (such for patch builds) it'd take me at 8 to 10 tries on my local machine to work out all the kinks ... even having done similar things before ... I'm sure the very first time took dozens of tries ... (there is a hint buried in there :) I certainly don't mind failed builds ... but ... build.eclipse.org is a production machine and some care is called for, so "test builds" should be possible on non-production machines.
Curiosity got the best of me, I corrected the two errors I mentioned in comment #31. It still fails ... but, now for understandable reasons. You'll need to work out how to "install" your prereqs of wtp 3.3.1 (at least). Off hand, it appears your build still depends on those old "wst" and "jst" things which no longer exist. The ideal would be to use the wtp repo, and install exactly those features you want (which, would be about all of them, except for the "jpt" ones). HTH
(In reply to comment #32) Thanks for your help David. I wasn't familiar with, but I guess I should look into installing the features from the WTP repository instead.
While investigating on how to install Dali prereq from a WTP repository, I think I found two errors in the patch for the bug 360129. A patch has been uploaded.
Created attachment 204909 [details] Proposed patch for dependency.xml I have added a target for getting dependencies from the WTP repository. I named it "prereq.wtprepo", I am not sure if you agree on the name David. The routine is generic, the features to be installed will depend on the definition of property "wtp.tobeinstalledfeaturegroups".
Why not use the existing "wtp" group ID? We already have wtp.file, etc. in the dependencies.properties, so why not just add wtp.repo? It wouldn't be obvious unless you dug deep, but you can see the pattern in things like "dtp" and elsewhere: dtp.file=dtp-sdk_1.9.0.zip dtp.repo=dtp-p2repo_1.9.0.zip dtp.tobeinstalledfeaturegroups=org.eclipse.datatools.enablement.feature.feature.group,org.eclipse.datatools.sqldevtools.feature.feature.group,org.eclipse.datatools.connectivity.feature.feature.group The general logic is "if the <groupid>.tobeinstalledfeaturegroups property exists, then make use of the <groupid>.repo and install those features with p2, if tobeinstalledfeaturegrops does not exist, then use the "file" form and just unzip it.
(In reply to comment #36) > Why not use the existing "wtp" group ID? We already have wtp.file, etc. in the > dependencies.properties, so why not just add wtp.repo? Interesting, I am using the same group ID, but I thought that we will have to replace at the very least "getAndInstallDropins" by "getAndInstallRepo"? So then, what is the difference between the two targets?
(In reply to comment #37) > (In reply to comment #36) > > Why not use the existing "wtp" group ID? We already have wtp.file, etc. in the > > dependencies.properties, so why not just add wtp.repo? > > Interesting, I am using the same group ID, but I thought that we will have to > replace at the very least "getAndInstallDropins" by "getAndInstallRepo"? > So then, what is the difference between the two targets? The ant script should select which is appropriate (based on logic I outlined above ... BUT ... I'm not sure the script in the "Dali standalone build" has "kept up" with all the other scripts. In some cases when I was making a change to wtp.build or wtp4x.build, I would also make it to dali.build .... but .... not necessarily always ... so I'm not saying it would work ... just telling you how the other scripts work.
(In reply to comment #38) > The ant script should select which is appropriate (based on logic I outlined > above ... BUT ... I'm not sure the script in the "Dali standalone build" has > "kept up" with all the other scripts. In some cases when I was making a change > to wtp.build or wtp4x.build, I would also make it to dali.build .... but .... > not necessarily always ... so I'm not saying it would work ... just telling you > how the other scripts work. It worked on my local machine. I am starting a build.
1. ERROR: ImportNotFound The import org.eclipse.jpt.common.ui cannot be resolved I am sure that there is a simple explanation for the compilation errors in the latest build, but I haven't found it yet. The compilation passed on my local machine and the build was able to complete, so I do not understand what is different on the build machine. Ps. No new code has been released this week
(In reply to comment #40) > 1. ERROR: ImportNotFound > The import org.eclipse.jpt.common.ui cannot be resolved I have been able to reproduce the compiling errors locally. I have been investigating and noticed that org.eclipse.jpt.common.ui has been fetched, but is not present in the generated script: compile.org.eclipse.jpt_sdk.assembly.feature.xml David, can you enlighten us on how the generation process of the compile script works? I guess it must base on the org.eclipse.jpt.common.feature, then why it could left out o.e.jpt.common.ui? Thanks.
> > David, can you enlighten us on how the generation process of the compile script > works? I guess it must base on the org.eclipse.jpt.common.feature, then why it > could left out o.e.jpt.common.ui? Thanks. My guess is that it is related to o.e.jpt.common.ui already being there, already compiled, from just unzipping the wtp zip file into dropins. That confuses PDE build, since for many things if it finds a version already exists, it just assumes its good to go already. This is why, in the past, your "stand alone build" got the "wst" and "jst" zips (instead of wtp zip) because those specifically did not have "jpt" bundles in them. So, the best solution would be to use the p2 approach, and install only what you need as pre-reqs, from wtp repo archive file. Or, suppose you could have some custom scripts that went into "dropins" and removed all the jpt stuff (but, sounds fragile and error prone, to me). You could "test" my guess if you have a local build running ... go into your "prereqsCache" folder and (after making a backup copy) go in and "hack" the wtp zip file you have as a prereq and remove all the jpt stuff. We would not want to do that on build.eclipse.org (IMHO) since others might use it, someday, but the wtp build scripts just go by filename, when looking for things in prereqsCache, so if the wtp zip has the right name (that is, the name you say is a prereq), it will happily unzip its contents without further checking. (but, don't delete too much, or eclipse might complain about missing bundles, even if you did not need them, per se). Again, the p2 approach is better, long term. Chances are, you'd only have to specify a few features, such as org.eclipse.jst.enterprise_sdk.feature and it would "drag in" everything you needed? Not sure of what your exact list would be, I am just saying you do not need a list of the 60 or so features we routinely build.
(In reply to comment #42) Thank you David, that explains it. And I could verify that the "hack" method confirms your suspicions. Now can we just upload these zips (the build fetch both of them: repo and dropins) to the "prereqsCache" folder on the build server? We could rename it to something like wtp-xxxx-NO-JPT.zip, and modify the dependencies.properties file accordingly. Because it looks to me that it will be a tedious job to find the exact list of features that will not pull in the JPT bundles and make the build pass. And given also that the WTP zips will never change for this release.
> Thank you David, that explains it. And I could verify that the "hack" method > confirms your suspicions. Now can we just upload these zips (the build fetch > both of them: repo and dropins) to the "prereqsCache" folder on the build > server? . I would not do this. I have already stated my advice. You are free to do what you want, but the more you hack, the more error prone you will be, and the more you will be on your own. The p2 approach should not be hard at all, probably just requiring you to specify 1 to 12 features from WTP ... and, honestly, you should know what those are anyway. I know that's not the answer you were looking for, but, that's my best advise. Good luck.
(In reply to comment #44) Oh, and thought of a good "principle" to state that goes with my advice ... others, anyone, should be able to reproduce our builds, with publically available materials. Ideally "check out and go". If we start to "require" tweaked zips, available only on our build machine, then no one else would be able to reproduce the build, easily, without making their own tweaks. I say this, knowing we already have one tweaked zip from graffiti ... or did, not sure of latest status ... so ... if we (you) release engineers keep this up, it'll get worse and worse. Better to "spec it" in the scripts. Just my 2 cents.
David, we will use the p2 approach. In comment #42 I don't understand: > My guess is that it is related to o.e.jpt.common.ui already being there, > already compiled, from just unzipping the wtp zip file into dropins. So if it has already unzipped the wtp zip file into dropins, no matter what I define in the wtp.tobeinstalledfeaturegroups property, the JPT bundles will be already in dropins and that will confuse the PDE build. Can we change the target "prereq.wtp" in dependency.xml to use "getAndInstallRepo" instead of "getAndInstallDropins" currently. That way, it won't fetch the wtp zip file and it will get the wtp repo only, then it will only install what is defined in wtp.tobeinstalledfeaturegroups.
> > Can we change the target "prereq.wtp" in dependency.xml to use > "getAndInstallRepo" instead of "getAndInstallDropins" currently. That way, it > won't fetch the wtp zip file and it will get the wtp repo only, then it will > only install what is defined in wtp.tobeinstalledfeaturegroups. I see now ... you have been _trying_ to use the repo (at least for a few days) and the script uses the zip instead. I looked at scripts/dependency/dependency.xml and it looks like the prereq.wtp task was ever updated, so your comment above is correct. But, instead of calling getAndInstallRepo, it should call getAndInstallFramework. That tasks, getAndInstallFramework, is the one that has "smarts" and will decide if it should call getAndInstallDropins or getAndInstallRepo, depending on how the properties are defined. So, yes, changing to getAndInstallRepo would work, but I'd recommend getAndInstallFramework so that way the process would be more controlled by properties rather than hard coded to use one method or the other.
(In reply to comment #47) Sorry for typo, "the prereq.wtp task was ever updated" ==> "the prereq.wtp task was never updated"
Yes I have been trying since comment #37 :) (In reply to comment #47) > I'd recommend getAndInstallFramework so that way the process would be more > controlled by properties rather than hard coded to use one method or the other. I have been experimenting with "getAndInstallFramework", I am getting now: [build-dali-dali-sdk] generateScript: [build-dali-dali-sdk] [eclipse.buildScript] Some inter-plug-in dependencies have not been satisfied. [build-dali-dali-sdk] [eclipse.buildScript] Bundle org.eclipse.jpt.jpadiagrameditor.ui.tests: [build-dali-dali-sdk] [eclipse.buildScript] Missing required plug-in org.easymock_[2.4.0,3.0.0). David, how dependencies on the Orbit bundles works? We don't have a target for that prereq in dependency.xml.
> Yes I have been trying since comment #37 :) Sorry, I think I misread that comment. > David, how dependencies on the Orbit bundles works? We don't have a target for > that prereq in dependency.xml. Yes, you don't need a target, but, you do (I think) need to "include" it in a feature. Your test feature, in this case. My guess is, in the past, you were being "lucky" that someone else was (xsl? jsf?) and hence it was there for you by the time your tests compiled. In general, all of the "p2 format" map entries (most, but not all, come from Orbit) are automatically fetched by PDE and put in the transformedRepo location (as a holding location). So, that's how I can see the basic mechanism is working, since there's a lot of other bundles being fetched and put in there, just not easy mock. Oh, and also, I guess you are not "installing" other "test features" from those lower level features (e.g. xsl or jsf) which might have easy mock already "included". If you do really depend on any others test bundles/features from WTP, that'd have to be a separate prereg with its own list of "to be installed features". (not recommended :) but not sure how you are set up, so that might be required, and that might alternatively fix your problem, if that's been part of your normal test-time dependencies.) = = = = list of current "p2 fetched bundles" = = = = javax.persistence_2.0.3.v201010191057.jar javax.wsdl_1.6.2.v201012040545.jar org.apache.commons.collections_3.2.0.v201005080500.jar org.apache.commons.lang_2.1.0.v201005080500.jar org.apache.oro_2.0.8.v201005080400.jar org.apache.velocity_1.5.0.v200905192330.jar org.eclipse.persistence.antlr_2.4.0.v20110824-r9956.jar org.eclipse.persistence.asm_2.4.0.v20110824-r9956.jar org.eclipse.persistence.core_2.4.0.v20110824-r9956.jar org.eclipse.persistence.dbws_2.4.0.v20110824-r9956.jar org.eclipse.persistence.dbws.builder_2.4.0.v20110824-r9956.jar org.eclipse.persistence.jpa_2.4.0.v20110824-r9956.jar org.eclipse.persistence.jpa.jpql_1.0.1.v20110824-r9956.jar org.eclipse.persistence.jpa.jpql.source_1.0.1.v20110824-r9956.jar org.eclipse.persistence.moxy_2.4.0.v20110824-r9956.jar org.jdom_1.0.0.v201005080400.jar
David, just to let you know that the build is passing now. Let me know if you want me to update releng in HEAD to pick up the latest wtpbuilder? Or wait until builds are declared. Thank you!
excellent news! Thanks for letting us know. As for when to update wtpBuilder, that's mostly up to you and Carl, but since it does not itself trigger a build, I'd think you could update head, tag it, use the tag in your dali branch, and update the head and maintenance branches after weekly builds declared.
(In reply to comment #51) > ... just to let you know that the build is passing now. I am curious, how did you solve the easymock issue?
(In reply to comment #53) > I am curious, how did you solve the easymock issue? I don't remember exactly the circumstance that lead to that error, I tried too many things :(. But I got that error yesterday when I began to build from a not fully clean workspace because I had networking issues, it took more than a few of hours to build.
Carl, just to let you know that I have made some corrections for typo and retagged wtpbuilder. I have also updated releng Head and the maintenance branches (HEAD, R3_3_maintenance, R3_2_maintenance, R3_3_1_maintenance_dali).
Dali 3.1.0 was released on Dec 23rd, 2011.
