Community
Participate
Working Groups
It is required that the EPP package repo and the Sim Release repo become "visible" at the same time when we do a release. It would be handy if this was automated, say via Hudson, where a "time to execute" could be entered beforehand. Not only would this make sure they were made visible at the same time, but also help avoid complications if one (or both!) of the release engineers were not available at the designated time. I have created such a job for the "Sim Rel repo" and am trying it today (5/20) for the first time. [I will, of course, be sitting here watching it, and correct it or do "by hand" if there are errors in the script.] That job is https://hudson.eclipse.org/simrel/view/Releng%20jobs/job/simrel.releng.makeVisible/ Perhaps EPP could "tie in" to that script. The script is actually very simple, see http://git.eclipse.org/c/simrel/org.eclipse.simrel.tools.git/plain/promoteUtils/makeVisible.sh The hard part, compared to the previous version, was making sure it would run correctly even when not ran from the "current directory" as was done when running from it was ran from a shell terminal.
Markus, I have taken the liberty of "setting this up". I have even set it up to work "this Friday" at 9:30 -- as long as I hear "ok" from you. And BTW, I actually opened this bug *before* last Friday's missed mail. :) The Hudson job is at https://hudson.eclipse.org/simrel/view/Releng%20jobs/job/simrel.releng.makeVisible/ The files that "do the work" are at For EPP: http://git.eclipse.org/c/simrel/org.eclipse.simrel.tools.git/tree/promoteUtils/makeEPPVisible.sh for SimRel: http://git.eclipse.org/c/simrel/org.eclipse.simrel.tools.git/tree/promoteUtils/makeVisible.sh The are almost identical, except for "REPO_ROOT" plus, the pattern we use for the "temporary names" are a bit different. I use names such as compositeContentRC2.jar but you use names such as compositeContent.jar.RC2. I have taken all that into account, of course, in the scripts, but some proofreading by you would not hurt :) If you are interested in changing your system, one reason I use compositeContentRC2.jar is because I can still say vi compositeContentRC2.jar and vi will unzip the jar, and show me the XML. With yours, I have to copy to my ~/temp/ area to change names, and check if contents are as expected. Not that I normally have to do that. But did last week. :) If we ever use the "same system" for naming those composites, I'd probably just use one script, and "pass in" the REPO_ROOT as a variable. You should have complete access to the Hudson config. Let me know if any questions after you get a chance to review, assuming you do. I also hope to put into place a tiny "auto check" which simply lists the IUs at the "site". I did this for some other cases after I made a typo in a "patch repository" and broke the "atomic" nature of the repository. So, that is currently the main purpose of the "test" ... make sure they are at least "atomic". I suppose eventually we could "count" the IU's listed and make sure they are within some certain range? Or, I guess, even set up other tests (using p2Director?) to "install" or "update" to latest version ... though doubt either of us have time to do that before Neon.
(In reply to David Williams from comment #1) > I have taken the liberty of "setting this up". I have even set it up to work > "this Friday" at 9:30 -- as long as I hear "ok" from you. I think that's a very good idea! (I won't have the time today to have a closer look into the script, but I will definitively look at it before our release tomorrow/Friday.)
(In reply to Markus Knauer from comment #2) > (In reply to David Williams from comment #1) > > I have taken the liberty of "setting this up". I have even set it up to work > > "this Friday" at 9:30 -- as long as I hear "ok" from you. > > I think that's a very good idea! > > (I won't have the time today to have a closer look into the script, but I > will definitively look at it before our release tomorrow/Friday.) Ok, great! I'll plan on it. Of course, ideally, you will be sitting there watching it the first few times. :) I have set it up to trigger another "sanity check" job, on just /releases/neon, to make sure it is "atomic". It just lists the IUs available. I realize there is more that could be done eventually ... but, I'm always concerned I am going to make a typo or delete the wrong "old one". The job is https://hudson.eclipse.org/simrel/view/Releng%20jobs/job/simrel.releng.sanityCheckComposites/ It actually only runs the script at http://git.eclipse.org/c/simrel/org.eclipse.simrel.tools.git/tree/promoteUtils/checkComposites.sh That script is actually capable of doing the check on a long list of repositories, but figure for tomorrow I'd set the default to be just that one. It only takes about 10 seconds per repo (after eclipse is installed, and the repo cloneed -- and I do not "clean each build" so they should all be ready. Let me know if you have any suggestions (or concerns!) Thanks again,
We need to solve an issue with the file permissions. The SimRel HIPP user has no permissions to do anything (copy, move, delete, ...) in the EPP download area /technology/epp which will lead to a failure of the script. In order to test this I set up a small temporary test job in the SimRel HIPP and it failed as expected: Started by user mknauer@eclipsesource.com [workspace] $ /bin/bash -xe /tmp/genie.simrel/hudson5348450516652637575.sh + test -s /home/hudson/genie.simrel/.alias + true + touch /home/data/httpd/download.eclipse.org/technology/epp/packages/neon/foobar touch: cannot touch `/home/data/httpd/download.eclipse.org/technology/epp/packages/neon/foobar': Permission denied Finished: FAILURE One quick and temporary workaround: I changed the group ownership from 'technology.packaging' to 'callistoadmin' of the EPP files. I hope that helps for today, but I will be watching the job running.
Adding webmasters for help in solving the issue in comment 4. Here's the solutions I'm aware of, each less than ideal. A. add the "genie.simrel" user to the technology.packaging group. - I don't think this is too good, since gives a little "too much" access "across projects". But, not sure in practice this is too bad, since that Hipp instance is fairly "tied down", as far as I know. -- That is, from memory, not just anyone is "callisto-dev" can create a job there -- but, I'd have to check to know for sure. B. Use some sort of ACL to give permission to a small area of technology.packaging download area. - that just seems like a long term maintenance problem. C. Me and Markus could have separate jobs, each on our own HIPP instance, and either by convention "time the jobs" well, or one "signal" the other to trigger its job. - the timing convention is probably ok in practice, but there could be an issue if "one of them failed" for whatever reason, then the repository would be "invalid" until fixed. - the "signal" approach is probably ok in practice, but I've heard that method is "deprecated" and may not be available in future versions of Hudson. = = = Webmasters, any better solutions you can think of?
Some other changes to consider: It would be nice if the "checkpoint" variable was a parameter in the job, but I am not sure what happens then for a "timed build" ... normally parameterized builds wait for the user to click a button before they run, but have never tried with a "timed build". It would be best if the trigger to the "sanity check" job could take parameters. The sanity check job can check a long list of repositories, which I would like it to do occasionally, but after the "make visible" job, there is really only one repository we want to sanity check. = = = = = = Another item Markus and I agreed to in IM is that he could change the naming convention of his "files on hold" so that way the exact same script could be used for both repositories, and just pass in the "path to download area". (that would be the only difference between the two).
Just to document it here, the "successful" run log from today is at https://hudson.eclipse.org/simrel/view/Releng%20jobs/job/simrel.releng.makeVisible/4/console There were a few failed attempts, but that was just due to a programming mistake on my part. I make "checkpoint" (such as "RC1", "RC2" a variable, and forgot to assign it a value!
(In reply to David Williams from comment #6) > Another item Markus and I agreed to in IM is that he could change the naming > convention of his "files on hold" so that way the exact same script could be > used for both repositories, and just pass in the "path to download area". > (that would be the only difference between the two). All file names in EPP are renamed following the SimRel naming convention (e.g. to compositeArtifactsRC2.jar); there is no need to keep two similar scripts up to date any more.
(In reply to David Williams from comment #5) I can't think of anything you haven't already suggested, and I agree with your assessments of the downsides. > - the "signal" approach is probably ok in practice, but I've heard that > method is "deprecated" and may not be available in future versions of > Hudson. This is probably the 'best' choice out of all the options. -M.
(In reply to Markus Knauer from comment #8) > All file names in EPP are renamed following the SimRel naming convention > (e.g. to compositeArtifactsRC2.jar); there is no need to keep two similar > scripts up to date any more. Ok, thanks. I now have one version, just "makeVisible", but given next comment, may need some tweaks. (I "hard coded" the paths for each repo, except did add train name ... if we have a job on simrel that signals epp, then some other parameter will need to be added). (In reply to Eclipse Webmaster from comment #9) > > - the "signal" approach is probably ok in practice, but I've heard that > > method is "deprecated" and may not be available in future versions of > > Hudson. > > This is probably the 'best' choice out of all the options. > > -M. Thanks Matt. Sound ok to you Markus? I think then, also, your job, upon success, would have to send a signal back to run the "sanity check job", which at least would give us some additional reassurance that BOTH jobs ran. If you haven't noticed yet, I created a "temptest" directory in your space, to test the script. I was hoping something like "rsync --group" might work, but still "permission denied". I would delete it, but maybe should leave there while we test the "new system" of signally jobs. If you don't mind, I'll just create a new job on EPP HIPP and (mostly) copy/paste what I have in sim rel. (If I have permission to create jobs there :)
(In reply to David Williams from comment #10) > Sound ok to you Markus? Yes, it's not perfect to manage multiple jobs over different Hudson instances, but we will make it work. Since I'm not sure if you have enough permissions on the EPP HIPP I created an empty job https://hudson.eclipse.org/packaging/job/epp.releng.makeVisible/ and added your user to it. If anything fails I could look into it some time tomorrow/Thursday.
(In reply to Markus Knauer from comment #11) > (In reply to David Williams from comment #10) > > Sound ok to you Markus? > > Yes, it's not perfect to manage multiple jobs over different Hudson > instances, but we will make it work. > > Since I'm not sure if you have enough permissions on the EPP HIPP I created > an empty job > > https://hudson.eclipse.org/packaging/job/epp.releng.makeVisible/ > > and added your user to it. If anything fails I could look into it some time > tomorrow/Thursday. It was a much larger pain than I imagined. But, now it works. I had already created a job before I saw your note here. You can rename it if you want, but ... then must rename the "trigger" job too! https://hudson.eclipse.org/packaging/job/epp-makeVisible/ I removed or commented out the "temptest/neon" part of the Hudson scripts, but left the "temptest" directory in place in case you want to see .. or in case we need to do more testing. Delete at will. The procedure then is you don't have to do anything Markus -- well, except get the "composite*RC3.jar" files ready! But in simrel job, I (or one of us) will have to update the "checkpoint" each milestone. I also added the "trainname", as a parameter so we can use the same jobs for oxygen or neon (or, we can copy another job version if we want). My point is I "send" those parameters to your job, and works like a charm, once I figured out (remembered) how to. The scripts "inside" Hudson are nearly identical. In one place, I specify SIMREL and in your version its says EPP. Plus, the curl commands at the end of each are a little different, since they trigger different jobs. (I used same "password token" throughout). Plus, you'll see a lot of "URGENT" mail from my testing, but in the end I have the sanityCheck job send us a note on success or failure. Failures say "URGENT" success will say "Good to go". A much larger pain! :) [Of course, anything that takes me 4 or 5 hours is a pain. :) ]
