Community
Participate
Working Groups
[ I've CC'd the cross-project since some people who may be interested in this thread -- I'll remove the CC after the initial bug to avoid spamming everyone ] In light of the build server outage, I'd like to plan ahead. Here are some tasks I'd like to see done sooner than later. 1. MATT: How to init a new master should the current master die. On a related note, we should have a backup copy of the image file somewhere, even if it's on another mirage. We probably already have one, but I'd have to go looking where it is ... I'm just thinking out loud here. 2. DENIS: shadow copy of /opt/public. It's ugly and huge, but I think having an rsync copy, even if it's a week old, would be hugely beneficial and help us (and the committers) recover much more quickly. We could even just get a 1.5T SATA drive, toss it in an existing server and rsync to it. Simple yet effective. 3. MATT: a quick how-to enable a new slave would be awesome should we experience a tremendous spike that the current Hudson servers cannot handle (doubt that, but still). 4. DENIS: backing up the new build server's config is important. Again, a weekly rsync to /home/data/common/backup would be sufficient. Having the hudson job info there has been insanely helpful.
May I suggest too, that we "document" any other tweaks, adjustments, installs, etc., that were required when moving to the new server, just as referenced bug URLs, so there is one spot that has sort of an accumulated history of "things to consider" if/when we ever change machines again ... though some are related to platform architecture change which probably will never happen again :) I'll add some I know about to the "See also" field, but if anyone knows of others, I think it'd be handy to have this "central list"? (Not sure that's exactly the right use of "see also" field ... so if anyone would prefer 'depends on' or something, let me know.
> 1. MATT: How to init a new master should the current master die. On a related > note, we should have a backup copy of the image file somewhere, even if it's on > another mirage. We probably already have one, but I'd have to go looking where > it is ... I'm just thinking out loud here. > > 3. MATT: a quick how-to enable a new slave would be awesome should we > experience a tremendous spike that the current Hudson servers cannot handle > (doubt that, but still). I've added docs to our KB for these. -M.
(In reply to comment #0) > 2. DENIS: shadow copy of /opt/public. We put a 1.4T disk in the server and mounted it as /opt/backup. A weekly rsync copies data from /opt/backup > 4. DENIS: backing up the new build server's config is important. Again, a > weekly rsync to /home/data/common/backup would be sufficient. It is there. > May I suggest too, that we "document" any other tweaks I've updated our internal docs to refer to this bug for hints. We're done here. Should we ever need to do this again, we should be in much better shape.
