Bugzilla – Full Text Bug Listing
| Summary: | Rename HIPP "mexx" instance to match one project | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Community | Reporter: | Denis Roy <denis.roy> | ||||||
| Component: | CI-Jenkins | Assignee: | CI Admin Inbox <ci.admin-inbox> | ||||||
| Status: | RESOLVED FIXED | QA Contact: | |||||||
| Severity: | normal | ||||||||
| Priority: | P3 | CC: | dennis.huebner, Ed.Merks, nobody, stepper, thanh.ha, webmaster | ||||||
| Version: | unspecified | ||||||||
| Target Milestone: | --- | ||||||||
| Hardware: | PC | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | |||||||||
| Bug Depends on: | |||||||||
| Bug Blocks: | 413102 | ||||||||
| Attachments: |
|
||||||||
|
Description
Denis Roy
Created attachment 237833 [details]
hipps
At the moment my HIPP list looks like this. I'm sure, MWE hudson does not exist Modeling Workflow Engine:
I would like to keep Mexx at least as HTTP URL. We already shared this URL with committers and our latest composite update site points to it. Also the buckminster build setup (in all the projects) is using /mexx/ as source for the latest builds. It will take some time and to change it all again. But if there are some technical reasons for the change, I would prefer Xtext over EMF as a new name. P.S.: Could you please restart mexx right now? We have some problems with gerrit trigger. It kicks 3 builds in the same time for one gerrit change and all the build have the same number. 8| One more. Can we setup Mexx HIPP to use a separate tmp directory. Our tests produce a lot of small tmp-files and sometimes, if e.g. vm abnormally exits (OutOfMemory or what ever), they remain in the tmp folder. I've installed https://wiki.jenkins-ci.org/display/JENKINS/Tmp+Cleaner+Plugin plugin and would like to clean up jenkins tmp at least weekly. Right now java's tmpdir points to system's tmp: java.io.tmpdir=/tmp see https://hudson.eclipse.org/mexx/systemInfo. Somewhere under /home/hudson/genie.modeling.mexx would be fine. IMHO, temp folder clean up, is in general a good idea, for all the HIPPs. The current /tmp directory was about 1,3 GB, maybe more, I have not enough persimmons to scan the whole directory. > At the moment my HIPP list looks like this. I'm sure, MWE hudson does not > exist Modeling Workflow Engine: I had attached the instance to MWE. I've since attached it to modeling.tmf.xtext. (In reply to Dennis Huebner from comment #2) > I would like to keep Mexx at least as HTTP URL. The URL is now https://hudson.eclipse.org/xtext but I've added some redirects to ensure all /mexx URLs still work. Please use /xtext from now on. > Also the > buckminster build setup (in all the projects) is using /mexx/ as source for > the latest builds. It will take some time and to change it all again. /mexx will continue to work for some time. Eventually we'll clean it up, but there is no rush. > But if there are some technical reasons for the change, I would prefer Xtext > over EMF as a new name. It is to allow you to start/stop/restart your own instance :) > P.S.: Could you please restart mexx right now? We have some problems with > gerrit trigger. It kicks 3 builds in the same time for one gerrit change and > all the build have the same number. 8| I've restarted it. I had noticed this yesterday, and your HIPP instance was consuming lots of CPU cycles. FYI: it may be possible that a single HIPP instance is not sufficient for all your projects. If we see that you must restart it often because of the gerrit trigger, it may be beneficial to isolate projects to their own instance. Other projects are experiencing great stability, even with the Gerrit plugin. (In reply to Dennis Huebner from comment #3) > IMHO, temp folder clean up, is in general a good idea, for all the HIPPs. It is. We have scripts to clean up /tmp. Matt, can you make sure these scripts are running every day on all the hipp servers? > At the moment my HIPP list looks like this.
That was helpful. When you get a moment can you show me what it looks like now, and perhaps test the stop/start functionality?
Created attachment 237856 [details]
hipps, looks better now
Restart seems to work. Works too good :| Hard restart, I expected some kind of "Please lock your security belts, we restart now" or something. All running jobs were just canceled. Good to know for the next time :)
Some issues: https://hudson.eclipse.org/xtext/manage says: The connection to Gerrit is down! Check your settings and the Gerrit server. New user is probably not ready yet: java.io.FileNotFoundException /shared/download-staging.priv/modeling/emf/base/site_277981523.zip (Permission denied) ... and so on (In reply to Dennis Huebner from comment #8) > Some issues: > https://hudson.eclipse.org/xtext/manage says: The connection to Gerrit is > down! Check your settings and the Gerrit server. > > New user is probably not ready yet: java.io.FileNotFoundException > /shared/download-staging.priv/modeling/emf/base/site_277981523.zip > (Permission denied) ... and so on I fixed this. The problem was that when mexx was renamed the file structure changed and the ssh rsa key was moved to a new path. I reconfigured Gerrit Trigger to look for it in the correct path now. (In reply to Thanh Ha from comment #9) > (In reply to Dennis Huebner from comment #8) > > Some issues: > > https://hudson.eclipse.org/xtext/manage says: The connection to Gerrit is > > down! Check your settings and the Gerrit server. > > > > New user is probably not ready yet: java.io.FileNotFoundException > > /shared/download-staging.priv/modeling/emf/base/site_277981523.zip > > (Permission denied) ... and so on > > I fixed this. The problem was that when mexx was renamed the file structure > changed and the ssh rsa key was moved to a new path. I reconfigured Gerrit > Trigger to look for it in the correct path now. Still fails: Caused by: java.io.FileNotFoundException: /shared/download-staging.priv/modeling/emft/mwe/site_265834590.zip (Permission denied) New xtext user needs the same permissions as mexx had. And could you please change the tmp folder? java.io.tmpdir (In reply to Thanh Ha from comment #9) > (In reply to Dennis Huebner from comment #8) > > Some issues: > > https://hudson.eclipse.org/xtext/manage says: The connection to Gerrit is > > down! Check your settings and the Gerrit server. > > > > New user is probably not ready yet: java.io.FileNotFoundException > > /shared/download-staging.priv/modeling/emf/base/site_277981523.zip > > (Permission denied) ... and so on > > I fixed this. The problem was that when mexx was renamed the file structure > changed and the ssh rsa key was moved to a new path. I reconfigured Gerrit > Trigger to look for it in the correct path now. EMF base is still locked for hudson: staging.priv/modeling/emf/base/site_264696964.zip due to java.io.FileNotFoundException /shared/download-staging.priv/modeling/emf/base/site_264696964.zip (Permission denied) (In reply to Dennis Huebner from comment #10) > Still fails: > Caused by: java.io.FileNotFoundException: > /shared/download-staging.priv/modeling/emft/mwe/site_265834590.zip > (Permission denied) > For some reason xtext HIPP user was not in the proper groups. I've readded the groups as follows: - signers - modeling.emf.mwe - modeling.tmf.xtext - modeling.emf.emf - modeling.m2t.xpand > And could you please change the tmp folder? java.io.tmpdir I'm not sure what you mean here. Isn't this a property you set in a job? (In reply to Thanh Ha from comment #12) > (In reply to Dennis Huebner from comment #10) > > Still fails: > > Caused by: java.io.FileNotFoundException: > > /shared/download-staging.priv/modeling/emft/mwe/site_265834590.zip > > (Permission denied) > > > > For some reason xtext HIPP user was not in the proper groups. I've readded > the groups as follows: > > - signers > - modeling.emf.mwe > - modeling.tmf.xtext > - modeling.emf.emf > - modeling.m2t.xpand > > Forgot to mention for these changes to take effect we need to restart HIPP. There is currently a job running so I did not want to restart it. I'll check again in a bit and restart it if the job finishes, or you can restart it if you notice the job's done before I do. (In reply to Thanh Ha from comment #12) > (In reply to Dennis Huebner from comment #10) > > > And could you please change the tmp folder? java.io.tmpdir > > I'm not sure what you mean here. Isn't this a property you set in a job? I think it's a java parameter which should be passed during the hudson start -Djava.io.tmpdir=/home/hudson/genie.modeling.tmf.xtext/tmp. I'm not quiet sure. see https://hudson.eclipse.org/xtext/systemInfo > 'll check again in a bit and restart it if the job finishes, or you can restart it if you notice the job's done before I do. I will cancel them now. Would be good if you can figure out how to change java.io.tmpdir before restart. (In reply to Denis Roy from comment #5) > It is. We have scripts to clean up /tmp. Matt, can you make sure these > scripts are running every day on all the hipp servers? Ok I've added a cron task that will clean up all files/folders in /tmp older than 7 days every morning at 1am. -M. (In reply to Dennis Huebner from comment #14) > (In reply to Thanh Ha from comment #12) > > (In reply to Dennis Huebner from comment #10) > > > > > And could you please change the tmp folder? java.io.tmpdir > > > > I'm not sure what you mean here. Isn't this a property you set in a job? > > I think it's a java parameter which should be passed during the hudson start > -Djava.io.tmpdir=/home/hudson/genie.modeling.tmf.xtext/tmp. I'm not quiet > sure. > see https://hudson.eclipse.org/xtext/systemInfo > Oh I see what you mean. I guess my next question is then does this property need to be different for every HIPP instance? I know the platform project typically sets this property as part of their individual job configs but I guess it is more convenient if it's set globally for the instance. Although on the other hand setting it globally for a Hudson instance rather than per job would mean there's still a chance for collision if 2 jobs are similar. > And could you please change the tmp folder? java.io.tmpdir
I would prefer we keep it to /tmp
That way we can clean up a single central /tmp location instead of having random temp directories everywhere.
(In reply to Thanh Ha from comment #16) > globally for the instance Sounds perfect. Hudson's TempClearPlugin will take care of it. And if I become a filling I have to clear it now, I can start a hudson job that sweeps it. (In reply to Denis Roy from comment #17) > > And could you please change the tmp folder? java.io.tmpdir > > I would prefer we keep it to /tmp > > That way we can clean up a single central /tmp location instead of having > random temp directories everywhere. Hudson automatically cleans it up. Right now there is about 1G crap under /tmp and this crap is not my :) (In reply to Dennis Huebner from comment #19) > (In reply to Denis Roy from comment #17) > > > And could you please change the tmp folder? java.io.tmpdir > > > > I would prefer we keep it to /tmp > > > > That way we can clean up a single central /tmp location instead of having > > random temp directories everywhere. > > Hudson automatically cleans it up. Right now there is about 1G crap under > /tmp and this crap is not my :) Is there any benefit to having a separate tmp now that Webmaster added a script to cleanup /tmp? I tend to agree with Denis, from a sysadmin perspective it's easier for us to manage a single /tmp than for us to manage a separate one for all HIPPs. (In reply to Thanh Ha from comment #20) > (In reply to Dennis Huebner from comment #19) > > (In reply to Denis Roy from comment #17) > > > > And could you please change the tmp folder? java.io.tmpdir > > > > > > I would prefer we keep it to /tmp > > > > > > That way we can clean up a single central /tmp location instead of having > > > random temp directories everywhere. > > > > Hudson automatically cleans it up. Right now there is about 1G crap under > > /tmp and this crap is not my :) > > Is there any benefit to having a separate tmp now that Webmaster added a > script to cleanup /tmp? > > I tend to agree with Denis, from a sysadmin perspective it's easier for us > to manage a single /tmp than for us to manage a separate one for all HIPPs. Also as far as I can tell everything in /tmp is created by Hudson from various projects. This leads me to believe that Hudson doesn't actually automatically clean it up. One compromise I can think of is what if we set the tmpdir to be. /tmp/<hipp> This way at least all the tmps are organized by HIPP instance and still exist in /tmp. Would this be an OK compromise? (In reply to Thanh Ha from comment #22) > One compromise I can think of is what if we set the tmpdir to be. > > /tmp/<hipp> > > > This way at least all the tmps are organized by HIPP instance and still > exist in /tmp. Would this be an OK compromise? +1 Hudson plays dead :( https://hudson.eclipse.org/xtext/ is the start/stop not working for you? (In reply to Dennis Huebner from comment #23) > (In reply to Thanh Ha from comment #22) > > One compromise I can think of is what if we set the tmpdir to be. > > > > /tmp/<hipp> > > > > > > This way at least all the tmps are organized by HIPP instance and still > > exist in /tmp. Would this be an OK compromise? > > +1 I updated the Hudson init script to do this now. I restarted the instance with the new init script that sets tmpdir. I noticed that Hudson creates the following files on startup that looks like they might be important: jna6227513637910289632.tmp libs_jetty-http.jar93712025487016308hudson libs_jetty-servlet-api.jar8922624718255558763hudson libs_jetty-web-app.jar327640656592650022hudson libs_hudson-jetty-war-executable.jar7027694174232890661hudson libs_jetty-io.jar7311781131116457990hudson libs_jetty-servlet.jar6444823446254209611hudson libs_jetty-xml.jar3555918629284705833hudson libs_jetty-continuation.jar6468021667645392634hudson libs_jetty-security.jar8684809987455048010hudson libs_jetty-util.jar8431801008838187209hudson libs_jetty.jar555671377540493576hudson Perhaps the cron job should ignore these files when it's cleaning things up? > Perhaps the cron job should ignore these files when it's cleaning things up?
The job will delete files that haven't had their mtime updated in more than 7 days. Do you think that will cause a problem?
(In reply to Denis Roy from comment #28) > > Perhaps the cron job should ignore these files when it's cleaning things up? > > The job will delete files that haven't had their mtime updated in more than > 7 days. Do you think that will cause a problem? I'm not entirely sure. Although I just checked the ones owned by cdt and noticed it was last touched today: ... Nov 29 10:48 libs_jetty-continuation.jar4076935379109827521hudson so maybe it does touch it once in awhile? I guess we can find out. > I guess we can find out.
Can you check the status of those files on the three hipp servers? If some date back to older than 7 days, we'll have to find another approach.
(In reply to Denis Roy from comment #30) > > I guess we can find out. > > Can you check the status of those files on the three hipp servers? If some > date back to older than 7 days, we'll have to find another approach. Hmm I found that hipp1 didn't have any of those files. hipp2 only had them owned by CDT and hipp3 has them for tools.tm and technology.packaging. The tools.tm version has an mtime of Nov 21st. Based on this data I'm guessing these files probably only exist if a plugin is installed, although I'm not sure which at this point. I think we'll be good. Is a plugin is silly enough to maintain critical state in a file in /tmp, it needs to have a bug reported against it. Otherwise, the mexx instance has been renamed, and I think all is well here. Thanks Dennis for working with us on this one. |