Community
Participate
Working Groups
We need a process to remove jobs that are no longer current. Each job consumes memory, and takes CPU cycles to render, and consumes bandwidth and disk space. Some jobs may no longer be needed -- perhaps the project has gone away.
You can delete these jobs that are owned by me eclipse-equinox-git-test-M eclipse-equinox-git-test-N
mylyn-heartbeat-connectors is no longer needed.
I agree we need a way to identify and delete unneeded jobs but I would expect the overhead to be minimal if the workspace was cleared and old builds were removed. Any info available on how much memory and disk an unused job consumes? More important for disk space would be ensuring all jobs are configured to clear old builds after a certain time interval, and wipe the workspace. Definitely if an Eclipse project gets archived removing its Hudson jobs should be part of that process.
I would be very apprehensive about a policy that systematically removes jobs for older maintenance lines on active projects. Even outside of paid support program, the project team may decide to roll out a service release on a line that hasn't seen activity in quite a while. Removing these jobs will make producing such releases more difficult as the build job in question would need to be re-created.
> I would expect the overhead to be minimal I didn't look at the source code, but I can't help but imagining a slew of Objects inside an ArrayList[] which gets .instantiated frequently but never properly gc'd. (In reply to comment #4) The job config is stored in files in /shared/jobs/(job-name). I can't see why archived jobs couldn't just sleep somewhere else and be revived as needed -- or even moved to a seldom used hudson instance for archived builds.
That sounds rather like premature optimization. I can understand wiping workspaces since we are always short of disk space, but absent concrete evidence that inactive jobs are causing a perf issue, why go through the hassle of archiving/un-archiving process? Also, any perf impact analysis should consider difference between jobs that are just inactive and those that are explicitly disabled.
> That sounds rather like premature optimization. That could be. Here is my reasoning: - Our Hudson instance is not running very well. If we didn't have any problem at all, I would agree with your assessment. - We've been running Hudson for years now, yet comments about sluggish UI response have only been surfacing for the last few months. - It is my understanding that re-building a past release (>8 month) is a use-case exception, not a use-case norm. - With the above in mind, the hassle of archiving old jobs (with seldom unarchiving) is a smaller hassle than actually performing a performance impact analysis (to me, anyway). - A performance impact analysis won't actually help a single thing if no inherent app performance problem is found; however, removing unused jobs will definitely make Hudson startups and "reload from disk" faster, reduce the amount of HTML sent to your browser and unclutter the "All Jobs" tab, which currently has 351 jobs. It all seemed like sound reasoning to me when I wrote this bug. I'm willing to close this as INVALID.
> - We've been running Hudson for years now, yet comments about sluggish UI > response have only been surfacing for the last few months. Doesn't this suggest that the cause isn't with the number of jobs? > - With the above in mind, the hassle of archiving old jobs (with seldom > unarchiving) is a smaller hassle than actually performing a performance impact > analysis (to me, anyway). On the other hand, the webmaster is now on the hook for additional work to regularly archive old jobs and respond to requests for un-archival in a timely fashion. Bringing back an archived job is also disruptive to a running server as you have to trigger a global reload from disk (I believe). Something to think about...
I think it makes complete sense to have some way to find and remove jobs that are no longer needed. I see quite a few jobs labeled "nightly" that haven't run in 8-10 months. It is quite likely many of these are no longer needed and their owners have forgotten about them. In fact looking through mine, I can say you can safely delete this one: https://hudson.eclipse.org/hudson/job/orion-integration-deploy/ I was just worrying about automatically deleting jobs that hadn't run in 8 months because I can imagine that causing unnecessary hassle for job owners when they realize the job is gone at the moment they need it. It would probably be ok if there was plenty of advance warning sent to the job owners, and an easy way to say "don't delete this one"
(In reply to comment #9) > I think it makes complete sense to have some way to find and remove jobs that > are no longer needed. I see quite a few jobs labeled "nightly" that haven't run > in 8-10 months. It is quite likely many of these are no longer needed and their > owners have forgotten about them. Please do not delete MWE-nightly-Maintenance (11 Month) and MWE-Language-nightly-Maintenance (11 Month) they are still needed. (workspace was cleaned) Thanks.
(In reply to comment #8) > > - We've been running Hudson for years now, yet comments about sluggish UI > > response have only been surfacing for the last few months. > > Doesn't this suggest that the cause isn't with the number of jobs? Au contraire... After many years of adequate performance, we could now be hitting the limits of memory consumption that are allowed by our Hudson instance (either by configuration, max. memory on the box or application's efficiency in using allocated RAM). Winston has confirmed (via cross-project) that Hudson tends to keep a lot of information in RAM -- actually, much much more than I would have thought. "Hudson has a tendency to keep the info about entire artifacts of a build. For example, if a job contains JUnit test results then this info is kept in the memory. Currently Hudson is holding about 360 jobs with 6500 builds and 510,000 JUnit Case results in the memory. Average memory consumed by each JUnit Test case is about 200kb. So net memory consumption by JUnit results should be about 100 MB, however memory profiler reports 250 MB of memory occupied by Test cases." http://dev.eclipse.org/mhonarc/lists/cross-project-issues-dev/msg07316.html The bug still stands -- stale jobs that are no longer used consume valuable resources and decrease our overall happiness. I'm not suggesting we prune jobs systematically -- but we do need a mechanism to clean up.
Does it make a difference to purge old builds (rather than inactive jobs)? For Mylyn we have configured most jobs to only keep the last five builds with the exception of release builds which we tag and keep "forever".
It sounds like the bulk of the memory overhead is in keeping builds around as opposed to the job itself. It would be good to check with Winston re memory consumption of a disabled job with all builds deleted. Perhaps it isn't that high? I postulate that that is the best way to archive maintenance line build jobs of active projects. Perhaps a better focus would be in cutting down the number of builds that jobs keep around... 6500/360 = 18 ... I doubt projects really need more than five latest builds.
> I postulate that that is the best way to archive maintenance line build > jobs of active projects. Clearly, I have not properly articulated the intent of this bug. From comment 0: "Some jobs may no longer be needed -- perhaps the project has gone away." Those jobs that you have described above are neither "no longer needed" nor has the project "gone away". The jobs I want to eliminate are those jobs that a committer uses for testing stuff, then moves on once the testing is complete. These jobs become orphaned and will never be used again. Ever. Prime examples of such jobs are listed in comment 1, comment 2 and comment 9. We need a reasonable way of removing those jobs -- they are not used, they are not needed, they add no value, yet they add clutter and consume resources.
(In reply to comment #14) > > I postulate that that is the best way to archive maintenance line build > > jobs of active projects. > > Clearly, I have not properly articulated the intent of this bug. From comment > 0: "Some jobs may no longer be needed -- perhaps the project has gone away." > > Those jobs that you have described above are neither "no longer needed" nor has > the project "gone away". > > The jobs I want to eliminate are those jobs that a committer uses for testing > stuff, then moves on once the testing is complete. These jobs become orphaned > and will never be used again. Ever. > > Prime examples of such jobs are listed in comment 1, comment 2 and comment 9. > > We need a reasonable way of removing those jobs -- they are not used, they are > not needed, they add no value, yet they add clutter and consume resources. Denis, https://hudson.eclipse.org/hudson/job/xtend-head/ and https://hudson.eclipse.org/hudson/job/xtend-maintenance/ can be removed.
> https://hudson.eclipse.org/hudson/job/xtend-maintenance/ Gone. > https://hudson.eclipse.org/hudson/job/xtend-head/ Really? You used it last week, and it has xtend-head-test as a downstream project. > https://hudson.eclipse.org/hudson/job/orion-integration-deploy/ Deleted. > mylyn-heartbeat-connectors is no longer needed. Deleted. > eclipse-equinox-git-test-M > eclipse-equinox-git-test-N Also deleted.
> The jobs I want to eliminate are those jobs that a committer uses for testing > stuff, then moves on once the testing is complete. These jobs become orphaned > and will never be used again. Ever. If that's the extent of the cleanup, then I don't see anyone objecting to this.
(In reply to comment #16) > > https://hudson.eclipse.org/hudson/job/xtend-maintenance/ > Gone. > > > https://hudson.eclipse.org/hudson/job/xtend-head/ > Really? You used it last week, and it has xtend-head-test as a downstream > project. Yes, II moved the build back to Xtext-nightly-HEAD, and xtend-head-test is now triggered by Xtext-nightly-HEAD. https://hudson.eclipse.org/hudson/job/z_denis/ looks also orphaned :D
Ok, I've removed your job. > https://hudson.eclipse.org/hudson/job/z_denis/ looks also orphaned :D :D
Hi Denis, the following jobs (for which I am responsible) are not used anymore. You can remove them: emffacet-integration modisco-integration modisco-integration-0.9
> emffacet-integration > modisco-integration > modisco-integration-0.9 Physical garbage collection > Java garbage collection Keep 'em coming!
At this point I'm happy where this is. I usually send out reminders after each June release reminding committers to 'clean up'.. I'll mention Hudson in that too.
