Bugzilla – Full Text Bug Listing
| Summary: | Investigate deploying HIPP instances inside of Docker containers | ||
|---|---|---|---|
| Product: | Community | Reporter: | Thanh Ha <thanh.ha> |
| Component: | CI-Jenkins | Assignee: | Mikaël Barbero <mikael.barbero> |
| Status: | RESOLVED WONTFIX | QA Contact: | |
| Severity: | normal | ||
| Priority: | P3 | CC: | cdtdoug, denis.roy, fwaibel, jeick, mike.milinkovich, mknauer, pwebster, webmaster |
| Version: | unspecified | ||
| Target Milestone: | --- | ||
| Hardware: | PC | ||
| OS: | Linux | ||
| See Also: | https://bugs.eclipse.org/bugs/show_bug.cgi?id=375350 | ||
| Whiteboard: | stalebug | ||
|
Description
Thanh Ha
On the surface, I think this is a good idea. There are some things to consider. > It appears even the HIPP instances are still interfering with each other > sometimes and projects HIPPs can still affect other projects which are on > the same physical server. Unless you dedicate a physical machine to one HIPP, two dockers can still interfere with each other. If one decides to start thrashing its virtual hard drive, it will slow down others. Same thing with CPU resources. Of course, you could probably impose CPU core limits to each docker, but then you'll slow down those build jobs that use multiple cores responsibly. > I think it would be worth investigating deploying HIPP instances inside of > self contained containers would be beneficial as we could easily limit > resources to the containers. So how much RAM would you allocate to each container? If you allocate 8 GB, then a 64G server can only host 7 HIPP instances, which drives up the overall cost of HIPP. Lower RAM allocation also means increased potential for swapping, which taxes the disk I/O of the physical host (thus affecting other instances). Another thing: in past tests, we've found virtualization to cost between 5% and 25% performance; mostly CPU, but even network and disk I/O. Even more penalty on fully virtualized environments, such as Windows, which is why we decided to go bare iron with HIPP. We'd have to measure the performance impact of having multiple containers all running build jobs. Then there's security -- each container needs access to mounted filesystems. Network interfaces. So again, I'm not opposed to trying something new, but will it really resolve the issues? A build job that erratically consumes 17/24 CPU cores will still behave erratically, and a test suite that creates 32,000 files in /tmp will still be broken eventually. (In reply to Denis Roy from comment #1) > On the surface, I think this is a good idea. There are some things to > consider. > > > It appears even the HIPP instances are still interfering with each other > > sometimes and projects HIPPs can still affect other projects which are on > > the same physical server. > > Unless you dedicate a physical machine to one HIPP, two dockers can still > interfere with each other. If one decides to start thrashing its virtual > hard drive, it will slow down others. Same thing with CPU resources. > > Of course, you could probably impose CPU core limits to each docker, but > then you'll slow down those build jobs that use multiple cores responsibly. > This is true and I wasn't planning on adding CPU core limits to each container. I was thinking more in terms of RAM and giving each container it's own /tmp so that if a project does create too many files in /tmp it would only affect their separate /tmp directory and not a global shared /tmp. I guess I see this more as a chroot jail for a HIPP instance. They can still affect other projects but at least we minimize the ways it can interact with the other project sharing the same server. > > > > I think it would be worth investigating deploying HIPP instances inside of > > self contained containers would be beneficial as we could easily limit > > resources to the containers. > > So how much RAM would you allocate to each container? If you allocate 8 GB, > then a 64G server can only host 7 HIPP instances, which drives up the > overall cost of HIPP. Lower RAM allocation also means increased potential > for swapping, which taxes the disk I/O of the physical host (thus affecting > other instances). > I am not 100% sure but I am pretty sure Docker's Memory limits is actually just a maximum limit (and by default it does not apply any limits). It does not actually reserve that much memory for each container causing the other processes to not be able to use it but you are right if a project hits this limit we impose it will start swapping. My idea for this was to decide on a reasonable maximum that we expect a project would ever need so that the default Java algorithm of 1/64th of total ram would be a lower number but it's possible to just not set one too then ram limits would remain exactly like it is today. Just to note Docker users cgroups (http://en.wikipedia.org/wiki/Cgroups) to limit resources to the container. > Another thing: in past tests, we've found virtualization to cost between 5% > and 25% performance; mostly CPU, but even network and disk I/O. Even more > penalty on fully virtualized environments, such as Windows, which is why we > decided to go bare iron with HIPP. We'd have to measure the performance > impact of having multiple containers all running build jobs. > Docker images run very close to bare iron since it's not true virtualization like vmware is. It's kernel level virtualization so it's actually just launching a new process on the host system. There is a hit on performance though depending on what services you start in the container as it will have it's own Distro stack minus the kernel (It uses the same kernel as the host system). Definitely would be worth measuring the performance hit though. > Then there's security -- each container needs access to mounted filesystems. > Network interfaces. > It actually wouldn't need direct access to mounted file systems because in Docker you can just mount the host system's directories. As long as we make sure to create the docker internal user with the same UID as the genie.project UID on the host system we can set it up to have exactly the same access restrictions as if HIPP wasn't running inside a Docker container and we don't have to worry about giving Docker permissions to NFS directly. > So again, I'm not opposed to trying something new, but will it really > resolve the issues? A build job that erratically consumes 17/24 CPU cores > will still behave erratically, and a test suite that creates 32,000 files in > /tmp will still be broken eventually. Yes, but at least when this happens there's a self contained container we can first limit resources to reduce their affect against other instances on the same server then investigate more easily. HIPP in Docker instances seems like a great idea... food for EclipseCon discussions about this topic! (I'm sure you've seen our talk next week: https://www.eclipsecon.org/na2014/session/lightweight-software-labs-eclipse-rt-cloud) Interesting Jenkins plugin [1] which allows Jenkins to dynamically provision Linux slaves. I don't think it's available for Hudson though but maybe can be ported. [1] https://wiki.jenkins-ci.org/display/JENKINS/Docker+Plugin Just putting this on Mikael's plate. One way I've seen this deployed is via dynamic slaves in Jenkins. The project I've seen this deployed on has a VM image defined using the Jenkins jClouds plugin. So when their job starts it launches a new VM based on the image. The image contains access to a Docker daemon that they can pull and run docker containers from. Inside their Job definition they have a build timeout to ensure that the job will eventually stop if it runs to long. This prevents the VM with Docker from staying in service for more than a couple hours and the VM gets destroyed. Docker instances for running job is definitely something interesting, but I think this bug is more about dockerizing the HIPP instance itself, isn't it? I will have a look at it shortly, but my priority is currently to offer a way for projects to upgrade their HIPP (bug 422507) (In reply to Mikael Barbero from comment #7) > Docker instances for running job is definitely something interesting, but I > think this bug is more about dockerizing the HIPP instance itself, isn't it? You're correct. I guess my comment about allowing projects to run jobs in Docker could be another potential future improvement. For what it's worth... we used to have some (Jenkins) build servers and migrated all of them to Docker. Now every team has its own Jenkins-could-be-Hudson running inside a container. We maintain a "base" image where we have pre-installed and pre-configured the common parts, and then we have some project/teams specific images that use the base image as a basis. If you are interested in our experiences with this just ask... Docker is definitely the way forward, especially for jobs that have special environment requirements such as our CDT jobs that need toolchains installed. And we wouldn't be tied to SuSE. But is Hudson capable of this? Are we starting to run in cases where we'd rather be using Jenkins. I know I have some other examples, like the Xvnc recorder. BTW, apparently the Docker plugin isn't ported to Hudson. https://bugs.eclipse.org/bugs/show_bug.cgi?id=457997 This bug hasn't had any activity in quite some time. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. If you have further information on the current state of the bug, please add it. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. -- The automated Eclipse Genie. Closing as WONTFIX. Running [HJ]ipp inside Docker is not what we want.
Having [HJ]IPP launch a container at build-time, however, could be interesting.
> > Docker instances for running job is definitely something interesting, but I
> > think this bug is more about dockerizing the HIPP instance itself, isn't it?
>
> You're correct. I guess my comment about allowing projects to run jobs in
> Docker could be another potential future improvement.
|