Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 322108

Summary: Xvfb leaks (grows?) memory on new build server
Product: Community Reporter: David Williams <david_williams>
Component: ServersAssignee: Eclipse Webmaster <webmaster>
Status: RESOLVED WORKSFORME QA Contact:
Severity: minor    
Priority: P3 CC: grant_gayed, pwebster
Version: unspecified   
Target Milestone: ---   
Hardware: PC   
OS: Windows 7   
See Also: https://bugs.eclipse.org/bugs/show_bug.cgi?id=314811
Whiteboard:
Attachments:
Description Flags
screen capture from Xvfb none

Description David Williams CLA 2010-08-09 04:01:31 EDT
As discussed in bug 302613, we need use the X Virtual Frame Buffer (Xvfb) to serve as a "fake" display, on systems with no display. This worked fine in the past, and as we moved to new server it was recommended I start my own, instead of using one root had set up for us (years and year ago). Fine worked great, but I went back after a few days to check my memory that it only takes 10-15m of memory. But it was at 200M! after a long weekend of tests and investigations, this seems to happen only on this new Suse 11 system. I'm using same scheme on my red hat and Ubuntu systems, and no problems there ... the cook along test after test, hovering around 10 to 15 m with no increases. 

So, what's the problem? I first tried several command line parameters to limit data, make sure the connection timeout was short (60 sec) and that the service would 'reset' itself, when all clients disconnected. but none seemed to make any difference. 

My current command looks something like 


# limit data space (ld) to 10m bytes. 
# limit stack space (ls) to 1m bytes.
# limit number of open files (lf) to 20
# set connection time out to 60 seconds
XVFBSCREEN=${BUILD_HOME}/tmp
mkdir -p  ${XVFBSCREEN}
/usr/bin/Xvfb :$DISPLAYNUMBER -screen 0 1600x1200x16 -fbdir "${XVFBSCREEN}" -fp "${XVFBFONTPATH}" -ld 10240 -ls 1024 -lf 20 -to 60 -reset  &
echo $! > ccxvfb.pid

My hope was to start this when starting our cruisecontrol server and just let it run, there in the background, and let all our tests use the same one, instead of stopping and starting for each test. And, I'm pretty sure it should run there as a steady service without many resources from system. 

I thought I should document the issue here, and maybe someone would have some ideas ... perhaps it is related to some other XServer settings on the new system? 

To make matters worse, I can not figure out how to see what version is running. It apparently has no "-version" command, and using 
rpm -qa | grep -i xvfb 
shows nothing ... as if it wasn't installed (I'm assuming installed as part of something else, since it is there, in /usr/bin/Xvfb

I searched the internet for possibilities, but couldn't find anything recent or relevant, but didn't know how to focus my search on specifics, especially since I don't know what version we are using.

I do get one odd error on startup, only on the SUSE OS machine ... not my others. The error message is 

(EE) config/hal: NewInputDeviceRequest failed (2)

Could the "leak" be related to that? Something not set up right? 
I know little about config/hal ... but, is there an easy way to say "allow wtpBuild to create new input device? (oh, maybe I'll try specifying "no keyboard). 

Anyway ... just documenting the problem ... maybe others will know something.
Comment 1 David Williams CLA 2010-08-09 04:13:00 EDT
(In reply to comment #0)


> 
> (EE) config/hal: NewInputDeviceRequest failed (2)
> 
> Could the "leak" be related to that? Something not set up right? 
> I know little about config/hal ... but, is there an easy way to say "allow
> wtpBuild to create new input device? (oh, maybe I'll try specifying "no
> keyboard). 
> 

Disabling the keyboard (-kb) didn't help, but I did run across one reference on the web that said (after explaining how normally started by 'init', etc :) 
... "The X server may also  be  started  directly  by  the  user,
     though  this  method  is usually reserved for testing and is
     not recommended for normal operation.   On  some  platforms,
     the user must have special permission to start the X server,
     often because access to certain devices (e.g. /dev/mouse) is
     restricted."

So ... maybe I (wtpBuild, that is) does need some special permissions to access "devices"?
Comment 2 David Williams CLA 2010-08-29 14:49:14 EDT
To give a small update, Following a tip from Paul Webster, I mimicked the way he does it for e4Build, and start a window manager after starting Xvfb. In particular, something similar to the following 2 lines of script: 

Xvfb :$DISPLAYNUMBER -screen 0 1600x1200x16  &

DISPLAY=:$DISPLAYNUMBER metacity --display=:$DISPLAYNUMBER --replace --sm-disable >/dev/null 2>&1 &

This seems to allow it to run without growing in memory. (Might still be growing a little ... but, not enough to be significant). 

I _think_ I even tried other window managers besides metacity (with out it helping the memory problem) but don't recall (and didn't really know what I was doing anyway) ... just saying it _might_ be related or specific to particular window managers. 

The fact that I've never had to do this before (that is, start my own window manager) implies that maybe before (on other machines) a window manager would be started automatically when needed? Some default? And the fact that the default window manager is not starting, as indicated by bug 314811, is perhaps what causes the apparent "memory leak"? 

I'll leave this open for now ... if the "default window manager" issue is ever solved, maybe I'll try my procedure again to see if that fixes memory leak issue .... but, this is not currently a problem for WTP. Feel free to close if its just noise for you and/or anyone thinks Xvfb processes _should_ always need a window manager started for just for it.
Comment 3 Denis Roy CLA 2010-09-22 10:00:41 EDT
I'll be interested in seeing how this pans out on the new(again) build server.
Comment 4 David Williams CLA 2010-09-22 18:53:50 EDT
(In reply to comment #3)
> I'll be interested in seeing how this pans out on the new(again) build server.

still grows. At least for me. :( 

Without starting my own window manager, anyway. 

Not sure why e4Build's memory isn't growing ... unless they just haven't done any UI tests yet (it does require actually using it ... doesn't grow just all by itself).

We both started off at about the 13 M usage level, but after several scores of UI tests, wtpBuild's use has grown to 52 M. I don't see "metacity" being used by e4Build (that is, I don't see the process running). Not sure what else could be different. 

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
wtpBuild  8827  0.0  0.2 106888 52196 pts/5    S    12:13   0:12 Xvfb :1001 -screen 0 1600x1200x24 -fbdir /shared/webtools/tmp -fp /usr/share/fonts/misc/
e4Build  22498  0.0  0.0  68168 13324 ?        S    Sep21   0:00 Xvfb :8 -screen 0 1280x1024x24 -auth auth.cfg
Comment 5 David Williams CLA 2010-10-14 20:17:44 EDT
Maybe not directly related, but while running our Eclipse unit tests, we sometimes get this message in our logs: 

Xlib:  extension "RANDR" missing on display "127.0.0.1:1001.0".


I wonder if this is an indication that something isn't installed or configured quite right for Xorg (X Windows?) 

I tried searching for that message, suse and xfvb but couldn't find anything simple, that I could understand. 

Just thought I'd post the info here, in case it means something to someone.
Comment 6 Denis Roy CLA 2010-10-14 20:22:17 EDT
> Xlib:  extension "RANDR" missing on display "127.0.0.1:1001.0".

http://en.wikipedia.org/wiki/RandR

I'd think this is unrelated to a memory leak, but thanks for posting the info.
Comment 7 David Williams CLA 2010-10-14 20:25:01 EDT
Grant, I'm adding you as CC on this bug, since you've always been so helpful about "Display" related things in the past. Anything in this bug sound familiar? Growing memory use of Xvfb? Does Xvfb normally require its own Windows Manager? Is RAndR important to Eclipse or XWindows? 

My intuition is this bug is not directly related to Eclipse, per se ... but would be nice to be reassured our "test environment" was simulating a Display correctly.
Comment 8 David Williams CLA 2010-10-14 22:13:25 EDT
Created attachment 180922 [details]
screen capture from Xvfb

Here's an X windows dump of the Xvfb_screen0 file that Xvfb uses to store "the screen" (so copying it, should be same as a screen capture) ... captured during one of our tests. 

It is the nastiest thing I've every seen. :) 
sure doesn't look like 1600x1200x24
may be even 8 bit color? 
Though there must be a 101 things I don't understand about X Windows and Xvfb, so not sure if its significant ... I just thought it was interesting. (On some of my "local" test machines ... the images I've captured in the past look pretty normal). 
 
[update: the xwd attachment was 7 meg which bugzilla (understandably) thought was too big, so I "saved as" a PNG file ... and there was nothing "lost" in the translation ... the original was just as ugly :)
Comment 9 David Williams CLA 2010-10-14 22:22:55 EDT
(In reply to comment #8)

> sure doesn't look like 1600x1200x24
> may be even 8 bit color? 

I've a theory already ... there is no 1600x1200 resolution screen defined in 
/etc/X11/xorg.conf 
so I bet somehow X Windows picks some fallback values ... and does end up with 8 bit color depth. 
I'll try using 
1280x1024x24
which I do see defined in Screen section

Section "Screen"
  DefaultDepth 16
  SubSection "Display"
    Depth      15
    Modes      "1280x1024" "1152x864" "1024x768" "800x600"
  EndSubSection
  SubSection "Display"
    Depth      16
    Modes      "1280x1024" "1152x864" "1024x768" "800x600"
  EndSubSection
  SubSection "Display"
    Depth      24
    Modes      "1280x1024" "1152x864" "1024x768" "800x600"
  EndSubSection
  SubSection "Display"
    Depth      8
    Modes      "1280x1024" "1152x864" "1024x768" "800x600"
  EndSubSection
  Device       "Device[0]"
  Identifier   "Screen[0]"
  Monitor      "Monitor[0]"
EndSection
Comment 10 David Williams CLA 2010-10-15 02:23:34 EDT
> I'll try using 
> 1280x1024x24

Didn't help. Must be something else causing the low color capture (or, a more complicated version of setting options).
Comment 11 David Williams CLA 2010-10-20 02:24:52 EDT
FWIW, I can connect the the Xvfb session (using x11vnc) and view it on a (real) display and the image/colors all looked pretty normal (at least, when using the hextile encoding algorithm). So I don't think the poor "screen shots" from Xvfb file copy are an indication of much. Some limit to exactly how and when copied I guess.  

Using x11vnc and a vncviewer is kind of interesting is that more "diagnostic" information about the xsession is displayed. The only very odd thing, though, was that on build.eclipse.org, the Xvfb reports that it has 32 mouse buttons! (normally, at least on my test systems, it reports 3 mouse buttons). 

I'm changed to "minor" as the current scheme works for what we need, and its just an interesting mystery at this point. It may not be that unusual to "require" a window manager since (on both production and test environments) when I do not use one, there is a little bit of the eclipse screen "lost" off the top of the display ... specifically, the title bar is not displayed ... it appears "off screen" at the top, but this means there is no way to grab it to move, minimize, restore, etc. 
And that sounds like exactly the kind of thing a window manager would manage. So, while I've not needed in past and not needed on other systems ... it must be some very subtle distinction. 

But, as I said ... currently is working ... so is just an interesting puzzle at this point.
Comment 12 Denis Roy CLA 2011-08-12 14:30:28 EDT
> But, as I said ... currently is working ... so is just an interesting puzzle at
> this point.

Works for me too?