Bugzilla – Full Text Bug Listing
| Summary: | Sapphire HIPP is unable to run builds | ||
|---|---|---|---|
| Product: | Community | Reporter: | Konstantin Komissarchik <konstantin> |
| Component: | CI-Jenkins | Assignee: | CI Admin Inbox <ci.admin-inbox> |
| Status: | RESOLVED FIXED | QA Contact: | |
| Severity: | blocker | ||
| Priority: | P3 | CC: | david_williams, denis.roy, mikael.barbero, webmaster |
| Version: | unspecified | ||
| Target Milestone: | --- | ||
| Hardware: | All | ||
| OS: | All | ||
| Whiteboard: | |||
|
Description
Konstantin Komissarchik
The upgrade eventually completed, but all builds are failing with a message like this: /jobs/genie.sapphire/10.0.x/workspace/releng/ant-library/library.xml:402: Could not create tempfile in Note that there is nothing after "in" I saw this happening before the upgrade and thought a restart/upgrade would cure it. Apparently not. The failure seems to be originating from within an implementation of an Ant task, so I don't have any further info. Has something changed recently related to HIPP's temp folder? I'm not aware of any changes and the permissions on both /tmp/genie.sapphire and /jobs/genie.sapphire look ok. Mikael, any suggestions? -M. Any chance that the instance is out of disk space or some such? Disk space seems fine right now. It's possible that if lots of other jobs were running we'd run out, but in that case I would expect more projects to let us know. -M. The issue reproduces consistently and on all jobs on this HIPP instance. We need to keep Sapphire builds working again soon. Could we try with a clean HIPP instance with jobs copied over? In the Hudson configuration (https://hudson.eclipse.org/sapphire/configure) all of the ARGS and OPTS refer to -Djava.io.tmpdir=$JAVA_TMPDIR For fun I replaced that with: -Djava.io.tmpdir=/tmp and the build seems to run just fine. https://hudson.eclipse.org/sapphire/job/10.0.x/68/console Using /tmp will almost guarantee problems down the road so we shouldn't leave it there. I'll see if we can use something else. Interesting. I wonder what would happen if we don't set java.io.tmpdir property? Is there something about these instances that makes Java's default behavior not suitable? The HIPP instances all share the same underlying OS, and many Java apps assume they own the box, so you'd have filename clashes, or someone else's build deleting your tempfiles while your build is running, etc. Matt, is the deploy script supposed to establish what the correct value of $JAVA_TMPDIR should be? (In reply to Konstantin Komissarchik from comment #7) > Interesting. I wonder what would happen if we don't set java.io.tmpdir > property? Is there something about these instances that makes Java's default > behavior not suitable? [After writing the long version, I see Denis already gave the short version ... but, I'll post my long version since it has a few additional details.] = = = (In reply to Konstantin Komissarchik from comment #7) > Interesting. I wonder what would happen if we don't set java.io.tmpdir > property? Is there something about these instances that makes Java's default > behavior not suitable? In the past, there has been issues using /tmp, I think for several reasons. A. Maven/Tycho builds generate huge "/tmp" directories. (admittedly, that may not concern you, since you still use PDE). B. The infrastructure typically has /tmp on a non-networked file system, though where the rest of the build is done, ${WORKSPACE} is NFS. I could be wrong, but I think perhaps /tmp on the virtual machines, is /tmp on the host? I think this led to some odd timing issues and "collisions" on /tmp. The best place to put "tmp" is ${WORKSPACE}/tmp That way, it is specific to each of your jobs. [And no one else's running on the same HIPP host machine, if my guess is correct.] Plus, when you do a "clean workspace" you know you are also cleaning anything "cached" in "tmp". As "evidence" of this importance, in recent versions of Hudson, with the "maven3 build task" they have an options to "use private tmp" and "use private maven repo" which automatically puts them in ${WORKSPACE}/tmp. and ${WORKSPACE}/.m2 (I guess, on that last one). And, of course, for Java to use any "tmp" directory, it must exist before Java (or, Ant, etc.) tries to use it. It is not automatically created by Java (for some weird reason :). > The best place to put "tmp" is ${WORKSPACE}/tmp
> That way, it is specific to each of your jobs. [And no one else's running on
> the same HIPP host machine, if my guess is correct.]
>
> Plus, when you do a "clean workspace" you know you are also cleaning
> anything "cached" in "tmp".
That is great advice, David, thank you. Does the temp directory then need to be set at the job level? How do we set this up?
(In reply to Denis Roy from comment #10) > > The best place to put "tmp" is ${WORKSPACE}/tmp > > That way, it is specific to each of your jobs. [And no one else's running on > > the same HIPP host machine, if my guess is correct.] > > > > Plus, when you do a "clean workspace" you know you are also cleaning > > anything "cached" in "tmp". > > > That is great advice, David, thank you. Does the temp directory then need to > be set at the job level? How do we set this up? Fundamentally, I do think it needs to be set at the job level. I thought this was "well documented" somewhere in the Wiki system, but I could not find it. 3 points: 1. I do not think the infrastructure can do it since the directory has to be created before java can use it. (And, in theory, some may want it in a slightly different place (such as if they have a custom 'clean' job, or several build steps, each needing a unique tmp directory.). And, besides, only Hudson knows where ${WORKSPACE} is. 2. I have never heard of $JAVA_TMPDIR but am nearly certain there is no standard environment variable (that covers all platforms). See http://stackoverflow.com/questions/1924136/environment-variable-to-control-java-io-tmpdir My guess is at some point, for some setups, JAVA_TMPDIR was defined on the HIPP instance, but, I do not know what it was defined to, and obviously not set on all HIPP instances. 3. Where the infrastructure CAN do something is to set up the HIPP instances so that the "private tmp" and "private maven repo" are "checked" by default. In bug 480803 I suggest this for our 'shared instance' and Platform HIPP. BUT, this only helps those that use the "Maven 3 plugin" (the maven embedded with Hudson). It is under "system setup" once that plugin is installed. Other than that, I'd suggest just documenting it well. But for sure do not want to set it to the empty string. :) I am still curious as to what happened that made the working configuration all of a sudden stop working. I am guessing that something is no longer setting JAVA_TMPDIR environment variable. I cannot recall a single time that I've seen a temp folder conflict. Per HIPP instance temp folders seems like an appropriate level of isolation unless you happen to know that some jobs on a particular instance misbehave in their temp folder usage. Then you can isolate just the problematic jobs with a per job setting. I did implement the per job temp folder approach as I wanted to get Sapphire builds back in order. Here is what I did: 1. Removed ANT_ARGS, ANT_OPTS, JAVA_ARGS, JVM_OPTS definitions from Hudson config. If temp folder is specified in these, it overrides the system property passed directly to the build. 2. Added java.io.tmpdir=$WORKSPACE/temp to the Properties block of the existing Invoke Ant build step in all of Sapphire build jobs. After some experimentation, I found that the build works without an explicit step of creating the temp folder ahead of time. (In reply to Konstantin Komissarchik from comment #12) > I cannot recall a single time that I've seen a temp folder conflict. It is probably more an issue with Maven/Tycho builds since I believe they use /tmp a lot -- at least used to? > Per > HIPP instance temp folders seems like an appropriate level of isolation > unless you happen to know that some jobs on a particular instance misbehave > in their temp folder usage. Then you can isolate just the problematic jobs > with a per job setting. I *think* that each HIPP Instance does not have their own /tmp folder. That is "shared" by all the HIPP instances running on the same host (hipp1 in your case, per https://hudson.eclipse.org/ ). But, honestly, have not checked. > After some experimentation, I found that the build works without an explicit > step of creating the temp folder ahead of time. And, sorry to ask the obvious, but was anything written to your /temp folder? Your build may not use it. Oh, but I guess you are saying it was created anyway, even if empty. Maybe things have changed with newer VMs? >> After some experimentation, I found that the build works without an explicit >> step of creating the temp folder ahead of time. > > And, sorry to ask the obvious, but was anything written to your /temp folder? > Your build may not use it. Oh, but I guess you are saying it was created > anyway, even if empty. Maybe things have changed with newer VMs? My build system doesn't explicitly use the temp folder, but various Ant tasks appear to leverage Java's File.createTempFile() facility, which was failing when temp folder was set to an empty string. Since the build is not failing any more and since all my build jobs clear the workspace before each build, I assume that Java is able to create temp folder as needed. I am using Java 8. This problem may exist in older versions. (In reply to Konstantin Komissarchik from comment #14) > This problem may exist in older versions. It may be that ant "fixes" the problem too. Anyway ... glad you are up and running again. And, I too am curious how and why -Djava.io.tmpdir=$JAVA_TMPDIR got introduced. My itself, it does not seem to make any sense. Even if $JAVA_TMPDIR "used to" be set to something and now isn't, I would be curious as to what it was set to. And, really, just curious. (In reply to David Williams from comment #15) > And, I too am curious how and why > -Djava.io.tmpdir=$JAVA_TMPDIR > got introduced. My itself, it does not seem to make any sense. > Even if $JAVA_TMPDIR "used to" be set to something and now isn't, I would be > curious as to what it was set to. And, really, just curious. It's been with us since we started creating HIPPs and is supposed to be set by the startup scripts to /tmp/projecthippuser. As I recall it was to try and separate the temp space used by the differing instances running on any given host. Perhaps there's something with the 3.3 upgrade(or a system patch) that's causing it to be empty(or ignored) now. -M. |