Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 571640

Summary: JDT Gerrit tests failing with "Cannot run program lsof: error=2"
Product: [Eclipse Project] JDT Reporter: Jay Arthanareeswaran <jarthana>
Component: CoreAssignee: Andrey Loskutov <loskutov>
Status: VERIFIED FIXED QA Contact:
Severity: major    
Priority: P3 CC: loskutov, manoj.palat, sravankumarl, stephan.herrmann, Vikas.Chandra
Version: 4.19   
Target Milestone: 4.20 M1   
Hardware: All   
OS: All   
See Also: https://git.eclipse.org/r/c/jdt/eclipse.jdt.core/+/177135
https://git.eclipse.org/r/c/jdt/eclipse.jdt.core/+/176825
https://bugs.eclipse.org/bugs/show_bug.cgi?id=543506
https://git.eclipse.org/c/jdt/eclipse.jdt.core.git/commit/?id=d2775025c44cae53dc1e67d02b419c4e6c80a72d
https://git.eclipse.org/r/c/jdt/eclipse.jdt.core/+/178002
https://git.eclipse.org/c/jdt/eclipse.jdt.core.git/commit/?id=2af7f7fe05757222d516240b39d11bc70398f1d3
https://git.eclipse.org/r/c/jdt/eclipse.jdt.core/+/178320
https://git.eclipse.org/c/jdt/eclipse.jdt.core.git/commit/?id=09bb69aa67a0339779e60595227112c004cfd25d
Whiteboard:
Bug Depends on:    
Bug Blocks: 571351    

Description Jay Arthanareeswaran CLA 2021-03-03 05:01:42 EST
The log has this:

java.io.IOException: Cannot run program "lsof": error=2, No such file or directory
	at org.eclipse.jdt.core.tests.builder.LeakTestsBefore9.testLeaksOnCleanBuild(LeakTestsBefore9.java:36)
Caused by: java.io.IOException: error=2, No such file or directory
	at org.eclipse.jdt.core.tests.builder.LeakTestsBefore9.testLeaksOnCleanBuild(LeakTestsBefore9.java:36)

This appears nothing related to the tests at all and most likely an infra issue. Something is missing in the OS, perhaps.

We had a similar problem (bug 560014) in the past.
Comment 1 Jay Arthanareeswaran CLA 2021-03-04 01:27:15 EST
Any chance of a resolution here? This is blocking our RC2 work.
Comment 2 Eclipse Genie CLA 2021-03-04 02:06:28 EST
New Gerrit change created: https://git.eclipse.org/r/c/jdt/eclipse.jdt.core/+/177135
Comment 3 Frederic Gurr CLA 2021-03-04 04:11:07 EST
Sorry, no resolution yet.

In a minimal build environment (centos-7) lsof can be found:
https://ci.eclipse.org/jdt/job/webmaster-test/1/console

Can you provide a minimal test case that reproduces the error?
Comment 4 Eclipse Genie CLA 2021-03-04 07:31:39 EST
New Gerrit change created: https://git.eclipse.org/r/c/jdt/eclipse.jdt.core/+/176825
Comment 5 Stephan Herrmann CLA 2021-03-06 14:43:02 EST
Recent test failure https://ci.eclipse.org/jdt/job/eclipse.jdt.core-Gerrit/4152/testReport/junit/org.eclipse.jdt.core.tests.builder/BuilderTests/testLeaksOnCleanBuild/ reads:

junit.framework.AssertionFailedError: Failed to read opened file descriptors
	at org.eclipse.jdt.core.tests.builder.LeakTestsBefore9.testLeaksOnCleanBuild(LeakTestsBefore9.java:36)

This indicates the the lsof invocation returned an empty string.

@Andrey: in bug 543506  you made the call with this command line
"lsof -F n -a -p " + pid + " / -b "

On my ubuntu system this causes: 
lsof: status error on -b: No such file or directory

Reason: there doesn't seem to be syntax for '/' in the given way. If I remove that char, no error occurs.

What is '/' intended to do?
Comment 6 Andrey Loskutov CLA 2021-03-06 18:24:03 EST
(In reply to Stephan Herrmann from comment #5)
> What is '/' intended to do?

It was supposed to be the root of the file system. That was used to work on RHEL 7.4, I can check it again.
Comment 7 Stephan Herrmann CLA 2021-03-06 19:46:29 EST
(In reply to Andrey Loskutov from comment #6)
> (In reply to Stephan Herrmann from comment #5)
> > What is '/' intended to do?
> 
> It was supposed to be the root of the file system. That was used to work on
> RHEL 7.4, I can check it again.

Then it should probably be the last argument after -b ? the "names" in:

lsof  [  -?abChlnNOPRtUvVX  ] [ -A A ] [ -c c ] [ +c c ] [ +|-d d ] [ +|-D D ] [ +|-e s ] [ +|-E ] [ +|-f [cfgGn] ] [ -F [f] ] [ -g [s] ] [ -i
       [i] ] [ -k k ] [ -K k ] [ +|-L [l] ] [ +|-m m ] [ +|-M ] [ -o [o] ] [ -p s ] [ +|-r [t[m<fmt>]] ] [ -s [p:s] ] [ -S [t] ] [ -T [t] ] [ -u s  ]
       [ +|-w ] [ -x [fl] ] [ -z [z] ] [ -Z [Z] ] [ -- ] [names]

right?
Comment 9 Andrey Loskutov CLA 2021-03-08 05:38:56 EST
(In reply to Andrey Loskutov from comment #6)
> (In reply to Stephan Herrmann from comment #5)
> > What is '/' intended to do?
> 
> It was supposed to be the root of the file system. That was used to work on
> RHEL 7.4, I can check it again.

I must correct me: this was used to select only regular (physical) files/directories.

Something like that was mentioned here: https://serverfault.com/questions/106398/lsof-restrict-output-to-physical-files-only-how

I guess that was some undocumented hack that is may be "fixed" in latest lsof versions, and still there on RHEL 7.4 where I'm working on.

So if that is not working on gerrit anymore, we can remove "/".
Comment 10 Mikaël Barbero CLA 2021-03-15 08:31:26 EDT
Can this one be closed or moved to JDT?
Comment 11 Mikaël Barbero CLA 2021-03-17 10:48:37 EDT
Closing as as INVALID as it is not an issue with Jenkins. Feel free to reopen/move if needed.
Comment 12 Stephan Herrmann CLA 2021-03-18 08:52:03 EDT
(In reply to Mikaël Barbero from comment #11)
> Closing as as INVALID as it is not an issue with Jenkins. Feel free to
> reopen/move if needed.

Over to JDT
Comment 13 Eclipse Genie CLA 2021-03-18 08:54:14 EDT
New Gerrit change created: https://git.eclipse.org/r/c/jdt/eclipse.jdt.core/+/178002
Comment 14 Stephan Herrmann CLA 2021-03-18 08:56:14 EDT
(In reply to Andrey Loskutov from comment #9)
> So if that is not working on gerrit anymore, we can remove "/".

"we" is a dangerous word in team organization, because that guy rarely creates any commits whatsoever. Ever heard of diffusion of responsibility? ;p

(In reply to Eclipse Genie from comment #13)
> New Gerrit change created:
> https://git.eclipse.org/r/c/jdt/eclipse.jdt.core/+/178002

Trying to prevent the death of relevant tests.
Comment 15 Andrey Loskutov CLA 2021-03-18 10:38:28 EDT
*** Bug 562681 has been marked as a duplicate of this bug. ***
Comment 16 Andrey Loskutov CLA 2021-03-18 10:39:53 EDT
(In reply to Stephan Herrmann from comment #14)
> (In reply to Andrey Loskutov from comment #9)
> > So if that is not working on gerrit anymore, we can remove "/".
> 
> "we" is a dangerous word in team organization, because that guy rarely
> creates any commits whatsoever. Ever heard of diffusion of responsibility? ;p
> 
> (In reply to Eclipse Genie from comment #13)
> > New Gerrit change created:
> > https://git.eclipse.org/r/c/jdt/eclipse.jdt.core/+/178002
> 
> Trying to prevent the death of relevant tests.

Thanks Stephan, I'm just not able to find free time for anything, too busy with my day work.
Comment 18 Stephan Herrmann CLA 2021-03-19 09:06:18 EDT
(In reply to Eclipse Genie from comment #17)
> Gerrit change https://git.eclipse.org/r/c/jdt/eclipse.jdt.core/+/178002 was
> merged to [master].
> Commit:
> http://git.eclipse.org/c/jdt/eclipse.jdt.core.git/commit/
> ?id=2af7f7fe05757222d516240b39d11bc70398f1d3

This should do.
Comment 19 Andrey Loskutov CLA 2021-03-24 09:59:30 EDT
Unfortunately this causes DNF in jdt.core.tests.builder on Linux

https://download.eclipse.org/eclipse/downloads/drops4/I20210323-1240/testResults.php

https://download.eclipse.org/eclipse/downloads/drops4/I20210323-1240/testresults/ep420I-unit-cen64-gtk3-java11_linux.gtk.x86_64_11/org.eclipse.jdt.core.tests.builder.BuilderTests.txt

java.lang.Exception: ThreadDump for thread "DumpStackTracesTimer"
	at java.base@11.0.10/java.lang.Thread.dumpThreads(Native Method)
	at java.base@11.0.10/java.lang.Thread.getAllStackTraces(Thread.java:1658)
	at org.eclipse.ui.testing.dumps.TimeoutDumpTimer.dumpStackTraces(TimeoutDumpTimer.java:238)
	at org.eclipse.ui.testing.dumps.TimeoutDumpTimer.logStackTraces(TimeoutDumpTimer.java:224)
	at org.eclipse.ui.testing.dumps.TimeoutDumpTimer.dump(TimeoutDumpTimer.java:161)
	at org.eclipse.ui.testing.dumps.TimeoutDumpTimer.run(TimeoutDumpTimer.java:138)
	at java.base@11.0.10/java.util.TimerThread.mainLoop(Timer.java:556)
	at java.base@11.0.10/java.util.TimerThread.run(Timer.java:506)
java.lang.Exception: ThreadDump for thread "Worker-1"
	at java.base@11.0.10/java.lang.Object.wait(Native Method)
	at org.eclipse.core.internal.jobs.WorkerPool.sleep(WorkerPool.java:200)
	at org.eclipse.core.internal.jobs.WorkerPool.startJob(WorkerPool.java:242)
	at org.eclipse.core.internal.jobs.Worker.run(Worker.java:58)
java.lang.Exception: ThreadDump for thread "main"
	at java.base@11.0.10/java.io.FileInputStream.readBytes(Native Method)
	at java.base@11.0.10/java.io.FileInputStream.read(FileInputStream.java:279)
	at java.base@11.0.10/java.io.BufferedInputStream.read1(BufferedInputStream.java:290)
	at java.base@11.0.10/java.io.BufferedInputStream.read(BufferedInputStream.java:351)
	at java.base@11.0.10/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
	at java.base@11.0.10/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
	at java.base@11.0.10/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
	at java.base@11.0.10/java.io.InputStreamReader.read(InputStreamReader.java:181)
	at java.base@11.0.10/java.io.BufferedReader.fill(BufferedReader.java:161)
	at java.base@11.0.10/java.io.BufferedReader.readLine(BufferedReader.java:326)
	at java.base@11.0.10/java.io.BufferedReader.readLine(BufferedReader.java:392)
	at org.eclipse.jdt.core.tests.builder.AbstractLeakTest.readLsofLines(AbstractLeakTest.java:191)
	at org.eclipse.jdt.core.tests.builder.AbstractLeakTest.getOpenDescriptors(AbstractLeakTest.java:174)
	at org.eclipse.jdt.core.tests.builder.AbstractLeakTest.checkOpenDescriptors(AbstractLeakTest.java:160)
	at org.eclipse.jdt.core.tests.builder.AbstractLeakTest.assertNotLeaked(AbstractLeakTest.java:135)
	at org.eclipse.jdt.core.tests.builder.AbstractLeakTest.internalTestUsedLibraryLeaks(AbstractLeakTest.java:83)
	at org.eclipse.jdt.core.tests.builder.AbstractLeakTest.testLeaksOnCleanBuild(AbstractLeakTest.java:61)
	at org.eclipse.jdt.core.tests.builder.LeakTestsBefore9.testLeaksOnCleanBuild(LeakTestsBefore9.java:36)
	at java.base@11.0.10/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Comment 20 Eclipse Genie CLA 2021-03-24 10:52:49 EDT
New Gerrit change created: https://git.eclipse.org/r/c/jdt/eclipse.jdt.core/+/178320
Comment 22 Andrey Loskutov CLA 2021-03-25 03:42:46 EDT
Works in I20210324-1800
Comment 23 Vikas Chandra CLA 2021-04-06 03:25:46 EDT
verified based on last comment.