Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 542419

Summary:	All sprotty builds failing
Product:	Community	Reporter:	Jan Koehnlein <jan>
Component:	CI-Jenkins	Assignee:	CI Admin Inbox <ci.admin-inbox>
Status:	RESOLVED FIXED	QA Contact:
Severity:	normal
Priority:	P3	CC:	mikael.barbero, webmaster
Version:	unspecified
Target Milestone:	---
Hardware:	PC
OS:	Mac OS X
Whiteboard:

Description Jan Koehnlein

2018-12-05 03:33:47 EST

Currently all sprotty builds are failing with something like:

> exec failed: container_linux.go:247: starting container process caused  "process_linux.go:110: decoding init error from pipe caused \"read parent: connection reset by peer\"

Jenkins UI is pretty unresponsive as well.

There seem to be a severe network problem, as when I restart the build I get 

> java.io.IOException: It seems https://api.github.com is unreachable

(https://jenkins.eclipse.org/sprotty/job/sprotty-theia/job/master/14/console)

Comment 1 Mikaël Barbero

2018-12-05 04:45:45 EST

I've tweaked the default pipeline durability and restarted your instance. Please give it another try. Thanks.

Comment 2 Jan Koehnlein

2018-12-05 04:56:59 EST

Still same issue on https://jenkins.eclipse.org/sprotty/job/sprotty/job/master/23/console

Haven't tried the others.

Comment 3 Mikaël Barbero

2018-12-05 05:06:38 EST

It seems to be related to https://bugzilla.redhat.com/show_bug.cgi?id=1655214 (we've updated the cluster yesterday to patch a CVE). The issue seems to happen only with some images. While we investigate further, could you please try to run your build with the alpine variant of the node image?

Comment 4 Jan Koehnlein

2018-12-05 05:18:37 EST

Still the same https://jenkins.eclipse.org/sprotty/job/sprotty/job/master/26/console

Comment 5 Jan Koehnlein

2018-12-05 05:21:04 EST

node:8.14-alpine

Comment 6 Jan Koehnlein

2018-12-05 05:23:48 EST

(In reply to Jan Koehnlein from comment #5)
> node:8.14-alpine

same with node:8.12-alpine

Comment 7 Mikaël Barbero

2018-12-05 05:32:16 EST

The current workaround seems to be downgrade docker RPM to docker-1.13.1-75 (https://stackoverflow.com/a/53612348) but I would like to double check how to do that properly with Denis first. Will keep you posted in the afternoon. Thanks for your patience.

Comment 8 Jan Koehnlein

2018-12-05 05:33:02 EST

Thank you for the prompt action!

Comment 9 Jan Koehnlein

2018-12-06 02:47:02 EST

Any news on that?

Comment 10 Mikaël Barbero

2018-12-06 04:04:21 EST

I'm on it. Redhat has released a new build of docker. See https://bugzilla.redhat.com/show_bug.cgi?id=1655214

/begin quote
Cause: 
The docker-runc binary used by docker shipped in 7.6.1 Extras was built with an older version of golang which had a bug leading to a crash in certain FIPS modes.

Consequence: docker didn't work with some images.

Fix: a new docker package with docker-runc has been built with an updated golang dependency to take care of FIPS compliance.

Result: docker works as expected.
/end quote

I'm currently trying the new version on a single node and will do others once I confirmed it is working as expected.

Comment 11 Mikaël Barbero

2018-12-06 04:17:52 EST

The new docker version RPM is still in QA and not widely available. We will need to wait a couple of hours before we can actually upgrade the package.

Comment 12 Mikaël Barbero

2018-12-06 16:24:36 EST

New RPM has been released and rolled out on the cluster. 

Builds are now green https://jenkins.eclipse.org/sprotty/job/sprotty-theia/job/master/16/console

Comment 13 Jan Koehnlein

2018-12-08 11:13:09 EST

The yarn builds are working again, thank you.

But the Java part (sprotty-server) doesn't build again yet:
https://jenkins.eclipse.org/sprotty/job/sprotty-server/job/master/12/console

Looks like the new pod template overrides what we set in the Jenkinsfile

```
def kubernetes_config = """
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: openjdk
    image: openjdk:8
    tty: true
"""
```

at least openjdk is not found during build.

Comment 14 Mikaël Barbero

2018-12-10 09:52:00 EST

This is just a guess, but it may very well be a issue/bug in Jenkins. Could you please try to put a labellabel for your agent https://github.com/eclipse/sprotty-server/blob/master/Jenkinsfile#L14 that is different from the label in https://github.com/eclipse/sprotty/blob/master/Jenkinsfile#L14?

Comment 15 Jan Koehnlein

2018-12-12 03:42:27 EST

That was it. Thanks a lot for your help!