Bug 539991

Comment 1 Mikaël Barbero 2018-10-10 15:21:52 EDT

I've added credentials so you can now ssh/scp to build.eclipse.org and put stuff in /home/data/httpd/download.eclipse.org 

Use the ssh agent plugin in your pipeline to inject the ssh bot credentials. Credential id is "project-storage.eclipse.org-bot-ssh"

steps {
    sshagent ( ['project-storage.eclipse.org-bot-ssh']) {
        sh '''
            ssh genie.wildwebdeveloper@build.eclipse.org ls -al /home/data/httpd/download.eclipse.org/wildwebdeveloper
        '''
    }
}

Note project-storage.eclipse.org does not exist yet. We eventually will use a dedicated host for accessing download.eclipse.org data. In the meantime, we ask projects to use build.eclipse.org.

Comment 2 Mickael Istria 2018-10-11 04:48:31 EDT

With
https://github.com/eclipse/wildwebdeveloper/blob/master/Jenkinsfile#L55

on
https://jenkins.eclipse.org/wildwebdeveloper/job/Wildwebdeveloper/job/master/8/console

I get
'''
[Wildwebdeveloper_master-ROJ4BPCU4Q5VTI7VBBSR4KWQVLVOSBZHJPOZ6FPJL3JEULICJSWQ] Running shell script
Executing shell script inside container [container] of pod [buildtestpod-t6pwt-sqrss]
+ ssh genie.wildwebdeveloper@build.eclipse.org rm -rf /home/data/httpd/download.eclipse.org/wildwebdeveloper/snapshots
No user exists for uid 1000100000
Executing command: "ssh-agent" "-k" 
'''

Did I miss anything?

Comment 3 Mickael Istria 2018-10-11 05:36:59 EDT

After error from previous comment, I understood that it could be an issue of sshagent not being able to run in the container I'm using. I've then removed the default container (to explicitly reference it only where it's useful:
https://github.com/eclipse/wildwebdeveloper/commit/eecd78b53b9d4c77f38f57b62de4756862841bd0 and https://github.com/eclipse/wildwebdeveloper/commit/a5b38aa354f4376ee88be2ca481a9c478912a9a1

and tried again: https://jenkins.eclipse.org/wildwebdeveloper/job/Wildwebdeveloper/job/master/11/console

And got a different error
"""
[Wildwebdeveloper_master-ROJ4BPCU4Q5VTI7VBBSR4KWQVLVOSBZHJPOZ6FPJL3JEULICJSWQ] Running shell script
+ ssh genie.wildwebdeveloper@build.eclipse.org rm -rf /home/data/httpd/download.eclipse.org/wildwebdeveloper/snapshots
Host key verification failed.

Executing shell script inside container [jnlp] of pod [buildtestpod-pgz15-jlhxp]
Executing command: "ssh-agent" "-k" 
exit
"""

It seems like overall, the coupling of SSH agent and Containers is way trickier than SSH on regular instances.
Would it be possible to disable the host key verification on server, or maybe just to get back to good old nfs?

Comment 4 Mikaël Barbero 2018-10-11 06:56:06 EDT

The initial issue is due to the fact we run containers using an arbitrarily assigned user ID (1000100000) in our OpenShift cluster. Most images are not ready to be run with arbitrary user ID (especially when != root), hence the issue.

ssh is particular. It looks for the the username of the current user in the /etc/passwd file. If no mapping exists, it fails with the error "No user exists for uid 1000100000
". To avoid that, you need to use a trick in your image. OpenShift publish guidelines with best practices (https://docs.openshift.com/container-platform/3.9/creating_images/guidelines.html) about how to create Docker images. More specifically, see the section about how to support running with arbitrary user ID (https://docs.openshift.com/container-platform/3.9/creating_images/guidelines.html#use-uid). 

The public default jnlp image we're using (https://github.com/eclipse-cbi/dockerfiles/blob/master/jenkins-jnlp-agent/3.25/Dockerfile) is already using such a trick (see https://github.com/eclipse-cbi/dockerfiles/blob/master/jenkins-jnlp-agent/3.25/Dockerfile#L4 and https://github.com/eclipse-cbi/dockerfiles/blob/master/scripts/uid_entrypoint.

Regarding the other issue, we know where it comes from. As you're using the default pod template, you're not inheriting the config maps volumes we're mounting in all the containers of the default pod template. We've have an existing config map for .ssh/known_hosts file. Just add a config map volume to your podtemplate (as shown here https://github.com/jenkinsci/kubernetes-plugin#container-configuration). The config map name is "known-hosts". 

I'll add a section to the wiki FAQ about new infra (https://wiki.eclipse.org/Jenkins#FAQ) for both issues shortly.

Finally, thank you for taking the time to report the issues you're facing. We're just in the beginning of the new infra, and we get that it's sometimes frustrating it does not work out of the box, or that it's just not as simple as one would desire. We do our best to eventually make it as simple as possible for everyone to use. We need your feedbacks, but please bear with us.

Comment 5 Mikaël Barbero 2018-10-11 07:20:22 EDT

(In reply to Mikaël Barbero from comment #4)
> I'll add a section to the wiki FAQ about new infra
> (https://wiki.eclipse.org/Jenkins#FAQ) for both issues shortly.

Done

https://wiki.eclipse.org/Jenkins#My_build_fails_with_.22No_user_exists_for_uid_1000100000.22.2C_what.27s_the_issue.3F

and

https://wiki.eclipse.org/Jenkins#My_build_fails_with_.22Host_key_verification_failed.22.2C_what_should_I_do.3F

Comment 6 Eclipse Genie 2018-10-11 07:34:51 EDT

GitHub Pull Request 80 created by [mickaelistria]
https://github.com/eclipse/wildwebdeveloper/pull/80

Comment 7 Mickael Istria 2018-10-12 05:23:09 EDT

Thanks to your guidance, that now works.
https://github.com/eclipse/wildwebdeveloper/blob/master/Jenkinsfile
The interesting part are the kubernetes configuration (in particular for the 'jnlp' container) and the Deploy stage with the sshagent block.
I would actually recommend this approach (a container for doing the actual build, another for doing the publication or other non-build steps) as the build environment can very much differ from what's necessary for the deploy environment. For instance, with WildWebDeveloper, the build container doesn't include SSH.

Comment 8 Mikaël Barbero 2018-10-12 16:28:21 EDT

I've added some documentation on the wiki https://wiki.eclipse.org/Jenkins#How_do_I_deploy_artifacts_to_download.eclipse.org.3F

I've also defined a very simple docker image (eclipsecbi/ssh-client https://github.com/eclipse-cbi/dockerfiles/blob/master/ssh-client/1.0/Dockerfile). This avoids having to know the gory details of the jnlp image (like args: ['\$(JENKINS_SECRET)', '\$(JENKINS_NAME)'] ...).