Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 547626

Summary: hipp3 JAVA_TOOL_OPTIONS assumes java newer than 1.7?
Product: Community Reporter: Achim Kraus <achim.kraus>
Component: CI-JenkinsAssignee: CI Admin Inbox <ci.admin-inbox>
Status: CLOSED FIXED QA Contact:
Severity: normal    
Priority: P3 CC: frederic.gurr, mikael.barbero, sbernard, webmaster
Version: unspecified   
Target Milestone: ---   
Hardware: PC   
OS: Windows 10   
Whiteboard:

Description Achim Kraus CLA 2019-05-24 08:56:51 EDT
Recently I receive always a e-Mail, reporting that a californium build failed (see below). Interestingly the build is shown as successful!

https://ci.eclipse.org/californium/job/master/312/

Maybe the cause of that is:

JAVA_TOOL_OPTIONS: -XX:+IgnoreUnrecognizedVMOptions -XX:+UseContainerSupport -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap Unrecognized VM option 'UseContainerSupport'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

Seems, this complains about
Unrecognized VM option 'UseContainerSupport'
even if configured to ignore it.



--------------------------------------------
See <https://ci-staging.eclipse.org/californium/job/master/312/display/redirect?page=changes>

Changes:

[achim.kraus] Update version references to 2.0.0-M15.

------------------------------------------
Started by an SCM change
Agent default-agent-zl65f is provisioned from template Kubernetes Pod Template Agent specification [Kubernetes Pod Template] (null): 
* [jnlp] eclipsecbijenkins/jenkins-agent:latest@sha256:13bbe1abbec678a31b8c6df906e6abb24689bf6e3b9f7caaadeaf2850a32d436(resourceRequestCpu: 1250m, resourceRequestMemory: 2560Mi, resourceLimitCpu: 2375m, resourceLimitMemory: 2560Mi)
yaml:
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: jnlp
    volumeMounts:
    - name: settings-xml
      mountPath: /home/jenkins/.m2/settings.xml
      subPath: settings.xml
      readOnly: true
    - name: toolchains-xml
      mountPath: /home/jenkins/.m2/toolchains.xml
      subPath: toolchains.xml
      readOnly: true
    - name: settings-security-xml
      mountPath: /home/jenkins/.m2/settings-security.xml
      subPath: settings-security.xml
      readOnly: true
  volumes:
  - name: settings-xml
    secret:
      secretName: m2-secret-dir
      items:
      - key: settings.xml
        path: settings.xml
  - name: toolchains-xml
    configMap:
      name: m2-dir
      items:
      - key: toolchains.xml
        path: toolchains.xml
  - name: settings-security-xml
    secret:
      secretName: m2-secret-dir
      items:
      - key: settings-security.xml
        path: settings-security.xml


Building remotely on default-agent-zl65f in workspace <https://ci-staging.eclipse.org/californium/job/master/ws/>
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/eclipse/californium.git
 > /usr/bin/git init <https://ci-staging.eclipse.org/californium/job/master/ws/> # timeout=10 Fetching upstream changes from https://github.com/eclipse/californium.git
 > /usr/bin/git --version # timeout=10
 > /usr/bin/git fetch --tags --progress https://github.com/eclipse/californium.git +refs/heads/*:refs/remotes/origin/*
 > /usr/bin/git config remote.origin.url https://github.com/eclipse/californium.git # timeout=10  > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10  > /usr/bin/git config remote.origin.url https://github.com/eclipse/californium.git # timeout=10 Fetching upstream changes from https://github.com/eclipse/californium.git
 > /usr/bin/git fetch --tags --progress https://github.com/eclipse/californium.git +refs/heads/*:refs/remotes/origin/*
 > /usr/bin/git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 5e2e8d984cf54650f7184fa015d119aa44b7501f (origin/master)  > /usr/bin/git config core.sparsecheckout # timeout=10  > /usr/bin/git checkout -f 5e2e8d984cf54650f7184fa015d119aa44b7501f
Commit message: "Update version references to 2.0.0-M15."
 > /usr/bin/git rev-list --no-walk 0693029ef14c3782c3eca68a691d13f93b33647c # timeout=10 [master] $ /opt/tools/apache-maven/latest/bin/mvn -f pom.xml -DcreateJavadoc=true clean install Picked up JAVA_TOOL_OPTIONS: -XX:+IgnoreUnrecognizedVMOptions -XX:+UseContainerSupport -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap Unrecognized VM option 'UseContainerSupport'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
Build step 'Invoke top-level Maven targets' marked build as failure
Comment 1 Frederic Gurr CLA 2019-05-24 11:16:52 EDT
The Californium JIPP is in the process of being migrated to our new cluster infrastructure, see https://bugs.eclipse.org/bugs/show_bug.cgi?id=547119.

Therefore you are seeing "duplicate" builds. So the build that is successful was executed on the old infra (https://ci.eclipse.org/californium), while on the new infra (https://ci-staging.eclipse.org/californium) it failed.

It failed, because - by default - we now set cluster environment specific Java options, that are not supported by Java versions < 8. I've tried to change the JDK version to JDK8 for the master job, but the maven-javadoc-plugin complains about some inconsistencies, that should be fixed (or ignored).
Comment 2 Achim Kraus CLA 2019-05-24 11:21:54 EDT
Good to know!

Currently we use java 1.7, because it's something as a promise for the 2.0.
We plan to release it hopefully in summer. After that migrating to java 1.8 should be no issue. 
Do you know, if the old hipp3 is planned to be kept until summer?

The javadoc issues on master should be fixed on 2.0.x. So, if I configure the 2.0.x to java 1.8 and start it, we will see, if it works on the new JIPP.
Comment 3 Frederic Gurr CLA 2019-05-24 11:24:27 EDT
(In reply to Achim Kraus from comment #2)
> Do you know, if the old hipp3 is planned to be kept until summer?
We are planning to shutdown hipp3 at the end of June. Therefore, we encourage projects to finish the migration to the new infra soon.
Comment 4 Achim Kraus CLA 2019-05-24 11:38:59 EDT
Seems I can't login into 

https://ci-staging.eclipse.org/californium

the last build there is #734, so I'm wondering, how I can test, if californium is build there proper. Any ideas?
Comment 5 Achim Kraus CLA 2019-05-24 11:45:11 EDT
Supi, I can login!

I will do my best to make the jobs running there successfully :-).
Comment 6 Achim Kraus CLA 2019-05-24 11:47:00 EDT
I would create a new bug, if I fail to manage to build on the new jipp.
Comment 7 Simon Bernard CLA 2019-06-06 09:40:07 EDT
@Frederic,

I suspect that any projects which are using java 7 to compile will face this issue.
(I don't know if there is lot of project which need that, at least Californium and Leshan)

Is there any solution on your side to workaround/fix this for all users using java 7 ?

On our side Achim found different workarounds like : 
 - using pipeline 
or
 - using shell script build step instead of maven build step.

Both way allow us to override _JAVA_OPTIONS and JAVA_TOOL_OPTIONS environment variable but we find no way to use classic maven build step...

A possible workaround to allow to use mvn build step, could be to activate https://wiki.jenkins.io/display/JENKINS/EnvInject+Plugin to easily override environment variable at job level.


Bonus question, should +IgnoreUnrecognizedVMOptions avoid this issue ? Does it means we face a JVM bug ?
Comment 8 Mikaël Barbero CLA 2019-06-07 04:45:06 EDT
You can also define parameters for you build job. Named them _JAVA_OPTIONS and JAVA_TOOL_OPTIONS but leave them to the empty string value. As those are injected as environment variable during the build, it should works for you.

Unfortunately, +IgnoreUnrecognizedVMOptions is not used for options provided by _JAVA_OPTIONS and JAVA_TOOL_OPTIONS. It's a known issue that won't be fixed in OpenJDK for backward compat issue IIRC.
Comment 9 Mikaël Barbero CLA 2019-06-07 04:48:26 EDT
Also, I highly recommend you to change the build configuration and use Java 8. You can configure javac/maven-compiler-plugin to read the source as Java 7 and generate Java 7 bytecode. For API usage, I recommend the usage of the animal-sniffer maven plugin.
Comment 10 Achim Kraus CLA 2019-06-07 05:05:34 EDT
FMPOC using java 8 and animal sniffer is not that compliant than to use java 7.

To use java 7 easily, I prepared a simple declarative pipeline for the ci-/cd stuff (tests on commit, deploy snapshot nightly). I will also create pipeines for the californium.tools and release jobs. But only, if it could be agreed to use them. (Otherwise it would be a waste of time.) 

My personal impression is, that Simon doesn't want the pipeline. I don't know why. Therefore it's hard to convince him. May be, because he would then have to build pipelines for leshan as well. But currently it's Simon, who should state, why he thinks, that there are better solutions than the pipelines.

See our discussion here:
https://github.com/eclipse/californium/pull/965
Comment 11 Achim Kraus CLA 2019-06-07 05:09:23 EDT
FMPOC => FMPOV :-)
Comment 12 Mikaël Barbero CLA 2019-06-07 05:48:38 EDT
(In reply to Achim Kraus from comment #10)
> FMPOC using java 8 and animal sniffer is not that compliant than to use java
> 7.

Gotcha

> To use java 7 easily, I prepared a simple declarative pipeline for the
> ci-/cd stuff (tests on commit, deploy snapshot nightly). I will also create
> pipeines for the californium.tools and release jobs. But only, if it could
> be agreed to use them. (Otherwise it would be a waste of time.) 
> 
> My personal impression is, that Simon doesn't want the pipeline. I don't
> know why. Therefore it's hard to convince him. May be, because he would then
> have to build pipelines for leshan as well. But currently it's Simon, who
> should state, why he thinks, that there are better solutions than the
> pipelines.
> 
> See our discussion here:
> https://github.com/eclipse/californium/pull/965

Job configurations along with the code is IMO a big advantage compared to having it only in Jenkins. But it's up to your team to decide.
Comment 13 Simon Bernard CLA 2019-06-07 08:54:15 EDT
@Achim why are you saying that :
> "Therefore it's hard to convince him."

I just try to understand what motivates the transition to pipeline ... and in this PR[1] I already said :
"Anyway I'm not against using pipeline, I don't know it so much and I just try to better understand the situation. :-)"

[1]: https://github.com/eclipse/californium/pull/965#issuecomment-499488058

> But currently it's Simon, who should state, why he thinks, 
> that there are better solutions than the pipelines.

If using pipelines is only motivated by "being able to set 2 environments variables", it's not so eccentric to ask/search if it exists more isolate way to do that.
If there are other reasons, it's fine. I just ask.

I share my concern with eclipse staff because I suspect other projects could be affected. (I'm sure for at least 1 : Leshan, as I'm committer on it and I will face the same issue)

I agree with Achim on this point :
> FMPOV using java 8 and animal sniffer is not that compliant than to use java 7.

@Mikaël, I tested to add JAVA_TOOL_OPTIONS and _JAVA_OPTIONS parameter for build and it works. Thx for the tips.
Comment 14 Mikaël Barbero CLA 2019-06-07 09:02:57 EDT
(In reply to Simon Bernard from comment #13)
> @Achim why are you saying that :
> > "Therefore it's hard to convince him."
> 
> I just try to understand what motivates the transition to pipeline ... and
> in this PR[1] I already said :
> "Anyway I'm not against using pipeline, I don't know it so much and I just
> try to better understand the situation. :-)"
> 
> [1]: https://github.com/eclipse/californium/pull/965#issuecomment-499488058
> 
> > But currently it's Simon, who should state, why he thinks, 
> > that there are better solutions than the pipelines.
> 
> If using pipelines is only motivated by "being able to set 2 environments
> variables", it's not so eccentric to ask/search if it exists more isolate
> way to do that.
> If there are other reasons, it's fine. I just ask.

You certainly don't need pipeline to define the 2 environment variables. The parameters do the job for this need. However, pipeline are IMO interesting mainly because you store the build config along with your code. Being able to version it, and let external contributors help you with it (they don't have access to the Jenkins job config) are the main advantages. 

Also, as specified on the official doc https://jenkins.io/doc/book/pipeline/:

Creating a Jenkinsfile and committing it to source control provides a number of immediate benefits:

* Automatically creates a Pipeline build process for all branches and pull requests.
* Code review/iteration on the Pipeline (along with the remaining source code).
* Audit trail for the Pipeline.
* Single source of truth for the Pipeline, which can be viewed and edited by multiple members of the project.

Finally, if you ever need to use custom docker images for your build environment (e.g. if you need a specific tool to be installed), you will need to use pipeline.

> I share my concern with eclipse staff because I suspect other projects could
> be affected. (I'm sure for at least 1 : Leshan, as I'm committer on it and I
> will face the same issue)
> 
> I agree with Achim on this point :
> > FMPOV using java 8 and animal sniffer is not that compliant than to use java 7.
> 
> @Mikaël, I tested to add JAVA_TOOL_OPTIONS and _JAVA_OPTIONS parameter for
> build and it works. Thx for the tips.

Glad to hear that.
Comment 15 Simon Bernard CLA 2019-06-07 09:44:50 EDT
@Mikaël thx again for this introduction about benefits of jenkins pipeline !
Comment 16 Achim Kraus CLA 2019-06-10 15:11:37 EDT
> You certainly don't need pipeline to define the 2 environment variables. 

Sure, "sh ... maven" will do it also, I tested that with the ci-jobs for "1.0.x" and "2.0.x". You must use a "sh" with a scripted "maven invocation" instead of a parametrized "maven" invocation. And that "sh ... maven" must be copied to every job, which executes maven. That would en up in about 10 copies of that "sh .. maven" in each of these jenkins jobs. 
Therefore tried to put that stuff in a declarative pipeline to load it into the jobs instead of copying it.