Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.

Bug 573166

Summary: Build stand-alone MAT packages with a newer Eclipse release such as 2021-03
Product: [Tools] MAT Reporter: Andrew Johnson <andrew_johnson>
Component: CoreAssignee: Andrew Johnson <andrew_johnson>
Status: RESOLVED FIXED QA Contact:
Severity: normal    
Priority: P3 CC: krum.tsvetkov
Version: 1.11   
Target Milestone: 1.12.0   
Hardware: All   
OS: All   
See Also: https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/179830
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=7a3b61bf69a181f7d3d1f31939603538a6f00249
https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/179847
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=299dc4b7e35a7829292e5f090ff41a92e0319cad
https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/179858
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=29c641dcad9e1d40f662e78481af3d44031f9a4f
https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/180087
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=61fad20db48c622dfba2bc0b19da0a791845d1eb
https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/180319
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=efc4e547cd96c1585ae26ada89036bbb5d46b942
https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/180328
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=3967578dd993cf7f91adcfcf1426d4bbd93ec77c
https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/180839
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=3df2ebf7eaeb0b5753733461de1a4ba0c60d468d
https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/180840
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=bbebf5881164211809bced676b6c172aa6dba597
https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/180852
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=8ce9641421a7df12cf0e38e9714611684c3d9f30
https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/180930
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=0dac064869114f493fb0370ec735bcc981ff3c7e
https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/182090
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=790dfa6f0e0b9222c47530abd058c073f321ad3f
https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/182099
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=db82f0f8fdf4fc3137d7eaea5db6d29902855f60
https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/182103
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=25353183cd99dad5136075d4fa5b91a94e686c2b
https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/182267
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=03ec7fd5d0d6e0d34cc854ba7a72eb4d131876ad
https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/182269
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=eb0fb1db6dfd987f8b8200111500b5d89f6b2a90
https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/182271
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=69fe85d9507dbc7c6f41ee672b60ff2752385973
https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/182272
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=1add2f701eae573a64edfcbf4c9cf0d24d9f5cc7
https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/182275
https://git.eclipse.org/c/mat/org.eclipse.mat.git/commit/?id=126304c4227fd0756f8d9c68576bab9d65195509
Whiteboard:
Bug Depends on: 574357    
Bug Blocks: 569438, 572718    

Description Andrew Johnson CLA 2021-04-26 08:01:12 EDT
Bug 567414 updated the Eclipse package for building standalone MAT to 2020-03.
This was the last Java 8 version.

There are some security problems with plugins with that version 2020-03 which are fixed in Eclipse 2021-03 or 2021-06. These versions require Java 11 though.

I think we should provide a version based on say 2021-06 but allow users of Memory Analyzer 1.11 based on 2020-03 to upgrade the MAT version so if they are running on a single-user system and are not worried about the security vulnerabilities to carry on with the older platform.
Comment 1 Eclipse Genie CLA 2021-04-26 08:45:38 EDT
New Gerrit change created: https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/179830
Comment 3 Andrew Johnson CLA 2021-04-26 09:58:31 EDT
I defined a new mat-2021-03.target file and did a build using this as an input argument, and also changed
the build to use adoptopenjdk-hotspot-jdk11-latest instead of adoptopenjdk-hotspot-jdk8-latest

The build worked. The resulting executable needs Java 11 to run, and gives an error pop-up with Java 8:
An error has occurred. See the log file ...55161.log.

mat.product needs to be fixed to have:
      <property name="osgi.requiredJavaVersion" value="11" />
but for the moment we want to be able to parameterize the build to continue to also build 2020-03 versions.
Comment 4 Andrew Johnson CLA 2021-04-26 09:59:56 EDT
See https://ci.eclipse.org/mat/job/tycho-mat-nightly/1167/
Comment 5 Eclipse Genie CLA 2021-04-26 13:47:36 EDT
New Gerrit change created: https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/179847
Comment 7 Andrew Johnson CLA 2021-04-26 16:44:11 EDT
Build #1169 https://ci.eclipse.org/mat/job/tycho-mat-nightly/1169/ added a build configuration option to select a product file other than mat.product, such as mat-2021-03.product which was changed to enforce Java 11.

However this build compared to 1168 is missing the UI themes and the choice of p2 update sites. These 
seem to be in mat.p2.inf so perhaps we need a special version of that as well.
Comment 8 Eclipse Genie CLA 2021-04-26 17:02:39 EDT
New Gerrit change created: https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/179858
Comment 10 Eclipse Genie CLA 2021-05-01 10:13:55 EDT
New Gerrit change created: https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/180087
Comment 12 Eclipse Genie CLA 2021-05-06 13:15:03 EDT
New Gerrit change created: https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/180319
Comment 14 Eclipse Genie CLA 2021-05-06 15:45:27 EDT
New Gerrit change created: https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/180328
Comment 16 Andrew Johnson CLA 2021-05-06 17:47:18 EDT
There is a problem building javadoc with Java 11 as it now fails to build the Javadoc, probably because of all  the import errors:

  [javadoc] Constructing Javadoc information...
  [javadoc] /home/jenkins/agent/workspace/tycho-mat-nightly/plugins/org.eclipse.mat.dtfj/src/org/eclipse/mat/dtfj/DTFJIndexBuilder.java:91: error: package com.ibm.dtfj.image does not exist
  [javadoc] import com.ibm.dtfj.image.CorruptData;
  [javadoc]                          ^
  [javadoc] /home/jenkins/agent/workspace/tycho-mat-nightly/plugins/org.eclipse.mat.dtfj/src/org/eclipse/mat/dtfj/DTFJIndexBuilder.java:92: error: package com.ibm.dtfj.image does not exist
  
   [javadoc] /home/jenkins/agent/workspace/tycho-mat-nightly/plugins/org.eclipse.mat.chart.ui/src/org/eclipse/mat/ui/internal/chart/PieChartPane.java:334: error: cannot find symbol
  [javadoc]         public Chart getRunTimeModel()
  [javadoc]                ^
  [javadoc]   symbol:   class Chart
  [javadoc]   location: class PieChartPane.CallBackListener
  [javadoc] 548 errors
     [copy] Copying 6 files to /home/jenkins/agent/workspace/tycho-mat-nightly/plugins/org.eclipse.mat.ui.help/doc
     [copy] Copying 1 file to /home/jenkins/agent/workspace/tycho-mat-nightly/plugins/org.eclipse.mat.ui.help/doc
     [copy] Copying 1 file to /home/jenkins/agent/workspace/tycho-mat-nightly/plugins/org.eclipse.mat.ui.help/doc
     [copy] Copying 3 files to /home/jenkins/agent/workspace/tycho-mat-nightly/plugins/org.eclipse.mat.ui.help/doc
     [copy] Copying 2 files to /home/jenkins/agent/workspace/tycho-mat-nightly/plugins/org.eclipse.mat.ui.help/doc
     [copy] Copying 2 files to /home/jenkins/agent/workspace/tycho-mat-nightly/plugins/org.eclipse.mat.ui.help/doc
[INFO] Executed tasks
Comment 17 Andrew Johnson CLA 2021-05-07 04:07:57 EDT
Fixed via Bug 327129 by adding MAT plugins as extra requirements to the org.eclipse.mat.ui.help pom.xml and this also makes the dependencies available, and then ant javadoc picks them up via test_classpath from maven.test.classpath com.ibm.dtfj etc do not need to be specified as they are dependencies of the mat plugins which are the only ones being compiled by javadoc
Not sure if this is quite the Maven way to do things, but it works so far and can be adjusted as needed (e.g. using a different varaible or extra requirements data).
Comment 18 Eclipse Genie CLA 2021-05-21 01:58:35 EDT
New Gerrit change created: https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/180839
Comment 20 Eclipse Genie CLA 2021-05-21 02:33:30 EDT
New Gerrit change created: https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/180840
Comment 22 Eclipse Genie CLA 2021-05-21 04:23:36 EDT
New Gerrit change created: https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/180852
Comment 24 Eclipse Genie CLA 2021-05-23 16:52:15 EDT
New Gerrit change created: https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/180930
Comment 26 Eclipse Genie CLA 2021-06-17 03:57:13 EDT
New Gerrit change created: https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/182090
Comment 28 Eclipse Genie CLA 2021-06-17 05:55:21 EDT
New Gerrit change created: https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/182099
Comment 30 Eclipse Genie CLA 2021-06-17 06:13:13 EDT
New Gerrit change created: https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/182103
Comment 32 Andrew Johnson CLA 2021-06-17 08:48:46 EDT
I had some trouble building with 2021-06 as the Tycho test initiation failed, on both nightly build and build standalone packages.

Moving up to Tycho 2.1.0 fixed that, but then required an update to the target platform atttribute includeSource="true"

org.eclipse.core.runtime.CoreException: Plug-in org.eclipse.tycho.surefire.osgibooter was unable to load class org.eclipse.tycho.surefire.osgibooter.HeadlessTestApplication.
	at org.eclipse.core.internal.registry.osgi.RegistryStrategyOSGI.throwException(RegistryStrategyOSGI.java:212)
	at org.eclipse.core.internal.registry.osgi.RegistryStrategyOSGI.createExecutableExtension(RegistryStrategyOSGI.java:198)
	at org.eclipse.core.internal.registry.ExtensionRegistry.createExecutableExtension(ExtensionRegistry.java:920)
	at org.eclipse.core.internal.registry.ConfigurationElement.createExecutableExtension(ConfigurationElement.java:246)
	at org.eclipse.core.internal.registry.ConfigurationElementHandle.createExecutableExtension(ConfigurationElementHandle.java:63)
	at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:198)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134)
	at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:401)
	at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:255)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:654)
	at org.eclipse.equinox.launcher.Main.basicRun(Main.java:591)
	at org.eclipse.equinox.launcher.Main.run(Main.java:1462)
	at org.eclipse.equinox.launcher.Main.main(Main.java:1435)
Caused by: java.lang.NoClassDefFoundError: org/eclipse/core/runtime/IPlatformRunnable
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
	at org.eclipse.osgi.internal.loader.ModuleClassLoader.defineClass(ModuleClassLoader.java:293)
	at org.eclipse.osgi.internal.loader.classpath.ClasspathManager.defineClass(ClasspathManager.java:718)
	at org.eclipse.osgi.internal.loader.classpath.ClasspathManager.findClassImpl(ClasspathManager.java:641)
	at org.eclipse.osgi.internal.loader.classpath.ClasspathManager.findLocalClassImpl(ClasspathManager.java:609)
	at org.eclipse.osgi.internal.loader.classpath.ClasspathManager.findLocalClassImpl(ClasspathManager.java:589)
	at org.eclipse.osgi.internal.loader.classpath.ClasspathManager.findLocalClass(ClasspathManager.java:568)
	at org.eclipse.osgi.internal.loader.ModuleClassLoader.findLocalClass(ModuleClassLoader.java:345)
	at org.eclipse.osgi.internal.loader.BundleLoader.findLocalClass(BundleLoader.java:401)
	at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:480)
	at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:170)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	at org.eclipse.osgi.internal.framework.EquinoxBundle.loadClass(EquinoxBundle.java:622)
	at org.eclipse.core.internal.registry.osgi.RegistryStrategyOSGI.createExecutableExtension(RegistryStrategyOSGI.java:196)
	... 16 more
Caused by: java.lang.ClassNotFoundException: org.eclipse.core.runtime.IPlatformRunnable cannot be found by org.eclipse.tycho.surefire.osgibooter_1.7.0
	at org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.java:519)
	at org.eclipse.osgi.internal.loader.ModuleClassLoader.loadClass(ModuleClassLoader.java:170)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	... 31 more

I also needed to parameterise the mat-standalone-packages build to allow selection of the target platform and product files.
Comment 33 Andrew Johnson CLA 2021-06-19 13:08:51 EDT
The productConfiguration user property doesn't seem to work for selecting the product file (and p2 file).

https://www.eclipse.org/tycho/sitedocs/tycho-packaging-plugin/product-export-mojo.html

Perhaps as we are using https://www.eclipse.org/tycho/sitedocs/tycho-p2/tycho-p2-director-plugin/materialize-products-mojo.html then this looks for all the product files with the right product id, and chooses the first, and all those
product files mat-2020-03 mat-2021-03 and mat-2021-06 all have the same id, so mat-2020-03 is chosen.

Perhaps create a mat.product file and mat.p2.inf file which is a copy of mat-2021-06.product etc., either in Git or as part of the
build. Or else somehow use tycho-packaging:product-export ?
This needs to work for both the https://ci.eclipse.org/mat/job/tycho-mat-nightly/ and https://ci.eclipse.org/mat/job/mat-standalone-packages/ builds.
Comment 34 Eclipse Genie CLA 2021-06-20 10:43:50 EDT
New Gerrit change created: https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/182267
Comment 36 Eclipse Genie CLA 2021-06-20 10:59:56 EDT
New Gerrit change created: https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/182269
Comment 38 Eclipse Genie CLA 2021-06-20 11:12:00 EDT
New Gerrit change created: https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/182271
Comment 40 Eclipse Genie CLA 2021-06-20 12:54:43 EDT
New Gerrit change created: https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/182272
Comment 42 Eclipse Genie CLA 2021-06-20 14:37:09 EDT
New Gerrit change created: https://git.eclipse.org/r/c/mat/org.eclipse.mat/+/182275
Comment 44 Andrew Johnson CLA 2021-06-20 16:53:00 EDT
I tried copying the appropriate product and p2.inf file to mat.product and mat.p2.inf as part of the build - but that didn't work (and also those would not be first in alphabetical order).
I then tried creating a mat-1.product and mat-1.p2.inf file with the 2021-06 changes. That works with the mat-standalone-packages build and gave a RCP with the 
osgi.requiredJavaVersion=11 so gave the right warning about the wrong Java.
It also had the correct reference to the Eclipse update site: http://download.eclipse.org/releases/2021-06/
though the update process did offer to upgrade to the MAT at the location - even though it should be the same as the RCP version.
Comment 45 Krum Tsvetkov CLA 2021-06-21 04:20:30 EDT
(In reply to Andrew Johnson from comment #44)


I tested the Mac version:
- with Java 1.8 I get the expected reasonable error message that I need at least 11
- with a newer Java the tool works fine

> though the update process did offer to upgrade to the MAT at the location -
> even though it should be the same as the RCP version.
In the past I created a separate feature to be used in the standalone version. While the content (the plugins) are the same, the features found in the simrel repository are different than the ones used in the RCP.
I believe I did this as a workaround for some issues with updating from the RCP years ago. It might be worth to verify again if this is needed - it does cause some confusion.

On Friday I already created all the content for the download pages.
I will replace it now with the newer version. However, I will wait one more day before updating the Homepage/Downloads page and announcing the availability of the standalone builds, in order to give some time to the mirrors to update. I believe if I just replace the files now, then some mirrors will still serve the old content.
Comment 46 Andrew Johnson CLA 2021-06-21 06:08:10 EDT
I have now notarized the latest stand-alone build. I had some problems doing that: bug 574357 but that is now fixed.
Comment 47 Andrew Johnson CLA 2021-06-22 11:53:00 EDT
It would be nice to have a better way of selecting the product files at build time, but what we have works for 1.12.