Bugzilla – Full Text Bug Listing
| Summary: | GTK compatibility problem following JIRO upgrade | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Community | Reporter: | Ed Willink <ed> | ||||
| Component: | CI-Jenkins | Assignee: | CI Admin Inbox <ci.admin-inbox> | ||||
| Status: | CLOSED INVALID | QA Contact: | |||||
| Severity: | normal | ||||||
| Priority: | P3 | CC: | akurtakov, denis.roy, frederic.gurr, mikael.barbero, sravankumarl, webmaster | ||||
| Version: | unspecified | ||||||
| Target Milestone: | --- | ||||||
| Hardware: | PC | ||||||
| OS: | Windows 10 | ||||||
| See Also: |
https://bugs.eclipse.org/bugs/show_bug.cgi?id=562329 https://bugs.eclipse.org/bugs/show_bug.cgi?id=563422 |
||||||
| Whiteboard: | |||||||
| Bug Depends on: | 563442 | ||||||
| Bug Blocks: | |||||||
| Attachments: |
|
||||||
|
Description
Ed Willink
(In reply to Mikaël Barbero from 562329#28) > Your issue looks like an issue because of some gtk version mismatch. It may > be caused by the addition we made after from bug 552165 comment #82 (see > https://github.com/eclipse-cbi/jiro-agents/commit/ > c5b829ab65a8ba26a342b3e3bc8f2978396eca74). > > We will need help from SWT-GTK experts to diagnose the issue though. I'll cc > appropriate people to the bug you will create to track that down. This seems unavoidable. I see a platform - CentOS, GTK, Java, Maven a system under test, Eclipse 4.12 and its plugins Sometimes a 'better' aspect of the platform will break/expose a latent bug in the system under test. Where this occurs in Java, we have the freedom to select a different Java version. Where this occurs in Maven, we have the freedom to backup the Maven version as was very necessary till recently. Where this occurs in GTK we do not seem to have the freedom to specify an older GTK / CentOS to ensure that one that works with Eclipse 4.12 is in use. Debugging SWT won't help. Possibly the GTK breakage might be fixed so that a future GTK is compatible again. Possibly a bug in Eclipse 4.12 will be identified, but Eclipse 4.12 isn't going to be fixed; just the same as many old Eclipses won't run on recent Java's tough; use an old Java. It seems that a GTK/CentOS version selection is necessary. Ideally via a job option, but (In reply to Ed Willink from comment #0) > Do I need to raise a bug for an ocl-lts JIPP that has more vintage support? (In reply to Ed Willink from comment #1) > I see > a platform - CentOS, GTK, Java, Maven > a system under test, Eclipse 4.12 and its plugins > > Sometimes a 'better' aspect of the platform will break/expose a latent bug > in the system under test. > > Where this occurs in Java, we have the freedom to select a different Java > version. > > Where this occurs in Maven, we have the freedom to backup the Maven version > as was very necessary till recently. > > Where this occurs in GTK we do not seem to have the freedom to specify an > older GTK / CentOS to ensure that one that works with Eclipse 4.12 is in use. AFAIK, this is hardly possible. It may be possible to do that by building gtk from sources, but it would force you to define a bunch of environment variables / paths to use the proper version. Also, you would hardly catch end users' issues with this as this is not the typical setup: people just use the gtk version from their Linux distribution (and centos 7 is very common, still maintained and not bleeding edge either) Nevertheless, now that we run on a container-based infrastructure, you can build your own build environment in the form of a docker image. This way, you will have a tight control on when it gets updated and what's in there. See https://wiki.eclipse.org/Jenkins#How_do_I_run_my_build_in_a_custom_container.3F for details. Of course, you will have to learn about Docker and how to build a docker image by yourself. > Debugging SWT won't help. Possibly the GTK breakage might be fixed so that a > future GTK is compatible again. Possibly a bug in Eclipse 4.12 will be > identified, but Eclipse 4.12 isn't going to be fixed; just the same as many > old Eclipses won't run on recent Java's tough; use an old Java. > > It seems that a GTK/CentOS version selection is necessary. Ideally via a job > option, but If Eclipse 2019-09 stopped working on centos 7 with only official packages, I'm pretty sure that the Eclipse/SWT team is interested in learning why and how to make it work as some of their users may have a similar environment. Asking end user to just change their environment is not an option IMO. @Sravan, @Alex, can you help there? In the meantime, I'm building a specific docker image without webkitgtk4 as suggested by Sravan in bug 562329 comment #30. Once setup, we can try your compat jobs with it to see if it fixes anything (i'll do the config changes to use this image). Would that work for you? Newer Gtk versions require fixes in SWT - this is unavoidable. Much like working with new Java version requires fixes and changes in JDT. One can check all the usages of GTK_VERSION in swt codebase and see them. E.g. Menu._setVisible is one example where SWT in order to provide richer support for users touches Gtk internals which do change. If there is issue with latest Eclipse/SWT we will look into it but old version no longer working after system/gtk update is inevitable and nothing we can do about it. gtk3 is 3.22.30-5.el7 on the build image. It's not expected to be supported by SWT from 2019.09? (In reply to Mikaël Barbero from comment #4) > gtk3 is 3.22.30-5.el7 on the build image. It's not expected to be supported > by SWT from 2019.09? My reply was mostly for the Oxygen task failing. Regarding 2019-09 it should be working but looking at https://ci.eclipse.org/ocl/job/ocl-compatibility-2019-09/24/consoleFull I only see that the build timed out. My gut feeling would be there is a test relying on focus - and these are totally unpredictable because not only they rely on the gtk version but also on the WM used e.g mutter/kwin/..., how are they started and etc. If you are able to make the problem go away that helps, but I'm not too keen on something that is special for me, because that is very likely to break with the next infrastructure improvement. If however you are pretty much offering an ocl-lts JIPP that uses the Eclipse ...-2019 compatible docker whereas the current JIPPs use the Eclipse 2020-... docker then that may be a necessary evil for a 'platform' breakage that other users who check compatibility may also be interested in. If the docker can be a job selection, just like the Java version, even better. I will treat the OCL JIPP as 'offline' for the next week so that you can experiment. (In reply to Alexander Kurtakov from comment #5) > Regarding 2019-09 it should be working but looking at > https://ci.eclipse.org/ocl/job/ocl-compatibility-2019-09/24/consoleFull I > only see that the build timed out. My gut feeling would be there is a test > relying on focus - and these are totally unpredictable because not only they > rely on the gtk version but also on the WM used e.g mutter/kwin/..., how are > they started and etc. Magic dialogs are certainly hazardous and can interact really badly with Tycho; different issue. These compatibility jobs usually run every Sunday, so job 24 suggests that the 2019-09 compatibility has been working for nearly six months. Since it is a clone of other jobs, it has not needed tweaking since the initial couple of runs that fixed the naming. The Job Config history shows only changes from OCL JIRO migration in April. So my instinct tells me that given a family of jobs that have been running well, pace infrastructure issues, for a couple of years, and then collapse after an infrastructure change, then the problem is the infrastructure change. wrt mutter etc, that brings back memories, there seemed to be only one combination that worked, but since moving to JIRO that combination seems to have evolved to 'migration'. I don't see mutter or metacity in my OCL or QVTd configs. Since OCL only migrated to the new infrastructure a few weeks ago, it may be that some new 'mutter' muttering is required. Ok, so I've spent *a lot* of time on this one. I've tried - same image without webkitgtk4 - with an old image (built a year ago - very similar to the one from your last succeeding build https://ci.eclipse.org/ocl/job/ocl-compatibility-2019-09/22/console), - with newer images based on centos 7.4, 7.5, 7.6, 7.7 and 7.8 (in addition to base 7) - with centos 8 None works, they all triggers the (SWT:832): Gtk-WARNING **: 08:38:35.940: Negative content height -1 (allocation 1, extents 1x1) while allocating gadget (node scrolledwindow, owner GtkScrolledWindow) Next try: a recent fedora. @Ed, if you could double check on your side that the actual target platform resolution doesn't bring newer SWT into play, that would be appreciated. The target platform is rather simple:
<?xml version="1.0" encoding="UTF-8" standalone="no"?><?pde version="3.8"?><target includeMode="feature" name="Eclipse OCL Oxygen Build Target Platform" sequenceNumber="16">
<locations>
<location includeAllPlatforms="false" includeConfigurePhase="false" includeMode="planner" includeSource="true" type="InstallableUnit">
<unit id="org.eclipse.sdk.feature.group" version="0.0.0"/>
<unit id="org.eclipse.emf.sdk.feature.group" version="0.0.0"/>
<unit id="org.eclipse.uml2.sdk.feature.group" version="0.0.0"/>
<unit id="org.eclipse.xtext.sdk.feature.group" version="0.0.0"/>
<repository location="https://download.eclipse.org/releases/oxygen/201804111000"/>
</location>
<location includeAllPlatforms="false" includeConfigurePhase="false" includeMode="planner" includeSource="true" type="InstallableUnit">
<unit id="org.eclipse.ocl.tests.feature.group" version="0.0.0"/>
<unit id="org.eclipse.ocl.examples.tests.feature.group" version="0.0.0"/>
<unit id="org.eclipse.ocl.examples.impactanalyzer.tests" version="0.0.0"/>
<repository location="https://ci.eclipse.org/ocl/job/ocl-master/lastSuccessfulBuild/artifact/targetPlatform/"/>
<!--repository location="https://ci.eclipse.org/ocl/job/ocl-branch-tests/lastSuccessfulBuild/artifact/targetPlatform/"/-->
<!--repository location="https://download.eclipse.org/modeling/mdt/ocl/updates/releases/6.3.0"/-->
</location>
<location includeAllPlatforms="false" includeConfigurePhase="false" includeMode="planner" includeSource="true" type="InstallableUnit">
<unit id="org.eclipse.emf.examples.library" version="0.0.0"/>
<repository location="https://download.eclipse.org/modeling/emf/emf/builds/release/2.13"/>
</location>
</locations>
</target>
Install Eclipse and friends from https://download.eclipse.org/releases/oxygen/201804111000 which should not permit a newer SWT.
Install the latest OCL build from https://ci.eclipse.org/ocl/job/ocl-master/lastSuccessfulBuild/artifact/targetPlatform/
Install one test model from https://download.eclipse.org/modeling/emf/emf/builds/release/2.13
It could be that the latest OCL build is somehow different to previous builds, and that is what the compatibility test is testing for, but I can only see this happening if a newer JDT generates functionally different byte code. This remote possibility could be eliminated by using https://ci.eclipse.org/ocl/job/ocl-master/464/artifact/targetPlatform/ but somehow despite this being a Keep-Build-Forever the build artefacts have been wiped. https://ci.eclipse.org/ocl/job/ocl-master/482/artifact/targetPlatform/ for M1 is older than the last successful ocl-compatibility-oxygen run.
Otherwise a very unlikely possibility is that https://download.eclipse.org/releases/oxygen/201804111000 and all other compatibility repos the other jobs use have been corrupted. Or https://download.eclipse.org/modeling/emf/emf/builds/release/2.13 is corrupted.
Otherwise it must be the JIRO platform that has changed.
when I compare build 112 and 114 I noticed java options missing from newer build specifically these two lines Picked up JAVA_TOOL_OPTIONS: -XX:+IgnoreUnrecognizedVMOptions -XX:+UseContainerSupport -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap Picked up _JAVA_OPTIONS: -XX:MaxRAMPercentage=64.0 Will this case any problem? Fedora 28 failed as well. (In reply to Ed Willink from comment #9) > The target platform is rather simple: > > It could be that the latest OCL build is somehow different to previous > builds, and that is what the compatibility test is testing for, but I can > only see this happening if a newer JDT generates functionally different byte > code. This is very unlikely. > This remote possibility could be eliminated by using > https://ci.eclipse.org/ocl/job/ocl-master/464/artifact/targetPlatform/ but > somehow despite this being a Keep-Build-Forever the build artefacts have > been wiped. Your job config says to only keep 1 job with artifacts. Artifacts are wiped by Jenkins for older builds in this case. Depending on when you set the build to be "kept forever", the artifacts could have been wiped by then. Please don't change that to a too high value. Jenkins is not meant to be a file server with backup for older builds. > https://ci.eclipse.org/ocl/job/ocl-master/482/artifact/targetPlatform/ for > M1 is older than the last successful ocl-compatibility-oxygen run. > > Otherwise a very unlikely possibility is that > https://download.eclipse.org/releases/oxygen/201804111000 and all other > compatibility repos the other jobs use have been corrupted. Or > https://download.eclipse.org/modeling/emf/emf/builds/release/2.13 is > corrupted. This unlikely as you say. > Otherwise it must be the JIRO platform that has changed. We changed the "migration" image, but the only changes are the addition of webkitgtk4, curl and lsof. I've tried with an image without these packages and it did not change anything. I've even managed to find the exact reference of the image you've been using for https://ci.eclipse.org/ocl/job/ocl-compatibility-2019-09/22/console. It did not work either. So I'm struggling at identifying what could be the root of the failure. Also, the images have been updated for many projects (incl. platform) and none have face similar issues. That's why I'm starting to look at if it could something from OCL. So far, I'm unable to find any answer to this issue. (In reply to Sravan Kumar Lakkimsetti from comment #10) > when I compare build 112 and 114 I noticed java options missing from newer > build specifically these two lines > > Picked up JAVA_TOOL_OPTIONS: -XX:+IgnoreUnrecognizedVMOptions > -XX:+UseContainerSupport -XX:+UnlockExperimentalVMOptions > -XX:+UseCGroupMemoryLimitForHeap > Picked up _JAVA_OPTIONS: -XX:MaxRAMPercentage=64.0 > > Will this case any problem? Very unlikely. If running on an older JDK8 (IIRC, before u192), it could cause OOME or trigger the kernel OOMKiller. Nothing else. (In reply to Mikaël Barbero from comment #11) > Also, the images have been updated for many projects (incl. platform) and > none have face similar issues. That's why I'm starting to look at if it > could something from OCL. So far, I'm unable to find any answer to this > issue. OCL has no problem for its current release on the current platform. How many other projects do as OCL does to confirm that the new build is installable and runnable on older platforms? and how many of those projects run those compatibility builds regularly? as often I'm probably the first to detect a problem that others may see as they test their 2020-06 release candidates. Most of the projects my team is involved in support only the latest version of their dependencies. This is the only viable solution we have found in the long term with the current workforce. I probably overstated the impact on 2020-06 tests. In the past OCL release notes made hopeful assertions about installable on X, which sometimes I even confirmed, but it burned too many releng cycles partoicularly with the cadence increasing fourfold. So I wrote the compatability jobs that confirm that the latest release really can be used on older platforms and standalone on Java 5.... My claims are no longer hopeful they are areto-tested every Sunday which gives an early warning when something breaks. Unfortunately I have too many users who ask about their planned upgrade to Oxygen! I really have to support older releases to ensure that the latest OCL release can be used. OCL might be the only project with this style of compatibility job (which had been working for 2 years). --- Not wishing to teach you your job, but are you really sure that your experiments really changed what you intended? I find that it is incredibly easy to test a 'different' configuration and so draw the 'wrong' conclusion. Long ago I observed a rule of thumb that while developing test harnesses 90% of the problems were in the defective test harness or its use and only 10% were in the system under test. (In reply to Mikaël Barbero from comment #8) > None works, they all triggers the > > (SWT:832): Gtk-WARNING **: 08:38:35.940: Negative content height -1 > (allocation 1, extents 1x1) while allocating gadget (node scrolledwindow, > owner GtkScrolledWindow) In principle if the JIRO is put back to precisely that for the https://ci.eclipse.org/ocl/job/ocl-compatibility-oxygen/112/ build, it should work. Hence my skepticism that finger trouble may mean you haven't successfully reverted what you were trying to revert. The -ve parameters in the messages suggest that either a widget is now being created that was not before, or that the initialization of that widget is relying on some different initialization code. Both seem more likely as genuine functionality changes rather than intermittencies. This change needs to be understood to provide the appropriate release note to warn old Eclipse users of new hazards. Prior to the JIRO migration of OCL a couple of weeks ago, this job was solid, but it has not really run often enough on the new JIRO to be sure. It is possible that there is a dialog timing variability and the earlier jobs were lucky and the new runs are unlucky. If I were developing this job now and hitting this GTK problem afresh, I would prune the JUnit test suite to discover which test/tests provoke the problem. If no platform/timing fix was forthcoming, I would have to add conditionalization to the test suite to exclude tests that fail in vintage scenarios. I can confirm the the UI tests have to take precautions to suppress unwanted GIT pop-ups and accommodate "Do you want to change perspective" dialogs so there is an opportunity for variant race conditions. However it remains the case that testing of both Eclipse Oxygen and 2019-09 now behaves in a different way after your upgrade to before. (In reply to Ed Willink from comment #6) > I will treat the OCL JIPP as 'offline' for the next week so that you can > experiment. Thank you. You have obviously tried various things without success, so I guess that this is now going too fall into the too-hard / limited-benefit category long grass. If you can restore the OCL JIPP to its standard configuration, I will endeavour to identify which test(s) cause the trouble and maybe conditionalize them / re-time them / ... (In reply to Ed Willink from comment #17) > If you can restore the OCL JIPP to its standard configuration... Done. A few notes: * All ocl-compatibility-* builds that ran successfully on May 3rd already showed Gtk-WARNINGS during the tests, but the tests did not time out. Therefore we don't know, when exactly the Gtk-WARNINGS started to appear. AFAICT, last known good builds (without Gtk-WARNINGS) were in the beginning of April on the old infra. * I've set up a build ocl-compatibility-2019-03 build on a Jenkins instance on the old infra. It did not show the GtK-WARNINGS, but the tests still timed out (after more than one hour). That is a strong indication that Gtk-WARNINGS are not the only issue with the compatibility tests. To make any further analysis easier for everyone, I'd suggest the following: * Try to isolate failing tests and create a minimal test case. A short turn-around time can speed up debugging a lot. * To reduce the number of potential Maven related corner-cases and issues, convert Maven jobs to Freestyle jobs. Maven jobs have been deprecated for a looong time. * Always use the "Publish JUnit test results report" post-build action (In reply to Frederic Gurr from comment #19) > * Try to isolate failing tests and create a minimal test case. A short > turn-around time can speed up debugging a lot. I'm trying, but the JIRO interface seems weird. It's almost as if there are two different interfaces with different historic statuses and as I go up/down to tests/configure it's almost random which I come out on. Only one of them offers Job History or Build With Parameters. > * To reduce the number of potential Maven related corner-cases and issues, > convert Maven jobs to Freestyle jobs. Maven jobs have been deprecated for a > looong time. How do I tell which I am using? How do I convert? Where is the deprecation marker? > * Always use the "Publish JUnit test results report" post-build action Indeed. I always do. But somehow the configure option doesn't even offer a decent Archiving Post-Build action. It's there on ocl-master so the plugin must be there ... I have a very strong suspicion that the various JIRO migrations trashed the jobs subtly. ---- Is it possible that both old and new infrastructure are there at once giving me these two options? (In reply to Ed Willink from comment #20) > > * To reduce the number of potential Maven related corner-cases and issues, > > convert Maven jobs to Freestyle jobs. Maven jobs have been deprecated for a > > looong time. > > How do I tell which I am using? How do I convert? Where is the deprecation > marker? Ah. It's in the job title. Just started creating a new job. Option 2 very attractively offers a Maven job. No indication of deprecated. On the contrary description indicates that it saves considerable effort, though on reflection what it saves seems to conflict with Tycho. (In reply to Mikaël Barbero from comment #11) > Fedora 28 failed as well. > > (In reply to Ed Willink from comment #9) > > The target platform is rather simple: > > > > It could be that the latest OCL build is somehow different to previous > > builds, and that is what the compatibility test is testing for, but I can > > only see this happening if a newer JDT generates functionally different byte > > code. > > This is very unlikely. Except that after starting to debug some magic inconsistencies, it looks as if JDT may now be throwing a NoSuchMethodError with a bad argument list, where it previously boxed a primitive boolean to a Boolean to find the method. For this to cause the regression breakage but no current breakage, JDT must now be boxing inside the method when it previously boxed outside. With a single Maven surefile tst
OK:
<repository location="https://download.eclipse.org/modeling/mdt/ocl/updates/releases/6.11.0"/>
<repository location="https://download.eclipse.org/modeling/mdt/ocl/updates/milestones/6.12.0/S202004130511"/>
<repository location="https://download.eclipse.org/modeling/mdt/ocl/updates/milestones/6.12.0/S202005041315/"/>
NoSuchMethodError fail:
<repository location="https://ci.eclipse.org/ocl/job/ocl-master/lastSuccessfulBuild/artifact/targetPlatform/"/>
<repository location="https://download.eclipse.org/modeling/mdt/ocl/updates/nightly/6.12.0"/>
This is running locally so it is nothing to do with JIRO at all.
The difference between the OK (M2 and earlier) and Bad Repos (post M2) is the build system that created the repo under test, which given that we are interested in the deliverable byte code can pretty much only be JDT.
To prove it we just need to change the 'current' build target to a pre-M2 JDT and all should be ok again.
Created attachment 282980 [details]
Disassembly
Attached disassembly shows that some calls to
setWithLastConsumed:(Lorg/eclipse/emf/ecore/EObject;Ljava/lang/String;ZLjava/lang/String;)V
have 3 arguments giving the NoSuchMethodError while others have 4
setWithLastConsumed:(Lorg/eclipse/emf/ecore/EObject;Ljava/lang/String;Ljava/lang/Object;Ljava/lang/String;)V
Comment on attachment 282980 [details] Disassembly Wrong fileand wrong bug. Correct file on Bug 563442 Problem solved. All OCL compatibility jobs now pass (after following advice to use FreeStyle jobs and archive tests.) Two tests have benign GTK warnings. This problem followed on from the Bug 562329 infrastructure change. Unfortunately Xtext chose exactly that moment to introduce a weird API incompatibility so the infrastructure change was blamed for the Xtext change. > This problem followed on from the Bug 562329 infrastructure change. > Unfortunately Xtext chose exactly that moment to introduce a weird API > incompatibility so the infrastructure change was blamed for the Xtext change. Glad you're resolved your issue; the timing was unfortunate. (In reply to Ed Willink from comment #9) > Otherwise it must be the JIRO platform that has changed. (In reply to Ed Willink from comment #15) > I find that it is incredibly > easy to test a 'different' configuration and so draw the 'wrong' conclusion. Looks like you were right :) Closing. |