Bugzilla – Full Text Bug Listing
| Summary: | After a Jenkins upgrade & restart, Webtools and Datsatools JIPPs are dead / do not restart | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | Community | Reporter: | Nick Boldt <nboldt> | ||||||
| Component: | CI-Jenkins | Assignee: | CI Admin Inbox <ci.admin-inbox> | ||||||
| Status: | RESOLVED FIXED | QA Contact: | |||||||
| Severity: | blocker | ||||||||
| Priority: | P3 | CC: | frederic.gurr, mikael.barbero, nboldt, webmaster | ||||||
| Version: | unspecified | ||||||||
| Target Milestone: | --- | ||||||||
| Hardware: | PC | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | |||||||||
| Bug Depends on: | |||||||||
| Bug Blocks: | 548354 | ||||||||
| Attachments: |
|
||||||||
|
Description
Nick Boldt
JIPP is back up. Sorry -- should have waited longer. Or maybe WM fixed it? I've fixed it. I assume you upgraded to LTS version 2.89.4? We have been experiencing some problems with auto-restart after doing that. Should work fine with usual plugin updates AFAICT. BTW: I have limited the number of executors to 2 for now since the machine seems to be under heavy load for the last days. Closing. I've done it again... updated some plugins, triggered a restart... and now more than 5 mins later, the service is still not responding. Additionally, the error message that says "Members of the project can restart this service using the CI Control tools in their Eclipse Foundation account (login required)." is invalid as there are no links in my portal to the JIPP. I'm a committer in wtp.common, servertools, and releng, but evidently none of those are associated with the JIPP instance. Could such an association be made? Created attachment 273085 [details]
screenshot of my portal view: no jipps associated with wtp subprojects common, servertools, or releng
WTP chose to use a single CI instance for all its sub-projects: https://ci.eclipse.org/webtools/ This is out of our regular rule being 1 Jenkins instance per project. The text you are seeing is then normal. There is no JIPP for common, servertools or releng. Regarding the restart feature, you're not a member of the "webtools" top-level project, only on some subproject. Currently, we cannot give you access to the Webttols JIPP management. This issue would be a dupe of bug 521906. In the meantime, I've restarted the JIPP. Here are the log I have before the crash: Mar 12 04:22:14 hipp10 org.a11y.Bus[13878]: Activating service name='org.a11y.atspi.Registry' Mar 12 04:22:14 hipp10 org.a11y.Bus[13878]: Successfully activated service 'org.a11y.atspi.Registry' Mar 12 04:22:14 hipp10 org.a11y.atspi.Registry[13885]: SpiRegistry daemon is running with well-known name - org.a11y.atspi.Registry Mar 12 04:22:14 hipp10 org.a11y.atspi.Registry[13885]: Xlib: extension "XEVIE" missing on display ":451". Mar 12 04:37:53 hipp10 org.a11y.Bus[13878]: g_dbus_connection_real_closed: Remote peer vanished with error: Underlying GIOStream returned 0 bytes on an async read (g-io-error-quark, 0). Exiting. (In reply to Mikaël Barbero from comment #6) > Regarding the restart feature, you're not a member of the "webtools" > top-level project, only on some subproject. Currently, we cannot give you > access to the Webttols JIPP management. This issue would be a dupe of bug > 521906. So in future, when I want to restart the JIPP, I need to keep reopening this BZ ? Because I can shut down the server with this: https://hudson.eclipse.org/webtools/restart but then it doesn't come back on its own, which SEEMS like a bug, not a permission-denied thing. (In reply to Nick Boldt from comment #7) > (In reply to Mikaël Barbero from comment #6) > > Regarding the restart feature, you're not a member of the "webtools" > > top-level project, only on some subproject. Currently, we cannot give you > > access to the Webttols JIPP management. This issue would be a dupe of bug > > 521906. > > So in future, when I want to restart the JIPP, I need to keep reopening this > BZ ? > > Because I can shut down the server with this: > > https://hudson.eclipse.org/webtools/restart > > but then it doesn't come back on its own, which SEEMS like a bug, not a > permission-denied thing. AFAICT, the Jenkins self restart feature is pretty buggy (see the number of related bugs https://issues.jenkins-ci.org/browse/JENKINS-41319?jql=status%20in%20(Untriaged%2C%20Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20text%20~%20%22restart%22). It's the reason why we provide a restart feature that works at the OS service level. The can shutdown the instance with the given /restart link because you've been added specifically in the Jenkins security pane as an admin of this instance. The control pane on the portal works with Foundation's data, ie which projects you're a committer on. We have no concept of project's releng admin in the Foundation database. Well, I guess I need to request another restart, so that I can remove the Naginator plugin that you've requested we not use. https://dev.eclipse.org/mhonarc/lists/cross-project-issues-dev/msg15407.html However it seems that plugin depends on Build Timeout plugin, which is preventing me from removing it. https://hudson.eclipse.org/webtools/pluginManager/installed So... how about I just don't use it, even though it's installed? (In reply to Nick Boldt from comment #9) > Well, I guess I need to request another restart, Done. > so that I can remove the > Naginator plugin that you've requested we not use. > > https://dev.eclipse.org/mhonarc/lists/cross-project-issues-dev/msg15407.html > > However it seems that plugin depends on Build Timeout plugin, which is > preventing me from removing it. > > https://hudson.eclipse.org/webtools/pluginManager/installed > > So... how about I just don't use it, even though it's installed? What a weird dependency... No problem to keep it as long as you don't use it. Thanks for your understanding. For some reason I can no longer run a pipeline job... it worked fine on Mar 15, and with minor changes it now no longer works [0]: [0] https://ci.eclipse.org/webtools/view/webtools_R3_10/job/WTP-Pipeline_R3_10/27/console Apparently the missing plugin is `Pipeline: Build Step`, which is installed to version 2.7 [1]. So that's maybe not the cause [2]. [1] https://ci.eclipse.org/webtools/pluginManager/installed [2] https://stackoverflow.com/questions/42118619/jenkins-pipeline-no-such-dsl-method-build I would simply restart Jenkins to see if that fixes things, since I see "Warning: This Jenkins instance requires a restart. Changing the state of plugins at this time is strongly discouraged. Restart Jenkins before proceeding." I've also downloaded 4 plugin updates and have updated to the latest Jenkins version available (2.107.1). But since I can't restart Jenkins without your doing some background magic to bring the server back up... Can you restart the WTP JIPP for me? (In reply to Nick Boldt from comment #11) > Can you restart the WTP JIPP for me? Done. (In reply to Nick Boldt from comment #9) > However it seems that plugin depends on Build Timeout plugin, which is > preventing me from removing it. This was bothering me enough to look into it again. The Build Timeout plugin has an optional dependency on the Naginator Plugin. Unfortunately this optional dependency still causes the error, that the Naginator Plugin can't be uninstalled from the Web UI. Hence I've removed it the hard way on the command line. I've also updated core and plugins to the latest versions while I was at it. Please re-open if restarting the Webtools JIPP does not work. At 15:12 I requested installation of an update for the SonaQube Scanner, then checked the box for "restart Jenkins when no jobs are running" so that it'd restart and activate the plugin immediately. At 15:34 https://hudson.eclipse.org/webtools/ was still showing: "This CI instance is currently unavailable. It may be turned off, or it may be unresponsive. Members of the project can restart this service using the CI Control tools in their Eclipse Foundation account (login required). "If the problem persists, please contact the project team on their forum or file a bug." Reopen'd. Webtools JIPP has been restarted. I just did an automated update to the latest Jenkins version in the WTP JIPP. At 11:55am ET, with no jobs running, I launched a restart by clicking the "restart when no jobs are running". At 11:58pm ET, https://ci.eclipse.org/webtools/ still shows: "This CI instance is currently unavailable. It may be turned off, or it may be unresponsive. Members of the project can restart this service using the CI Control tools in their Eclipse Foundation account (login required). If the problem persists, please contact the project team on their forum or file a bug." By 12:01pm, the server had returned. So looks like this is actually fixed. :D I just restarted it @ 12:01pm :) https://ci.eclipse.org/datatools/ restart triggered over an hour ago. "This CI instance is currently unavailable. It may be turned off, or it may be unresponsive. Members of the project can restart this service using the CI Control tools in their Eclipse Foundation account (login required). "If the problem persists, please contact the project team on their forum or file a bug." Still not up. So... yeah, this is STILL a problem. https://hudson.eclipse.org/webtools/ is also down, please restart it as well. https://ci.eclipse.org/datatools/ is still dead. Created attachment 274912 [details] account page claims DTP CI JIPP is running And to add insult to injury, the CI CONTROL panel on my account page [1] says it's up. [1] https://accounts.eclipse.org/users/nickb Now getting 502 proxy error: Proxy Error The proxy server received an invalid response from an upstream server. The proxy server could not handle the request GET /datatools/. Reason: Error reading from remote server -- https://ci.eclipse.org/datatools/ Webtools JIPP is up and I've restarted the datatools JIPP. Restart capabilities of Jenkins became unreliable sometime after LTS 2.73.x At the moment there is not much we can do about it. We expect this to improve again with JIPPs being provisioned in the cluster environment (CJE). Thanks, it's working for me now. Sign me up for being an early candidate to move to CJE, for DTP and TM JIPPs. If that goes smoothly, same for WTP. |