Community
Participate
Working Groups
Created attachment 277546 [details] Full stack trace of this NoClassDefFoundError Our application is based on Eclipse RCP and Equinox Target Components. It consists of a equinox/osgi-driven server and a eclipse-rcp-driven client. The bundle org.eclipse.equinox.console.ssh is used server-side to enable login via ssh for monitoring purposes. After our upgrade from Eclipse Oxygen to Eclipse 2018-12 and it’s corresponding Equinox Target Components, we hit a java.lang.NoClassDefFoundError: org/apache/sshd/server/auth/AsyncAuthException as soon our server starts up. The stack trace is attached. As i understand the trace, the problems seems to be the following: As of org.eclipse.equinox.console.ssh_1.0.200 the class org.eclipse.equinox.console.ssh.SshServ imported (among others) org.apache.sshd.server.PasswordAuthenticator; As of org.eclipse.equinox.console.ssh_1.2.0.v20181107-1024 this changed to org.apache.sshd.server.auth.password.PasswordAuthenticator of the corresponding newer version of sshd core. While the old PasswordAuthenticator-Interface (part of sshd core 0.8.0) demanded the following boolean authenticate(String username, String password, ServerSession session); the new one from sshd core 2.0.0 now demands boolean authenticate(String username, String password, ServerSession session) throws PasswordChangeRequiredException, AsyncAuthException; As i understand the problem, loading SshServ now triggers Java into this class, seeing the new signature and wondering what AsyncAuthException is (it's derived from RuntimeException but Java has no way to see that beforehand) As AsyncAuthException is part of the package org.apache.sshd.server.auth that org.eclipse.equinox.console.ssh currently does not import in the osgi manifest, this will fail. Therefore our current solution would be to add the following to Import-Package of org.eclipse.equinox.console.ssh’s MANIFEST.MF: org.apache.sshd.server.auth;version="2.0.0"
(In reply to Josef Härtl from comment #0) > Therefore our current solution would be to add the following to > Import-Package of org.eclipse.equinox.console.ssh’s MANIFEST.MF: > org.apache.sshd.server.auth;version="2.0.0" Have you confirmed that adding the import works?
As of now, hacking the Manifest.mf this way is our current workaround, yes. This gets our server up and running and accepting ssh connections. However login itself (using a Custom JAAS Authentication Login Module as described in https://help.eclipse.org/2018-12/index.jsp?topic=%2Forg.eclipse.platform.doc.isv%2Fguide%2Fconsole_shell.htm) still does not work (in the end, JaasPasswordAuthenticator.authenticate fails to loginContext.login() because our custom module is not found on the classpath anymore) I'm still investigating on that, but guess this will become a new ticket.
I constructed a minimal example project to demonstrate the error. While constructing the example it got clear both problems somehow originate in having osgi security enabled. More specifically, by providing -Dorg.osgi.framework.security=osgi and -Djava.security.policy. For our scenario, osgi security is mission critical. My example project is a single Plug-in Project that uses the currently running Eclipse as its target platform. Normally we would not do this, but this keeps the example very small and easy to build while showing our problem. The example project does not use any external libraries. It implements a minimum LoginModule simply accepting anyone. Again, our setup is much more complex, but it shows our problem. Import my sample project as an “Existing project”. Afterwards Start.launch should provide an OSGi Framework launcher of the same name. [Both environments use Oracle Java 8u191] My first test configuration is an environment as used in our current build: 1. Download Oxygen.2 RCP (we use https://www.eclipse.org/downloads/download.php?file=/technology/epp/downloads/release/oxygen/2/eclipse-rcp-oxygen-2-win32-x86_64.zip) 2. Therein install Equinox Target Components (with the version that was current back then = 3.13.2.v20171108-1834) (via install new Software) In this environment everything works fine. My second test configuration is the environment we’re currently porting to: 1. Download Eclipse 2018-12 RCP (we use https://www.eclipse.org/downloads/download.php?file=/technology/epp/downloads/release/2018-12/R/eclipse-rcp-2018-12-R-win32-x86_64.zip) 2. Therein install Equinox Target Components (in my case 3.16.0.v20181130-2106)(via install new Software) In this environment, we receive the manifest problem as soon the launcher is started. As mentioned before, this may yet be workarounded by inserting the additional Import-Package org.apache.sshd.server.auth;version="2.0.0" into the org.eclipse.equinox.console.ssh_1.2.0.v20181107-1024 of this eclipse installation. After restarting eclipse with that, the server boots up. But as soon a connection with putty/kitty is initiated at localhost:4222, it fires the second error: Our LoginModule is no longer to be found. I attached the log of this error to Classloader.log. At first glance this seems to originate in a wrong classloader being used: The error starts in javax.security.auth.login.LoginContext.invoke(String), more precisely in Class<?> c = Class.forName(moduleStack[i].entry.getLoginModuleName(),true, contextClassLoader); (around line 710) The contextClassLoader used herein is determined upon construction of the LoginContext in JaasPasswordAuthenticator.authenticate. Indirectly, Java initializes it with the currently present classloader. In our case, however, this is a sun.misc.Launcher$AppClassLoader, whereas in Oxygen.2 (and by not using osgi security) it was org.eclipse.osgi.internal.framework.ContextFinder. In this classloader (not even using osgi anymore??), our custom loginmodule does not exist, therefore resulting in the error in Classloader.log.
Created attachment 277572 [details] Classloader.log
Created attachment 277573 [details] Sample project for reproducing both bugs
Anjum, can you have a look.
Hi Josef, I tried to reproduce the problem and I was successful with Eclipse 2018-12, but I am unable to test that the example works on Oxygen. I am getting the following error when doing ssh. Unable to negotiate with ::1 port 4250: no matching key exchange method found. Their offer: diffie-hellman-group1-sha1 Do you find this error familiar?
(In reply to Anjum Fatima from comment #7) > Hi Josef, > I tried to reproduce the problem and I was successful with Eclipse 2018-12, > but I am unable to test that the example works on Oxygen. I am getting the > following error when doing ssh. > > Unable to negotiate with ::1 port 4250: no matching key exchange method > found. Their offer: diffie-hellman-group1-sha1 > > Do you find this error familiar? Hi Anjum, the specific error does not sound that familiar to me. But to me it looks like a typical case where your ssh client is just using incompatible settings. Also, what about the port? Out of the box, the example should run on port 4222, not 4250. To reproduce my ssh client environment: We are using PuTTY 0.7. Mostly standard settings with an IUTF8 do not send flag (don't know if this is still required). I've attached our configuration, in reg format as described in https://stackoverflow.com/questions/13023920/how-to-export-import-putty-sessions-list.
Created attachment 277600 [details] Our configuration for PuTTY 0.70
Hi Josef, I debugged very deeply, it looks like changes in the org.apache.sshd bundle is causing an innocuous thread to be created which is not inheriting the contextClassLoader(ContextFinder) set by equinox container. The below thread gets created when you do ssh to the server. Daemon System Thread [anInnocuousThread] (Suspended (breakpoint at line 248 in ThreadUtils$SshdThreadFactory)) ThreadUtils$SshdThreadFactory.newThread(Runnable) line: 248 ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor, Runnable) line: 619 ThreadPoolExecutor.addWorker(Runnable, boolean) line: 932 ThreadPoolExecutor.execute(Runnable) line: 1367 KQueuePort(AsynchronousChannelGroupImpl).executeOnPooledThread(Runnable) line: 188 Invoker.invokeIndirectly(AsynchronousChannel, CompletionHandler<V,? super A>, A, V, Throwable) line: 212 Invoker.invoke(AsynchronousChannel, CompletionHandler<V,? super A>, A, V, Throwable) line: 188 UnixAsynchronousServerSocketChannelImpl.onEvent(int, boolean) line: 193 KQueuePort$EventHandlerTask.run() line: 301 InnocuousThread(Thread).run() line: 748 InnocuousThread.run() line: 74 You can also verify this by adding a breakpoint at line 248 in org.apache.sshd.common.util.threads.ThreadUtils$SshdThreadFactory
New Gerrit change created: https://git.eclipse.org/r/137408
Thanks for the analysis, Anjum i picked up there and investigated a bit further. I guess it’s rather not a sshd bug. It's rather difficult to explain ... so one thing after another: Start the application; wait until loaded. Now, set the breakpoint on ThreadUtils.class:248 Add an expression on “Thread.currentThread()” – it shows that the inheritedAccessControlContext uses not the osgi context but an empty protection domain (so, Java itself) Furthermore, consider the stack trace up to this breakpoint as it halts: ThreadUtils$SshdThreadFactory.newThread(Runnable) line: 248 ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor, Runnable) line: 619 ThreadPoolExecutor.addWorker(Runnable, boolean) line: 932 ThreadPoolExecutor.execute(Runnable) line: 1367 Iocp(AsynchronousChannelGroupImpl).executeOnPooledThread(Runnable) line: 188 Invoker.invokeIndirectly(AsynchronousChannel, CompletionHandler<V,? super A>, A, V, Throwable) line: 212 Invoker.invokeIndirectly(PendingFuture<V,A>) line: 313 WindowsAsynchronousServerSocketChannelImpl$AcceptTask.completed(int, boolean) line: 272 Iocp$EventHandlerTask.run() line: 397 [local variables unavailable] InnocuousThread(Thread).run() line: 748 [local variables unavailable] InnocuousThread.run() line: 74 [local variables unavailable] So, it’s basically beginning *somewhere* outside of sshd. sshd wants a new thread t (the thing we breakpointed on), yes, but it seems the thread/context information is flawed even before that. Looking up what an InnocuousThread is (numerous github-Results show up on google): It’s a class described as the following: “A thread that has no permissions, is not a member of any user-defined ThreadGroup and supports the ability to erase ThreadLocals.” Numerous functions in it do not hide the fact that this essentially means: A thread that does not care about security and always uses the system class loader (probably kicking osgi out by doing so) org.apache.sshd.core.source_2.0.0.v20181102-1323 does not seem to create an InnocuousThread itself. At least, not willingly / explicitly (searched the source files). Tracking back it’s creation suggests Nio2 is involved / it’s coming from Nio2ServiceFactory. As that, i guess it could be related to https://issues.apache.org/jira/browse/SSHD-332 where it’s implied that such effects exists inside Nio2. https://github.com/apache/mina-sshd/commit/bb2eb2b5f3192943ff9c23b391b44cad9899aeeb#diff-0923b58bc72c7eeda0b7c65559d3ead5 suggests that sshd 0.9.0 (remember: oxygen 2 used 0.7.0, eclipse 2018-12 uses 2.0) moved from using mina to mina or nio2. Currently it seems to use nio2 (seems to be the default) whereas as of oxygen2 it did not do so. The question would be: why? Mina would be present as a bundle (whether it’s already correctly tethered to sshd is another question) Much further investigation later: The option introduced in https://github.com/apache/mina-sshd/commit/bb2eb2b5f3192943ff9c23b391b44cad9899aeeb#diff-0923b58bc72c7eeda0b7c65559d3ead5 still exists, although under a slightly different name. As of now, it’s usuable via the VM argument (just add to the Start launcher) -Dorg.apache.sshd.common.io.IoServiceFactoryFactory=org.apache.sshd.common.io.mina.MinaServiceFactoryFactory However, for me this results in a Exception in thread "equinox ssh" java.lang.IllegalStateException: Unable to create instance of class org.apache.sshd.common.io.mina.MinaServiceFactoryFactory at org.apache.sshd.common.io.DefaultIoServiceFactoryFactory.newInstance(DefaultIoServiceFactoryFactory.java:140) at org.apache.sshd.common.io.DefaultIoServiceFactoryFactory.newInstance(DefaultIoServiceFactoryFactory.java:73) at org.apache.sshd.common.io.DefaultIoServiceFactoryFactory.getIoServiceProvider(DefaultIoServiceFactoryFactory.java:59) at org.apache.sshd.common.io.DefaultIoServiceFactoryFactory.create(DefaultIoServiceFactoryFactory.java:49) at org.apache.sshd.common.helpers.AbstractFactoryManager.getIoServiceFactory(AbstractFactoryManager.java:107) at org.apache.sshd.server.SshServer.createAcceptor(SshServer.java:386) at org.apache.sshd.server.SshServer.start(SshServer.java:289) at org.eclipse.equinox.console.ssh.SshServ.run(SshServ.java:69) … hinting at a problem in the integration in the mina/sshd/eclipse console ssh-Bundles eclipse 2018-12 uses. Further investigation showed that this class is now part of https://mvnrepository.com/artifact/org.apache.sshd/sshd-mina/2.2.0, which is not part of Eclipse 2018-12 / Equinox. As seen in the version history of this jar, it was introduced in 2.0.0, whereas sshd in oxygen2 was 0.7.0, so before https://github.com/apache/mina-sshd/commit/bb2eb2b5f3192943ff9c23b391b44cad9899aeeb#diff-0923b58bc72c7eeda0b7c65559d3ead5 and therefore always used mina.
PS: The mina parameter should become more clear if viewing BuiltinIoServiceFactoryFactories of sshd core 2.0.0 that's essentially interpreted in org.apache.sshd.common.io.IoServiceFactoryFactory#newInstance
Gerrit change https://git.eclipse.org/r/137408 was merged to [master]. Commit: http://git.eclipse.org/c/equinox/rt.equinox.bundles.git/commit/?id=f00ebea9045c979ba07104fd25befa5853711fc1
is this fixed?
(In reply to Dani Megert from comment #15) > is this fixed? Hello Dani, sadly this problem is not yet fixed. Here is some additional information, which I found regarding the sshd-jar: The old sshd ticket from around 0.12 seems to suggest that initially mina was used instead nio2 because nio2 misbehaved if confronted with security and oracle marked it "won't fix". 0.9.0 introduced an abstraction that allowed to choose the implementation and it seems 2.0.0 pushed the mina one into a separate jar that serves as a link between sshd core and mina core. Could you please look into it, maybe you could find some clues in there.
(In reply to Josef Härtl from comment #16) > (In reply to Dani Megert from comment #15) > > is this fixed? > > Hello Dani, > sadly this problem is not yet fixed. > Here is some additional information, which I found regarding the sshd-jar: > The old sshd ticket from around 0.12 seems to suggest that initially mina > was used instead nio2 because nio2 misbehaved if confronted with security > and oracle marked it "won't fix". 0.9.0 introduced an abstraction that > allowed to choose the implementation and it seems 2.0.0 pushed the mina one > into a separate jar that serves as a link between sshd core and mina core. > Could you please look into it, maybe you could find some clues in there. Tom, can you help?
(In reply to Dani Megert from comment #17) > (In reply to Josef Härtl from comment #16) > > (In reply to Dani Megert from comment #15) > > > is this fixed? > > > > Hello Dani, > > sadly this problem is not yet fixed. > > Here is some additional information, which I found regarding the sshd-jar: > > The old sshd ticket from around 0.12 seems to suggest that initially mina > > was used instead nio2 because nio2 misbehaved if confronted with security > > and oracle marked it "won't fix". 0.9.0 introduced an abstraction that > > allowed to choose the implementation and it seems 2.0.0 pushed the mina one > > into a separate jar that serves as a link between sshd core and mina core. > > Could you please look into it, maybe you could find some clues in there. > Tom, can you help? I don't think we have a known solution to fixing this in time for 4.11. Will need to wait for 4.12.
Meanwhile i found some time to investigate this a bit further. # My Environment For my explanations i am still referring to the previous environment consisting of Eclipse 2018-12, Equinox Target Components 3.16.0.v20181130-2106 and the modified Manifest. Working with Eclipse 2019-03 is more or less possible when starting up my second sample directly, but replicating my changes below definately is no fun because of https://bugs.eclipse.org/bugs/show_bug.cgi?id=545614. # Result SSH can be made to work with osgi security enabled # What i changed I uploaded an updated version of my sample. Here's what i needed to change in my code and/or the jar of Equinox. First, as shown above, NIO2 seems to have known problems with security. However it was declared default sometime between Oxygen.2 (sshd 0.7.0) and 2018-12 (sshd 2.0.0), causing the problems with ssh while osgi security is enabled. As outlined above, one can still force sshd into Mina mode. To achieve that, add the following to the VM Arguments of the Launcher: > -Dorg.apache.sshd.common.io.IoServiceFactoryFactory=org.apache.sshd.common.io.mina.MinaServiceFactoryFactory Then add the jars of a matching version of https://mvnrepository.com/artifact/org.apache.sshd/sshd-mina containing this MinaServiceFactoryFactory. I chose https://mvnrepository.com/artifact/org.apache.sshd/sshd-mina/2.0.0 to be compatible with eclipse’s sshd-core 2.0.0. More precisely, i put sshd-mina-2.0.0.jar and sshd-mina-2.0.0-sources.jar into either the plugins or the dropins directory to make them known to our target platform. After restarting eclipse, add sshd.mina to the launch configuration. At this point, starting the server will halt at SshServer:109 and afterwards fail with > Exception in thread "equinox ssh" java.lang.IllegalStateException: Unable to create instance of class org.apache.sshd.common.io.mina.MinaServiceFactoryFactory > at org.apache.sshd.common.io.DefaultIoServiceFactoryFactory.newInstance(DefaultIoServiceFactoryFactory.java:140) > at org.apache.sshd.common.io.DefaultIoServiceFactoryFactory.newInstance(DefaultIoServiceFactoryFactory.java:73) > at org.apache.sshd.common.io.DefaultIoServiceFactoryFactory.getIoServiceProvider(DefaultIoServiceFactoryFactory.java:59) > at org.apache.sshd.common.io.DefaultIoServiceFactoryFactory.create(DefaultIoServiceFactoryFactory.java:49) > at org.apache.sshd.common.helpers.AbstractFactoryManager.getIoServiceFactory(AbstractFactoryManager.java:107) > at org.apache.sshd.server.SshServer.createAcceptor(SshServer.java:386) > at org.apache.sshd.server.SshServer.start(SshServer.java:289) > at org.eclipse.equinox.console.ssh.SshServ.run(SshServ.java:69) ... because we still have to import sshd.mina to something. Inspecting the function of DefaultIoServiceFactoryFactory.java:140 shows that the classloader of the current thread is used. More precisely, in the end, it uses the classloader of sshd-core. As both our sample and our application are fragments of sshd-core anyway, i chose to simply import it there, so add the following to import-package of MANIFEST.MF of the sample project: > ,org.apache.sshd.common.io.mina;version="2.0.0" Furthermore, add -clean to the Launcher’s "Program arguments" to prevent some weird caching errors like "Unresolved requirement: Fragment-Host: org.apache.sshd.core" Starting the server up, now results in a different exception: > Exception in thread "equinox ssh" java.lang.NoSuchMethodError: org.apache.mina.core.service.SimpleIoProcessorPool.<init>(Ljava/lang/Class;Ljava/util/concurrent/Executor;ILjava/nio/channels/spi/SelectorProvider;)V > at org.apache.sshd.common.io.mina.MinaServiceFactory.<init>(MinaServiceFactory.java:47) > at org.apache.sshd.common.io.mina.MinaServiceFactoryFactory.create(MinaServiceFactoryFactory.java:49) > at org.apache.sshd.common.io.DefaultIoServiceFactoryFactory.create(DefaultIoServiceFactoryFactory.java:50) > at org.apache.sshd.common.helpers.AbstractFactoryManager.getIoServiceFactory(AbstractFactoryManager.java:107) > at org.apache.sshd.server.SshServer.createAcceptor(SshServer.java:386) > at org.apache.sshd.server.SshServer.start(SshServer.java:289) > at org.eclipse.equinox.console.ssh.SshServ.run(SshServ.java:69) Looking closely at https://mvnrepository.com/artifact/org.apache.sshd/sshd-mina/2.0.0 reveals that sshd-mina was built for mina-core 2.0.17. Eclipse 2018-12 (as well as 2019-03)’s Equinox Target Components however only provide mina-core 2.0.7. In our project we currently use mina-core 2.0.18, but 2.0.17 would suffice. So put the jar and sources from https://mvnrepository.com/artifact/org.apache.mina/mina-core/2.0.18 (or .17) into dropins or plugins and changed the start configuration to use this version instead of 2.0.7. Now, finally, the server starts up and accepts logins via SSH. Although at this point, PuTTY 0.7.1 tended to kick me out very quickly, saying "invalid MAC". After some research, this seems to originate from the sample logging everything out it can (DEBUG). Following the PuTTY log this sends a lot of packages to PuTTY with PuTTY seemingly unable to keep up. To prevent this from happening i reduced the logging to INFO in the project by specifying VM Argument -Dlogback.configurationFile=../../logback.xml and putting a minimalistic logback.xml there. After that the sample worked smoothly for me. Could be a problem with PuTTY / KiTTY. Could be a problem with the old (2.0.0) version of sshd-core managing the transmission (PuTTY / KiTTY: "Event Log: Server version: SSH-2.0-SSHD-CORE-2.0.0"). Hard to tell, i could not swap out the old sshd-core just like that. Wasn't that relevant to us anyway as putting a productive server into DEBUG mode isn't that realistic anyway. Probably it wouldn't hurt to update to a recent version of sshd-core sooner or later. # Suggestions - As for the required jars sshd-mina and the newer version of mina-core, it would be much better if Equinox delivered them from the start. - As for the import of org.apache.sshd.common.io.mina to the project fragmenting sshd-core: https://help.eclipse.org/2019-03/index.jsp?topic=%2Forg.eclipse.platform.doc.isv%2Fguide%2Fconsole_shell.htm already mentions "Also, it will be necessary to create a fragment to sshd-core bundle, with which to import the package of the custom login module", so one could simply add documentation that this import might be necessary as well. A possible alternative could be adding the import as optional to sshd-core itself (as sshd-core is part of eclipse-orbit's custom jar anyway) - As for the rest (selecting sshd-mina into the start configuration, adding the IoServiceFactoryFactory-VM-Parameter): This could be included into https://help.eclipse.org/2019-03/index.jsp?topic=%2Forg.eclipse.platform.doc.isv%2Fguide%2Fconsole_shell.htm (or a newer release of this site) - Perhaps consider updating sshd 2.0.0 to 2.2.0 to get sshd and respective sshd-mina fixes.
Created attachment 277947 [details] Updated sample for new comment 2019-03-21 08:29:11
Any update on this?
(In reply to Josef Härtl from comment #21) > Any update on this? No, unfortunately I think we are being faced with an issue with the console ssh getting some serious bit rot. The original maintainers of the support are no longer involved in the project and the remaining active contributors to the project don't have the necessary interest or resources to change the current state of the code. I'll be happy to review any contributing fixes to this. But without that I may have to recommend we deprecate the ssh bundle and mark it for future removal from Equinox. Marking for 4.13 to either get to the bottom of this or consider deprecation of the bundle.
Given the fact that ssh is mission critical for our application, i will try to create such a proposal - if my management agrees with it. My employer has no policy for contributing to foreign projects as resources already barely cover our own projects (and current eclipse migrations already did hit us hard). That said - technically, i already implanted a workaround for this into our application. Perhaps it can be extended into a proposal. Much of the expected additional effort would be getting started with equinox contribution. I would be grateful for input on how to quickstart a contribution to Equinox. Ideally also an explanation to these ECAs as i lack access to a legal advisor. Are there any links/articles that might help me?
(In reply to Josef Härtl from comment #23) > Ideally also an explanation to these ECAs as i lack access to a > legal advisor. Are there any links/articles that might help me? See https://www.eclipse.org/legal/ecafaq.php
Thanks for the link on the ECA explanation. I also found https://projects.eclipse.org/projects/eclipse.equinox/developer and https://github.com/eclipse/rt.equinox.p2/blob/master/CONTRIBUTING regarding the Equinox code base and https://www.eclipse.org/projects/handbook/ on Eclipse in general. Also, https://www.eclipse.org/equinox/documents/coding.php on coding style. I was very short on time in the last weeks and will be away until early July. I am on my own on this, therefore the frequent longer pauses in this ticket. Hopefully, i will find time to continue by then. Is there some resource on getting the Equinox code base up and running? "Getting Started" on https://projects.eclipse.org/projects/eclipse.equinox leads nowhere and https://wiki.eclipse.org/Category:Equinox at first glance only seems to address very particular problems. The code base sure is a start, but just looking at our own Eclipse-based application, i guess it's just that.
At last i found some time to get back to this. I wasn't able to find enough information to create a fully working development environment for equinox; it was good enough to do experiments on console.ssh and related projects, but i doubt it was the way meant to be. Even so, it became clear that this problem probably isn't solveable within the code of console.ssh. Technically, it's not a bug in sshd, sshd-mina or mina either. It's a problem within the dependency chain, versions used and configuration (which mostly would be up to the one using it). # Analysis of the problem I retraced the problem again to see if there could be an alternative solution that could be made to work out of the box without correcting dependencies. In brief: no. In detail: org.apache.sshd.common.io.DefaultIoServiceFactoryFactory.getIoServiceProvider() from org.apache.sshd.core has to choose an implementation. By default, it uses NIO2. It can be switched by specificing -Dorg.apache.sshd.common.io.IoServiceFactoryFactory=org.apache.sshd.common.io.mina.MinaServiceFactoryFactory. Comparing the implementations of NIO2 and Mina, the problem is how Nio2ServiceFactory uses an AsynchronousChannelGroup to create threads working on incoming ssh requests. Whereas Mina uses the SimpleIoProcessorPool of sshd itself. In the end, NIO2 ends up in an AccessController.doPrivileged(sun.nio.ch.Iocp$EventHandlerTask) inside sun.nio.ch.ThreadPool.defaultThreadFactory() which then decides to create an InnocuousThread rather than a normal one. This kind of thread doesn't carry over much, which in turn leads to the creation of ssh working threads that do no longer know their osgi heritage, hence cannot access a LoginModule that's not part of their own classloader. For this reason, the problem surfaces as ClassNotFound -the LoginModule-. As mentioned further above, the problems start with -Dorg.osgi.framework.security=osgi set. But this isn't specific to osgi-Security, but Security in general: As soon a SecurityManager is set, org.apache.sshd.common.util.threads.ThreadUtils.SshdThreadFactory.newThread(Runnable) has to create the thread by a AccessController.doPrivileged (which decides on only creating an InnocuousThread) whereas with no security set, a normal Thread can be created directly. While the normal thread inherits osgi, InnocuousThread does not (or, at least, does not for its' children). This confirms https://issues.apache.org/jira/browse/SSHD-332 where someone states that NIO2 won't work with security and Oracle does not intend to fix it. Checked myself: The piece of code mentioned there was introduced in a beta of Java 8 and remains there mostly unchanged, even in Java 12. SSHD solved the problem by creating/re-introducing the Mina implementation. As i saw no code switches one could use and Oracle seemingly does not want to fix NIO2, this seems to be the only way. # Further look on the dependencies - As of sshd.core 0.8.0 this problem did not exist (as above ticket already suggests and source confirms): NIO2 was not present as of 0.8.0. - As of sshd.core 1.2.0 NIO2 was used, but MinaServiceFactoryFactory was introduced into sshd.core. - As of sshd.core 2.0.0, MinaServiceFactoryFactory was split from the main project org.apache.sshd.core, moved to the sshd-mina extension. So, sshd.core alone is no longer sufficient. Furthermore, this new sshd-mina dependency requires a more recent version of org.apache.mina.core. Currently, Equinox comes with version 2.0.7. This version is as old as 2012 and is not compatible with sshd-mina from 2018. Enabling sshd-mina on that constellation results in a NoSuchMethodError [as also found in https://lists.opendaylight.org/pipermail/bugs/2016-August/027401.html]. So, technically, this is a subtle dependency problem introduced by upgrading from sshd.core 0.8.0 to sshd.core 2, which seems to have happened somehere between Eclipse 2018-09 ~ Equinox 3.15.0 and Eclipse 2018-12 ~ Equinox 3.16.0. Sources: - https://mvnrepository.com/artifact/org.apache.sshd/sshd-core - https://mvnrepository.com/artifact/org.apache.mina/mina-core/2.0.7 - https://mvnrepository.com/artifact/org.apache.sshd/sshd-mina/2.0.0 # Further look on the parameter With the sshd-mina-Implementation present, you need to advertise it to org.apache.sshd.common.io.DefaultIoServiceFactoryFactory.getIoServiceProvider(). This happens by specifying the vm parameter -Dorg.apache.sshd.common.io.IoServiceFactoryFactory=org.apache.sshd.common.io.mina.MinaServiceFactoryFactory I guess this parameter can or should not be hardcoded into equinox. I would opt for mentioning it in https://help.eclipse.org/2019-06/index.jsp?topic=%2Forg.eclipse.platform.doc.isv%2Fguide%2Fconsole_shell.htm instead. - First, because coding it would be prone to loading order issues. The property needs to be set before sshd.core decides *what* to initialize. So, better leave it to the launcher than setting it halfway through runtime. - Second, because hardcoding it might be a risk itself. Applications without security can also use the NIO2 default. Perhaps there are use-cases requiring NIO2 rather than Mina. # Possible solution for the dependencies In general, the solution would be to update to a compatible combination of sshd-core, mina-core and sshd-mina. A rather easy solution would be a combination of - sshd-core 2.0.0 (as currently present) - mina-core 2.0.21 (from 2.0.7 as of now) - sshd-mina 2.0.0 (new) as this - only moves the fix-version of mina-core (so, minimal risk of breaking other manifest references / worked out of the box for me) - does not change API. No code-change required. This would also address CVE-2019-0231 as mentioned in https://mina.apache.org/mina-project/index.html. Importing sshd-mina can be left up to the user and simply be put into documentation. More details below. # Optional: Further simplified solution It already became apparent that you are rather short on resources. The same goes for me. De facto i am the last maintainer of my project and already heavily struggling to keep up. https://wiki.eclipse.org/Orbit/Adding_Bundles_to_Orbit is beyond what i can afford. So, another possibility would be to leave sshd-mina up to the user/developer. sshd-mina already is a valid osgi bundle. The effort of integrating it into one's own project is minimal. Including the jar, setting the manifest-import. Done. Perhaps that could also enable a major simplification of the dependency process in eclipse. Still, i could not afford to maintain a dependency in eclipse, so, generally, i would opt for documentation. In this case, the fix would boil down to: 1. Update mina-core from 2.0.7 to 2.0.21, fixing CVE-2019-0231 by the way. 2. https://help.eclipse.org/2019-06/index.jsp?topic=%2Forg.eclipse.platform.doc.isv%2Fguide%2Fconsole_shell.htm: New section "osgi security": The user/developer has to provide the sshd-mina jar and has to set an import to org.apache.sshd.common.io.mina. His launcher must also include the vm argument -Dorg.apache.sshd.common.io.IoServiceFactoryFactory=org.apache.sshd.common.io.mina.MinaServiceFactoryFactory to use Mina instead of NIO2. The updated sample is still good for this. Adjust the launcher to use the updated mina-core and a newly inserted sshd-mina and go.
This bug hasn't had any activity in quite some time. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're closing this bug. If you have further information on the current state of the bug, please add it and reopen this bug. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. -- The automated Eclipse Genie.
