Community
Participate
Working Groups
Created attachment 278043 [details] Debug log After upgrading from 5.2.0 to 5.3.0 (today), our JGit no longer is able to connect to our server. We also cannot roll back as the old EGit/JGit repository has disappeared. SshSessionFactory.getInstance().getSession() never returns and does not throw an exception. The debug log is attached. The connection worked code worked immediately prior to the upgrade today. At the very least, please provide an means of reverting to the prior release (5.2.0). We are in ECLIPSE Mars.2 and have suffered a complete outage as a result of upgrading through this trap door.
1. Are you using Egit, or is this a JGit stand-alone? 2. I notice you are using the Apache MINA sshd client. If you're using EGit, you can switch back to Jsch in the preferences (Team->Git). BTW, I take exception at your labelling this switch a "trap door"; it was mentioned in the New&Noteworthy at https://wiki.eclipse.org/EGit/New_and_Noteworthy/5.3#SSH_Library , and it's mentioned in the release notes at https://projects.eclipse.org/projects/technology.egit/releases/5.3.0 . If that caused a "complete outage" that's regrettable, but it just means you blindly upgraded and rolled out without testing. 3. Interestingly, in your trace it looks as if this is a problem between Apache MINA sshd and whatever server you are using. The welcome banner is correctly received and logged, EGit then does nothing with it, and the very next thing is that the ssh client gets a SSH_MSG_USERAUTH_FAILURE from the server. Can you reproduce this with plain JGit from the command line using the option "--ssh apache" (without the double quotes)? Can you reproduce it with a supported Eclipse version (Eclipse Neon or newer)? JGit stand-alone shouldn't use the Apache MINA ssh client by default. I'll try to reproduce this locally, but I'll first have to figure out how to give the sshd on my Mac a banner in the first place. 4. Eclipse Mars is not supported anymore by EGit 5.3.0. Minimum supported platform is Eclipse Neon.0. (Though I don't think this has anything to do with this problem.) This is mentioned on the update site itself: https://download.eclipse.org/egit/updates/ and again in the Egit New&Noteworthy: https://wiki.eclipse.org/EGit/New_and_Noteworthy/5.3#Requires_Eclipse_Neon . 5. The old repository still exists. See https://wiki.eclipse.org/EGit/FAQ#Where_can_I_find_older_releases_of_EGit.3F
1. I am using both EGit and JGit directly. The fault we encountered was when using the SshSessionFactory. 2. Very good to know. I can verify that this did correct the problem. I am very relieved and this moves the situation way down from a Blocker. 3. I don't have a standalone JGit instance so that's difficult. 4. Understood. I am constrained by Mars because of other plugins that are not yet upgraded, but will try the function on a more recent version without those and report back - it might take a few days to move the infrastructure over to Neon as I have other problems with test plugin not building properly there because of a dependency on something in Mars. 5. Thanks. The ECLIPSE Revert function did not work because the old repository, where 5.2.0 came from was not available when I tried to revert (multiple times over an hour) That's why I used the bad phrase "trap door", for which I sincerely apologize. I just tested the Revert again and it does work so perhaps it was a load issue on the server. I do keep backup configurations so it was not a blind upgrade. The specific SSH server has been rock-solid for years with JSch and rigorously tested. It might not be header-related but a protocol difference from MINA. The server is configured to only allow key pairs. keyboard auth is prohibited. If I might suggest, there should be a minimum version dependency that prevents 5.3.0 from installing on Mars, if it is not supported.
(In reply to Randall Becker from comment #2) > If I might suggest, there should be a minimum version dependency that > prevents 5.3.0 from installing on Mars, if it is not supported. Agreed, but we didn't get around to this for 5.3.0. We just did so in 5.4.0, though I still have to test that trying to install 5.4.0 on Mars will indeed fail.
(In reply to Randall Becker from comment #2) > The specific SSH server has been rock-solid for years with JSch and > rigorously tested. It might not be header-related but a protocol difference > from MINA. The server is configured to only allow key pairs. keyboard auth > is prohibited. I don't doubt the server, but I suspect that Apache MINA sshd may have a bug. Maybe indeed a "protocol difference". Pubkey auth should not be the problem; I use the MINA implementation since it was introduced in EGit/JGit 5.2.0 with key pairs and never had a problem. I'll keep looking. Just gave my Mac's sshd a banner...
Cannot reproduce on my Mac, cloning a repo from localhost via ssh. I do see the banner being received and being ignored by EGit, and then everything continues normally. So it might be a problem with the client-side configuration, or a Windows-specific problem in Apache MINA sshd. I looked again in the log and in the Apache MINA sshd sources, and the log is in fact completely normal: 14:36:54.276 sendInitialServiceRequest() Send SSH_MSG_SERVICE_REQUEST for ssh-userauth Client tells server it wants to authenticate the user 14:36:54.276 writePacket() Writing 64 bytes 14:36:54.277 handleCompletedWriteCycle() finished writing len=64 These 64 bytes are that SSH_MSG_SERVICE_REQUEST 14:36:54.277 writePacket() Writing 80 bytes 14:36:54.277 handleNewKeys() sent 1 pending packets 14:36:54.277 handleCompletedWriteCycle() finished writing len=80 Pending message relating to the host key exchange also got written 14:36:54.450 handleReadCycleCompletion() read 48 bytes 14:36:54.451 handleServiceAccept() SSH_MSG_SERVICE_ACCEPT service=ssh-userauth Client got 48 bytes SSH_MSG_SERVICE_ACCEPT from the server 14:36:54.471 handleReadCycleCompletion() read 320 bytes Client got 320 bytes SSH_MSG_USERAUTH_BANNER message 14:36:54.471 process() Welcome banner(lang=): This system is a private system for use by Connect Worldwide, Your Independent HP Business Technology Community Any unauthorized access to this system is not allowed and will be audited. If you have accessed this site without authorization, please disconnect immediately. Client logs the welcome message; EGit ignores it. 14:36:54.474 handleReadCycleCompletion() read 48 bytes 14:36:54.474 processUserAuth() Received SSH_MSG_USERAUTH_FAILURE - partial=false, methods=publickey This is normal. Apache MINA sshd first sends a USERAUTh request with a "none" authentication method to get this failure, with which the server will send its accepted authentication mechanisms. So now the client will determine if it has a mechanism that the server can accept. 14:36:54.474 tryNext() starting authentication mechanisms: client=[publickey], server=[publickey] Client determined that server accepts (only) publickey. Client also can do publickey. 14:36:54.477 tryNext() attempting method=publickey So now client will try publickey. It'll try to get key pairs as configured and use them until the server accepts one or there are none left to try. I suppose the log ending here means that's where things got hung? Which is rather strange. Without debugging I cannot fathom what might cause this, but since I cannot reproduce the problem, I cannot do so myself. (I also don't have a Windows machine to try.) I'd start with a breakpoint at org.apache.sshd.client.session.ClientUserAuthService, line 283 (reads "} else if (!userAuth.process(null)) {", then step into when that breakpoint is hit.
That's where the hang occurs yes. Is it possible that MINA is not honouring the full OpenSSH ~/.ssh/config semantics and is not finding a reasonable identity? I can try the debug and report back - might take a day or so.
(In reply to Randall Becker from comment #6) > That's where the hang occurs yes. Is it possible that MINA is not honouring > the full OpenSSH ~/.ssh/config semantics and is not finding a reasonable > identity? We hook our own parser for ~/.ssh/config into Apache MINA sshd. It's the same that we already used for JSch. It doesn't support all features of the OpenSSH parser, but the ones that are missing have always been missing, so I doubt that's a problem. Especially if it works if you switch back to JSch. One difference is that with Apache MINA sshd we _do_ actually honor the SSH home directory as configured in the Eclipse preferences, while the JSch client always used ~/.ssh/config. So check that in Eclipse that directory is also configured as ~/.ssh/. Also, if it didn't find an identity, I'd expect the connection to fail, but not hang. > I can try the debug and report back - might take a day or so. That'd be great.
Created attachment 278056 [details] Debug log 2 Debug log after adding org.apache.sshd as an explicit project dependency.
I explicitly added apache.sshd as a dependency of the project to permit the breakpoint to be set - not my long-term plan. This should be added as a computed dependency (or an optional one, I suppose, and JSch selected if it's not there). I ran through a debug and ended up here (unsurprisingly from the debug trace): Daemon Thread [sshd-SshClient[4b2faf08]-nio2-thread-7] (Suspended) 1732714736.run() line: not available [local variables unavailable] AccessController.doPrivileged(PrivilegedAction<T>) line: not available [native method] Nio2Session$1(Nio2CompletionHandler<V,A>).completed(V, A) line: 37 Invoker.invokeUnchecked(CompletionHandler<V,? super A>, A, V, Throwable) line: not available Invoker$2.run() line: not available AsynchronousChannelGroupImpl$1.run() line: not available ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker) line: not available ThreadPoolExecutor$Worker.run() line: not available Thread.run() line: not available The daemon thread simply exits without further interaction. The complete trace is a bit different this time, with the dependency added. Instead of hanging, the session fails without any indication to the caller. I have attached the trace, which now includes exceptions.
(In reply to Randall Becker from comment #9) > I explicitly added apache.sshd as a dependency of the project to permit the > breakpoint to be set - not my long-term plan. This should be added as a > computed dependency (or an optional one, I suppose, and JSch selected if > it's not there). Shouldn't be necessary, it should have been installed automatically. The second log doesn't help much, it shows that the client successfully initiates authentication with an RSA key and then gets a timeout after 120 seconds. The timeout may be due to your debugging. In any case it appears that EGit does find your key pair. So whatever the problem is it doesn't appear to be there. Can you reproduce this with EGit Neon? Easiest way to get set up for debugging this is probably to set up an Oomph-provisioned Eclipse as described at https://wiki.eclipse.org/EGit/Contributor_Guide#Automated_Developer_Setup . Then start a runtime workbench and try to clone your repo in there.
A jstack trace when that happens might also be useful. I've seen a number of bug reports about deadlocks in Apache MINA sshd recently. Maybe you hit one of those or yet another deadlock. :-( [1] https://issues.apache.org/jira/browse/SSHD-911 [2] https://issues.apache.org/jira/browse/SSHD-912 [3] https://issues.apache.org/jira/browse/SSHD-916
Now that JSch is deprecated - and not really accessible, we moved to Apache MINA. In that situation, our code: session = SshSessionFactory.getInstance().getSession(new URIish(remoteURL), CredentialsProvider.getDefault(), repo.getFS(), timeout); works fine with a banner and key-pair authentication but does not support keyboard-authentication or password. I tried replacing the CredentialsProvider with a UserNameAndPasswordCredentialProvider but that does, according to the log, not ever attempt to access the password. The operation just times out. This was tried all of the numerous different URIish git repository forms with no difference. I also went into ~/.ssh/config to force the user id and password authentication, which is accepted (I can see the user name change in the log), but the password method is still not used. I think there may be a gap in one of the components in the path that I cannot identify. I'm now on EGit/JGit 5.13 under ECLIPSE 2021-09.
(In reply to Randall Becker from comment #12) > session = SshSessionFactory.getInstance().getSession(new URIish(remoteURL), > CredentialsProvider.getDefault(), repo.getFS(), timeout); > > works fine with a banner and key-pair authentication but does not support > keyboard-authentication or password. Cannot reproduce in tests with JGit. Perhaps some misconfiguration in ~/.ssh/config? What is set as default CredentialsProvider? (Note that JGit doesn't set one.) I also don't quite understand what you are doing. You write "our code", but then you mention EGit and Eclipse 2021-09. Is "your code" in an Eclipse plug-in that you have installed?
>Cannot reproduce in tests with JGit. Perhaps some misconfiguration in ~/.ssh/config? What is set as default CredentialsProvider? (Note that JGit doesn't set one.) Ok, that's interesting. We don't set one either but it works fine with private keys. We tried setting UserNameAndPasswordCredentialsProvider as the default, without success - and also passed a separate instance to getSession() - also without success. >I also don't quite understand what you are doing. You write "our code", but then you mention EGit and Eclipse 2021-09. Is "your code" in an Eclipse plug-in that you have installed? Sorry, the "our code" is an ECLIPSE plugin we are testing with 2021-09. It worked with earlier versions of JGit from the Mars days, but when Jsch disappeared, we lost capability. Is there a sample that works with 2021-09 and 5.13?
(In reply to Randall Becker from comment #14) > Is there a sample that works with 2021-09 and 5.13? A sample plugin using JGit and working on all Eclipses since Neon.3 is of course EGit 5.13. Note that the banner is never even passed up to JGit; it is already swallowed (only logged at debug level) in Apache MINA sshd.
I noticed that the banner is no longer an issue. Maybe we should close this case because of that. Banners were an issue with Jsch but MINA seems fine to hide it.
(In reply to Thomas Wolf from comment #15) > (In reply to Randall Becker from comment #14) > > Is there a sample that works with 2021-09 and 5.13? > > A sample plugin using JGit and working on all Eclipses since Neon.3 is of > course EGit 5.13. Note that the banner is never even passed up to JGit; it > is already swallowed (only logged at debug level) in Apache MINA sshd. The EGit example at 5.13 could use a bit of documentation. I'm having a hard time following it, particularly around the TransportConfigurator.
I think I figured this out. The EGitCredentialProvider, which comes back from CredentialProvider.getDefault() is being initialized with null user and pass even if the URI used contains both. With Jsch, the initialization was done and prompts happened at a different time for user/pass if they were null. With Mina, we have to detect this condition and deal with the prompt ourselves. Now that I know, it's fine. This can be closed.
Sorry about there not being a lot of code documentation on how EGit is structured and coded internally. OK, to be honest, there's nothing. Nada. Zilch. I had started once a couple of years ago, but soon was overwhelmed and lost in details. Maybe I should try again now that I know it a bit better. (There are still a lot of corners I haven't even seen yet, though.) EGit was never intended as a "how to code with JGit" example, but if one needs to see how something is done in a working example, it's still a good place to look. Even if some of the code might be a bit dated, and more modern ways of doing things might exist nowadays. But anyway: I guess you found your problem; compare [1] from the commit that added Apache MINA sshd to EGit. Sorry, I didn't remember that arcane bit; it's been three years. [1] https://git.eclipse.org/r/c/egit/egit/+/131893/8/org.eclipse.egit.ui/src/org/eclipse/egit/ui/internal/credentials/EGitCredentialsProvider.java
