Key mismastch error when the key should actually work fine

[Drugoy]

$: cat ~/.ssh/config ~/.config/boring/.boring.toml
Host myhost
  HostName 1.2.3.4
  User myuser
  IdentityFile ~/.ssh/my.key

[[tunnels]]
name = 'to_myhost'
local = '2025'
host = 'myhost'

$: boring o -a
[01:02:34] ERROR Could not open tunnel 'to_myhost': could not setup SSH client: ssh: handshake failed: knownhosts: key mismatch
$: ssh myhost echo 123
This account is currently not available.

ssh command proves that ssh works (there is no '123' only because the remote user has shell set to /bin/false, so my keys are of course correct.

[alebeck]

Hi, what version of boring are you running (boring v)?

[Drugoy]

The latest available, 0.11.7

[alebeck]

Hmm, that means the key the server returns does not match the one in your known_hosts. Sometimes Golang's ssh/known_host implementation is more strict or checks in different order than ssh(1), which is why the latter can still work. Can you check in your ~/.ssh/known_hosts whether the entries for that host look fine, and that they specify an IP address (since you appear to be dialing an IP address based on your ssh_config)?

[Drugoy]

I've tried wiping ~/.ssh/known_hosts, then connecting once via ssh to that host (to get a single record added to the known_hosts file), and then retrying boring o -a - the error persists.
The record in the known_hosts file doesn't contain IP address, it's just

|1|blahblahblahblah=|blahblahblahblah= ssh-ed25519 blahblahblahblahblahblahblahblah

[alebeck]

Hostname hashing could be the culprit -- can you for debugging check if a cleartext line

1.2.3.4 ssh-ed25519 blahblahblahblahblahblahblahblah

with IP and key replaced by the correct values works? If there's a hostname in your ssh_config for "myhost", there should be the host name instead of the IP.

[Drugoy]

Nope, it's not that: I've added HashKnownHosts No to the ~/.ssh/config to make ~/.ssh/known_hosts get non-hashed hostname.
Still the same error.

[alebeck]

Hi, sorry for the delay. I tried to reproduce this again given the information you provided, but wasn't successful so far. Have you been able to spot the problem by chance?

In summary, Go's ssh/knownhosts seems to find the key in your known_hosts but cannot match it to the server's ("key mismatch"). My best guess is that the key is duplicated in known_hosts and only the first (stale) key is checked, which is the behavior of ssh/knownhosts; but you wiped your known_hosts and it still doesn't work.

Is it possible that you recently upgraded boring by some other means than homebrew and there's still the old daemon running in the background? Since 0.11.5 we have fixed a problem originating in Go's ssh/knownhosts that would sometimes lead to the issue you described.

[Drugoy]

I'm on linux, I don't use homebrew here.
No, I haven't figured the issue out.

[alebeck]

I see. Did you use any prior version of boring before upgrading to 0.11.7, or was this the first version you used? If the former is the case, you might be running an old daemon. Can you try killall boring to kill it? Any new interaction with boring will then automatically spawn the new daemon.

[Drugoy]

As far as I remember - I did, but I have no recollection about what was the version and honestly I doubt it matters. I know my process list, there are no older versions running or something. It is not running at all.

[alebeck]

Hmm, there should be a daemon instance running as soon as you interact with boring (e.g. list tunnels). For me it looks like this (though yours should be in a different location):

~ $ pgrep -fl boring
8189 /opt/homebrew/bin/boring --daemon

Can you confirm that?

[Drugoy]

I mean before I try boring o -a - there are no running processes of possible old instances of boring. As soon as I run said command I get the error and the process gets daemonized (well, I kill it afterwards anyways, since it's of no use to me).
If you have no idea why the problem occurs - maybe the program needs more debug lines / verbose mode?