I have an sshd server running on a Raspberry Pi (Raspbian 10), which has worked pretty flawlessly for over a year, but has suddenly started acting up in the last few weeks. I'll attempt a connection, and get a message like:
Connection closed by xxx.xxx.xxx.xxx port 22
or
client_loop: send disconnect: Broken pipe
I've found other answers on here which suggest restarting the sshd service, or rebooting the machine entirely, both of which have usually solved the problem.
This time though, it won't budge. I've restarted several times now, and I still get the Connection closed error every time I try.
I've also seen answers (like this one) that suggest recreating the host keys. But this didn't help either.
I can't see anything in the client or server logs that has helped, but then again I'm not an expert. Both are below (hostnames and public addresses purposely obscured). Any ideas?
sshd (server) log
# sshd service starts
Nov 11 08:01:05 murchu27 sshd[1319]: debug1: Bind to port 22 on 0.0.0.0.
Nov 11 08:01:05 murchu27 sshd[1319]: Server listening on 0.0.0.0 port 22.
Nov 11 08:01:05 murchu27 sshd[1319]: debug1: Bind to port 22 on ::.
Nov 11 08:01:05 murchu27 sshd[1319]: Server listening on :: port 22.
# attempting connection
Nov 11 08:01:16 murchu27 sshd[1319]: debug1: Forked child 1321.
Nov 11 08:01:16 murchu27 sshd[1321]: debug1: Set /proc/self/oom_score_adj to 0
Nov 11 08:01:16 murchu27 sshd[1321]: debug1: rexec start in 5 out 5 newsock 5 pipe 7 sock 8
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: inetd sockets after dupping: 3, 3
Nov 11 08:01:17 murchu27 sshd[1321]: Connection from 192.168.0.33 port 48940 on 192.168.0.18 port 22
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: Client protocol version 2.0; client software version OpenSSH_8.8
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: match: OpenSSH_8.8 pat OpenSSH* compat 0x04000000
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: Local version string SSH-2.0-OpenSSH_7.9p1 Raspbian-10+deb10u2+rpt1
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: permanently_set_uid: 107/65534 [preauth]
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: SSH2_MSG_KEXINIT sent [preauth]
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: SSH2_MSG_KEXINIT received [preauth]
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: kex: algorithm: curve25519-sha256 [preauth]
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: kex: host key algorithm: ssh-ed25519 [preauth]
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none [preauth]
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none [preauth]
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: expecting SSH2_MSG_KEX_ECDH_INIT [preauth]
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: rekey after 134217728 blocks [preauth]
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: SSH2_MSG_NEWKEYS sent [preauth]
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: expecting SSH2_MSG_NEWKEYS [preauth]
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: SSH2_MSG_NEWKEYS received [preauth]
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: rekey after 134217728 blocks [preauth]
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: KEX done [preauth]
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: userauth-request for user murchu27 service ssh-connection method none [preauth]
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: attempt 0 failures 0 [preauth]
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: PAM: initializing for "murchu27"
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: PAM: setting PAM_RHOST to "192.168.0.33"
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: PAM: setting PAM_TTY to "ssh"
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: monitor_read_log: child log fd closed
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: do_cleanup
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: PAM: cleanup
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: Killing privsep child 1322
Nov 11 08:01:17 murchu27 sshd[1321]: debug1: audit_event: unhandled event 12
ssh (client) log
❯ ssh pi -vvv
OpenSSH_8.8p1, OpenSSL 1.1.1l 24 Aug 2021
debug1: Reading configuration data /home/murchu27/.ssh/config
debug1: /home/murchu27/.ssh/config line 8: Applying options for pi
debug1: Reading configuration data /etc/ssh/ssh_config
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/murchu27/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/murchu27/.ssh/known_hosts2'
debug2: resolving "{hostname}" port 22
debug3: resolve_host: lookup {hostname}:22
debug3: ssh_connect_direct: entering
debug1: Connecting to {hostname} [xxx.xxx.xxx.xxx] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x48
debug1: Connection established.
debug1: identity file /home/murchu27/usb/keys/id_rsa type -1
debug1: identity file /home/murchu27/usb/keys/id_rsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.8
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.9p1 Raspbian-10+deb10u2+rpt1
debug1: compat_banner: match: OpenSSH_7.9p1 Raspbian-10+deb10u2+rpt1 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to {hostname}:22 as 'murchu27'
debug3: record_hostkey: found key type ED25519 in file /home/murchu27/.ssh/known_hosts:7
debug3: load_hostkeys_file: loaded 1 keys from {hostname}
debug1: load_hostkeys: fopen /home/murchu27/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: order_hostkeyalgs: have matching best-preference key type ssh-ed25519-cert-v01@openssh.com, using HostkeyAlgorithms verbatim
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:uJQTWMF40CT0aGtTOGFqxM6snsjBssUbhYDojBfGKr0
debug3: record_hostkey: found key type ED25519 in file /home/murchu27/.ssh/known_hosts:7
debug3: load_hostkeys_file: loaded 1 keys from {hostname}
debug1: load_hostkeys: fopen /home/murchu27/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '{hostname}' is known and matches the ED25519 host key.
debug1: Found key in /home/murchu27/.ssh/known_hosts:7
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/murchu27/usb/keys/id_rsa explicit
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
