steipete/ssh-doctor

SSH Doctor

Use when SSH connects then closes before auth, Remote Login seems advertised but unusable, or local/remote Mac SSH needs diagnosis.

Rules

• Do not print secrets, tokens, full env, or broad secret grep output.
• Validate locally first: loopback failure means server-side sshd/launchd/config; loopback success plus remote failure means network/firewall/filter/listen path.
• Report suspicious config lines before changing /etc/ssh/sshd_config.
• Prefer non-interactive SSH:

ssh -o RequestTTY=no -o RemoteCommand=none HOST 'hostname; id -un'

Baseline

hostname; id -un; sw_vers
ipconfig getifaddr en0
ipconfig getifaddr en1 2>/dev/null || true
ipconfig getifaddr en7 2>/dev/null || true
sudo systemsetup -getremotelogin
sudo systemsetup -setremotelogin on
sudo launchctl print system/com.openssh.sshd 2>&1 | head -80
sudo launchctl kickstart -k system/com.openssh.sshd
sudo lsof -nP -iTCP:22 -sTCP:LISTEN
nc -vz 127.0.0.1 22
ssh -4 -F /dev/null -o RequestTTY=no -o RemoteCommand=none [email protected] 'hostname; id -un'

Use BatchMode=yes only when password fallback would hang or prompt.

Config

sudo sshd -T 2>&1 | egrep -i '^(allowusers|denyusers|allowgroups|denygroups|listenaddress|maxstartups|logingracetime|usepam|passwordauthentication|pubkeyauthentication|authenticationmethods)'
sudo egrep -n '^[[:space:]]*(AllowUsers|DenyUsers|AllowGroups|DenyGroups|Match|MaxStartups|LoginGraceTime|ListenAddress|AuthenticationMethods|UsePAM|PasswordAuthentication|PubkeyAuthentication)\b' /etc/ssh/sshd_config /etc/ssh/sshd_config.d/* 2>/dev/null || true

Suspicious:

• DenyUsers matching target user
• restrictive AllowUsers / AllowGroups
• Match block accidentally applying
• tiny MaxStartups
• tiny LoginGraceTime
• ListenAddress missing target interface

Logs

sudo log show --last 30m --predicate 'process == "sshd" OR process == "launchd"' --style compact | tail -160

Important Mac symptom:

• client: kex_exchange_identification: Connection closed by remote host
• server log: Could not create new instance of inetd service: 67: Too many processes
• launchctl print system/com.openssh.sshd: high copy count
• many sshd-session: USER processes parented by PID 1

This means launchd accepted TCP but refused to spawn more sshd inetd copies.

Stale sshd-session Fix

Inspect first:

sudo launchctl print system/com.openssh.sshd 2>&1 | egrep 'active count|copy count|state =|last exit code|runs ='
ps -axo pid,ppid,uid,user,state,lstart,etime,comm,args | awk '/sshd-session:/ && !/awk/ {print}'
sudo lsof -nP -c sshd-session -iTCP 2>/dev/null | head -120

If stale sessions are clearly stranded and blocking new SSH, terminate by selected command-line match:

ps -axo pid=,args= | awk '/sshd-session: / && !/awk/ {print $1}' | xargs sudo kill -TERM
sleep 2
ps -axo pid=,args= | awk '/sshd-session: / && !/awk/ {print}'

If TERM leaves blockers, re-check ownership and active shells before using KILL.

Firewall

Only after loopback works but remote fails:

sudo /usr/libexec/ApplicationFirewall/socketfilterfw --getglobalstate
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --listapps | grep -i ssh -A2 -B2 || true
sudo pfctl -sr 2>/dev/null | head -80
sudo pfctl -si 2>/dev/null | head -80

Also check listen address and target interface:

ifconfig | awk '/^[a-z0-9]+:/{iface=$1; sub(":","",iface)} iface ~ /^en[0-9]+$/ && /inet / {print iface, $2}'
sudo lsof -nP -iTCP:22 -sTCP:LISTEN

OP Profile Block

If asked to ensure ~/.profile has a Codex-managed OP_SERVICE_ACCOUNT_TOKEN copied from another host:

• verify exact variable/markers without printing value
• copy only the matching line/block
• redirect through a chmod 600 temp file
• never echo the token

Presence check:

awk 'BEGIN{b=0;e=0;x=0} /BEGIN Codex-managed OP_SERVICE_ACCOUNT_TOKEN/ {b=1} /END Codex-managed OP_SERVICE_ACCOUNT_TOKEN/ {e=1} /^[[:space:]]*(export[[:space:]]+)?OP_SERVICE_ACCOUNT_TOKEN=/ {x=1} END{print "marker_begin", b; print "marker_end", e; print "exact_var", x}' ~/.profile

Append from remote host:

tmpfile=$(mktemp /tmp/codex-op-token.XXXXXX)
chmod 600 "$tmpfile"
ssh -o RequestTTY=no -o RemoteCommand=none HOST 'awk '\''/^[[:space:]]*(export[[:space:]]+)?OP_SERVICE_ACCOUNT_TOKEN=/ {print; exit}'\'' ~/.profile' > "$tmpfile"
if [ -s "$tmpfile" ]; then
  {
    printf '\n# BEGIN Codex-managed OP_SERVICE_ACCOUNT_TOKEN\n'
    sed -n '1p' "$tmpfile"
    printf '# END Codex-managed OP_SERVICE_ACCOUNT_TOKEN\n'
  } >> ~/.profile
fi
rm -f "$tmpfile"

Closeout

Report:

• root cause
• exact commands changed
• validation output, redacted as needed
• whether remote should retry