Structure In A Void

Zero-code Reverse SSH Jumphost

Introduction

SSH (Secure Shell Protocol) is a widely-used protocol for securely accessing a remote shell.

In order for client to connect to an SSH server, the server's TCP port 22 must be directly reachable by the client.

Unfortunately, due to widespread use of NAT (Network Address Translation), most computers connected to the internet do not have a publicly-reachable IP address. The simplest way to overcome this limitation is to use a single SSH server with a publicly-reachable IP address and use it as a reverse-jumphost for connecting to all other hosts behind NAT:

Jumphost Jumphost Jumphost Client Client Jumphost -> Client host2 Host2 Host2 Jumphost -> Host2 port 22 Host1 Host1 Host1 -> Jumphost Host2 -> Jumphost Host3 Host3 Host3 -> Jumphost

Overview

SSH has a cool feature - it can forward connections from a UNIX socket on one host to a TCP socket on another.

Since all we need to connect to an SSH server is access to TCP port 22, this feature will be the building block for our reverse SSH jumphost - since UNIX sockets are just files on the filesystem, we can create a new UNIX socket for each host we want to access through the jumphost. If we name each socket by the hostname of the host it connects to, we will have a convenient way to connect to hosts by their hostname:

jumphost_detailed cluster_host_1 Host1 cluster_host_2 Host2 cluster_host_3 Host3 cluster_jumphost Jumphost Host1_Port22 TCP port 22 Host2_Port22 TCP port 22 Host3_Port22 TCP port 22 Host1_UnixSocket ~/Host1.sock Host1_UnixSocket -> Host1_Port22 Host2_UnixSocket ~/Host2.sock Host2_UnixSocket -> Host2_Port22 Host3_UnixSocket ~/Host3.sock Host3_UnixSocket -> Host3_Port22 Client Client Client -> Host2_UnixSocket

Another cool feature of SSH is ProxyCommand - the argument of this option will be used as a command to connect to the server. An excerpt from the manpage:

[...] The command can be basically anything, and should read from its standard input and write to its standard output. It should eventually connect an sshd server running on some machine. [...]

Since each connected host forwards connections from its UNIX socket to its TCP port 22, we can use netcat (nc -U /path/to/socket) as ProxyCommand to connect to the host.

Server configuration

Setting up reverse jumphost on our SSH server requires the following steps:

  1. Install netcat
  2. Create a new "host" user
  3. Copy authorized_keys file to /home/host/.ssh/ directory
  4. Configure SSH server for "host" user:
    • Disable password login (PasswordAuthentication=no)
    • Allow clients to overwrite UNIX sockets (StreamLocalBindUnlink=yes)
    • Allow anyone to connect to the UNIX sockets (StreamLocalBindMask=0111)

The configuration that should be put in /etc/ssh/sshd_config is the following:

Match User host
    PasswordAuthentication no
    StreamLocalBindUnlink yes
    StreamLocalBindMask 0111

Shell commands are provided below:

# Install "netcat" package (assuming Debian-based distribution)
sudo apt-get -y install netcat

# Create a new "host" user with disabled password login
sudo adduser --disabled-password host

# Copy authorized_keys file to "host" user to allow client login
sudo -u host mkdir -p /home/host/.ssh
cat .ssh/authorized_keys | sudo -u host tee /home/host/.ssh/authorized_keys

# Append configuration to /etc/ssh/sshd_config
printf 'Match User host\n\tPasswordAuthentication no\n\tStreamLocalBindUnlink true\n\tStreamLocalBindMask 0111\n' | sudo tee -a /etc/ssh/sshd_config

# Restart SSH server
sudo systemctl restart ssh

Host configuration

In order to be accesible through the server, each host must forward connections from a UNIX socket on the server to its own TCP port 22. In order for the connection to be persistent, we can write a systemd service.

For convenience, we can use the hostname of a host as a UNIX socket identificator.

Setting up persistent host tunnel requires the following steps:

  1. Authorize host SSH key with server "host" account
  2. Write a systemd service which:
    • Restarts on every failure (Restart=always) with 5 second timeout (RestartSec=5)
    • Does not rate-limit - we expect network interruptions (StartLimitIntervalSec=0)
    • Runs as the user whose public key we have authorized on the server
    • Runs an SSH command which:
      • Connects to server "server" without opening an interactive session (-N)
      • Forwards TCP connections from UNIX socket on server to its own TCP port 22 (-R /home/host/HOSTNAME.sock:localhost:22)
      • Disconnect on socket forwarding failure (-o ExitOnForwardFailure=true)
      • Check connection every 10 seconds (-o ServerAliveInterval=10)

For the step 1. we need to copy host's SSH public key (.pub file) to the client machine (the one that can access the server) in order to authorize the host's public key with server's "host" user.

Assuming the host public key is copied to a client's working directory (./id_rsa.pub), the shell command for step 1. is the following (NOTE: this needs to be executed on the client machine):

# Force-authorize new key for the user "host" on the server "SERVER"
ssh-copy-id -f -i ./id_rsa.pub host@SERVER

NOTE: Replace the "SERVER" with your server hostname or address.

For step 2. we need to write a systemd unit file. The configuration for the unit file is the following:

[Unit]
Description=Run reverse SSH jumphost
StartLimitIntervalSec=0

[Service]
User=USERNAME
Restart=always
RestartSec=5
ExecStart=/usr/bin/ssh -N -R /home/host/HOSTNAME.sock:localhost:22 -o ExitOnForwardFailure=true -o ServerAliveInterval=10 host@SERVER

[Install]
WantedBy=multi-user.target

NOTE: Replace the "USERNAME" with the appropriate username of the host, "HOSTNAME" with the appropriate hostname of the host, and "SERVER" with your server hostname or address.

To enable the systemd service, write the above configuration to /etc/systemd/system/reverse-jumphost.service on the host machine, then execute:

# Reload configuration files
sudo systemctl daemon-reload

# Start reverse-jumphost service
sudo systemctl start reverse-jumphost.service

# Enable reverse-jumphost service to run on reboot
sudo systemctl enable reverse-jumphost.service

Client configuration

Now, assuming we have properly set up our hosts and server, the only thing left is to configure the client.

In order to automatically use our jumphost when connection to any of the hosts (e.g. Host1, Host2, Host3, ...), we can use the following SSH options:

Host Host1 Host2 Host3 ...
    ProxyCommand ssh SERVER nc -U /home/host/%h
    ServerAliveInterval 10
    Hostname %h

Host Host1
    User User1

Host Host2
    User User2

Host Host3
    User User3

...

NOTE: Replace the "SERVER" with your server hostname or address, and "Host1/User1", "Host2/User2", ... with the appropriate Hostname/Username pairs.

If everything is configured correctly, we can now connect to our hosts as if they were on a local network:

$ ssh Host2
The authenticity of host 'Host2 (<no hostip for proxy command>)' can't be established.
ECDSA key fingerprint is SHA256:${FINGERPRINT}.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added 'Host2' (ECDSA) to the list of known hosts.
User2@Host2's password:

Conclusion

The main idea lies in the UNIX-socket-to-TCP-port-22 forwarding. Since UNIX sockets are files on the filesystem and can have proper filenames, this scheme does not require any port-mapping bookkeeping, and should be able handle thousands of devices without issues, assuming high enough network bandwidth.

For reduced host/server configuration in trusted environments, all other hosts can use the same SSH key as the one we previously configured.

For reduced client configuration, all hosts can have the same username and have the same common prefix in the hostname (e.g. host-1, host-2, ...), making the client configuration simple:

Host host-*
    ProxyCommand ssh SERVER nc -U /home/host/%h
    ServerAliveInterval 10
    Hostname %h
    User USERNAME

For better security, the host user's SSH shell could be disabled by adding the following lines to the /etc/sshd_config file:

Match User host
    ForceCommand /sbin/nologin