Warning: Potential Security Risk Ahead

When we communicate either via the web or from SSH, we want to be sure that the person (or computer) we’re talking to is who they say they are. 

We know if we visit a website that displays a security error that we should take additional steps to validate it’s the server we hoped to reach.

But what about when we use SSH?

If you’re a regular user of SSH, you will be intimately familiar with the message; The authenticity of host ‘Alice’ can’t be established.

ECDSA key fingerprint is SHA256:T6wfsUVsnopIBkqQ5SP7ntpTFW8IXbP1QBqhaqBikKs. Are you sure you want to continue connecting (yes/no/[fingerprint])? Bob knows if he calls Alice again, he should get the same fingerprint (simplification) but he can’t validate that this fingerprint belongs to Alice.

This is analogous to the HTTPS security warning:

Of course, you wish to continue connecting, you issued the ‘ssh user@alice command after all. But how trusted is that host? A nefarious actor may have entered your network, perhaps they’ve hit you with some poison routes or hijacked your DNS. 

This is a dangerous habit, sending usernames and passwords and not checking if they’re going to a trusted resource. We’d much prefer some evidence. Bob trusts a Certificate Authority (CA) to validate identities, and Bob can check the certificate to confirm that it was granted to Alice.

Great, just like the HTTPS padlock we see on the web, we know this host is trusted.  Now replace the word ‘fingerprint’ in this simplified description with ‘machine private key’ and we’re a step closer to the complete picture.

The solution consists of three components, a Certificate Authority server, SSH client and SSH target servers A Certificate Authority is mechanism for determining authenticity of hosts by the issuance of certificates, traditionally this is a stand-alone server, but it can exist as simply a public & private key pair. The Certificate Authority uses its private key to generate a certificate, hosts are able to validate host identities with this certificate by comparing it to the certificate authorities public key. The SSH Client is where the connection is initiated from, it will receive the SSH Target Server’s public key and certificate. It will use the Certificate Authorities’ public key to validate the certificate, then it will use the hosts public key to encrypt its response. The SSH Target Server sends its machine public key to the certificate authority server and receives a certificate in return. It’s then configured to send not only it’s public key, but additionally its certificate when a host initiates a connection.

There are two options, a Certificate Authority or OpenSSH; We use Hashicorp Vault at Advent One as our Certificate Authority as this enables us to rotate our secrets frequently and utilise its API to leverage automation. Obviously, time must be invested in setting up a secret server but ensuring the security of our data is not something we want to take the easy road on. The lowest barrier to entry option is OpenSSH, in just a few commands you can utilise a public/private key pair as a Certificate Authority, its power lies in its simplicity.  

The Setup: 

Certificate Authority Server The CA server will host the key pair that signs our certificates, in reality this can just be a public/private key pair stored securely.

For OpenSSH generate your key pair with ssh-keygen and place it in a (very) secure location. 

SSH Client The public key from our CA server is distributed to all SSH client machines, this will live as a ‘@certauthority’ entry in your ssh known_hosts file.

The default location is /etc/ssh/ssh_known_hosts, no configuration in your ssh config is required, it checks here by default. 

SSH Target Servers There are just two steps to setting up host-key validation on our target servers;

  • Generate a certificate for your hostkeyBy default, OpenSSH will have generated a public/private key pair for your server:/etc/ssh/ssh_host_rsa_key/etc/ssh/ssh_host_rsa_key.pub

  • We send the CA server our public key, it is then signed (by the CA servers private key) and a certificate is generated. We will save it as; /etc/ssh/ssh_host_rsa_key-cert.pub Generate a certificate on your existing CA or on OpenSSH using the command;

ssh-keygen -s ca_private_key -I Certificate_ID -h -n example.com \ ssh_host_rsa_key.pub Note: The Certificate_ID is for logging purposes, and will not alter certificate behaviour

  • Update your sshd config to provide that certificate to hosts We then add the following entries into your /etc/ssh/sshd_config: HostKey /etc/ssh/ssh_host_rsa_key HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pubAnd reload the SSH daemon.

  • That’s it! Now you’ll no longer see the authenticity warning on your hosts, and you can breathe a sigh of relief knowing you’re a little more secure than you were yesterday.


Automating the deployment.

We’re heavy users of Automation and our tool of choice is Ansible, as you can see below the automation workflow is very simple, this enables us to deploy this solution at scale. 

Can Advent One help you improve your security posture? We’d love to help.

Reach out to us at https://www.adventone.com/contact