I often see the "never do this!" approach when connecting to one ec2 instance thru another of copy-pasting keys. Obviously it’s a bad idea…. which you just teached to everybody…. during a security training.
But I never see explanations of how to do it properly, using pass-through authentication. But that is not explained, so what do you estimate the average user is going to do in real life?
3 Answers
Good observation ObiJan, though in defense of the course, I think the focus is on the AWS facilities and what is required to pass the exam. The use of SSH in AWS and in general is often overlooked and not well understood. In AWS, the key pairs are meant to be temporary during the EC2 provisioning process to give root/admin access to the instance. Ideally the instance will be joined to some identity management system, the initial keys are changed and the system is managed using company policies and procedures.
The use of SSH could easily be a course unto itself and often implementations are very prescriptive depending on the environment. In secure environments, I will even replace the ssh server with a version compiled not to to accept root logins or allow tunnels. All of this is well beyond the scope of this course and the requirements of the exam. You can find some good information and best practices on ssh.com
Why would you need to recompile sshd for that? Both options are just a switch in sshd.conf.
Use SSH agent forwarding:
This should be required reading.
Agreed wholeheartedly and have intended to come post some discussion points. For the purpose of real-world work, you should be looking at this from the perspective of managing multiple users, at which point you have a variety of options depending on the identity management patterns employed by the enterprise.
For a small group managed through IAM, I’d probably go with the style of solution proposed by Cloudonaut (https://cloudonaut.io/manage-aws-ec2-ssh-access-with-iam/). Keymaker may also come in handy (https://keymaker.readthedocs.io/en/latest/). These solutions automate the process and give you efficient mechanisms to revoke access. I’m not sure, however, how scalable they’d be for really large applications. Likewise, they may take some reengineering to integrate with federated identity.
Another option, which works with either IAM or Federated IdP, is to set up a lightweight SSH Certificate Authority. This requires a bit of knowledge about key management to do properly, but Netflix has handled the rest of the heavy-lifting in their BLESS project (https://github.com/Netflix/bless). Note that this runs in Lambda, so you won’t have to deploy or manage any servers. I believe that BLESS is intended to integrate with a bastion such that the certificates aren’t actually stored on the developer’s computer. If you want to go that route, you can take a look at Lyft’s open source client (https://eng.lyft.com/blessing-your-ssh-at-lyft-a1b38f81629d) in conjunction with SSH agent forwarding.
If you want an ready-to-go solution, you might look at SSH servers that incorporate identity and access management. Teleport (https://gravitational.com/teleport/) comes to mind as does Keybox (https://www.sshkeybox.com) and
There’s much more to the topic and some important policy decisions to be made. E.g., Do you trust SSH keys/certificates for server access to developer computers or do you generate certificates dynamically from a Bastion. If using certificates, where/how do you run your CA and what’s the lifespan of your certs. Get a taste for these design decisions by reading https://code.facebook.com/posts/365787980419535/scalable-and-secure-access-with-ssh/.
Oh yeah, and one more thing, I’m always looking for a way to skip SSH in production. Get familiar with EC2 Simple Systems Manager. With run command and other features, your team can run commands and scripts via API calls.
Good luck!
P.S. Okay I lied … please don’t ever let me catch you running sudo su in practice. I cringe every time I see this in a security-focused training video. For a security professional, the proper strategy would be to sort out the file and directory permissions so that you elevate privileges only on a per-command, as-needed basis.
Long-term solution: Go Serverless 🙂
There are options allowing access to AWS. For eg:, you can control access to unix resources using centrify . Another one using EC2 run command –https://aws.amazon.com/blogs/aws/manage-instances-at-scale-without-ssh-access-using-ec2-run-command/
Yes, there are other options, but this was specifically "how to ssh to your box behind a bastion host".
Here’s an AWS QuickStart: https://aws.amazon.com/quickstart/architecture/linux-bastion/