We released a new example that makes it easy for teams to set up just in time EC2 SSH access with Sym. As Mathew Pregasen walked us through recently, you have tons of options for how to manage SSH access to EC2. You also may be on a journey to remove this access entirely with GitOps or other automation approaches.
What we see in talking to customers, prospects and friends is that teams still need a reliable way to get onto boxes - be it for break-glass scenarios, maintenance of legacy systems, or day-to-day operations. You may have workflows that you just haven’t been able to automate yet for one of many good (or bad!) reasons.
The good news is that Sym provides a fast and flexible approach to help you with EC2 access that can mature with you as your access requirements evolve. You can start teams on their access maturity journey with peer-reviewed access, perhaps evolve towards on-call access, and maybe finish up with break-glass access. Maybe you'll also add an approval workflow for your CI system once you’ve got more automations in place.
Sym flows let you set up access and approval flows in Terraform, and then customize your routing rules and guardrails using our Python SDK. You can use our flows for lots of infrastructure and application use cases. AWS access is a common starting point for new implementations, and that is what we build on to provision the aws_ec2_ssm flow.
In this example, we’re going to use AWS Systems Manager Session Manager (Session Manager) to provide SSH access. Session Manager is a solid choice when you are running on AWS, as it is both free and already mostly set up for you when using default AMIs. Session Manager also supports access to ECS and EKS, which we’ll cover in a future post.
One of the neat things about Session Manager is that your instances don't need to support network ingress, either directly or via a bastion or VPN. You can put your instances in private subnets and still reach them, as long as those instances are able to connect to AWS’s Session Manager APIs (either through a NAT or a VPC Endpoint). This simplifies the public surface area you need to manage for your infrastructure.
Session Manager relies on the SSM Agent being installed on your instances. This is already the case for many AMIs that you will be starting off with. Here are the instructions if you need to get this installed yourself.
Once you have the agent installed, you need to make sure your EC2 instances have the necessary Session Manager IAM permissions in their instance profile. The AmazonSSMManagedInstanceCore managed policy defines the permissions you need for your instance to work with SSM.
Note: This managed policy allows ssm:GetParameter on *. If you want to limit instances to access only a subset of your SSM parameters, you can either use your own custom version of the managed policy, or consider using KMS keys to subdivide access to different groups of SSM parameters.
Your users need IAM permissions to work with Session Manager just as your target EC2 instances do. In the case of users, there’s not a great managed policy to start with. We recommend that you constrain the instances that a user can connect to by tag (replace tag-key and tag-value with an appropriate tag combination):
Beyond instance access, you'll need to provide access to the following additional permissions. The TerminateSession action is constrained so that only the current federated IAM user can terminate the session. We've got it set up this way since we are going to be using Session Manager with AWS IAM Identity Center. If you are using regular IAM users then the TerminateSession configuration is slightly different.
Once your instances and user permissions are set up, you can provision a Sym Flow to manage temporary SSH access! We use an AWS IAM Identity Center (AWS SSO) Strategy in our example but the same concept applies for our AWS IAM or Okta strategies.
Please refer to our step-by-step tutorial for all the specifics on getting an AWS IAM Identity Center Flow set up. In this post we're just going to highlight the specifics that we configure to get this working with Session Manager.
First we’re going to provision a permission set with a customer-managed policy. The policy allows Session Manager access to EC2 instances tagged with Department=FrontEnd;
Then we use the permission set as a Sym AWS SSO Target:
We create a Sym AWS SSO Strategy that uses our new Target:
Finally, we configure a Sym Flow that uses our SSO Strategy:
You can provision our example Flow and test with existing EC2 instances in your target account, or configure the example to provision test instances for you. If you set the bastions_enabled variable to true and configure a private_subnet_id, we'll use CloudPosse’s ec2-bastion-server module to provision two Session Manager enabled EC2 instances - one tagged with Department=FrontEnd and one with Department=BackEnd. the FrontEndSSH permission set will let you access the FrontEnd instance, but not the Backend one.
Once you've got everything configured, you need to authenticate with both AWSand Sym. Then you can provision your Sym Flow by runningterraform apply:
You'll need to set up your test users so that they can access the new FrontEndSSH permission set when they are escalated by Sym. You can use the AWS CLI's baked in support for AWS SSO profiles to create a configuration like the following (replace the sso_start_url and sso_acount_id with the proper values):
Users will also need to have the Session Manager plugin for the AWS CLI installed in order to connect.
Once you have the Sym Flow provisioned, users will have a secure, Slack-enabled way to manage JIT access to EC2!
First they'll make a request using the Sym Slack App:
Once approved, they will be able to access the frontend-ssh profile weconfigured above:
The profile won't work on the backend instance since it does not have the rightdescription: tags:
You can use Session Manager port forwarding to tunnel to private services in your VPC, like a database or an internal app. Built-in port forwarding support was announced in May 2022… until then you needed to monkey around with ProxyCommand in order to get this to work. Now you can enable port forwarding by starting a session with a different Systems Manager Document.
When we started a session before, we were implicitly using the default SSM-SessionManagerRunShell Document. AWS provides many additional Systems Manager Documents that you can use for sessions. You can also create your own. In order to forward a port from a remote service, you’ll use the AWS-StartPortForwardingSessionToRemoteHost document.
With this document, you specify the endpoint, remote port, and local port of the remote service that you want to be able access locally:
Note that you’ll need to ensure that the EC2 instance you are going to tunnel through is able to reach your target service. This might mean opening up security group ingress and egress between your target service and your EC2 instance.
You’ll also have to ensure your users have IAM permissions to use the AWS-StartPortForwardingSessionToRemoteHost document. In our example, we gave users permissions to use all Systems Manager Documents. If you want to constrain users to specific documents, you can update your policy so that the ssm:StartSession and ssm:SendCommand actions only work on specific document resources. In the following policy, we continue to allow access to all EC2 instances tagged with the appropriate resource tag, but only allow access to the port forwarding document.
Note that by default, users can always use the SSM-SessionManagerRunShell documenteven if you don't give that permission explicitly. You can turn this behavioroff if you want to manage all document access explicitly.
Session Manager supports other important controls that you should consider enabling in a production environment. You can enable logging of session activity to both CloudWatch and S3. If you are working in a VPC that does not have outbound access to the internet, you can enable a VPC endpoint so that your EC2 instances can reach the Session Manager APIs privately. By editing your Session Manager documents, you can restrict what commands users can run in a session, edit the shell profiles, and more.
Bridgecrew's session-manager module handles creation of Session Manager documents with KMS-encrypted logging enabled as well as a VPC endpoint. Note: Check the pricing on VPC Endpoints before getting started with them!
In this post we introduced Sym's new JIT Access To AWS EC2 with Session Manager example. As discussed above, we’ll be publishing examples of how to use Sym and Session Manager to provide temporary access to ECS and EKS. We’re also working on templates for RDS for teams that are interested in temporary access to RDS with an IAM-only solution.
Please let us know if you have questions on this example and/or want to try it out!
