Just in Time Access to EC2 with Sym

Just in Time Access to EC2 with Sym

We released a new example that makes it easy for teams to set up just in time EC2 SSH access with Sym. As Mathew Pregasen walked us through recently, you have tons of options for how to manage SSH access to EC2. You also may be on a journey to remove this access entirely with GitOps or other automation approaches.

What we see in talking to customers, prospects and friends is that teams still need a reliable way to get onto boxes - be it for break-glass scenarios, maintenance of legacy systems, or day-to-day operations. You may have workflows that you just haven’t been able to automate yet for one of many good (or bad!) reasons.

JIT Access With Sym

The good news is that Sym provides a fast and flexible approach to help you with EC2 access that can mature with you as your access requirements evolve. You can start teams on their access maturity journey with peer-reviewed access, perhaps evolve towards on-call access, and maybe finish up with break-glass access. Maybe you’ll also add an approval workflow for your CI system once you’ve got more automations in place.

Sym flows let you set up access and approval flows in Terraform, and then customize your routing rules and guardrails using our Python SDK. You can use our flows for lots of infrastructure and application use cases. AWS access is a common starting point for new implementations, and that is what we build on to provision the aws_ec2_ssm flow.

Session Manager FTW

In this example, we’re going to use AWS Systems Manager Session Manager (Session Manager) to provide SSH access. Session Manager is a solid choice when you are running on AWS, as it is both free and already mostly set up for you when using default AMIs. Session Manager also supports access to ECS and EKS, which we’ll cover in a future post.

One of the neat things about Session Manager is that your instances don’t need to support network ingress, either directly or via a bastion or VPN. You can put your instances in private subnets and still reach them, as long as those instances are able to connect to AWS’s Session Manager APIs (either through a NAT or a VPC Endpoint). This simplifies the public surface area you need to manage for your infrastructure.

Enabling Session Manager On Your Instances

Session Manager relies on the SSM Agent being installed on your instances. This is already the case for many AMIs that you will be starting off with. Here are the instructions if you need to get this installed yourself.

Once you have the agent installed, you need to make sure your EC2 instances have the necessary Session Manager IAM permissions in their instance profile. The AmazonSSMManagedInstanceCore managed policy defines the permissions you need for your instance to work with SSM.

Note: This managed policy allows ssm:GetParameter on *. If you want to limit instances to access only a subset of your SSM parameters, you can either use your own custom version of the managed policy, or consider using KMS keys to subdivide access to different groups of SSM parameters.

Enabling Session Manager For Your Users

Your users need IAM permissions to work with Session Manager just as your target EC2 instances do. In the case of users, there’s not a great managed policy to start with. We recommend that you constrain the instances that a user can connect to by tag (replace tag-key and tag-value with an appropriate tag combination):

{
    "Action": [
        "ssm:StartSession",
        "ssm:SendCommand"
    ],
    "Condition": {
        "StringLike": {
            "ssm:resourceTag/tag-key": "tag-value"
        }
    },
    "Effect": "Allow",
    "Resource": "*"
}

Beyond instance access, you’ll need to provide access to the following additional permissions. The TerminateSession action is constrained so that only the current federated IAM user can terminate the session. We’ve got it set up this way since we are going to be using Session Manager with AWS IAM Identity Center. If you are using regular IAM users then the TerminateSession configuration is slightly different.

{
    "Action": [
        "ssm:GetConnectionStatus",
        "ssm:DescribeSessions",
        "ssm:DescribeInstanceProperties",
        "ssm:DescribeInstanceInformation",
        "ec2:DescribeInstances"
    ],
    "Effect": "Allow",
    "Resource": "*"
},
{
    "Action": [
        "ssm:TerminateSession",
        "ssm:ResumeSession"
    ],
    "Condition": {
        "StringLike": {
            "ssm:resourceTag/aws:ssmmessages:session-id": [
                "${aws:userid}"
            ]
        }
    },
    "Effect": "Allow",
    "Resource": "*"
}

Configure a Sym Flow for AWS IAM Identity Center

Once your instances and user permissions are set up, you can provision a Sym Flow to manage temporary SSH access! We use an AWS IAM Identity Center (AWS SSO) Strategy in our example but the same concept applies for our AWS IAM or Okta strategies.

Please refer to our step-by-step tutorial for all the specifics on getting an AWS IAM Identity Center Flow set up. In this post we’re just going to highlight the specifics that we configure to get this working with Session Manager.

First we’re going to provision a permission set with a customer-managed policy. The policy allows Session Manager access to EC2 instances tagged with Department=FrontEnd;

data "aws_ssoadmin_instances" "this" {
  provider = aws.sso
}

# Create an AWS SSO PermissionSet that allows Session Manager access
# to EC2 instances tagged with Department=FrontEnd
resource "aws_ssoadmin_permission_set" "frontend_ssh" {
  name             = "FrontEndSSHAccess"
  description      = "Access to SSH to FrontEnd instances"
  instance_arn     = tolist(data.aws_ssoadmin_instances.this.arns)[0]
  session_duration = "PT2H"

  provider = aws.sso

  tags = var.tags
}

# Customer managed policies are attached using their name. You need to make
# sure the managed policy actually exists in any account you try to provision
# this into
resource "aws_ssoadmin_customer_managed_policy_attachment" "frontend_ssh" {
  instance_arn       = tolist(data.aws_ssoadmin_instances.this.arns)[0]
  permission_set_arn = aws_ssoadmin_permission_set.frontend_ssh.arn

  customer_managed_policy_reference {
    name = aws_iam_policy.frontend_ssh.name
    path = aws_iam_policy.frontend_ssh.path
  }

  provider = aws.sso
}

Then we use the permission set as a Sym AWS SSO Target:

# A Sym Target that will grant users FrontEnd SSH access in the given AWS account
resource "sym_target" "frontend_ssh" {
  type = "aws_sso_permission_set"

  name  = "frontend-ssh"
  label = "FrontEnd SSH"

  settings = {
    # `type=aws_sso_permission_set` sym_targets need both an AWS Permission
    # Set ARN and an AWS Account ID to make an sso account assignment
    permission_set_arn = aws_ssoadmin_permission_set.frontend_ssh.arn
    account_id         = data.aws_caller_identity.main.account_id
  }
}

We create a Sym AWS SSO Strategy that uses our new Target:

# The Strategy your Flow uses to escalate to AWS SSO Permission Sets
resource "sym_strategy" "aws_sso" {
  type           = "aws_sso"
  name           = "main-aws-sso"
  integration_id = sym_integration.sso_context.id

  # This must be a list of `aws_sso_permission_set` sym_targets that users can
  # request to be escalated to
  targets = [sym_target.frontend_ssh.id]

  settings = {
    instance_arn = module.sso_connector.settings["instance_arn"]
  }
}

Finally, we configure a Sym Flow that uses our SSO Strategy:

resource "sym_flow" "this" {
  name  = "aws_sso"
  label = "AWS SSO Access"

  template       = "sym:template:approval:1.0.0"
  implementation = "${path.module}/impl.py"
  environment_id = sym_environment.this.id

  params = {
    strategy_id = sym_strategy.aws_sso.id

    prompt_fields_json = jsonencode([
      {
        name     = "reason"
        label    = "Why do you need access?"
        type     = "string"
        required = true
      },
      {
        name           = "duration"
        type           = "duration"
        allowed_values = ["30m", "1h"]
        required       = true
      }
    ])
  }
}

Enabling Test Instances

You can provision our example Flow and test with existing EC2 instances in your target account, or configure the example to provision test instances for you. If you set the bastions_enabled variable to true and configure a private_subnet_id, we’ll use CloudPosse’s ec2-bastion-server module to provision two Session Manager enabled EC2 instances - one tagged with Department=FrontEnd and one with Department=BackEnd. the FrontEndSSH permission set will let you access the FrontEnd instance, but not the Backend one.

Provisioning Your Flow

Once you’ve got everything configured, you need to authenticate with both AWS and Sym. Then you can provision your Sym Flow by running terraform apply:

$ aws sso login --profile my-profile
$ symflow login --org-slug my-org
$ terraform apply

...

Apply complete! Resources: 30 added, 0 changed, 0 destroyed.

Outputs:

backend_bastion_id = "i-05b7516a19f826be9"
frontend_bastion_id = "i-060ee0879f3ec4eb7"

Configuring User Access

You’ll need to set up your test users so that they can access the new FrontEndSSH permission set when they are escalated by Sym. You can use the AWS CLI’s baked in support for AWS SSO profiles to create a configuration like the following (replace the sso_start_url and sso_acount_id with the proper values):

[profile frontend-ssh]
sso_start_url=https://myco.awsapps.com/start
sso_region=us-east-1
sso_account_id=123456789012
sso_role_name=FrontEndSSHAccess
region = us-east-1
output=json

Users will also need to have the Session Manager plugin for the AWS CLI installed in order to connect.

Testing Out JIT Access With Sym

Once you have the Sym Flow provisioned, users will have a secure, Slack-enabled way to manage JIT access to EC2!

First they’ll make a request using the Sym Slack App:

Once approved, they will be able to access the frontend-ssh profile we configured above:

$ aws --profile=frontend-ssh ssm start-session --target i-060ee0879f3ec4eb7

Starting session with SessionId: [email protected]
sh-4.2$ whoami
ssm-user

The profile won’t work on the backend instance since it does not have the right tags:

$ aws --profile=frontend-ssh ssm start-session --target i-05b7516a19f826be

An error occurred (AccessDeniedException) when calling the StartSession operation: User: arn:aws:sts::111111111111:assumed-role/AWSReservedSSO_FrontEndSSHAccess_62bd274a20760ba0/[email protected] is not authorized to perform: ssm:StartSession on resource: arn:aws:ec2:us-east-1:111111111111:instance/i-05b7516a19f826be9 because no identity-based policy allows the ssm:StartSession action

Setting up Port Forwarding (and An Aside On Systems Manager Documents)

You can use Session Manager port forwarding to tunnel to private services in your VPC, like a database or an internal app. Built-in port forwarding support was announced in May 2022… until then you needed to monkey around with ProxyCommand in order to get this to work. Now you can enable port forwarding by starting a session with a different Systems Manager Document.

When we started a session before, we were implicitly using the default SSM-SessionManagerRunShell Document. AWS provides many additional Systems Manager Documents that you can use for sessions. You can also create your own. In order to forward a port from a remote service, you’ll use the AWS-StartPortForwardingSessionToRemoteHost document.

With this document, you specify the endpoint, remote port, and local port of the remote service that you want to be able access locally:

$ aws ssm start-session \
    --target instance-id \
    --document-name AWS-StartPortForwardingSessionToRemoteHost \
    --parameters '{"host":["mydb.example.us-east-1.rds.amazonaws.com"],"portNumber":["3306"], "localPortNumber":["3306"]}'

Starting session with SessionId: [email protected]
Port 3306 opened for sessionId [email protected]
Waiting for connections...

Note that you’ll need to ensure that the EC2 instance you are going to tunnel through is able to reach your target service. This might mean opening up security group ingress and egress between your target service and your EC2 instance.

Checking Document Permissions

You’ll also have to ensure your users have IAM permissions to use the AWS-StartPortForwardingSessionToRemoteHost document. In our example, we gave users permissions to use all Systems Manager Documents. If you want to constrain users to specific documents, you can update your policy so that the ssm:StartSession and ssm:SendCommand actions only work on specific document resources. In the following policy, we continue to allow access to all EC2 instances tagged with the appropriate resource tag, but only allow access to the port forwarding document.

{
    "Action": [
        "ssm:StartSession",
        "ssm:SendCommand"
    ],
    "Condition": {
        "StringLike": {
            "ssm:resourceTag/Department": "FrontEnd"
        }
    },
    "Effect": "Allow",
    "Resource": [
        "arn:aws:ec2:*:*:instance/*",
        "arn:aws:ssm:*:*:document/AWS-StartPortForwardingSessionToRemoteHost"
    ]
}

Note that by default, users can always use the SSM-SessionManagerRunShell document even if you don’t give that permission explicitly. You can turn this behavior off if you want to manage all document access explicitly.

More SSM Knobs and Levers

Session Manager supports other important controls that you should consider enabling in a production environment. You can enable logging of session activity to both CloudWatch and S3. If you are working in a VPC that does not have outbound access to the internet, you can enable a VPC endpoint so that your EC2 instances can reach the Session Manager APIs privately. By editing your Session Manager documents, you can restrict what commands users can run in a session, edit the shell profiles, and more.

Bridgecrew’s session-manager module handles creation of Session Manager documents with KMS-encrypted logging enabled as well as a VPC endpoint. Note: Check the pricing on VPC Endpoints before getting started with them!

Next Steps

In this post we introduced Sym’s new JIT Access To AWS EC2 with Session Manager example. As discussed above, we’ll be publishing examples of how to use Sym and Session Manager to provide temporary access to ECS and EKS. We’re also working on templates for RDS for teams that are interested in temporary access to RDS with an IAM-only solution.

Please let us know if you have questions on this example and/or want to try it out!

Related Posts