The very first leadership principle at Amazon is “Customer Obsession.” You may have heard that before if you’ve attended an AWS event or read an article about the company. They talk about it a lot.
The best companies figure out how to incorporate their customer’s perspective into all functions of the business. That could mean deeply understanding customer pain points in order to provide the right solution. It could mean being extra responsive and empathetic when a customer has a problem. But for cloud security engineers, customer obsession means - above all - protecting their data.
I fight for the ~~users~~ customers!
Building a web application today requires a different set of technologies and security practices than twenty years ago when I started out. I don’t think I know anyone who has been in a data center in the past ten years and I’m pretty sure no one needs to worry about IE4 anymore. There is one thing that has remained very much the same: most SaaS companies are built atop relational databases. Protecting customer data still requires careful consideration of the who, when and how for access to your production database. The tricky part is balancing controls to keep out CLU (you’ll notice from the images that I was listening to the Tron soundtrack when I wrote this) with convenient paths for developers to get the access they need.
You probably arrived here because you're interested in learning the options available to you for managing access to Amazon's Relational Database Service (AWS RDS). If so, you probably have a good understanding of RDS’ capabilities as well as the benefits of managed services, so I won’t belabor them here.
RDS is special among its managed-database peers because of its wide and incredibly powerful array of security features. The focus of this article will be on the features that protect access to the data itself vs managing access to RDS API actions. Sometimes developers need to get into a SQL session (or Rails console or Django console or Prisma query console) for debugging, maintenance or to test system changes. Administrators carry the burden of providing that access to developers while keeping attackers out. The goal of this article is to provide a high-level understanding of the options to do so.
RDS supports several database engines but the features in this article are not available to all of them. We’ll use PostgreSQL as our engine and provide a couple working configurations using Terraform and Bash. The intent is to explain RDS access management capabilities through code and provide working configurations the reader can use to build on.
The first step will be to create a baseline RDS configuration for reference. We’ll use an example that is simple and illustrative but wildly insecure. It’s based on Hashicorp's Terraform tutorial: Manage AWS RDS Instances. As Hashicorp notes: this configuration in its current form has security vulnerabilities. The reader should - and I cannot stress this enough - NEVER USE THIS CONFIGURATION TO STORE CUSTOMER DATA.
The Terraform below will create a publicly accessible PostgreSQL database. It first defines a VPC with a public endpoint, a public subnet and a security group configured to allow traffic from the internet. It will then create an RDS DB Instance configured for public access.
To follow along you’ll need Terraform (version 1.3+), psql, jq, the AWS CLI and an AWS profile configured with the permissions to apply the Terraform below (e.g. AWSAdministratorAccess).
The necessary network components are defined in the first half of our main.tf
In this example, we’re using a feature of AWS VPC that makes it easy to connect to our instance from the internet. Resources provisioned into a VPC get private DNS hostnames by default. If we add the two DNS attributes above to a VPC configuration, our RDS Instance will also get a public DNS hostname. If we then associate a public subnet and properly configured security group, we’ll have a convenient way to connect directly to our instance.
Now that our network components are configured, we can define our RDS Instance and connect it to the network. We’ll do that in the rest of main.tf.
The code in main.tf is supported by the files: output.tf, variables.tf and versions.tf, the contents of which can be found in Hashicorp’s example repository. Once everything is in place, we can initialize our configuration.
Before we apply this Terraform, we need to define the db_password variable required to create the DB Instance. Note that this value will end up in the terraform state file.
Now that our configuration is initialized and our environment is set up, we can apply the Terraform.
Now we can connect to the instance using our outputs to construct the connection information.
That was fast and easy, thanks to the very useful example provided by Hashicorp. BUT this is just an example to help someone new to RDS+Terraform get started. It should never be used to store actual customer data.
A handy way to take stock of our current security situation is to view it through the lens of the AWS Foundational Security Best Practices controls for RDS. It may not surprise you to learn that we are in fact, in violation of all of them. A clean sweep! Let’s just say that if we began using this configuration to hold customer data, we would be doing a very poor job of protecting our customers.
Our customers deserve a better hero
What is particularly helpful about the controls found in the AWS Foundational Security Best Practices is that they not only help you gauge your relative security posture, they also help you understand your options to address shortcomings. Each control maps to a specific setting or feature to improve security. The primary focus of the rest of this article will be on two of the most important controls that also happen to be related to access management.
RDS.2 Amazon RDS DB instances should prohibit public access, as determined by the PubliclyAccessible configuration
If your database is on the public internet you could be at risk of brute force and DDOS attacks. Scanners are constantly probing the internet for databases to come online. If your database is publicly available, uses the default port and the default master username with a weak password, you could find yourself in trouble.
RDS.10 IAM authentication should be configured for RDS instances
Ditch static database credentials and use IAM to manage access directly to your database.
If you have a PostgreSQL instance sitting naked on the public internet there are a couple of quick wins to mitigate your risks. You could change the default port number, change the default administrator username and use a strong password. This will help deter brute force attacks but a solution that allows us to avoid them all together is preferable.
We could update the CIDR range in our security group to limit the IP addresses. If there are a small number of IPs from which you expect to receive connections, it could be manageable and help avoid attackers from discovering your DB Instance. But using this configuration to support several remote employees would quickly become untenable.
Ideally, we would disable public access.
If we refer to our baseline example, reconfiguring the RDS instance (by removing publicly_accessible) and making our subnet private will accomplish the task. Our database will be off the internet … but now no one can connect to it.
Before we discuss how to provide your team access to a database that is not publicly available, let’s talk about how to configure an EC2 instance to connect to our database. After all, we’re a SaaS startup. We need a web app!
To support a public facing web application connected to a not-publicly-access database, we need to add a new public subnet, provision the EC2 instance in there, and enable traffic from the public subnet to the database’s private subnet. The application server’s subnet can accept data from the public internet and the database subnet can only accept traffic from the subnet that the EC2 instance is in.
There is still no explicit way for developers to get to the database. Ideally you’d figure out a way of avoiding the need for them to do this, perhaps by restoring a sanitized copy into a development environment. But realistically, developers at fast growing startups will need to get to the data. Developers are creative and can usually find a way to get what they need.
One pattern that can emerge is that developers SSH into the application server and from there, use production credentials to connect from the EC2 instance to the database. This seems like a hack but it allows you to leverage all the thought and effort you’ve put into designing access to EC2 instances to also protect your database.
Now we’re starting get serious about security
As your team and customers grow, you may want to apply dedicated, more stringent policies to your production database. Also, you may get a maturity level where developers are no longer SSH-ing into your application servers.
One option is to provision a bastion host so developers can connect a SQL client to the database via an SSH tunnel. You could also use a PAM provider (like CyberArk) to provide extra security features (like MFA) and central management of SSH access policies. Tailscale is a modern and popular service that creates tunnels between developer workstations and databases. It also provides advanced security features and makes it easy to centralize access management across many types of cloud resources. Plus, Tailscale integrates with Sym!
Recall RDS.10 from above. It’s the control that states “IAM authentication should be configured for RDS instances.” IAM database authentication is a capability of RDS that allows your team to use their IAM users to authenticate directly to the database instance. It’s an extremely cool feature that gives database administrators IAM super powers. You can leverage the IAM groups, roles and conditions you’re already using to manage access to the rest of your AWS resources. Not to mention the myriad of security features that come with IAM including MFA, and logging and alerting via CloudTrail. IAM database authentication is currently supported for MariaDB, MySQL, and PostgreSQL.
IAM makes it easy to provide safe and scalable access for the whole team
To demonstrate configuration to get IAM database authentication off the ground, I’m going to build from our baseline terraform example. By the end of this next example our database will still be on the internet and it will still be in violation of many best practice controls. But building from our original example will help isolate the specific steps to get IAM database authentication working. Hopefully this will help you determine what steps to take to enable IAM database authentication on your own RDS Instance.
Here are the high level steps:
Let’s start with our original main.tf where our RDS instance is defined. We need to set the iam_database_authentication_enabled argument to true and add apply_immediately so we won’t need to wait for a maintenance window.
Next, we’ll create the necessary IAM entities. Most organizations should manage user/permissions associations via groups and roles. But to keep our example as simple as possible, we’re going to create a user (db_user) and attach a policy directly to it. The specific IAM action our user needs is rds-db:connect. Let’s put our IAM-related terraform in a separate file called iam.tf
We’ll need our IAM user name handy for constructing our database connection information so let’s add it to outputs.tf. I also added it to variables.tf to make it easier to test and experiment.
After we’ve applied the terraform updates, we should create a database user we can then associate with our new IAM user. We’ll do this by granting the rds_iam role to our database user. That tells RDS to map the database user to an IAM user of the same name.
Since I didn’t want anyone following these steps to end up with an active AWS access key in their state file, I opted to create one as a separate step on the command line. We need to use the credentials for our new IAM user to generate a temporary token for database access.
Connecting to the database is a two step process. Step one is generating an auth token and step two is using that token to connect to the database.
We can now harness the power of IAM to manage access to our database!
This example builds on our insecure baseline and still is in violation of several best practices. The intent is to provide a conceptual overview of the building blocks available to you as you consider how to secure your RDS instance. As you are building out your own configuration, it may be helpful to refer to example github repos (like this one) that provide fully working examples of RDS with IAM database authentication and no public access. Also if you are starting out from scratch, consider using an existing module (like CloudPosse’s RDS terraform module) which can help bootstrap your RDS Instance with several best practices baked in. If you are using other people’s code, you should make sure to understand what is happening under the hood. Hopefully this article has provided an understanding of foundational concepts to help you navigate more examples.
RDS provides the weapons we need to be a worthy protector of our customer’s data
Anyone following along now has a database password in their state file and an active access key on their workstation. Everything should get cleaned up by running terraform destroy. If something goes wrong unwinding your changes, you can ClickOps your way to safety. To make double sure you’ve deleted your access key, feel free to use the following command:
Sym helps provide just-in-time access to databases using native database user management or to RDS instances via IAM. Check out how to use Sym to manage just-in-time access to MySQL or just-in-time access to to Postgres. Also, check out how we can support just-in-time access to EC2 to see an example of how Sym can be used to manage access to AWS resources via IAM.
