Working with multiple AWS accounts at Ticketea

Working with multiple AWS accounts is often a good practice. People use multiple accounts for different purposes. At Ticketea, we map our production, staging and sandbox environments to distinct AWS accounts. You might use multiple AWS accounts to compartmentalize environments, services, or even development teams. Let's explore why and how to introduce multiple AWS accounts in your organization.

Goals

In this article we'll set up 3 different AWS accounts. We'll call them auth, staging and production.

Switch between AWS accounts

Let's imagine a group of developers who need to work with both the staging and production accounts. The auth account will contain the IAM users corresponding to our developers.

We'll set things up so that our developers will be able to log into the AWS console, and then switch between the auth, staging and production accounts seamlessly. We'll also make sure that when a developer switches to the staging or production account, appropriate restrictions are put into place to prevent accidental manipulation of sensible resources.

Switch between AWS accounts

Benefits of using multiple accounts

Before we jump to the technical implementation, let's talk about the benefits of using multiple accounts:

  • Environment separation: At Ticketea we use at least 3 accounts on a daily basis: production, staging and sandbox. This is standard software engineering hygiene.
  • Blast radius reduction: Having separate AWS accounts allows for a clean separation between your environments and resources, thus limiting the accidental damage of certain actions. Cross-account access is possible, but specific permissions must be granted explicitly.
  • Map your organizational structure to your AWS infrastructure: Do you have multiple teams and/or departments? Do you want to keep things separate? Use multiple AWS accounts and grant cross-account access only in case of specific needs.
  • Cost control:
    • Easily identify where your money is spent. Even though the same result can be achieved using tags, separating things by accounts feels easier than cramming all of you resources into one account and then filtering.
    • At Ticketea, we have a few mechanisms that shut down all non-production resources at night. This guarantees nice savings on our AWS bill. Applying these things only to certain AWS accounts prevents shutting down production resources accidentally.

Terminology

Let's clarify a few concepts which we'll use throughout the rest of the article:

  • AWS account: The container of your AWS resources, such as EC2 instances or S3 buckets. An account is identified by a 12-digit number which you can normally find browsing to the AWS support pages.
  • Root user: A user who has complete administrative power over a certain AWS account. The root user is normally created alongside the AWS account, and its use should be avoided in favor of IAM users.
  • IAM user: A user whose privileges are granted by the root user. In contrast with the root user, a IAM user has no intrinsic privileges, and any access to any AWS resource must be granted explicitly by the root user, or by another IAM user with appropriate administrative rights.
  • IAM role: A role is an identity associated with a set of permissions. The difference between a role and a user is that a role can be assumed by many users, or even ec2 instances or other AWS services. In short, when a user or a piece of software assumes a role, it is granted certain permissions specified by the role. Think of it as a hat: when you wear this hat, you are automatically given certain permissions associated with it.
  • Policy: A policy is a JSON document specifying a list of permissions to grant or revoke. This is effectively how you grant permissions. Policies can be attached to roles, user and groups.

Accounts set-up - Auth Account

Let's start working on the accounts configuration. The first account we'll work on is the Auth account.

This account is just a container for our IAM users. We won't create any other resource in here.

  1. We'll need a fresh, empty AWS account. Create one if you don't have one already.
  2. Create a group called developers. You will need to attach a few policies to this group.

The first policy allows developers to self-manage their credentials, multi-factor-authentication devices (aka MFA - more on this later), and a few other IAM bits.
Please make sure to replace <AUTH_ACCOUNT_ID> with the Auth account 12-digit identifier.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowAllUsersToListAccounts",
            "Effect": "Allow",
            "Action": [
                "iam:ListAccountAliases",
                "iam:ListUsers",
                "iam:GetAccountSummary"
            ],
            "Resource": "*"
        },
        {
            "Sid": "AllowIndividualUserToSeeAndManageTheirOwnAccountInformation",
            "Effect": "Allow",
            "Action": [
                "iam:ChangePassword",
                "iam:CreateAccessKey",
                "iam:CreateLoginProfile",
                "iam:DeleteAccessKey",
                "iam:DeleteLoginProfile",
                "iam:GetAccountPasswordPolicy",
                "iam:GetLoginProfile",
                "iam:ListAccessKeys",
                "iam:UpdateAccessKey",
                "iam:UpdateLoginProfile",
                "iam:ListSigningCertificates",
                "iam:DeleteSigningCertificate",
                "iam:UpdateSigningCertificate",
                "iam:UploadSigningCertificate",
                "iam:ListSSHPublicKeys",
                "iam:GetSSHPublicKey",
                "iam:DeleteSSHPublicKey",
                "iam:UpdateSSHPublicKey",
                "iam:UploadSSHPublicKey"
            ],
            "Resource": "arn:aws:iam::<AUTH_ACCOUNT_ID>:user/${aws:username}"
        },
        {
            "Sid": "AllowIndividualUserToListTheirOwnMFA",
            "Effect": "Allow",
            "Action": [
                "iam:ListVirtualMFADevices",
                "iam:ListMFADevices"
            ],
            "Resource": [
                "arn:aws:iam::<AUTH_ACCOUNT_ID>:mfa/*",
                "arn:aws:iam::<AUTH_ACCOUNT_ID>:user/${aws:username}"
            ]
        },
        {
            "Sid": "AllowIndividualUserToManageTheirOwnMFA",
            "Effect": "Allow",
            "Action": [
                "iam:CreateVirtualMFADevice",
                "iam:DeactivateMFADevice",
                "iam:DeleteVirtualMFADevice",
                "iam:EnableMFADevice",
                "iam:ResyncMFADevice"
            ],
            "Resource": [
                "arn:aws:iam::<AUTH_ACCOUNT_ID>:mfa/${aws:username}",
                "arn:aws:iam::<AUTH_ACCOUNT_ID>:user/${aws:username}"
            ]
        }
    ]
}

You will also need policies that allow users of our developers group to "switch role". Switching role will effectively make possible for a user to jump to another account. At a low level, this is done by obtaining a set of temporary credentials through STS (AWS Security Token Service).

You need to add a policy for each account you want to jump to. Please make sure to replace <TARGET_ACCOUNT_ID> with the 12-digit identifier of the account you want to be able to jump to:

{
    "Version": "2012-10-17",
    "Statement": {
        "Effect": "Allow",
        "Action": "sts:AssumeRole",
        "Resource": "arn:aws:iam::<TARGET_ACCOUNT_ID>:role/DeveloperAccess"
    }
}

DeveloperAccess is the role which the user will assume upon jumping to the target AWS account. This role will define set of permissions.

If you have chosen to enable MFA for your accounts, users should configure a MFA device before they can log in. This can be done in the auth account, choosing your own user and going to the "security credentials" tab. Don't worry if the AWS console gives you a few warnings telling you that you don't have permission to do certain things. We've set up our Auth account in a very restrictive way, which means that each user will only be able to manage his/her credentials. Everything else will be restricted.

Accounts set-up - Staging Account

In this account (which, again, you can create if you don't have one already) you'll need to create the DeveloperAccess role. With this role you do two things:
1. You declare that this account will accept users coming from a third-party account
2. You grant these users certain permissions

To create the DeveloperAccess role, you need to browse to IAM > Roles > Create new role > Role for cross-account access > Provide access between AWS accounts you own.

You will need to provide a few details:

  1. The 12-digit Auth account id you're expecting users from
  2. Whether or not you require users to have logged in the auth account using two factor authentication (we use it at Ticketea, and we highly suggest you to do the same!)
  3. A policy specifying which permissions you'll grant to those users coming from the Auth account. For instance, you might use the built-in ReadOnlyAccess policy for your production environment, whereas you might want to relax things a little for a sandbox/development account.

Trying it out

After adding a user to the developer group in the Auth account, you should be able to log in with the corresponding credentials. Once you have logged in, click in the top-right menu:

Switch between AWS accounts

You will be brought to a new page which will ask you for:

  1. The 12-digit AWS account id your're trying to switch to
  2. The role you want to assume (DeveloperAccess in our case)
  3. A name which will be shown in the account's drop-down menu next time you use it.

Switch between AWS accounts

After submitting the form, you will be brought automatically to the new account, and you will be given the permission defined by the DeveloperAccess role. You can switch back at any time using the same menu, clicking on "back to ".

Tip: There's a very handy Chrome plugin which gives you more control over the switch role menu.

Command line

Once you've successfully set things up, you can switch accounts using a menu in the AWS dashboard. Now, how do we do the same thing in a console session?

The answer is setting up a profile for your Auth account, and a profile for each of your environments. These profiles will refer to the auth account for the authentication procedure. Please make sure to replace <AUTH_ACCOUNT_ID> and the other parameters with your own values.

$ cat ~/.aws/config
[profile auth]
output = json
region = eu-west-1

[profile staging]
role_arn = arn:aws:iam::<TARGET_ACCOUNT_ID>:role/DeveloperAccess
mfa_serial = arn:aws:iam::<AUTH_ACCOUNT_ID>:mfa/<AUTH_ACCOUNT_MFA_ID>
source_profile = auth
output = json
region = eu-west-1


$ cat ~/.aws/credentials
[auth]
aws_access_key_id = <AUTH_ACCOUNT_KEY_ID>
aws_secret_access_key = <AUTH_ACCOUNT_ACCESS_KEY>

That's it! Now try launching a sample command:

~$ aws ec2 describe-regions --profile=staging
Enter MFA code:  
...
[snip]

The MFA token will be required to perform any action (as specified by the policy embedded in the DeveloperAccess role). The credentials will be cached for a few minutes, after which you'll be required to enter your MFA code again. This makes the MFA mechanism effectively unsuitable for non-interactive use cases.

Non-interactive / programmatic access

For such cases, you may want to define a different cross-account access role which doesn't enforce the multi-factor authentication (since you can't enter the MFA token interactively). An example might be an external continuous integration service which has to access your AWS account programmatically to deploy an application. Please note that non-interactive access should imply that the user is not given access to the AWS management console. In other words: if you create a role that doesn't enforce MFA for programmatic access purposes, that role should probably never be assumed by a human being.

Conclusions

It certainly takes a bit of effort to set multiple accounts up, but once you do it, you will enjoy all of the benefits of the boundaries you establish.

Please leave us a comment if you've found this article useful or want to contribute to it. You can also follow Ticketea's engineering team at @ticketeaeng to get notified about interesting articles in the future.