Secrets management banner

Secrets management in AWS with SM Parameter Store and KMS

Securely storing and retrieving secrets should be considered a core service in any environment. Secret management should be implemented early alongside other core services such as authentication and network controls.

Parameter Store and Hashicorp options

CMD Solutions evaluated AWS Systems Manager (SM) Parameter Store and Hashicorp Vault as options for secret management. We really like Vault, however, most of the features weren’t necessary for our simple use case, and it didn’t fit with our general rule of minimising pets and management overhead in our solutions, so we forged ahead with SM Parameter Store. It’s now been a year, no regrets! We’ve also had time to optimise the way we utilise Systems Manager Parameter store and thought it might be nice to share some of our learnings.

SSM Parameter Store learnings

SSM Parameter Store is a pretty simple service to implement. You don’t really need to do anything, it’s already there ready to use. However, implementing proper security controls also means adding complexity. Access to parameters is controlled with a combination of KMS Key Policies and IAM User policies. Combining the two policies is difficult to get right and finding a balance between enabling automation while ensuring secrets are adequately protected is the main challenge this blog hopes to address.

Our SM needs to work in various scenarios:

  • Storing and retrieving secretsfrom a machine/infrastructure code
  • RBAC should bemanaged by infrastructure code
  • Ability to split encrypt/decrypt to different users – This allows less secure servers (eg. in a public subnet) to create the secret but not read it.  Allowing only heavily protected servers to decrypt it.
  • Short-lived access– In most cases access to a secret should be allocated on an as-needed temporary basis.  Automation often runs once when an instance is first provisioned, leaving access open for the life of the instance is dangerous and unnecessary.
  • Long-lived access– In some cases access needs to be maintained for long periods.
  • Ability to delegateaccess (typically to a machine) to grant temporary access when it’s needed.
  • Key rotation– PCI-DSS dictates that keys used for encrypting should be rotated.
  • Audit trailand visibilityinto who currently has access to a secret.

Controlling access to SSM parameter secrets

When automating the deployment of an application such as a database containing credit-card information, or a critical Identity service like Active Directory, it can be convenient to generate and store a highly-sensitive password in an SSM Parameter. Secure SSM parameters use a KMS DEK to encrypt the parameter value. You can specify a KMS key to use, or you can use the default KMS key generated by AWS for the SSM service.

There are three options available to control access to SSM parameter secrets:

  1. Allow AWS Root account in the KMS policy and hand off access control to IAM user profiles
  2. Use KMS Key policies to control access to the parameter
  3. A combination of 1. and 2.  No AWS Root account permit In KMS Key Policy, only allow specific roles.

To explain each option in a bit more detail.

1. Allow AWS Root account in the KMS policy and hand off access control to IAM user profiles

  • Access to KMS Key is handed off to IAM user policies
  • An IAM user policy must grant access to both the KMS key and the SSM parameter
  • The default ssm key can be used, however it is best practice to create your own key.  Just copy the key policy from the aws/ssm key and tweak as needed
  • The Administrator role and any IAM User/Role with kms:*, ssm:*, resource:* will have access to all parameters created with this KMS Key
  • Access to SSM params can be controlled by ssm namespace
  • Difficult to provision short-lived temporary access as IAM User policies are generally static

2. Use KMS Key policies to control access to the parameter

  • Enables the use of KMS Grants when accessing SSM
  • IAM Users with kms:* or ssm:* are still not granted access
  • Unable to control access based on SSM namespace
  • Unnecessary complexity for long-lived less-sensitive secrets that don’t need Grants.

3. A combination of 1. and 2.  No AWS Root account permit In KMS Key Policy, only allow specific roles.

  • Enables the use of KMS Grants when accessing SSM
  • Can ensure accounts with kms:* or ssm:* are still not granted access
  • Access to SSM params can be controlled by ssm namespace where Key Policy permits
  • Unnecessary complexity for long-lived less-sensitive secrets that don’t need Grants.
CMD recommend employing multiple KMS Keys for use by SM Params:
  • Option 3 – for more sensitive secrets with short-lived access.
  • Option 1 – for long lived secrets. Just select the appropriate key when creating the secret.

The “Default KMS Key Policy” and orphaned KMS Keys

KMS Key Policies are different from other service policies like S3 Key Policies.  In S3 it’s often necessary to set Deny statements in an S3 Key Policy to ensure an IAM user with s3:* on any resource isn’t granted access to buckets they shouldn’t be accessing. With KMS, even if an IAM user has a policy allowing kms:* on resource:* it still doesn’t allow them access to a KMS key.

The exception to this rule is when the root account is granted full access, as it is in the Default KMS Key Policy.  KMS Keys created in the console without a Key policy are given the “Default KMS Key Policy”. The below KMS Key will now allow any user with kms:* resource:* full access.

Example Default KMS Key Policy Snippet – Root Account

{
  "Version": "2012-10-17",
  "Id": "key-consolepolicy-3",
  "Statement": 
  [{
    "Sid": "Enable IAM User Permissions",
    "Effect": "Allow",
    "Principal": {
    "AWS": "arn:aws:iam::240696153881:root"
  },
    "Action": "kms:*",
    "Resource": "*"
  },

    ...

  ]
}

AWS do this to help prevent orphaned keys. If you remove all access to a key, in particular to kms:PutKeyPolicy, the key is orphaned and un-usable until AWS support fixes it.

CMD recommended Key Policy Roles

Combining KMS Key policies and SSM IAM policies is a complex task at the best of times. CMD recommend using a standard set of statements in all Key policies to simplify the process and create a point of reference for KMS discussions, planning and implementation.

The following roles should be defined in each KMS Key Policy as its own statement.

Role  
DecryptorA consumer of the key permitted to perform decrypt tasks and retrieve ssm secrets and decrypting data keys (DEK).
Optional. You may choose to omit this statement when using KMS Grants KMS Grants to allocate decrypt permits.
EncryptorA consumer of the key permitted to perform encrypt tasks such as creating ssm secrets and creating Data Keys (DEK). Often both Key Encryptor and Key Decryptor are the same user, however they are sometimes split across different roles. For example a set of Internet-facing servers can be given permission to encrypt, but not decrypt. A separate set of servers not directly connected to the Internet can then perform decryption, reducing the data's attack surface.
Optional. You may choose to omit this statement when using KMS Grants to allocate encrypt permits.
GrantorPermitted to delegate access by creating grants on a temporary basis to other users (Grantees).
CreatorThe creator of the key is often not the KMS Key administrator. This role requires minimum access for creating the key, and possibly deletion as well. It must allow kms:PutKeyPolicy or the creation will fail (unless BypassPolicyLockoutSafetyCheck is enabled). Be careful not to create orphan keys that no one can access or manage.
  • kms:PutKeyPolicy - This is required to prevent orphaned keys that no-one has permission to manage. Bypassed with --bypass-policy-lockout-safety-check.
  • kms:EnableKeyRotation - Most likely you will want to enable key rotation to meet compliance.
  • kms:CreateAlias - Usually you will want to associate an alias with your CMK for convenience and rotation behind the alias.
  • kms:ScheduleKeyDeletion. - Optional. Eg. when creating in CloudFormation allows you to delete the stack/key afterwards when testing.
  • kms:DescribeKey - Required when managing the key with Cloudformation.
  • kms:ListKeys - Needed to view key in the AWS console
  • kms:DeleteAlias - Optional. Eg. when creating in CloudFormation allows you to delete the alias along with the key when testing.
    MFA should be enforced.
AdministratorThey have access to manage all keys but not consume (encrypt/decrypt). They do have access to allocate themselves permissions to encrypt/decrypt, and monitoring should be in place to notify when encrypt/decrypt permissions are allocated.

MFA should be enforced.

KMS Grants

Infrastructure automation tasks are often a configure once, and leave it type scenario.  In this situation it is ideal to grant and revoke access to KMS Keys with a specific set of actions for a short-lived time period.

For example, it’s not ideal for an instance profile to be granted long lived access to highly-sensitive secrets, in-case the instance is compromised at a later date.

KMS Grants are perfect for this scenario. They enable you to programmatically delegate the task of granting access to a KMS Key to other AWS principals. Typically, delegation is given to a machine, like an instance profile or lambda.

This is ideal for break-glass type access, or an automation task where access only needs to exist for the period of the script execution.

A “Grantor” is someone with kms:CreateGrant priviliges, and is able to grant a subset of kms actions to a “Grantee”:

– Decrypt
– Encrypt
– GenerateDataKey
– GenerateDataKeyWithoutPlaintext
– ReEncryptFrom
– ReEncryptTo
– RetireGrant
– DescribeKey

Note: You can also restrict the Create Grant delegation even further by using Encryption Context. Encryption context is basically a set of key/value pairs that can be set when encrypting data, and must match when decrypting data. It’s then possible to set constraints specifying EncryptionContextEquals and/or EncryptionContextSubset constraints. Encryption Context is only relevant when using KMS encryption within your application code. As we’re only discussing access to KMS Keys by Amazon services (SSM), we do not require encryption context in these examples.

Grants do not expire themselves over time. They must be revoked or retired. Typically, you will setup a lambda to revoke Grants to a schedule, or have the Grantee retire their own grant when they are finished with it.

Automation scenarios

Scenario 1: Short-lived Secrets with Autoscale Events

Scenario 1: Short-lived Secrets with Autoscale Events diagram

Scenario 2: Short-lived Secrets with Configuration Management

Scenario 2: Short-lived Secrets with Configuration Management Diagram

Note: Beware of the current Ansible module for KMS Grants. It doesn’t use grants, and instead updates the KMS Key policy directly. This is far from ideal as it removes one of the main reasons for grants: delegation, and means the person running the automation needs kms:PutKeyPolicy permission which should only be allocated to High-Privileged users. CMD have a custom KMS Grant ansible module we hope to add to Ansible soon.

Key takeaways

  • Utilise Grants to enable automation, temporary access and separate key policy management and key access delegation.
  • Use a defined set of Roles in KMS Key Policies, see above.
  • Use multiple KMS Keys for use by SM Params. Option 3 for more sensitive secrets with short-lived access. Option 1 for long lived secrets.  Just select the appropriate key when creating the secret.
  • Monitor the main KMS actions with CloudTrail/SIEM. Encrypt/Decrypt/PutKeyPolicy/ScheduleKeyDeletion/CreateGrant.
  • Monitor active grants and schedule revoking grants.
  • Generate your own KMS Keys. Don’t use the default keys.

For more strategic IT and cloud insights visit the CMD Solutions blog or contact CMD Solutions.

No Comments

Sorry, the comment form is closed at this time.