Last updated April 22, 2025

This article describes how to use AWS Key Management Service (KMS) to create a customer managed key (CMK) to encrypt Apache Kafka on Heroku in Private and Shield Spaces. This process involves three high-level steps:

1. Create a customer managed key (CMK) in your AWS KMS
2. Apply an identity and access management (IAM) policy to that CMK to permit Heroku Data to use the key on your behalf
3. Create a Kafka cluster with the encryption key

AWS Prerequisites

You perform the steps in this section from your Amazon KMS dashboard. Alternatively, you can use the AWS CLI.

Step 1: Create a Customer Managed Key

When logged into the AWS web console, navigate to Key Management Service and click Create Key.

Step 2: Select Symmetric Key

Select Symmetric for the key type, Encrypt and decrypt for the key usage, and click Next.

We use S3 to store backups and it only supports symmetric CMKs.

Step 3: Add Details and Set Permissions

Add an alias, such as heroku-data, and click Next. You can skip configuring key administrative permissions.

When defining key usage permissions, enter the Heroku Data AWS Account ID 021876802972 in the Other AWS accounts section, and click Next.

Step 4: Review and Complete

Review the key policy and complete the creation. Make note of the Amazon resource name (ARN) of your CMK.

Step 5: Enable Automatic Key Rotation

You can enable automatic key rotation by navigating to the info page for your key. In the Automatic key rotation section, select Enable, and click Save.

AWS automatically handles key rotations every year without any intervention. This rotation for symmetric keys generated within AWS KMS doesn’t require re-encrypting your data. AWS KMS manages the previous versions of keys used for the decryption of data encrypted under an old version of a key. All new encryption requests use the new version of the key. Your Apache Kafka on Heroku cluster doesn’t experience downtime when the key is rotated.

Alternative: Use the AWS CLI

You can use the AWS CLI to create your customer managed key (CMK) with the appropriate key policy.

$ export AWS_ACCOUNT_ID=`aws sts get-caller-identity --output text --query 'Account'`
$ export HEROKU_DATA_ACCOUNT_ID=021876802972
$ curl -s -o key-policy.json https://gist.githubusercontent.com/jdowning/8d146cd238de828141e81b458dc546f0/raw/fc2a69603dc1364f1bc2fd2b5beb0af210150444/key-policy.json
$ aws kms create-key --description 'heroku-data-test' --policy $(envsubst < key-policy.json)

The output of the create-key command includes the key’s ARN which you need during provisioning. This is referred to as CMK_ARN in later steps.

We recommend you enable automatic key rotation on your CMK:

$ aws enable-key-rotation --key-id CMK_ARN

Create an Apache Kafka on Heroku Cluster with an Encryption Key

Now that you have a customer managed key with the appropriate permissions configured, you can use that key to encrypt your data managed by Heroku Data. You need the full Amazon resource name (ARN) of your CMK. You can use the --encryption-key provisioning flag when creating your cluster.

$ heroku addons:create heroku-kafka:private-extended-0 --encryption-key CMK_ARN --app your-app-name

$ heroku addons:create heroku-kafka:private-extended-0 --app your-app-name -- --encryption-key CMK_ARN

Disabling Your Encryption Key

As part of the lifecycle of your encryption key, you may need to disable the key. When performing this action, all data encrypted with the key will be rendered inaccessible.

You can disable your encryption key via the AWS web console or the AWS CLI.

$ aws kms disable-key --key-id CMK_ARN

When we receive notification of the key disablement, there’s a 10 minute waiting period before action is taken. This waiting period is to ensure accidental changes to the key status don’t unnecessarily alter resources that depend on the key.

After the 10 minute waiting period, we shut down all services that use the encryption key. Next, we stop all servers that run those services. Finally, we send an email notification to application administrators notifying them of the disabled action.

Enabling Your Encryption Key

If you need to regain access to data that is encrypted with your encryption key, you can enable that key to restore. You can enable your encryption key via the AWS web console or the AWS CLI.

$ aws kms enable-key --key-id CMK_ARN

After we receive notification of the key enablement, we start previously stopped servers. When they’re running, we start affected services. Finally, we send an email notification to application administrators notifying them of the enabled action.

Limitations

There are limitations that apply to databases encrypted with a customer encryption key.

• This feature is only available for Apache Kafka on Heroku add-ons in Private or Shield plans.