Last updated August 20, 2024

This article describes how to configure Change Data Capture (CDC) for Heroku Postgres events and stream them to your Apache Kafka on Heroku add-on provisioned in a Private Space or a Shield Private Space. This process involves three high-level steps:

1. Creating an app in Private Space or Shield Private Space.
2. Provisioning a Private or Shield Heroku Postgres add-on and a Private or Shield Apache Kafka on Heroku add-on on your new app.
3. Creating a streaming data connector to enable CDC events from your Postgres to your Kafka.

For more information about how to best configure a streaming data connector, see Best Practices for Heroku’s Streaming Data Connectors.

Heroku App Setup

To begin, create a Private or Shield Private Space. When your Space is available, you can create an app in your Space.

$ heroku spaces:create --region virginia --team my-team-name --space myspace
$ heroku spaces:wait --space myspace
$ heroku apps:create --space myspace my-cdc-app

Heroku Add-ons Setup

Next, you need two Private or Shield data add-ons attached to your app.

$ heroku addons:create heroku-postgresql:private-7 --as DATABASE --app my-cdc-app
$ heroku addons:create heroku-kafka:private-extended-2 --as KAFKA --app my-cdc-app

You can monitor the add-on provisioning progress:

$ heroku addons:wait --app my-cdc-app

When your add-ons are available, import your schema and your data into your Postgres database.

Heroku’s Streaming Data Connector Setup

When you have a Private or Shield Private Space App with Heroku Postgres and Apache Kafka on Heroku add-ons configured, you can provision a connector.

First, install the CLI plugin:

$ heroku plugins:install data

To create a connector, you must gather several pieces of information.

1. The name of the Kafka add-on
2. The name of the Postgres add-on
3. The names of the Postgres tables from which you want to capture events
4. (Optionally) The names of the columns you wish to exclude from capture events

In order to capture events in your Postgres database, a few requirements must be met:

• The database encoding must be UTF-8
• The tables must currently exist
• The tables must have a primary key
• The tables must not be partitioned
• The table names must only contain the characters [a-z,A-Z,0–9,\_]
• The Kafka Formation must have direct Zookeeper access disabled

Next, you can create the connector. You need the names of your Postgres and Kafka add-ons, as well as a list of fully qualified tables you want to include in your database capture events:

$ heroku data:connectors:create \
    --source postgresql-neato-98765 \
    --store kafka-lovely-12345 \
    --table public.posts --table public.users

Provisioning can take approximately 15–20 minutes to complete. You can monitor the connector provisioning progress:

$ heroku data:connectors:wait gentle-connector-1234

Streaming Data Connectors creates a topic in your Kafka cluster for each table you’ve chosen to capture data changes. When your connector is available, you can view the details including newly created Kafka topics:

$ heroku data:connectors:info gentle-connector-1234
=== Data Connector status for gentle_connector_1234
Name:   gentle_connector_1234
Status: available

=== Configuration
Table Name   Topic Name
public.posts gentle_connector_1234.public.posts
public.users gentle_connector_1234.public.users

These topics are configured with the following parameters:

• partition_count: 32
• replication_factor: 3
• cleanup_policy: delete
• retention_time_ms: 1209600000

Your principal (Kafka user) has Read and Describe access on the table topics created by Streaming Data Connectors. Write, Delete, and Alter operations are denied.

This feature also creates a heartbeat topic for each connector. Your principal also has Read and Describe access to the heartbeat topic.

Managing a Connector

After you’ve created your connector, there a few options available for managing it.

Pause or Resume

You can pause processing of new events. While a connector is paused, it stops polling for additional records until you resume it. Alternating between these two states is simple:

# to pause
$ heroku data:connectors:pause gentle-connector-1234

# to resume
$ heroku data:connectors:resume gentle-connector-1234

Update Configuration

You can modify certain properties associated with your connector via the CLI. These properties include:

For example, you can update the tombstones.on.delete to false:

$ heroku data:connectors:update gentle-connector-1234 \
  --setting tombstones.on.delete=false

Configuration managed by Heroku

Most configuration properties are entirely managed by Heroku and are modified as needed.

Update Tables and Excluded Columns

You can also modify the connector’s Postgres tables, as well as excluded columns.

For example, you can add the table public.parcels and remove the table public.posts:

$ heroku data:connectors:update gentle-connector-1234 \
  --add-table public.parcels \
  --remove-table public.posts

New tables must adhere to the same requirements as outlined in the Setup.

Likewise, you can add and remove excluded columns:

$ heroku data:connectors:update gentle-connector-1234 \
  --exclude-column public.parcels.address \
  --remove-excluded-column public.posts.keys

Destroying a Connector

You can destroy a connector via the CLI.

$ heroku data:connectors:destroy gentle-connector-1234