Why it's needed
Change Data Capture (CDC) is a streaming product on the Lightning Platform that enables you to efficiently integrate your Salesforce data with external systems.
CDC publishes events for changes in Salesforce records — create, update, delete, and undelete operations — so you can update data in an external system instead of doing periodic exports or API polling.
When to use it
- Receive notifications of Salesforce record changes including create, update, delete, and undelete operations.
- Capture all field changes for all records.
- Get broad access to all data regardless of sharing rules.
- Get information about each change in the event header — such as the origin of the change — so you can ignore changes your own client generated.
The challenge — two distinct data streams
When dealing with two distinct types of data streams:
- Bulk Data — large volumes pulled from Salesforce using the Bulk API.
- Change Data Capture (CDC) — real-time event-driven updates sourced via Salesforce's Streaming API.
The challenge: these two streams are structurally different. The Bulk API payload is optimized for high-volume extraction; the CDC payload focuses on granular, event-based updates. The differences make it difficult to work with the data in a unified, meaningful way.
The pipeline I designed
I've designed a data integration pipeline that addresses this by:
- Using Confluent Kafka Connect to source both the Bulk API and CDC data from Salesforce.
- Leveraging Snowflake's Sink Connector to ingest the data into Snowflake — and most importantly, unifying the two structurally distinct payloads into a single cohesive object using Snowflake's Dynamic Tables.
The architecture diagram below depicts the pattern used to unify the CDC object with the snapshot object using Snowflake Dynamic Tables.
